2023-10-10T17:20:24.365 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-10T17:20:24.366 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} --owner scheduled_choffman@teuthology --verbose --archive /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361 --name choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi -- /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/orig.config.yaml 2023-10-10T17:20:24.424 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-10T17:20:24.520 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361 branch: wip-choffman-62793 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} email: choffman@redhat.com first_in_suite: false job_id: '7420361' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-choffman-62793 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 05a43c8fb3988914c879a50970f9054a098e0612 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 05a43c8fb3988914c879a50970f9054a098e0612 kclient: mntopts: - ms_mode=crc - wsync syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-testing-62793 sha1: 48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf owner: scheduled_choffman@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5469 sha1: 05a43c8fb3988914c879a50970f9054a098e0612 sleep_before_teardown: 0 suite: fs:workload suite_branch: wip-testing-62793 suite_path: /home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa suite_relpath: qa suite_repo: https://github.com/chrisphoffman/ceph.git suite_sha1: 48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - fs/misc teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-10_17:11:00 tube: smithi user: choffman verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-10T17:20:24.520 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa; will attempt to use it 2023-10-10T17:20:24.594 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa/tasks 2023-10-10T17:20:24.594 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-10T17:20:24.596 INFO:teuthology.task.internal:Checking packages... 2023-10-10T17:20:24.620 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '05a43c8fb3988914c879a50970f9054a098e0612' 2023-10-10T17:20:24.621 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-10T17:20:24.621 INFO:teuthology.packaging:ref: None 2023-10-10T17:20:24.621 INFO:teuthology.packaging:tag: None 2023-10-10T17:20:24.622 INFO:teuthology.packaging:branch: wip-choffman-62793 2023-10-10T17:20:24.622 INFO:teuthology.packaging:sha1: 05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:20:24.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-choffman-62793 2023-10-10T17:20:24.843 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3908.g05a43c8f 2023-10-10T17:20:24.844 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-10T17:20:24.851 INFO:teuthology.task.internal:no buildpackages task found 2023-10-10T17:20:24.851 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-10T17:20:24.888 INFO:teuthology.task.internal:Saving configuration 2023-10-10T17:20:24.909 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-10T17:20:24.915 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-10T17:20:24.949 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 17:14:32.194213', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-10T17:20:24.991 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 17:14:32.195393', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-10T17:20:25.021 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 17:14:32.192674', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-10T17:20:25.022 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-10T17:20:25.029 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-10-10T17:20:25.029 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-10-10T17:20:25.029 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-10-10T17:20:25.030 INFO:teuthology.run_tasks:Running task console_log... 2023-10-10T17:20:25.186 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8b66065ca0>, signals=[15]) 2023-10-10T17:20:25.188 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-10T17:20:25.195 INFO:teuthology.task.internal:Opening connections... 2023-10-10T17:20:25.195 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:20:25.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T17:20:25.273 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2023-10-10T17:20:25.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T17:20:25.351 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2023-10-10T17:20:25.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T17:20:25.445 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-10T17:20:25.453 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2023-10-10T17:20:25.474 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2023-10-10T17:20:25.475 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2023-10-10T17:20:25.535 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Red Hat Enterprise Linux" 2023-10-10T17:20:25.535 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8.6 (Ootpa)" 2023-10-10T17:20:25.536 INFO:teuthology.orchestra.run.smithi078.stdout:ID="rhel" 2023-10-10T17:20:25.536 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="fedora" 2023-10-10T17:20:25.536 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8.6" 2023-10-10T17:20:25.536 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2023-10-10T17:20:25.536 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-10T17:20:25.537 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2023-10-10T17:20:25.537 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-10T17:20:25.538 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.redhat.com/" 2023-10-10T17:20:25.538 INFO:teuthology.orchestra.run.smithi078.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-10T17:20:25.538 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-10T17:20:25.538 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:20:25.539 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-10T17:20:25.539 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-10T17:20:25.539 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-10T17:20:25.539 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-10T17:20:25.540 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2023-10-10T17:20:25.569 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2023-10-10T17:20:25.589 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2023-10-10T17:20:25.589 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2023-10-10T17:20:25.649 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Red Hat Enterprise Linux" 2023-10-10T17:20:25.649 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="8.6 (Ootpa)" 2023-10-10T17:20:25.649 INFO:teuthology.orchestra.run.smithi157.stdout:ID="rhel" 2023-10-10T17:20:25.650 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE="fedora" 2023-10-10T17:20:25.650 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="8.6" 2023-10-10T17:20:25.650 INFO:teuthology.orchestra.run.smithi157.stdout:PLATFORM_ID="platform:el8" 2023-10-10T17:20:25.650 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-10T17:20:25.651 INFO:teuthology.orchestra.run.smithi157.stdout:ANSI_COLOR="0;31" 2023-10-10T17:20:25.651 INFO:teuthology.orchestra.run.smithi157.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-10T17:20:25.651 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.redhat.com/" 2023-10-10T17:20:25.651 INFO:teuthology.orchestra.run.smithi157.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-10T17:20:25.651 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-10T17:20:25.652 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:20:25.652 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-10T17:20:25.652 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-10T17:20:25.652 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-10T17:20:25.652 INFO:teuthology.orchestra.run.smithi157.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-10T17:20:25.653 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2023-10-10T17:20:25.685 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2023-10-10T17:20:25.707 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2023-10-10T17:20:25.707 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2023-10-10T17:20:25.766 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="Red Hat Enterprise Linux" 2023-10-10T17:20:25.766 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="8.6 (Ootpa)" 2023-10-10T17:20:25.767 INFO:teuthology.orchestra.run.smithi190.stdout:ID="rhel" 2023-10-10T17:20:25.767 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="fedora" 2023-10-10T17:20:25.767 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="8.6" 2023-10-10T17:20:25.767 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el8" 2023-10-10T17:20:25.767 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-10T17:20:25.768 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2023-10-10T17:20:25.768 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-10T17:20:25.768 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://www.redhat.com/" 2023-10-10T17:20:25.768 INFO:teuthology.orchestra.run.smithi190.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-10T17:20:25.768 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-10T17:20:25.769 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:20:25.769 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-10T17:20:25.769 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-10T17:20:25.769 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-10T17:20:25.769 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-10T17:20:25.770 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2023-10-10T17:20:25.820 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-10T17:20:25.830 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-10T17:20:25.847 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-10T17:20:25.848 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2023-10-10T17:20:25.851 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2023-10-10T17:20:25.853 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2023-10-10T17:20:25.872 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-10T17:20:25.879 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-10T17:20:25.879 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2023-10-10T17:20:25.910 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2023-10-10T17:20:25.913 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2023-10-10T17:20:26.144 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-10T17:20:26.257 INFO:teuthology.run_tasks:Running task kernel... 2023-10-10T17:20:26.272 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-10-10T17:20:26.273 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-10-10T17:20:26.273 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-10-10T17:20:26.273 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-10T17:20:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T17:20:26.293 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-10-10T17:20:26.352 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-10T17:20:26.353 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.18.0-372.9.1.el8.x86_64 2023-10-10T17:20:26.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2023-10-10T17:20:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-10T17:20:27.684 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-10-10T17:20:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 8.9 kB/s | 3.5 kB 00:00 2023-10-10T17:20:30.968 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 16 MB 00:01 2023-10-10T17:20:34.443 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 405 kB/s | 24 kB 00:00 2023-10-10T17:20:35.137 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-10T17:20:35.181 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-10-10T17:20:35.182 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-10-10T17:20:35.182 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-10-10T17:20:35.299 DEBUG:teuthology.orchestra.run.smithi078:> echo no | sudo yum reinstall kernel || true 2023-10-10T17:20:35.627 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-10T17:20:35.627 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-10-10T17:20:35.890 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:01 ago on Tue 10 Oct 2023 05:20:34 PM UTC. 2023-10-10T17:20:35.934 INFO:teuthology.orchestra.run.smithi078.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-10T17:20:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:Error: No packages marked for reinstall. 2023-10-10T17:20:36.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum reinstall -y kernel || true 2023-10-10T17:20:36.347 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-10T17:20:36.348 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-10-10T17:20:36.604 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:02 ago on Tue 10 Oct 2023 05:20:34 PM UTC. 2023-10-10T17:20:36.647 INFO:teuthology.orchestra.run.smithi078.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-10T17:20:36.672 INFO:teuthology.orchestra.run.smithi078.stderr:Error: No packages marked for reinstall. 2023-10-10T17:20:36.718 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2023-10-10T17:20:36.798 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-10T17:20:36.798 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-10T17:20:36.798 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-10T17:20:36.799 DEBUG:teuthology.parallel:result is None 2023-10-10T17:20:36.799 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-10T17:20:36.807 INFO:teuthology.task.internal:Creating test directory... 2023-10-10T17:20:36.807 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-10T17:20:36.810 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-10T17:20:36.812 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-10T17:20:36.832 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-10T17:20:36.839 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-10T17:20:36.863 INFO:teuthology.task.internal:Creating archive directory... 2023-10-10T17:20:36.864 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-10T17:20:36.866 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-10T17:20:36.874 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-10T17:20:36.908 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-10T17:20:36.915 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-10T17:20:36.916 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-10T17:20:36.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T17:20:36.960 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-10T17:20:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T17:20:36.980 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-10T17:20:36.982 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-10T17:20:37.003 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-10T17:20:37.013 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T17:20:37.029 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T17:20:37.047 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T17:20:37.067 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T17:20:37.076 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T17:20:37.097 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-10T17:20:37.099 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-10T17:20:37.124 INFO:teuthology.task.internal:Configuring sudo... 2023-10-10T17:20:37.125 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-10T17:20:37.130 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-10T17:20:37.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-10T17:20:37.169 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-10T17:20:37.183 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-10T17:20:37.184 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-10T17:20:37.200 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-10T17:20:37.211 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-10T17:20:37.231 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T17:20:37.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T17:20:37.364 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T17:20:37.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T17:20:37.490 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:20:37.491 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-10T17:20:37.581 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T17:20:37.649 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T17:20:37.693 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T17:20:37.787 DEBUG:teuthology.orchestra.run.smithi157:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T17:20:37.825 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:20:37.826 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-10T17:20:37.895 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T17:20:37.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-10T17:20:38.000 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T17:20:38.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-10T17:20:38.131 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:20:38.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-10T17:20:38.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2023-10-10T17:20:38.202 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2023-10-10T17:20:38.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2023-10-10T17:20:38.262 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T17:20:38.262 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T17:20:38.304 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T17:20:38.741 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-10T17:20:38.749 INFO:teuthology.task.internal:Starting timer... 2023-10-10T17:20:38.750 INFO:teuthology.run_tasks:Running task pcp... 2023-10-10T17:20:38.792 INFO:teuthology.run_tasks:Running task selinux... 2023-10-10T17:20:38.816 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-10-10T17:20:38.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2023-10-10T17:20:38.885 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2023-10-10T17:20:38.887 DEBUG:teuthology.orchestra.run.smithi157:> sudo service auditd rotate 2023-10-10T17:20:38.944 INFO:teuthology.orchestra.run.smithi157.stdout:Rotating logs: 2023-10-10T17:20:38.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2023-10-10T17:20:39.009 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2023-10-10T17:20:39.011 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-10T17:20:39.011 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2023-10-10T17:20:39.037 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2023-10-10T17:20:39.038 DEBUG:teuthology.orchestra.run.smithi157:> /usr/sbin/getenforce 2023-10-10T17:20:39.074 INFO:teuthology.orchestra.run.smithi157.stdout:Permissive 2023-10-10T17:20:39.075 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2023-10-10T17:20:39.097 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2023-10-10T17:20:39.097 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi157.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2023-10-10T17:20:39.098 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T17:20:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T17:20:39.127 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T17:20:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T17:20:39.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T17:20:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T17:20:39.183 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-10T17:20:39.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2023-10-10T17:20:39.210 DEBUG:teuthology.orchestra.run.smithi157:> sudo /usr/sbin/setenforce permissive 2023-10-10T17:20:39.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2023-10-10T17:20:39.266 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-10T17:20:39.275 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-10T17:20:39.276 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-10T17:20:39.293 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-10T17:20:39.294 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi157.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-10T17:30:04.417 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2023-10-10T17:30:04.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-10T17:30:04.420 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T17:30:04.489 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-10-10T17:30:04.561 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-10T17:30:04.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-10-10T17:30:04.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T17:30:04.631 DEBUG:teuthology.orchestra.run.smithi157:> true 2023-10-10T17:30:04.700 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2023-10-10T17:30:04.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-10-10T17:30:04.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-10T17:30:04.769 DEBUG:teuthology.orchestra.run.smithi190:> true 2023-10-10T17:30:04.835 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2023-10-10T17:30:04.835 INFO:teuthology.run_tasks:Running task clock... 2023-10-10T17:30:04.845 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-10T17:30:04.845 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-10T17:30:04.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T17:30:04.849 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-10T17:30:04.849 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T17:30:04.851 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-10T17:30:04.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T17:30:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-10T17:30:04.883 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-10T17:30:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-10T17:30:04.898 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-10T17:30:04.905 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-10T17:30:04.919 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-10T17:30:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2023-10-10T17:30:04.979 INFO:teuthology.orchestra.run.smithi157.stderr:sudo: ntpd: command not found 2023-10-10T17:30:04.985 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2023-10-10T17:30:04.985 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2023-10-10T17:30:04.992 INFO:teuthology.orchestra.run.smithi157.stdout:506 Cannot talk to daemon 2023-10-10T17:30:04.996 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2023-10-10T17:30:05.000 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-10T17:30:05.007 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-10T17:30:05.010 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-10T17:30:05.014 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-10T17:30:05.021 INFO:teuthology.orchestra.run.smithi157.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-10T17:30:05.024 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-10T17:30:05.083 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-10-10T17:30:05.089 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T17:30:05.089 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-10-10T17:30:05.089 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.090 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.090 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.090 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.099 INFO:teuthology.orchestra.run.smithi190.stderr:bash: ntpq: command not found 2023-10-10T17:30:05.100 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-10-10T17:30:05.102 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T17:30:05.102 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2023-10-10T17:30:05.103 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.103 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.103 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.103 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.105 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T17:30:05.105 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-10-10T17:30:05.105 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.106 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.106 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.106 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T17:30:05.107 INFO:teuthology.run_tasks:Running task pexec... 2023-10-10T17:30:05.116 INFO:teuthology.task.pexec:Executing custom commands... 2023-10-10T17:30:05.117 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:30:05.117 DEBUG:teuthology.orchestra.run.smithi078:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-10T17:30:05.118 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi157.front.sepia.ceph.com 2023-10-10T17:30:05.118 DEBUG:teuthology.orchestra.run.smithi157:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-10T17:30:05.119 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi190.front.sepia.ceph.com 2023-10-10T17:30:05.119 DEBUG:teuthology.orchestra.run.smithi190:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-10T17:30:05.505 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:05.517 INFO:teuthology.orchestra.run.smithi190.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:05.518 INFO:teuthology.orchestra.run.smithi157.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:06.382 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Oct 2023 05:28:28 PM UTC. 2023-10-10T17:30:06.414 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Oct 2023 05:28:23 PM UTC. 2023-10-10T17:30:06.418 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Oct 2023 05:28:33 PM UTC. 2023-10-10T17:30:08.102 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-10-10T17:30:08.103 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-10-10T17:30:08.103 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-10-10T17:30:08.104 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-10-10T17:30:08.104 INFO:teuthology.orchestra.run.smithi078.stdout:Resetting modules: 2023-10-10T17:30:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools 2023-10-10T17:30:08.104 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:30:08.104 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-10-10T17:30:08.105 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-10-10T17:30:08.105 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:30:08.135 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-10-10T17:30:08.136 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-10-10T17:30:08.136 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2023-10-10T17:30:08.136 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-10-10T17:30:08.136 INFO:teuthology.orchestra.run.smithi190.stdout:Resetting modules: 2023-10-10T17:30:08.136 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools 2023-10-10T17:30:08.137 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:30:08.137 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-10-10T17:30:08.137 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2023-10-10T17:30:08.137 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:30:08.294 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-10-10T17:30:08.295 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-10-10T17:30:08.295 INFO:teuthology.orchestra.run.smithi157.stdout: Package Architecture Version Repository Size 2023-10-10T17:30:08.295 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-10-10T17:30:08.296 INFO:teuthology.orchestra.run.smithi157.stdout:Resetting modules: 2023-10-10T17:30:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: container-tools 2023-10-10T17:30:08.296 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:30:08.296 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-10-10T17:30:08.297 INFO:teuthology.orchestra.run.smithi157.stdout:================================================================================ 2023-10-10T17:30:08.297 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:30:09.086 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Oct 2023 05:28:28 PM UTC. 2023-10-10T17:30:09.130 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Oct 2023 05:28:23 PM UTC. 2023-10-10T17:30:09.286 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Oct 2023 05:28:33 PM UTC. 2023-10-10T17:30:12.014 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-10-10T17:30:12.468 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:13.207 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-10-10T17:30:13.289 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-10-10T17:30:13.684 INFO:teuthology.orchestra.run.smithi190.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:13.765 INFO:teuthology.orchestra.run.smithi157.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:13.916 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Oct 2023 05:28:28 PM UTC. 2023-10-10T17:30:15.094 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Oct 2023 05:28:33 PM UTC. 2023-10-10T17:30:15.118 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:51 ago on Tue 10 Oct 2023 05:28:23 PM UTC. 2023-10-10T17:30:15.775 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-10-10T17:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================================ 2023-10-10T17:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-10-10T17:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================================ 2023-10-10T17:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout:Installing group/module packages: 2023-10-10T17:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-10-10T17:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-10-10T17:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-10-10T17:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-10-10T17:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-10-10T17:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-10-10T17:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-10-10T17:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-10-10T17:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-10-10T17:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-10-10T17:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-10-10T17:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout:Installing module profiles: 2023-10-10T17:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools/common 2023-10-10T17:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2023-10-10T17:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools rhel8 2023-10-10T17:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-10-10T17:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================================ 2023-10-10T17:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout:Install 10 Packages 2023-10-10T17:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 18 M 2023-10-10T17:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 65 M 2023-10-10T17:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-10-10T17:30:15.990 INFO:teuthology.orchestra.run.smithi078.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 208 kB/s | 25 kB 00:00 2023-10-10T17:30:16.024 INFO:teuthology.orchestra.run.smithi078.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 267 kB/s | 41 kB 00:00 2023-10-10T17:30:16.099 INFO:teuthology.orchestra.run.smithi078.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 411 kB/s | 94 kB 00:00 2023-10-10T17:30:16.183 INFO:teuthology.orchestra.run.smithi078.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 257 kB/s | 49 kB 00:00 2023-10-10T17:30:16.275 INFO:teuthology.orchestra.run.smithi078.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.0 MB/s | 524 kB 00:00 2023-10-10T17:30:16.358 INFO:teuthology.orchestra.run.smithi078.stdout:(6/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.5 MB/s | 213 kB 00:00 2023-10-10T17:30:16.392 INFO:teuthology.orchestra.run.smithi078.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 715 kB/s | 149 kB 00:00 2023-10-10T17:30:16.634 INFO:teuthology.orchestra.run.smithi078.stdout:(8/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 25 MB/s | 7.0 MB 00:00 2023-10-10T17:30:16.759 INFO:teuthology.orchestra.run.smithi078.stdout:(9/10): buildah-1.26.2-1.module+el8.6.0+16771+2 12 MB/s | 8.0 MB 00:00 2023-10-10T17:30:16.803 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-10-10T17:30:16.809 INFO:teuthology.orchestra.run.smithi078.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 5.3 MB/s | 2.2 MB 00:00 2023-10-10T17:30:16.810 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-10-10T17:30:16.810 INFO:teuthology.orchestra.run.smithi078.stdout:Total 19 MB/s | 18 MB 00:00 2023-10-10T17:30:16.818 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-10-10T17:30:16.895 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================================ 2023-10-10T17:30:16.896 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-10-10T17:30:16.896 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================================ 2023-10-10T17:30:16.896 INFO:teuthology.orchestra.run.smithi157.stdout:Installing group/module packages: 2023-10-10T17:30:16.896 INFO:teuthology.orchestra.run.smithi157.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-10-10T17:30:16.897 INFO:teuthology.orchestra.run.smithi157.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-10-10T17:30:16.897 INFO:teuthology.orchestra.run.smithi157.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-10-10T17:30:16.897 INFO:teuthology.orchestra.run.smithi157.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-10-10T17:30:16.897 INFO:teuthology.orchestra.run.smithi157.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-10-10T17:30:16.897 INFO:teuthology.orchestra.run.smithi157.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-10-10T17:30:16.898 INFO:teuthology.orchestra.run.smithi157.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-10-10T17:30:16.898 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-10-10T17:30:16.898 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-10-10T17:30:16.898 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-10-10T17:30:16.899 INFO:teuthology.orchestra.run.smithi157.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-10-10T17:30:16.899 INFO:teuthology.orchestra.run.smithi157.stdout:Installing module profiles: 2023-10-10T17:30:16.899 INFO:teuthology.orchestra.run.smithi157.stdout: container-tools/common 2023-10-10T17:30:16.899 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:30:16.899 INFO:teuthology.orchestra.run.smithi157.stdout:Enabling module streams: 2023-10-10T17:30:16.899 INFO:teuthology.orchestra.run.smithi157.stdout: container-tools rhel8 2023-10-10T17:30:16.900 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:30:16.900 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-10-10T17:30:16.900 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================================================ 2023-10-10T17:30:16.900 INFO:teuthology.orchestra.run.smithi157.stdout:Install 10 Packages 2023-10-10T17:30:16.901 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:30:16.901 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 18 M 2023-10-10T17:30:16.902 INFO:teuthology.orchestra.run.smithi157.stdout:Installed size: 65 M 2023-10-10T17:30:16.902 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-10-10T17:30:16.910 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================================ 2023-10-10T17:30:16.910 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-10-10T17:30:16.910 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================================ 2023-10-10T17:30:16.910 INFO:teuthology.orchestra.run.smithi190.stdout:Installing group/module packages: 2023-10-10T17:30:16.911 INFO:teuthology.orchestra.run.smithi190.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-10-10T17:30:16.911 INFO:teuthology.orchestra.run.smithi190.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-10-10T17:30:16.911 INFO:teuthology.orchestra.run.smithi190.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-10-10T17:30:16.911 INFO:teuthology.orchestra.run.smithi190.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-10-10T17:30:16.912 INFO:teuthology.orchestra.run.smithi190.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-10-10T17:30:16.912 INFO:teuthology.orchestra.run.smithi190.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-10-10T17:30:16.912 INFO:teuthology.orchestra.run.smithi190.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-10-10T17:30:16.912 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-10-10T17:30:16.912 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-10-10T17:30:16.913 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-10-10T17:30:16.913 INFO:teuthology.orchestra.run.smithi190.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-10-10T17:30:16.913 INFO:teuthology.orchestra.run.smithi190.stdout:Installing module profiles: 2023-10-10T17:30:16.914 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools/common 2023-10-10T17:30:16.914 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:30:16.914 INFO:teuthology.orchestra.run.smithi190.stdout:Enabling module streams: 2023-10-10T17:30:16.915 INFO:teuthology.orchestra.run.smithi190.stdout: container-tools rhel8 2023-10-10T17:30:16.915 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:30:16.915 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-10-10T17:30:16.915 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================================ 2023-10-10T17:30:16.915 INFO:teuthology.orchestra.run.smithi190.stdout:Install 10 Packages 2023-10-10T17:30:16.916 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:30:16.916 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 18 M 2023-10-10T17:30:16.917 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 65 M 2023-10-10T17:30:16.917 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-10-10T17:30:17.009 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-10-10T17:30:17.027 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-10-10T17:30:17.028 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-10-10T17:30:17.100 INFO:teuthology.orchestra.run.smithi157.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 466 kB/s | 94 kB 00:00 2023-10-10T17:30:17.128 INFO:teuthology.orchestra.run.smithi190.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 435 kB/s | 94 kB 00:00 2023-10-10T17:30:17.200 INFO:teuthology.orchestra.run.smithi157.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 135 kB/s | 41 kB 00:00 2023-10-10T17:30:17.210 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-10-10T17:30:17.214 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-10-10T17:30:17.237 INFO:teuthology.orchestra.run.smithi190.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 126 kB/s | 41 kB 00:00 2023-10-10T17:30:17.241 INFO:teuthology.orchestra.run.smithi157.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 348 kB/s | 49 kB 00:00 2023-10-10T17:30:17.295 INFO:teuthology.orchestra.run.smithi190.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 65 kB/s | 25 kB 00:00 2023-10-10T17:30:17.325 INFO:teuthology.orchestra.run.smithi157.stdout:(4/10): python3-pytoml-0.1.14-5.git7dea353.el8. 58 kB/s | 25 kB 00:00 2023-10-10T17:30:17.337 INFO:teuthology.orchestra.run.smithi190.stdout:(4/10): cockpit-podman-49.1-1.module+el8.6.0+16 5.1 MB/s | 524 kB 00:00 2023-10-10T17:30:17.367 INFO:teuthology.orchestra.run.smithi157.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.1 MB/s | 524 kB 00:00 2023-10-10T17:30:17.396 INFO:teuthology.orchestra.run.smithi190.stdout:(5/10): udica-0.2.6-4.module+el8.6.0+16771+28df 185 kB/s | 49 kB 00:00 2023-10-10T17:30:17.418 INFO:teuthology.orchestra.run.smithi157.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.6 MB/s | 149 kB 00:00 2023-10-10T17:30:17.488 INFO:teuthology.orchestra.run.smithi190.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 994 kB/s | 149 kB 00:00 2023-10-10T17:30:17.530 INFO:teuthology.orchestra.run.smithi190.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.6 MB/s | 213 kB 00:00 2023-10-10T17:30:17.601 INFO:teuthology.orchestra.run.smithi157.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 22 MB/s | 8.0 MB 00:00 2023-10-10T17:30:17.626 INFO:teuthology.orchestra.run.smithi157.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 823 kB/s | 213 kB 00:00 2023-10-10T17:30:17.630 INFO:teuthology.orchestra.run.smithi190.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-10-10T17:30:17.722 INFO:teuthology.orchestra.run.smithi190.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 11 MB/s | 2.2 MB 00:00 2023-10-10T17:30:17.737 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-10-10T17:30:17.760 INFO:teuthology.orchestra.run.smithi157.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 21 MB/s | 7.0 MB 00:00 2023-10-10T17:30:17.810 INFO:teuthology.orchestra.run.smithi157.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 11 MB/s | 2.2 MB 00:00 2023-10-10T17:30:17.811 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-10-10T17:30:17.811 INFO:teuthology.orchestra.run.smithi157.stdout:Total 20 MB/s | 18 MB 00:00 2023-10-10T17:30:17.813 INFO:teuthology.orchestra.run.smithi190.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 22 MB/s | 7.0 MB 00:00 2023-10-10T17:30:17.814 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-10-10T17:30:17.814 INFO:teuthology.orchestra.run.smithi190.stdout:Total 20 MB/s | 18 MB 00:00 2023-10-10T17:30:17.908 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-10-10T17:30:18.011 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-10-10T17:30:18.012 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-10-10T17:30:18.029 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-10-10T17:30:18.030 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-10-10T17:30:18.031 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-10-10T17:30:18.031 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-10-10T17:30:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-10T17:30:18.211 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-10-10T17:30:18.214 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-10-10T17:30:18.215 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-10-10T17:30:18.218 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-10-10T17:30:18.346 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-10-10T17:30:18.572 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-10-10T17:30:18.719 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-10-10T17:30:18.731 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-10-10T17:30:18.903 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-10-10T17:30:18.913 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-10-10T17:30:18.955 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-10-10T17:30:19.090 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-10T17:30:19.108 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-10T17:30:19.293 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-10-10T17:30:19.309 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-10-10T17:30:19.496 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-10-10T17:30:19.501 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-10-10T17:30:19.758 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-10-10T17:30:19.820 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-10-10T17:30:19.885 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-10-10T17:30:20.581 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-10-10T17:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-10-10T17:30:20.675 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-10-10T17:30:20.892 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-10-10T17:30:21.066 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-10-10T17:30:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-10T17:30:21.438 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-10-10T17:30:21.550 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-10T17:30:21.551 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-10-10T17:30:21.551 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-10T17:30:21.551 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-10-10T17:30:21.551 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-10-10T17:30:21.552 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-10-10T17:30:21.552 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-10-10T17:30:21.552 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-10-10T17:30:21.552 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-10-10T17:30:21.552 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-10-10T17:30:21.571 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-10-10T17:30:21.651 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-10-10T17:30:21.815 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-10-10T17:30:21.821 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-10-10T17:30:21.997 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-10-10T17:30:22.000 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-10T17:30:22.207 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-10T17:30:22.267 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-10T17:30:22.267 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-10-10T17:30:22.268 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-10T17:30:22.268 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-10-10T17:30:22.268 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-10-10T17:30:22.268 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-10-10T17:30:22.269 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-10-10T17:30:22.269 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-10-10T17:30:22.269 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-10-10T17:30:22.269 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-10-10T17:30:22.473 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-10-10T17:30:22.473 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-10-10T17:30:22.473 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-10-10T17:30:22.474 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-10-10T17:30:22.474 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-10-10T17:30:22.474 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-10-10T17:30:22.475 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-10-10T17:30:22.475 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-10-10T17:30:22.475 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-10-10T17:30:22.475 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-10-10T17:30:22.696 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-10-10T17:30:22.696 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:54 ago on Tue 10 Oct 2023 05:28:28 PM UTC. 2023-10-10T17:30:23.473 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-10-10T17:30:23.473 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:00 ago on Tue 10 Oct 2023 05:28:23 PM UTC. 2023-10-10T17:30:23.640 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-10-10T17:30:23.640 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:50 ago on Tue 10 Oct 2023 05:28:33 PM UTC. 2023-10-10T17:30:28.338 INFO:teuthology.orchestra.run.smithi078.stdout:Installed products updated. 2023-10-10T17:30:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:30:28.526 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-10-10T17:30:28.526 INFO:teuthology.orchestra.run.smithi078.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-10-10T17:30:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-10-10T17:30:28.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-10T17:30:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-10T17:30:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-10-10T17:30:28.528 INFO:teuthology.orchestra.run.smithi078.stdout: yajl-2.1.0-10.el8.x86_64 2023-10-10T17:30:28.529 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:30:28.529 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-10-10T17:30:28.669 DEBUG:teuthology.parallel:result is None 2023-10-10T17:30:30.160 INFO:teuthology.orchestra.run.smithi190.stdout:Installed products updated. 2023-10-10T17:30:30.355 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:30:30.356 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-10-10T17:30:30.356 INFO:teuthology.orchestra.run.smithi190.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:30.356 INFO:teuthology.orchestra.run.smithi190.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-10-10T17:30:30.356 INFO:teuthology.orchestra.run.smithi190.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:30.357 INFO:teuthology.orchestra.run.smithi190.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-10-10T17:30:30.357 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-10T17:30:30.357 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-10T17:30:30.357 INFO:teuthology.orchestra.run.smithi190.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:30.357 INFO:teuthology.orchestra.run.smithi190.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:30.357 INFO:teuthology.orchestra.run.smithi190.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-10-10T17:30:30.358 INFO:teuthology.orchestra.run.smithi190.stdout: yajl-2.1.0-10.el8.x86_64 2023-10-10T17:30:30.358 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:30:30.358 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-10-10T17:30:30.501 DEBUG:teuthology.parallel:result is None 2023-10-10T17:30:31.302 INFO:teuthology.orchestra.run.smithi157.stdout:Installed products updated. 2023-10-10T17:30:31.503 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:30:31.503 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-10-10T17:30:31.504 INFO:teuthology.orchestra.run.smithi157.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:31.504 INFO:teuthology.orchestra.run.smithi157.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-10-10T17:30:31.504 INFO:teuthology.orchestra.run.smithi157.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:31.504 INFO:teuthology.orchestra.run.smithi157.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-10-10T17:30:31.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-10T17:30:31.505 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-10T17:30:31.505 INFO:teuthology.orchestra.run.smithi157.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:31.505 INFO:teuthology.orchestra.run.smithi157.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-10-10T17:30:31.505 INFO:teuthology.orchestra.run.smithi157.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-10-10T17:30:31.506 INFO:teuthology.orchestra.run.smithi157.stdout: yajl-2.1.0-10.el8.x86_64 2023-10-10T17:30:31.506 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:30:31.506 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-10-10T17:30:31.640 DEBUG:teuthology.parallel:result is None 2023-10-10T17:30:31.640 INFO:teuthology.run_tasks:Running task install... 2023-10-10T17:30:31.649 DEBUG:teuthology.task.install:project ceph 2023-10-10T17:30:31.649 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612'}} 2023-10-10T17:30:31.650 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612'} 2023-10-10T17:30:31.650 INFO:teuthology.task.install:Using flavor: default 2023-10-10T17:30:31.667 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-10T17:30:31.668 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-10T17:30:31.669 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'tag': None, 'wait_for_package': False} 2023-10-10T17:30:31.670 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:30:31.674 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'tag': None, 'wait_for_package': False} 2023-10-10T17:30:31.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:30:31.676 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'tag': None, 'wait_for_package': False} 2023-10-10T17:30:31.676 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:30:31.820 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/ 2023-10-10T17:30:31.820 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3908.g05a43c8f 2023-10-10T17:30:31.850 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/ 2023-10-10T17:30:31.851 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3908.g05a43c8f 2023-10-10T17:30:31.951 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-10T17:30:31.952 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:30:31.952 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-10T17:30:31.955 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/ 2023-10-10T17:30:31.955 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3908.g05a43c8f 2023-10-10T17:30:31.989 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-10T17:30:31.990 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/05a43c8fb3988914c879a50970f9054a098e0612/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-10T17:30:32.015 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-10T17:30:32.015 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:30:32.016 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-10T17:30:32.043 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-10T17:30:32.044 DEBUG:teuthology.orchestra.run.smithi157:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/05a43c8fb3988914c879a50970f9054a098e0612/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-10T17:30:32.060 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-10T17:30:32.060 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:30:32.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-10T17:30:32.063 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-10T17:30:32.093 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-10T17:30:32.094 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/05a43c8fb3988914c879a50970f9054a098e0612/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-10T17:30:32.110 DEBUG:teuthology.orchestra.run.smithi157:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-10T17:30:32.148 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-10T17:30:32.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-10T17:30:32.179 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2023-10-10T17:30:32.181 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2023-10-10T17:30:32.186 DEBUG:teuthology.orchestra.run.smithi157:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-10T17:30:32.247 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-10T17:30:32.252 INFO:teuthology.orchestra.run.smithi157.stdout:check_obsoletes = 1 2023-10-10T17:30:32.253 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum clean all 2023-10-10T17:30:32.314 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2023-10-10T17:30:32.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2023-10-10T17:30:32.575 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:32.624 INFO:teuthology.orchestra.run.smithi157.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:32.688 INFO:teuthology.orchestra.run.smithi190.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:33.642 INFO:teuthology.orchestra.run.smithi078.stdout:44 files removed 2023-10-10T17:30:33.673 INFO:teuthology.orchestra.run.smithi190.stdout:44 files removed 2023-10-10T17:30:33.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-10T17:30:33.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-10T17:30:33.787 INFO:teuthology.orchestra.run.smithi157.stdout:44 files removed 2023-10-10T17:30:33.841 DEBUG:teuthology.orchestra.run.smithi157:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-10T17:30:34.028 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:34.059 INFO:teuthology.orchestra.run.smithi190.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:34.176 INFO:teuthology.orchestra.run.smithi157.stdout:Updating Subscription Management repositories. 2023-10-10T17:30:35.446 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 242 kB/s | 80 kB 00:00 2023-10-10T17:30:35.630 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 167 kB/s | 80 kB 00:00 2023-10-10T17:30:35.722 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 38 kB/s | 9.4 kB 00:00 2023-10-10T17:30:35.770 INFO:teuthology.orchestra.run.smithi157.stdout:ceph packages for x86_64 188 kB/s | 80 kB 00:00 2023-10-10T17:30:35.914 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 37 kB/s | 9.4 kB 00:00 2023-10-10T17:30:36.073 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 4.5 kB/s | 1.5 kB 00:00 2023-10-10T17:30:36.079 INFO:teuthology.orchestra.run.smithi157.stdout:ceph noarch packages 34 kB/s | 9.4 kB 00:00 2023-10-10T17:30:36.156 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 6.7 kB/s | 1.5 kB 00:00 2023-10-10T17:30:36.224 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-10-10T17:30:36.299 INFO:teuthology.orchestra.run.smithi190.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-10T17:30:36.421 INFO:teuthology.orchestra.run.smithi157.stdout:ceph source packages 4.6 kB/s | 1.5 kB 00:00 2023-10-10T17:30:36.564 INFO:teuthology.orchestra.run.smithi157.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-10T17:30:37.216 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-10-10T17:30:37.556 INFO:teuthology.orchestra.run.smithi157.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-10-10T17:30:38.233 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 8.3 MB/s | 16 MB 00:01 2023-10-10T17:30:40.741 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 422 kB/s | 24 kB 00:00 2023-10-10T17:30:41.123 INFO:teuthology.orchestra.run.smithi157.stdout:lab-extras 532 kB/s | 24 kB 00:00 2023-10-10T17:30:41.716 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 361 kB/s | 24 kB 00:00 2023-10-10T17:30:41.743 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 51 MB/s | 47 MB 00:00 2023-10-10T17:30:42.277 INFO:teuthology.orchestra.run.smithi157.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 44 MB/s | 47 MB 00:01 2023-10-10T17:30:42.843 INFO:teuthology.orchestra.run.smithi190.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-10-10T17:30:49.878 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 53 MB 00:01 2023-10-10T17:30:50.425 INFO:teuthology.orchestra.run.smithi157.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 53 MB 00:01 2023-10-10T17:30:51.019 INFO:teuthology.orchestra.run.smithi190.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 52 MB/s | 53 MB 00:01 2023-10-10T17:31:01.052 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T17:31:01.053 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T17:31:01.067 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-10T17:31:01.068 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-10T17:31:01.069 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-10T17:31:01.070 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-10T17:31:01.072 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-10T17:31:01.076 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-10T17:31:01.077 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-10T17:31:01.207 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-10-10T17:31:01.220 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-10-10T17:31:01.220 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-10-10T17:31:01.220 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-10-10T17:31:01.221 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-10-10T17:31:01.221 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-10T17:31:01.221 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-10T17:31:01.221 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.4 k 2023-10-10T17:31:01.221 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 5.3 M 2023-10-10T17:31:01.222 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 848 k 2023-10-10T17:31:01.222 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 166 k 2023-10-10T17:31:01.222 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 1.5 M 2023-10-10T17:31:01.222 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 127 k 2023-10-10T17:31:01.222 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 1.7 M 2023-10-10T17:31:01.223 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 7.4 M 2023-10-10T17:31:01.223 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 47 k 2023-10-10T17:31:01.223 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 13 M 2023-10-10T17:31:01.223 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 41 M 2023-10-10T17:31:01.223 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 265 k 2023-10-10T17:31:01.224 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 83 k 2023-10-10T17:31:01.224 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 258 k 2023-10-10T17:31:01.224 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 18 k 2023-10-10T17:31:01.224 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-10T17:31:01.225 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-10T17:31:01.225 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-10T17:31:01.225 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 31 k 2023-10-10T17:31:01.225 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 770 k 2023-10-10T17:31:01.225 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 139 k 2023-10-10T17:31:01.226 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-10T17:31:01.226 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-10T17:31:01.226 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-10T17:31:01.226 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-10T17:31:01.226 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-10T17:31:01.227 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-10T17:31:01.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 198 k 2023-10-10T17:31:01.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 371 k 2023-10-10T17:31:01.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 364 k 2023-10-10T17:31:01.227 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 106 k 2023-10-10T17:31:01.228 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 94 k 2023-10-10T17:31:01.228 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.4 M 2023-10-10T17:31:01.228 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 179 k 2023-10-10T17:31:01.229 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-10T17:31:01.229 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2023-10-10T17:31:01.230 INFO:teuthology.orchestra.run.smithi078.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-10T17:31:01.230 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.7 M 2023-10-10T17:31:01.230 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.6 M 2023-10-10T17:31:01.230 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-10-10T17:31:01.230 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-10T17:31:01.231 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 22 M 2023-10-10T17:31:01.231 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 23 k 2023-10-10T17:31:01.231 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 2.3 M 2023-10-10T17:31:01.231 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 240 k 2023-10-10T17:31:01.231 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 4.6 M 2023-10-10T17:31:01.232 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 18 M 2023-10-10T17:31:01.232 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 15 k 2023-10-10T17:31:01.232 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 24 k 2023-10-10T17:31:01.232 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-10T17:31:01.232 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-10T17:31:01.233 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-10T17:31:01.233 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-10T17:31:01.233 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-10T17:31:01.233 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 137 k 2023-10-10T17:31:01.233 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-10T17:31:01.234 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-10T17:31:01.234 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-10T17:31:01.234 INFO:teuthology.orchestra.run.smithi078.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-10T17:31:01.234 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-10T17:31:01.235 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-10T17:31:01.235 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 473 k 2023-10-10T17:31:01.235 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-10T17:31:01.235 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.5 M 2023-10-10T17:31:01.235 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-10T17:31:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-10T17:31:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-10T17:31:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-10T17:31:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-10T17:31:01.236 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-10T17:31:01.237 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-10T17:31:01.237 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-10T17:31:01.237 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-10T17:31:01.237 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-10T17:31:01.237 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-10T17:31:01.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-10T17:31:01.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-10T17:31:01.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-10T17:31:01.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-10T17:31:01.238 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-10T17:31:01.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 45 k 2023-10-10T17:31:01.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 119 k 2023-10-10T17:31:01.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-10T17:31:01.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-10T17:31:01.239 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-10T17:31:01.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-10T17:31:01.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-10T17:31:01.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-10T17:31:01.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-10T17:31:01.240 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-10T17:31:01.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-10T17:31:01.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-10T17:31:01.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-10T17:31:01.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T17:31:01.241 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-10T17:31:01.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-10T17:31:01.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-10T17:31:01.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-10T17:31:01.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-10T17:31:01.242 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-10T17:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-10T17:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-10T17:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-10T17:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-10T17:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-10T17:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-10T17:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-10T17:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-10T17:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-10T17:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-10T17:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-10T17:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-10T17:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T17:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-10T17:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-10T17:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-10T17:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-10T17:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-10T17:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-10T17:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-10T17:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-10T17:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-10T17:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-10T17:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-10T17:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-10T17:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-10T17:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-10T17:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-10T17:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-10T17:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-10T17:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-10T17:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2023-10-10T17:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql 10 2023-10-10T17:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-10-10T17:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-10-10T17:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout:Install 128 Packages 2023-10-10T17:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 3 Packages 2023-10-10T17:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 197 M 2023-10-10T17:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-10-10T17:31:01.512 INFO:teuthology.orchestra.run.smithi078.stdout:(1/131): ceph-18.0.0-3908.g05a43c8f.el8.x86_64. 29 kB/s | 6.4 kB 00:00 2023-10-10T17:31:01.666 INFO:teuthology.orchestra.run.smithi157.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T17:31:01.667 INFO:teuthology.orchestra.run.smithi157.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T17:31:01.681 INFO:teuthology.orchestra.run.smithi157.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-10T17:31:01.682 INFO:teuthology.orchestra.run.smithi157.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-10T17:31:01.683 INFO:teuthology.orchestra.run.smithi157.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-10T17:31:01.684 INFO:teuthology.orchestra.run.smithi157.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-10T17:31:01.686 INFO:teuthology.orchestra.run.smithi157.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-10T17:31:01.689 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-10T17:31:01.690 INFO:teuthology.orchestra.run.smithi157.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-10T17:31:01.823 INFO:teuthology.orchestra.run.smithi157.stdout:Dependencies resolved. 2023-10-10T17:31:01.836 INFO:teuthology.orchestra.run.smithi157.stdout:=========================================================================================================================== 2023-10-10T17:31:01.836 INFO:teuthology.orchestra.run.smithi157.stdout: Package Arch Version Repository Size 2023-10-10T17:31:01.836 INFO:teuthology.orchestra.run.smithi157.stdout:=========================================================================================================================== 2023-10-10T17:31:01.837 INFO:teuthology.orchestra.run.smithi157.stdout:Installing: 2023-10-10T17:31:01.837 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-10T17:31:01.837 INFO:teuthology.orchestra.run.smithi157.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-10T17:31:01.837 INFO:teuthology.orchestra.run.smithi157.stdout: ceph x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.4 k 2023-10-10T17:31:01.837 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 5.3 M 2023-10-10T17:31:01.838 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 848 k 2023-10-10T17:31:01.838 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 166 k 2023-10-10T17:31:01.838 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 1.5 M 2023-10-10T17:31:01.838 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 127 k 2023-10-10T17:31:01.838 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 1.7 M 2023-10-10T17:31:01.839 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 7.4 M 2023-10-10T17:31:01.839 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 47 k 2023-10-10T17:31:01.839 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 13 M 2023-10-10T17:31:01.839 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 41 M 2023-10-10T17:31:01.839 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 265 k 2023-10-10T17:31:01.840 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 83 k 2023-10-10T17:31:01.840 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 258 k 2023-10-10T17:31:01.840 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 18 k 2023-10-10T17:31:01.840 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-10T17:31:01.841 INFO:teuthology.orchestra.run.smithi157.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-10T17:31:01.841 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-10T17:31:01.841 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 31 k 2023-10-10T17:31:01.841 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 770 k 2023-10-10T17:31:01.841 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 139 k 2023-10-10T17:31:01.842 INFO:teuthology.orchestra.run.smithi157.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-10T17:31:01.842 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-10T17:31:01.842 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-10T17:31:01.842 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-10T17:31:01.842 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-10T17:31:01.843 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-10T17:31:01.843 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 198 k 2023-10-10T17:31:01.843 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 371 k 2023-10-10T17:31:01.843 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 364 k 2023-10-10T17:31:01.843 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 106 k 2023-10-10T17:31:01.844 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 94 k 2023-10-10T17:31:01.844 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.4 M 2023-10-10T17:31:01.844 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 179 k 2023-10-10T17:31:01.844 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-10T17:31:01.846 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrading: 2023-10-10T17:31:01.846 INFO:teuthology.orchestra.run.smithi157.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-10T17:31:01.847 INFO:teuthology.orchestra.run.smithi157.stdout: librados2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.7 M 2023-10-10T17:31:01.847 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.6 M 2023-10-10T17:31:01.847 INFO:teuthology.orchestra.run.smithi157.stdout:Installing dependencies: 2023-10-10T17:31:01.847 INFO:teuthology.orchestra.run.smithi157.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-10T17:31:01.847 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 22 M 2023-10-10T17:31:01.848 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 23 k 2023-10-10T17:31:01.848 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 2.3 M 2023-10-10T17:31:01.848 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 240 k 2023-10-10T17:31:01.848 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 4.6 M 2023-10-10T17:31:01.848 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 18 M 2023-10-10T17:31:01.849 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 15 k 2023-10-10T17:31:01.849 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 24 k 2023-10-10T17:31:01.849 INFO:teuthology.orchestra.run.smithi157.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-10T17:31:01.849 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-10T17:31:01.849 INFO:teuthology.orchestra.run.smithi157.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-10T17:31:01.850 INFO:teuthology.orchestra.run.smithi157.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-10T17:31:01.850 INFO:teuthology.orchestra.run.smithi157.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-10T17:31:01.850 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 137 k 2023-10-10T17:31:01.850 INFO:teuthology.orchestra.run.smithi157.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-10T17:31:01.850 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-10T17:31:01.851 INFO:teuthology.orchestra.run.smithi157.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-10T17:31:01.851 INFO:teuthology.orchestra.run.smithi157.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-10T17:31:01.851 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-10T17:31:01.851 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-10T17:31:01.851 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 473 k 2023-10-10T17:31:01.852 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-10T17:31:01.852 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.5 M 2023-10-10T17:31:01.852 INFO:teuthology.orchestra.run.smithi157.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-10T17:31:01.852 INFO:teuthology.orchestra.run.smithi157.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-10T17:31:01.852 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-10T17:31:01.853 INFO:teuthology.orchestra.run.smithi157.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-10T17:31:01.853 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-10T17:31:01.853 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-10T17:31:01.853 INFO:teuthology.orchestra.run.smithi157.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-10T17:31:01.853 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-10T17:31:01.854 INFO:teuthology.orchestra.run.smithi157.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-10T17:31:01.854 INFO:teuthology.orchestra.run.smithi157.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-10T17:31:01.854 INFO:teuthology.orchestra.run.smithi157.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-10T17:31:01.854 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-10T17:31:01.854 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-10T17:31:01.855 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-10T17:31:01.855 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-10T17:31:01.855 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-10T17:31:01.855 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 45 k 2023-10-10T17:31:01.855 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 119 k 2023-10-10T17:31:01.856 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-10T17:31:01.856 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-10T17:31:01.856 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-10T17:31:01.857 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-10T17:31:01.857 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-10T17:31:01.857 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-10T17:31:01.857 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-10T17:31:01.857 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-10T17:31:01.858 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-10T17:31:01.858 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-10T17:31:01.858 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-10T17:31:01.858 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T17:31:01.858 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-10T17:31:01.859 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-10T17:31:01.859 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-10T17:31:01.859 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-10T17:31:01.859 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-10T17:31:01.859 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-10T17:31:01.859 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-10T17:31:01.860 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-10T17:31:01.860 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-10T17:31:01.860 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-10T17:31:01.860 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-10T17:31:01.860 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-10T17:31:01.861 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-10T17:31:01.861 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-10T17:31:01.861 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-10T17:31:01.861 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-10T17:31:01.861 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-10T17:31:01.862 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-10T17:31:01.862 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T17:31:01.862 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-10T17:31:01.862 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-10T17:31:01.862 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-10T17:31:01.863 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-10T17:31:01.863 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-10T17:31:01.863 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-10T17:31:01.863 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-10T17:31:01.863 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-10T17:31:01.864 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-10T17:31:01.864 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-10T17:31:01.864 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-10T17:31:01.864 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-10T17:31:01.864 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-10T17:31:01.865 INFO:teuthology.orchestra.run.smithi157.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-10T17:31:01.865 INFO:teuthology.orchestra.run.smithi157.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-10T17:31:01.865 INFO:teuthology.orchestra.run.smithi157.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-10T17:31:01.865 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-10T17:31:01.865 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-10T17:31:01.865 INFO:teuthology.orchestra.run.smithi157.stdout:Enabling module streams: 2023-10-10T17:31:01.866 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql 10 2023-10-10T17:31:01.866 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:31:01.866 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction Summary 2023-10-10T17:31:01.866 INFO:teuthology.orchestra.run.smithi157.stdout:=========================================================================================================================== 2023-10-10T17:31:01.866 INFO:teuthology.orchestra.run.smithi157.stdout:Install 128 Packages 2023-10-10T17:31:01.867 INFO:teuthology.orchestra.run.smithi157.stdout:Upgrade 3 Packages 2023-10-10T17:31:01.867 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:31:01.868 INFO:teuthology.orchestra.run.smithi157.stdout:Total download size: 197 M 2023-10-10T17:31:01.868 INFO:teuthology.orchestra.run.smithi157.stdout:Downloading Packages: 2023-10-10T17:31:02.283 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T17:31:02.284 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-10T17:31:02.298 INFO:teuthology.orchestra.run.smithi190.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-10-10T17:31:02.299 INFO:teuthology.orchestra.run.smithi190.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-10-10T17:31:02.300 INFO:teuthology.orchestra.run.smithi190.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-10-10T17:31:02.301 INFO:teuthology.orchestra.run.smithi190.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-10T17:31:02.304 INFO:teuthology.orchestra.run.smithi190.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-10T17:31:02.307 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-10-10T17:31:02.308 INFO:teuthology.orchestra.run.smithi190.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-10-10T17:31:02.443 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2023-10-10T17:31:02.445 INFO:teuthology.orchestra.run.smithi078.stdout:(2/131): ceph-fuse-18.0.0-3908.g05a43c8f.el8.x8 909 kB/s | 848 kB 00:00 2023-10-10T17:31:02.455 INFO:teuthology.orchestra.run.smithi190.stdout:=========================================================================================================================== 2023-10-10T17:31:02.455 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2023-10-10T17:31:02.455 INFO:teuthology.orchestra.run.smithi190.stdout:=========================================================================================================================== 2023-10-10T17:31:02.455 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2023-10-10T17:31:02.456 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-10-10T17:31:02.456 INFO:teuthology.orchestra.run.smithi190.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-10T17:31:02.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.4 k 2023-10-10T17:31:02.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 5.3 M 2023-10-10T17:31:02.456 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 848 k 2023-10-10T17:31:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 166 k 2023-10-10T17:31:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 1.5 M 2023-10-10T17:31:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 127 k 2023-10-10T17:31:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 1.7 M 2023-10-10T17:31:02.457 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 7.4 M 2023-10-10T17:31:02.458 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 47 k 2023-10-10T17:31:02.458 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 13 M 2023-10-10T17:31:02.458 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 41 M 2023-10-10T17:31:02.458 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 265 k 2023-10-10T17:31:02.459 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 83 k 2023-10-10T17:31:02.459 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 258 k 2023-10-10T17:31:02.459 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 18 k 2023-10-10T17:31:02.459 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-10-10T17:31:02.459 INFO:teuthology.orchestra.run.smithi190.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-10T17:31:02.460 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-10T17:31:02.460 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 31 k 2023-10-10T17:31:02.460 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 770 k 2023-10-10T17:31:02.460 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 139 k 2023-10-10T17:31:02.460 INFO:teuthology.orchestra.run.smithi190.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-10T17:31:02.461 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-10-10T17:31:02.461 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-10T17:31:02.461 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-10-10T17:31:02.461 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-10-10T17:31:02.461 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-10-10T17:31:02.462 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 198 k 2023-10-10T17:31:02.462 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 371 k 2023-10-10T17:31:02.462 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 364 k 2023-10-10T17:31:02.462 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 106 k 2023-10-10T17:31:02.462 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 94 k 2023-10-10T17:31:02.463 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.4 M 2023-10-10T17:31:02.463 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 179 k 2023-10-10T17:31:02.463 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-10-10T17:31:02.463 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2023-10-10T17:31:02.463 INFO:teuthology.orchestra.run.smithi190.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-10-10T17:31:02.464 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.7 M 2023-10-10T17:31:02.464 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 3.6 M 2023-10-10T17:31:02.464 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2023-10-10T17:31:02.464 INFO:teuthology.orchestra.run.smithi190.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-10T17:31:02.464 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 22 M 2023-10-10T17:31:02.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 23 k 2023-10-10T17:31:02.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 2.3 M 2023-10-10T17:31:02.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 240 k 2023-10-10T17:31:02.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 4.6 M 2023-10-10T17:31:02.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 18 M 2023-10-10T17:31:02.465 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3908.g05a43c8f.el8 ceph-noarch 15 k 2023-10-10T17:31:02.466 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 24 k 2023-10-10T17:31:02.466 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-10T17:31:02.466 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-10T17:31:02.466 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-10T17:31:02.466 INFO:teuthology.orchestra.run.smithi190.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-10T17:31:02.467 INFO:teuthology.orchestra.run.smithi190.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-10-10T17:31:02.467 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 137 k 2023-10-10T17:31:02.467 INFO:teuthology.orchestra.run.smithi190.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-10-10T17:31:02.467 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-10T17:31:02.467 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-10T17:31:02.468 INFO:teuthology.orchestra.run.smithi190.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-10-10T17:31:02.468 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-10T17:31:02.468 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-10T17:31:02.468 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 473 k 2023-10-10T17:31:02.468 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-10T17:31:02.469 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 6.5 M 2023-10-10T17:31:02.469 INFO:teuthology.orchestra.run.smithi190.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-10T17:31:02.469 INFO:teuthology.orchestra.run.smithi190.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-10-10T17:31:02.469 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-10T17:31:02.469 INFO:teuthology.orchestra.run.smithi190.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-10T17:31:02.470 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-10T17:31:02.470 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-10T17:31:02.470 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-10T17:31:02.470 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-10T17:31:02.471 INFO:teuthology.orchestra.run.smithi190.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-10T17:31:02.471 INFO:teuthology.orchestra.run.smithi190.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-10T17:31:02.471 INFO:teuthology.orchestra.run.smithi190.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-10T17:31:02.471 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-10T17:31:02.471 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-10T17:31:02.472 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-10T17:31:02.472 INFO:teuthology.orchestra.run.smithi190.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-10T17:31:02.472 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-10T17:31:02.472 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 45 k 2023-10-10T17:31:02.472 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:18.0.0-3908.g05a43c8f.el8 ceph 119 k 2023-10-10T17:31:02.472 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-10T17:31:02.473 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-10T17:31:02.473 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-10T17:31:02.473 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-10T17:31:02.473 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-10T17:31:02.473 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-10T17:31:02.474 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-10T17:31:02.474 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-10T17:31:02.474 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-10T17:31:02.474 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-10T17:31:02.474 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-10T17:31:02.475 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T17:31:02.475 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-10T17:31:02.475 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-10T17:31:02.475 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-10T17:31:02.475 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-10T17:31:02.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-10T17:31:02.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-10T17:31:02.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-10T17:31:02.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-10T17:31:02.476 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-10T17:31:02.477 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-10T17:31:02.477 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-10T17:31:02.477 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-10T17:31:02.477 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-10T17:31:02.477 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-10T17:31:02.478 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-10T17:31:02.478 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-10T17:31:02.478 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-10T17:31:02.478 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-10T17:31:02.478 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-10T17:31:02.479 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-10T17:31:02.479 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-10T17:31:02.479 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-10T17:31:02.479 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-10T17:31:02.479 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-10T17:31:02.480 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-10T17:31:02.480 INFO:teuthology.orchestra.run.smithi190.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-10T17:31:02.480 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-10T17:31:02.480 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-10T17:31:02.480 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-10T17:31:02.481 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-10T17:31:02.481 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-10T17:31:02.481 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-10T17:31:02.481 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-10T17:31:02.481 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-10T17:31:02.482 INFO:teuthology.orchestra.run.smithi190.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-10-10T17:31:02.482 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-10T17:31:02.482 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-10T17:31:02.482 INFO:teuthology.orchestra.run.smithi190.stdout:Enabling module streams: 2023-10-10T17:31:02.482 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql 10 2023-10-10T17:31:02.483 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:31:02.483 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2023-10-10T17:31:02.483 INFO:teuthology.orchestra.run.smithi190.stdout:=========================================================================================================================== 2023-10-10T17:31:02.483 INFO:teuthology.orchestra.run.smithi190.stdout:Install 128 Packages 2023-10-10T17:31:02.483 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 3 Packages 2023-10-10T17:31:02.483 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:31:02.484 INFO:teuthology.orchestra.run.smithi157.stdout:(1/131): ceph-18.0.0-3908.g05a43c8f.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-10-10T17:31:02.485 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 197 M 2023-10-10T17:31:02.485 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2023-10-10T17:31:02.587 INFO:teuthology.orchestra.run.smithi078.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 1.1 MB/s | 166 kB 00:00 2023-10-10T17:31:03.003 INFO:teuthology.orchestra.run.smithi190.stdout:(1/131): ceph-18.0.0-3908.g05a43c8f.el8.x86_64. 34 kB/s | 6.4 kB 00:00 2023-10-10T17:31:03.121 INFO:teuthology.orchestra.run.smithi078.stdout:(4/131): ceph-base-18.0.0-3908.g05a43c8f.el8.x8 2.9 MB/s | 5.3 MB 00:01 2023-10-10T17:31:03.304 INFO:teuthology.orchestra.run.smithi078.stdout:(5/131): ceph-mds-18.0.0-3908.g05a43c8f.el8.x86 3.2 MB/s | 2.3 MB 00:00 2023-10-10T17:31:03.355 INFO:teuthology.orchestra.run.smithi078.stdout:(6/131): ceph-mgr-18.0.0-3908.g05a43c8f.el8.x86 6.6 MB/s | 1.5 MB 00:00 2023-10-10T17:31:04.035 INFO:teuthology.orchestra.run.smithi157.stdout:(2/131): ceph-base-18.0.0-3908.g05a43c8f.el8.x8 3.0 MB/s | 5.3 MB 00:01 2023-10-10T17:31:04.068 INFO:teuthology.orchestra.run.smithi157.stdout:(3/131): ceph-fuse-18.0.0-3908.g05a43c8f.el8.x8 527 kB/s | 848 kB 00:01 2023-10-10T17:31:04.110 INFO:teuthology.orchestra.run.smithi157.stdout:(4/131): ceph-immutable-object-cache-18.0.0-390 2.2 MB/s | 166 kB 00:00 2023-10-10T17:31:04.113 INFO:teuthology.orchestra.run.smithi078.stdout:(7/131): ceph-mon-18.0.0-3908.g05a43c8f.el8.x86 5.7 MB/s | 4.6 MB 00:00 2023-10-10T17:31:04.519 INFO:teuthology.orchestra.run.smithi157.stdout:(5/131): ceph-mgr-18.0.0-3908.g05a43c8f.el8.x86 3.8 MB/s | 1.5 MB 00:00 2023-10-10T17:31:05.039 INFO:teuthology.orchestra.run.smithi078.stdout:(8/131): ceph-osd-18.0.0-3908.g05a43c8f.el8.x86 11 MB/s | 18 MB 00:01 2023-10-10T17:31:05.094 INFO:teuthology.orchestra.run.smithi190.stdout:(2/131): ceph-fuse-18.0.0-3908.g05a43c8f.el8.x8 406 kB/s | 848 kB 00:02 2023-10-10T17:31:05.114 INFO:teuthology.orchestra.run.smithi078.stdout:(9/131): ceph-selinux-18.0.0-3908.g05a43c8f.el8 327 kB/s | 24 kB 00:00 2023-10-10T17:31:05.311 INFO:teuthology.orchestra.run.smithi190.stdout:(3/131): ceph-immutable-object-cache-18.0.0-390 765 kB/s | 166 kB 00:00 2023-10-10T17:31:05.452 INFO:teuthology.orchestra.run.smithi157.stdout:(6/131): ceph-mon-18.0.0-3908.g05a43c8f.el8.x86 4.9 MB/s | 4.6 MB 00:00 2023-10-10T17:31:06.198 INFO:teuthology.orchestra.run.smithi078.stdout:(10/131): ceph-radosgw-18.0.0-3908.g05a43c8f.el 6.2 MB/s | 13 MB 00:02 2023-10-10T17:31:06.332 INFO:teuthology.orchestra.run.smithi078.stdout:(11/131): cephfs-mirror-18.0.0-3908.g05a43c8f.e 1.9 MB/s | 258 kB 00:00 2023-10-10T17:31:06.524 INFO:teuthology.orchestra.run.smithi078.stdout:(12/131): libcephfs-devel-18.0.0-3908.g05a43c8f 162 kB/s | 31 kB 00:00 2023-10-10T17:31:06.699 INFO:teuthology.orchestra.run.smithi078.stdout:(13/131): libcephfs2-18.0.0-3908.g05a43c8f.el8. 4.3 MB/s | 770 kB 00:00 2023-10-10T17:31:06.744 INFO:teuthology.orchestra.run.smithi157.stdout:(7/131): ceph-common-18.0.0-3908.g05a43c8f.el8. 4.8 MB/s | 22 MB 00:04 2023-10-10T17:31:06.799 INFO:teuthology.orchestra.run.smithi078.stdout:(14/131): libcephsqlite-18.0.0-3908.g05a43c8f.e 1.3 MB/s | 137 kB 00:00 2023-10-10T17:31:06.908 INFO:teuthology.orchestra.run.smithi078.stdout:(15/131): librados-devel-18.0.0-3908.g05a43c8f. 1.3 MB/s | 139 kB 00:00 2023-10-10T17:31:07.136 INFO:teuthology.orchestra.run.smithi190.stdout:(4/131): ceph-mds-18.0.0-3908.g05a43c8f.el8.x86 1.3 MB/s | 2.3 MB 00:01 2023-10-10T17:31:07.233 INFO:teuthology.orchestra.run.smithi078.stdout:(16/131): libradosstriper1-18.0.0-3908.g05a43c8 1.4 MB/s | 473 kB 00:00 2023-10-10T17:31:07.828 INFO:teuthology.orchestra.run.smithi190.stdout:(5/131): ceph-mgr-18.0.0-3908.g05a43c8f.el8.x86 2.2 MB/s | 1.5 MB 00:00 2023-10-10T17:31:08.750 INFO:teuthology.orchestra.run.smithi078.stdout:(17/131): librgw2-18.0.0-3908.g05a43c8f.el8.x86 4.3 MB/s | 6.5 MB 00:01 2023-10-10T17:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout:(18/131): python3-ceph-argparse-18.0.0-3908.g05 448 kB/s | 45 kB 00:00 2023-10-10T17:31:09.037 INFO:teuthology.orchestra.run.smithi190.stdout:(6/131): ceph-mon-18.0.0-3908.g05a43c8f.el8.x86 3.8 MB/s | 4.6 MB 00:01 2023-10-10T17:31:09.170 INFO:teuthology.orchestra.run.smithi157.stdout:(8/131): ceph-osd-18.0.0-3908.g05a43c8f.el8.x86 4.8 MB/s | 18 MB 00:03 2023-10-10T17:31:09.322 INFO:teuthology.orchestra.run.smithi157.stdout:(9/131): ceph-radosgw-18.0.0-3908.g05a43c8f.el8 5.1 MB/s | 13 MB 00:02 2023-10-10T17:31:09.378 INFO:teuthology.orchestra.run.smithi157.stdout:(10/131): ceph-mds-18.0.0-3908.g05a43c8f.el8.x8 440 kB/s | 2.3 MB 00:05 2023-10-10T17:31:09.395 INFO:teuthology.orchestra.run.smithi157.stdout:(11/131): ceph-selinux-18.0.0-3908.g05a43c8f.el 108 kB/s | 24 kB 00:00 2023-10-10T17:31:09.437 INFO:teuthology.orchestra.run.smithi157.stdout:(12/131): libcephfs-devel-18.0.0-3908.g05a43c8f 753 kB/s | 31 kB 00:00 2023-10-10T17:31:09.509 INFO:teuthology.orchestra.run.smithi078.stdout:(19/131): ceph-test-18.0.0-3908.g05a43c8f.el8.x 9.2 MB/s | 41 MB 00:04 2023-10-10T17:31:09.534 INFO:teuthology.orchestra.run.smithi078.stdout:(20/131): python3-ceph-common-18.0.0-3908.g05a4 173 kB/s | 119 kB 00:00 2023-10-10T17:31:09.612 INFO:teuthology.orchestra.run.smithi157.stdout:(13/131): libcephfs2-18.0.0-3908.g05a43c8f.el8. 4.3 MB/s | 770 kB 00:00 2023-10-10T17:31:09.643 INFO:teuthology.orchestra.run.smithi078.stdout:(21/131): python3-cephfs-18.0.0-3908.g05a43c8f. 1.4 MB/s | 198 kB 00:00 2023-10-10T17:31:09.718 INFO:teuthology.orchestra.run.smithi078.stdout:(22/131): python3-rados-18.0.0-3908.g05a43c8f.e 2.0 MB/s | 371 kB 00:00 2023-10-10T17:31:09.742 INFO:teuthology.orchestra.run.smithi157.stdout:(14/131): libcephsqlite-18.0.0-3908.g05a43c8f.e 1.0 MB/s | 137 kB 00:00 2023-10-10T17:31:09.771 INFO:teuthology.orchestra.run.smithi157.stdout:(15/131): cephfs-mirror-18.0.0-3908.g05a43c8f.e 687 kB/s | 258 kB 00:00 2023-10-10T17:31:09.794 INFO:teuthology.orchestra.run.smithi078.stdout:(23/131): python3-rbd-18.0.0-3908.g05a43c8f.el8 2.4 MB/s | 364 kB 00:00 2023-10-10T17:31:09.813 INFO:teuthology.orchestra.run.smithi157.stdout:(16/131): librados-devel-18.0.0-3908.g05a43c8f. 1.9 MB/s | 139 kB 00:00 2023-10-10T17:31:09.819 INFO:teuthology.orchestra.run.smithi078.stdout:(24/131): python3-rgw-18.0.0-3908.g05a43c8f.el8 1.0 MB/s | 106 kB 00:00 2023-10-10T17:31:09.903 INFO:teuthology.orchestra.run.smithi078.stdout:(25/131): rbd-fuse-18.0.0-3908.g05a43c8f.el8.x8 870 kB/s | 94 kB 00:00 2023-10-10T17:31:10.094 INFO:teuthology.orchestra.run.smithi078.stdout:(26/131): rbd-nbd-18.0.0-3908.g05a43c8f.el8.x86 936 kB/s | 179 kB 00:00 2023-10-10T17:31:10.211 INFO:teuthology.orchestra.run.smithi078.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3908.g 197 kB/s | 23 kB 00:00 2023-10-10T17:31:10.328 INFO:teuthology.orchestra.run.smithi078.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3908.g05a43c8 1.1 MB/s | 127 kB 00:00 2023-10-10T17:31:10.571 INFO:teuthology.orchestra.run.smithi078.stdout:(29/131): rbd-mirror-18.0.0-3908.g05a43c8f.el8. 4.6 MB/s | 3.4 MB 00:00 2023-10-10T17:31:10.573 INFO:teuthology.orchestra.run.smithi157.stdout:(17/131): libradosstriper1-18.0.0-3908.g05a43c8 590 kB/s | 473 kB 00:00 2023-10-10T17:31:10.595 INFO:teuthology.orchestra.run.smithi190.stdout:(7/131): ceph-osd-18.0.0-3908.g05a43c8f.el8.x86 11 MB/s | 18 MB 00:01 2023-10-10T17:31:10.763 INFO:teuthology.orchestra.run.smithi157.stdout:(18/131): python3-ceph-argparse-18.0.0-3908.g05 235 kB/s | 45 kB 00:00 2023-10-10T17:31:10.796 INFO:teuthology.orchestra.run.smithi078.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3908.g05a43 3.7 MB/s | 1.7 MB 00:00 2023-10-10T17:31:10.896 INFO:teuthology.orchestra.run.smithi078.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3908.g05 2.3 MB/s | 240 kB 00:00 2023-10-10T17:31:11.004 INFO:teuthology.orchestra.run.smithi078.stdout:(32/131): ceph-mgr-rook-18.0.0-3908.g05a43c8f.e 438 kB/s | 47 kB 00:00 2023-10-10T17:31:11.005 INFO:teuthology.orchestra.run.smithi157.stdout:(19/131): python3-ceph-common-18.0.0-3908.g05a4 491 kB/s | 119 kB 00:00 2023-10-10T17:31:11.088 INFO:teuthology.orchestra.run.smithi078.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3908.g0 174 kB/s | 15 kB 00:00 2023-10-10T17:31:11.181 INFO:teuthology.orchestra.run.smithi157.stdout:(20/131): librgw2-18.0.0-3908.g05a43c8f.el8.x86 4.8 MB/s | 6.5 MB 00:01 2023-10-10T17:31:11.230 INFO:teuthology.orchestra.run.smithi078.stdout:(34/131): ceph-volume-18.0.0-3908.g05a43c8f.el8 1.8 MB/s | 265 kB 00:00 2023-10-10T17:31:11.330 INFO:teuthology.orchestra.run.smithi078.stdout:(35/131): cephadm-18.0.0-3908.g05a43c8f.el8.noa 836 kB/s | 83 kB 00:00 2023-10-10T17:31:11.331 INFO:teuthology.orchestra.run.smithi157.stdout:(21/131): python3-rados-18.0.0-3908.g05a43c8f.e 2.4 MB/s | 371 kB 00:00 2023-10-10T17:31:11.356 INFO:teuthology.orchestra.run.smithi157.stdout:(22/131): python3-cephfs-18.0.0-3908.g05a43c8f. 564 kB/s | 198 kB 00:00 2023-10-10T17:31:11.414 INFO:teuthology.orchestra.run.smithi078.stdout:(36/131): cephfs-top-18.0.0-3908.g05a43c8f.el8. 215 kB/s | 18 kB 00:00 2023-10-10T17:31:11.482 INFO:teuthology.orchestra.run.smithi157.stdout:(23/131): python3-rbd-18.0.0-3908.g05a43c8f.el8 2.4 MB/s | 364 kB 00:00 2023-10-10T17:31:11.523 INFO:teuthology.orchestra.run.smithi157.stdout:(24/131): python3-rgw-18.0.0-3908.g05a43c8f.el8 638 kB/s | 106 kB 00:00 2023-10-10T17:31:11.557 INFO:teuthology.orchestra.run.smithi157.stdout:(25/131): rbd-fuse-18.0.0-3908.g05a43c8f.el8.x8 1.2 MB/s | 94 kB 00:00 2023-10-10T17:31:11.641 INFO:teuthology.orchestra.run.smithi157.stdout:(26/131): rbd-nbd-18.0.0-3908.g05a43c8f.el8.x86 2.1 MB/s | 179 kB 00:00 2023-10-10T17:31:11.648 INFO:teuthology.orchestra.run.smithi078.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-10-10T17:31:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-10-10T17:31:11.716 INFO:teuthology.orchestra.run.smithi157.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3908.g 307 kB/s | 23 kB 00:00 2023-10-10T17:31:11.745 INFO:teuthology.orchestra.run.smithi190.stdout:(8/131): ceph-base-18.0.0-3908.g05a43c8f.el8.x8 610 kB/s | 5.3 MB 00:08 2023-10-10T17:31:11.765 INFO:teuthology.orchestra.run.smithi078.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-10-10T17:31:11.797 INFO:teuthology.orchestra.run.smithi157.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3908.g05a43c8 1.6 MB/s | 127 kB 00:00 2023-10-10T17:31:11.866 INFO:teuthology.orchestra.run.smithi078.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.6 MB/s | 476 kB 00:00 2023-10-10T17:31:11.904 INFO:teuthology.orchestra.run.smithi190.stdout:(9/131): ceph-radosgw-18.0.0-3908.g05a43c8f.el8 10 MB/s | 13 MB 00:01 2023-10-10T17:31:11.929 INFO:teuthology.orchestra.run.smithi190.stdout:(10/131): ceph-selinux-18.0.0-3908.g05a43c8f.el 133 kB/s | 24 kB 00:00 2023-10-10T17:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout:(41/131): ceph-mgr-diskprediction-local-18.0.0- 5.3 MB/s | 7.4 MB 00:01 2023-10-10T17:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 326 kB/s | 44 kB 00:00 2023-10-10T17:31:12.058 INFO:teuthology.orchestra.run.smithi078.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 566 kB/s | 33 kB 00:00 2023-10-10T17:31:12.117 INFO:teuthology.orchestra.run.smithi078.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 280 kB/s | 16 kB 00:00 2023-10-10T17:31:12.150 INFO:teuthology.orchestra.run.smithi157.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3908.g05a43 4.8 MB/s | 1.7 MB 00:00 2023-10-10T17:31:12.175 INFO:teuthology.orchestra.run.smithi078.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 923 kB/s | 185 kB 00:00 2023-10-10T17:31:12.188 INFO:teuthology.orchestra.run.smithi190.stdout:(11/131): cephfs-mirror-18.0.0-3908.g05a43c8f.e 1.0 MB/s | 258 kB 00:00 2023-10-10T17:31:12.201 INFO:teuthology.orchestra.run.smithi078.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2023-10-10T17:31:12.276 INFO:teuthology.orchestra.run.smithi078.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.7 MB/s | 384 kB 00:00 2023-10-10T17:31:12.279 INFO:teuthology.orchestra.run.smithi190.stdout:(12/131): libcephfs-devel-18.0.0-3908.g05a43c8f 339 kB/s | 31 kB 00:00 2023-10-10T17:31:12.301 INFO:teuthology.orchestra.run.smithi078.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 810 kB/s | 81 kB 00:00 2023-10-10T17:31:12.335 INFO:teuthology.orchestra.run.smithi078.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2023-10-10T17:31:12.360 INFO:teuthology.orchestra.run.smithi078.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 299 kB/s | 18 kB 00:00 2023-10-10T17:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 650 kB/s | 49 kB 00:00 2023-10-10T17:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout:(52/131): python3-kubernetes-11.0.0-6.el8.noarc 7.3 MB/s | 1.2 MB 00:00 2023-10-10T17:31:12.528 INFO:teuthology.orchestra.run.smithi078.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 638 kB/s | 59 kB 00:00 2023-10-10T17:31:12.595 INFO:teuthology.orchestra.run.smithi078.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 601 kB/s | 60 kB 00:00 2023-10-10T17:31:12.687 INFO:teuthology.orchestra.run.smithi078.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2023-10-10T17:31:12.762 INFO:teuthology.orchestra.run.smithi078.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 450 kB/s | 34 kB 00:00 2023-10-10T17:31:13.162 INFO:teuthology.orchestra.run.smithi078.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 446 kB/s | 283 kB 00:00 2023-10-10T17:31:13.167 INFO:teuthology.orchestra.run.smithi157.stdout:(30/131): ceph-mgr-diskprediction-local-18.0.0- 7.3 MB/s | 7.4 MB 00:01 2023-10-10T17:31:13.212 INFO:teuthology.orchestra.run.smithi078.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 436 kB/s | 196 kB 00:00 2023-10-10T17:31:13.226 INFO:teuthology.orchestra.run.smithi157.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3908.g05 4.1 MB/s | 240 kB 00:00 2023-10-10T17:31:13.229 INFO:teuthology.orchestra.run.smithi078.stdout:(59/131): python3-rsa-4.9-2.el8.noarch.rpm 962 kB/s | 64 kB 00:00 2023-10-10T17:31:13.276 INFO:teuthology.orchestra.run.smithi157.stdout:(32/131): ceph-mgr-rook-18.0.0-3908.g05a43c8f.e 949 kB/s | 47 kB 00:00 2023-10-10T17:31:13.280 INFO:teuthology.orchestra.run.smithi078.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 280 kB/s | 19 kB 00:00 2023-10-10T17:31:13.305 INFO:teuthology.orchestra.run.smithi078.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 318 kB/s | 24 kB 00:00 2023-10-10T17:31:13.318 INFO:teuthology.orchestra.run.smithi157.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3908.g0 349 kB/s | 15 kB 00:00 2023-10-10T17:31:13.346 INFO:teuthology.orchestra.run.smithi190.stdout:(13/131): libcephfs2-18.0.0-3908.g05a43c8f.el8. 722 kB/s | 770 kB 00:01 2023-10-10T17:31:13.347 INFO:teuthology.orchestra.run.smithi078.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 436 kB/s | 29 kB 00:00 2023-10-10T17:31:13.377 INFO:teuthology.orchestra.run.smithi157.stdout:(34/131): ceph-volume-18.0.0-3908.g05a43c8f.el8 4.4 MB/s | 265 kB 00:00 2023-10-10T17:31:13.381 INFO:teuthology.orchestra.run.smithi078.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 364 kB/s | 27 kB 00:00 2023-10-10T17:31:13.406 INFO:teuthology.orchestra.run.smithi078.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 4.2 MB/s | 255 kB 00:00 2023-10-10T17:31:13.419 INFO:teuthology.orchestra.run.smithi157.stdout:(35/131): cephadm-18.0.0-3908.g05a43c8f.el8.noa 2.0 MB/s | 83 kB 00:00 2023-10-10T17:31:13.449 INFO:teuthology.orchestra.run.smithi078.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 3.7 MB/s | 251 kB 00:00 2023-10-10T17:31:13.490 INFO:teuthology.orchestra.run.smithi157.stdout:(36/131): cephfs-top-18.0.0-3908.g05a43c8f.el8. 252 kB/s | 18 kB 00:00 2023-10-10T17:31:13.513 INFO:teuthology.orchestra.run.smithi190.stdout:(14/131): libcephsqlite-18.0.0-3908.g05a43c8f.e 824 kB/s | 137 kB 00:00 2023-10-10T17:31:13.515 INFO:teuthology.orchestra.run.smithi078.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 1.3 MB/s | 85 kB 00:00 2023-10-10T17:31:13.582 INFO:teuthology.orchestra.run.smithi078.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 345 kB/s | 23 kB 00:00 2023-10-10T17:31:13.636 INFO:teuthology.orchestra.run.smithi157.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 2.1 MB/s | 306 kB 00:00 2023-10-10T17:31:13.661 INFO:teuthology.orchestra.run.smithi157.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 2.3 MB/s | 59 kB 00:00 2023-10-10T17:31:13.663 INFO:teuthology.orchestra.run.smithi190.stdout:(15/131): librados-devel-18.0.0-3908.g05a43c8f. 928 kB/s | 139 kB 00:00 2023-10-10T17:31:13.695 INFO:teuthology.orchestra.run.smithi157.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 2.2 MB/s | 75 kB 00:00 2023-10-10T17:31:13.745 INFO:teuthology.orchestra.run.smithi157.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 9.3 MB/s | 476 kB 00:00 2023-10-10T17:31:13.766 INFO:teuthology.orchestra.run.smithi078.stdout:(68/131): thrift-0.13.0-2.el8.x86_64.rpm 9.5 MB/s | 1.7 MB 00:00 2023-10-10T17:31:13.770 INFO:teuthology.orchestra.run.smithi157.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.7 MB/s | 44 kB 00:00 2023-10-10T17:31:13.791 INFO:teuthology.orchestra.run.smithi078.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-10-10T17:31:13.804 INFO:teuthology.orchestra.run.smithi157.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 5.4 MB/s | 185 kB 00:00 2023-10-10T17:31:13.830 INFO:teuthology.orchestra.run.smithi157.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 1.3 MB/s | 33 kB 00:00 2023-10-10T17:31:13.872 INFO:teuthology.orchestra.run.smithi157.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 392 kB/s | 16 kB 00:00 2023-10-10T17:31:13.930 INFO:teuthology.orchestra.run.smithi157.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2023-10-10T17:31:13.972 INFO:teuthology.orchestra.run.smithi157.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 9.0 MB/s | 384 kB 00:00 2023-10-10T17:31:14.006 INFO:teuthology.orchestra.run.smithi157.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 2.4 MB/s | 81 kB 00:00 2023-10-10T17:31:14.022 INFO:teuthology.orchestra.run.smithi190.stdout:(16/131): libradosstriper1-18.0.0-3908.g05a43c8 1.3 MB/s | 473 kB 00:00 2023-10-10T17:31:14.039 INFO:teuthology.orchestra.run.smithi157.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 336 kB/s | 11 kB 00:00 2023-10-10T17:31:14.073 INFO:teuthology.orchestra.run.smithi157.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 528 kB/s | 18 kB 00:00 2023-10-10T17:31:14.133 INFO:teuthology.orchestra.run.smithi078.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 40 MB/s | 14 MB 00:00 2023-10-10T17:31:14.157 INFO:teuthology.orchestra.run.smithi157.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 14 MB/s | 1.2 MB 00:00 2023-10-10T17:31:14.191 INFO:teuthology.orchestra.run.smithi157.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 1.4 MB/s | 49 kB 00:00 2023-10-10T17:31:14.193 INFO:teuthology.orchestra.run.smithi078.stdout:(71/131): python3-websocket-client-0.56.0-5.el8 78 kB/s | 61 kB 00:00 2023-10-10T17:31:14.225 INFO:teuthology.orchestra.run.smithi157.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.7 MB/s | 59 kB 00:00 2023-10-10T17:31:14.258 INFO:teuthology.orchestra.run.smithi157.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.8 MB/s | 60 kB 00:00 2023-10-10T17:31:14.329 INFO:teuthology.orchestra.run.smithi157.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.9 MB/s | 283 kB 00:00 2023-10-10T17:31:14.367 INFO:teuthology.orchestra.run.smithi157.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 415 kB/s | 16 kB 00:00 2023-10-10T17:31:14.401 INFO:teuthology.orchestra.run.smithi157.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 1.0 MB/s | 34 kB 00:00 2023-10-10T17:31:14.442 INFO:teuthology.orchestra.run.smithi157.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 4.6 MB/s | 196 kB 00:00 2023-10-10T17:31:14.476 INFO:teuthology.orchestra.run.smithi157.stdout:(58/131): python3-rsa-4.9-2.el8.noarch.rpm 1.9 MB/s | 64 kB 00:00 2023-10-10T17:31:14.483 INFO:teuthology.orchestra.run.smithi078.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 186 kB/s | 54 kB 00:00 2023-10-10T17:31:14.509 INFO:teuthology.orchestra.run.smithi078.stdout:(73/131): python3-pyasn1-modules-0.3.7-6.el8.no 293 kB/s | 110 kB 00:00 2023-10-10T17:31:14.510 INFO:teuthology.orchestra.run.smithi157.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 562 kB/s | 19 kB 00:00 2023-10-10T17:31:14.543 INFO:teuthology.orchestra.run.smithi157.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 721 kB/s | 24 kB 00:00 2023-10-10T17:31:14.577 INFO:teuthology.orchestra.run.smithi157.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 877 kB/s | 29 kB 00:00 2023-10-10T17:31:14.611 INFO:teuthology.orchestra.run.smithi157.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 828 kB/s | 27 kB 00:00 2023-10-10T17:31:14.653 INFO:teuthology.orchestra.run.smithi157.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 6.0 MB/s | 255 kB 00:00 2023-10-10T17:31:14.684 INFO:teuthology.orchestra.run.smithi078.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2023-10-10T17:31:14.695 INFO:teuthology.orchestra.run.smithi157.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 5.9 MB/s | 251 kB 00:00 2023-10-10T17:31:14.762 INFO:teuthology.orchestra.run.smithi157.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 921 kB/s | 61 kB 00:00 2023-10-10T17:31:14.795 INFO:teuthology.orchestra.run.smithi157.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 2.5 MB/s | 85 kB 00:00 2023-10-10T17:31:14.829 INFO:teuthology.orchestra.run.smithi157.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 689 kB/s | 23 kB 00:00 2023-10-10T17:31:14.869 INFO:teuthology.orchestra.run.smithi078.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 901 kB/s | 323 kB 00:00 2023-10-10T17:31:14.904 INFO:teuthology.orchestra.run.smithi157.stdout:(68/131): thrift-0.13.0-2.el8.x86_64.rpm 23 MB/s | 1.7 MB 00:00 2023-10-10T17:31:14.930 INFO:teuthology.orchestra.run.smithi157.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-10-10T17:31:14.969 INFO:teuthology.orchestra.run.smithi078.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 137 kB/s | 39 kB 00:00 2023-10-10T17:31:15.194 INFO:teuthology.orchestra.run.smithi078.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 2.3 MB/s | 538 kB 00:00 2023-10-10T17:31:15.253 INFO:teuthology.orchestra.run.smithi078.stdout:(78/131): bison-3.0.4-10.el8.x86_64.rpm 1.8 MB/s | 688 kB 00:00 2023-10-10T17:31:15.313 INFO:teuthology.orchestra.run.smithi157.stdout:(70/131): ceph-test-18.0.0-3908.g05a43c8f.el8.x 6.8 MB/s | 41 MB 00:05 2023-10-10T17:31:15.380 INFO:teuthology.orchestra.run.smithi157.stdout:(71/131): rbd-mirror-18.0.0-3908.g05a43c8f.el8. 916 kB/s | 3.4 MB 00:03 2023-10-10T17:31:15.486 INFO:teuthology.orchestra.run.smithi078.stdout:(79/131): oniguruma-6.8.2-2.el8.x86_64.rpm 803 kB/s | 187 kB 00:00 2023-10-10T17:31:15.497 INFO:teuthology.orchestra.run.smithi157.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 489 kB/s | 54 kB 00:00 2023-10-10T17:31:15.511 INFO:teuthology.orchestra.run.smithi078.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 494 kB/s | 157 kB 00:00 2023-10-10T17:31:15.539 INFO:teuthology.orchestra.run.smithi157.stdout:(73/131): python3-pyasn1-modules-0.3.7-6.el8.no 501 kB/s | 110 kB 00:00 2023-10-10T17:31:15.581 INFO:teuthology.orchestra.run.smithi157.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.0 MB/s | 259 kB 00:00 2023-10-10T17:31:15.723 INFO:teuthology.orchestra.run.smithi157.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 274 kB/s | 39 kB 00:00 2023-10-10T17:31:15.757 INFO:teuthology.orchestra.run.smithi157.stdout:(76/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.5 MB/s | 323 kB 00:00 2023-10-10T17:31:15.762 INFO:teuthology.orchestra.run.smithi078.stdout:(81/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-10-10T17:31:15.795 INFO:teuthology.orchestra.run.smithi078.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 2.3 MB/s | 713 kB 00:00 2023-10-10T17:31:15.924 INFO:teuthology.orchestra.run.smithi157.stdout:(77/131): python3-scipy-1.0.0-21.module+el8.5.0 14 MB/s | 14 MB 00:00 2023-10-10T17:31:15.966 INFO:teuthology.orchestra.run.smithi157.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 2.5 MB/s | 538 kB 00:00 2023-10-10T17:31:15.989 INFO:teuthology.orchestra.run.smithi190.stdout:(17/131): ceph-common-18.0.0-3908.g05a43c8f.el8 1.6 MB/s | 22 MB 00:13 2023-10-10T17:31:15.991 INFO:teuthology.orchestra.run.smithi157.stdout:(79/131): bison-3.0.4-10.el8.x86_64.rpm 2.5 MB/s | 688 kB 00:00 2023-10-10T17:31:16.021 INFO:teuthology.orchestra.run.smithi078.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 1.4 MB/s | 320 kB 00:00 2023-10-10T17:31:16.046 INFO:teuthology.orchestra.run.smithi078.stdout:(84/131): python3-jsonpatch-1.21-2.el8.noarch.r 93 kB/s | 27 kB 00:00 2023-10-10T17:31:16.048 INFO:teuthology.orchestra.run.smithi190.stdout:(18/131): python3-ceph-argparse-18.0.0-3908.g05 772 kB/s | 45 kB 00:00 2023-10-10T17:31:16.058 INFO:teuthology.orchestra.run.smithi157.stdout:(80/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2023-10-10T17:31:16.115 INFO:teuthology.orchestra.run.smithi190.stdout:(19/131): python3-ceph-common-18.0.0-3908.g05a4 1.7 MB/s | 119 kB 00:00 2023-10-10T17:31:16.133 INFO:teuthology.orchestra.run.smithi157.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-10-10T17:31:16.167 INFO:teuthology.orchestra.run.smithi157.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 4.0 MB/s | 713 kB 00:00 2023-10-10T17:31:16.205 INFO:teuthology.orchestra.run.smithi078.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 239 kB/s | 44 kB 00:00 2023-10-10T17:31:16.275 INFO:teuthology.orchestra.run.smithi157.stdout:(83/131): python3-babel-2.5.1-7.el8.noarch.rpm 22 MB/s | 4.8 MB 00:00 2023-10-10T17:31:16.301 INFO:teuthology.orchestra.run.smithi157.stdout:(84/131): python3-jsonpatch-1.21-2.el8.noarch.r 159 kB/s | 27 kB 00:00 2023-10-10T17:31:16.306 INFO:teuthology.orchestra.run.smithi078.stdout:(86/131): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2023-10-10T17:31:16.326 INFO:teuthology.orchestra.run.smithi157.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 878 kB/s | 44 kB 00:00 2023-10-10T17:31:16.351 INFO:teuthology.orchestra.run.smithi157.stdout:(86/131): flex-2.6.1-9.el8.x86_64.rpm 1.7 MB/s | 320 kB 00:00 2023-10-10T17:31:16.447 INFO:teuthology.orchestra.run.smithi078.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 1.8 MB/s | 457 kB 00:00 2023-10-10T17:31:16.468 INFO:teuthology.orchestra.run.smithi157.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 22 MB/s | 3.7 MB 00:00 2023-10-10T17:31:16.532 INFO:teuthology.orchestra.run.smithi190.stdout:(20/131): ceph-test-18.0.0-3908.g05a43c8f.el8.x 8.8 MB/s | 41 MB 00:04 2023-10-10T17:31:16.543 INFO:teuthology.orchestra.run.smithi157.stdout:(88/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.1 MB/s | 457 kB 00:00 2023-10-10T17:31:16.556 INFO:teuthology.orchestra.run.smithi190.stdout:(21/131): python3-cephfs-18.0.0-3908.g05a43c8f. 447 kB/s | 198 kB 00:00 2023-10-10T17:31:16.632 INFO:teuthology.orchestra.run.smithi190.stdout:(22/131): python3-rados-18.0.0-3908.g05a43c8f.e 3.6 MB/s | 371 kB 00:00 2023-10-10T17:31:16.713 INFO:teuthology.orchestra.run.smithi157.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 264 kB/s | 64 kB 00:00 2023-10-10T17:31:16.715 INFO:teuthology.orchestra.run.smithi190.stdout:(23/131): python3-rbd-18.0.0-3908.g05a43c8f.el8 2.2 MB/s | 364 kB 00:00 2023-10-10T17:31:16.741 INFO:teuthology.orchestra.run.smithi190.stdout:(24/131): python3-rgw-18.0.0-3908.g05a43c8f.el8 981 kB/s | 106 kB 00:00 2023-10-10T17:31:16.766 INFO:teuthology.orchestra.run.smithi190.stdout:(25/131): rbd-fuse-18.0.0-3908.g05a43c8f.el8.x8 1.8 MB/s | 94 kB 00:00 2023-10-10T17:31:16.789 INFO:teuthology.orchestra.run.smithi078.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 465 kB/s | 225 kB 00:00 2023-10-10T17:31:16.794 INFO:teuthology.orchestra.run.smithi157.stdout:(90/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 509 kB/s | 225 kB 00:00 2023-10-10T17:31:16.814 INFO:teuthology.orchestra.run.smithi078.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 174 kB/s | 64 kB 00:00 2023-10-10T17:31:16.825 INFO:teuthology.orchestra.run.smithi190.stdout:(26/131): rbd-nbd-18.0.0-3908.g05a43c8f.el8.x86 3.0 MB/s | 179 kB 00:00 2023-10-10T17:31:16.884 INFO:teuthology.orchestra.run.smithi190.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3908.g 397 kB/s | 23 kB 00:00 2023-10-10T17:31:16.942 INFO:teuthology.orchestra.run.smithi190.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3908.g05a43c8 2.1 MB/s | 127 kB 00:00 2023-10-10T17:31:16.948 INFO:teuthology.orchestra.run.smithi078.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 798 kB/s | 126 kB 00:00 2023-10-10T17:31:16.952 INFO:teuthology.orchestra.run.smithi157.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 309 kB/s | 126 kB 00:00 2023-10-10T17:31:16.973 INFO:teuthology.orchestra.run.smithi078.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 126 kB/s | 20 kB 00:00 2023-10-10T17:31:16.978 INFO:teuthology.orchestra.run.smithi157.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 563 kB/s | 103 kB 00:00 2023-10-10T17:31:17.028 INFO:teuthology.orchestra.run.smithi157.stdout:(93/131): python3-jsonpointer-1.10-11.el8.noarc 63 kB/s | 20 kB 00:00 2023-10-10T17:31:17.040 INFO:teuthology.orchestra.run.smithi078.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.1 MB/s | 103 kB 00:00 2023-10-10T17:31:17.201 INFO:teuthology.orchestra.run.smithi190.stdout:(29/131): rbd-mirror-18.0.0-3908.g05a43c8f.el8. 7.5 MB/s | 3.4 MB 00:00 2023-10-10T17:31:17.249 INFO:teuthology.orchestra.run.smithi078.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 2.5 MB/s | 709 kB 00:00 2023-10-10T17:31:17.253 INFO:teuthology.orchestra.run.smithi157.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 880 kB/s | 198 kB 00:00 2023-10-10T17:31:17.276 INFO:teuthology.orchestra.run.smithi190.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3908.g05a43 5.1 MB/s | 1.7 MB 00:00 2023-10-10T17:31:17.295 INFO:teuthology.orchestra.run.smithi157.stdout:(95/131): libtool-2.4.6-25.el8.x86_64.rpm 2.0 MB/s | 709 kB 00:00 2023-10-10T17:31:17.341 INFO:teuthology.orchestra.run.smithi078.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2023-10-10T17:31:17.351 INFO:teuthology.orchestra.run.smithi190.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3908.g05 3.1 MB/s | 240 kB 00:00 2023-10-10T17:31:17.412 INFO:teuthology.orchestra.run.smithi157.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 11 MB/s | 4.9 MB 00:00 2023-10-10T17:31:17.438 INFO:teuthology.orchestra.run.smithi157.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.4 MB/s | 202 kB 00:00 2023-10-10T17:31:17.443 INFO:teuthology.orchestra.run.smithi190.stdout:(32/131): librgw2-18.0.0-3908.g05a43c8f.el8.x86 1.9 MB/s | 6.5 MB 00:03 2023-10-10T17:31:17.468 INFO:teuthology.orchestra.run.smithi190.stdout:(33/131): ceph-mgr-rook-18.0.0-3908.g05a43c8f.e 405 kB/s | 47 kB 00:00 2023-10-10T17:31:17.500 INFO:teuthology.orchestra.run.smithi078.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 11 MB/s | 4.9 MB 00:00 2023-10-10T17:31:17.502 INFO:teuthology.orchestra.run.smithi190.stdout:(34/131): ceph-prometheus-alerts-18.0.0-3908.g0 249 kB/s | 15 kB 00:00 2023-10-10T17:31:17.544 INFO:teuthology.orchestra.run.smithi190.stdout:(35/131): ceph-volume-18.0.0-3908.g05a43c8f.el8 3.4 MB/s | 265 kB 00:00 2023-10-10T17:31:17.569 INFO:teuthology.orchestra.run.smithi190.stdout:(36/131): cephadm-18.0.0-3908.g05a43c8f.el8.noa 1.2 MB/s | 83 kB 00:00 2023-10-10T17:31:17.596 INFO:teuthology.orchestra.run.smithi157.stdout:(98/131): openblas-0.3.15-3.el8.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2023-10-10T17:31:17.608 INFO:teuthology.orchestra.run.smithi078.stdout:(96/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-10-10T17:31:17.611 INFO:teuthology.orchestra.run.smithi190.stdout:(37/131): cephfs-top-18.0.0-3908.g05a43c8f.el8. 269 kB/s | 18 kB 00:00 2023-10-10T17:31:17.661 INFO:teuthology.orchestra.run.smithi078.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-10-10T17:31:17.671 INFO:teuthology.orchestra.run.smithi157.stdout:(99/131): librdkafka-0.11.4-3.el8.x86_64.rpm 1.3 MB/s | 354 kB 00:00 2023-10-10T17:31:17.750 INFO:teuthology.orchestra.run.smithi078.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-10-10T17:31:17.867 INFO:teuthology.orchestra.run.smithi078.stdout:(99/131): postgresql-server-10.21-2.module+el8. 24 MB/s | 5.1 MB 00:00 2023-10-10T17:31:17.901 INFO:teuthology.orchestra.run.smithi078.stdout:(100/131): postgresql-contrib-10.21-2.module+el 5.2 MB/s | 808 kB 00:00 2023-10-10T17:31:17.976 INFO:teuthology.orchestra.run.smithi078.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 14 MB/s | 1.5 MB 00:00 2023-10-10T17:31:18.013 INFO:teuthology.orchestra.run.smithi157.stdout:(100/131): postgresql-server-10.21-2.module+el8 8.8 MB/s | 5.1 MB 00:00 2023-10-10T17:31:18.097 INFO:teuthology.orchestra.run.smithi157.stdout:(101/131): postgresql-contrib-10.21-2.module+el 1.6 MB/s | 808 kB 00:00 2023-10-10T17:31:18.111 INFO:teuthology.orchestra.run.smithi190.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 8.1 MB/s | 7.4 MB 00:00 2023-10-10T17:31:18.126 INFO:teuthology.orchestra.run.smithi078.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 208 kB/s | 47 kB 00:00 2023-10-10T17:31:18.137 INFO:teuthology.orchestra.run.smithi190.stdout:(39/131): liboath-2.6.2-3.el8.x86_64.rpm 112 kB/s | 59 kB 00:00 2023-10-10T17:31:18.147 INFO:teuthology.orchestra.run.smithi157.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 3.2 MB/s | 1.5 MB 00:00 2023-10-10T17:31:18.172 INFO:teuthology.orchestra.run.smithi157.stdout:(103/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 295 kB/s | 47 kB 00:00 2023-10-10T17:31:18.231 INFO:teuthology.orchestra.run.smithi157.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.5 MB/s | 200 kB 00:00 2023-10-10T17:31:18.295 INFO:teuthology.orchestra.run.smithi190.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 422 kB/s | 306 kB 00:00 2023-10-10T17:31:18.373 INFO:teuthology.orchestra.run.smithi157.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 189 kB/s | 43 kB 00:00 2023-10-10T17:31:18.377 INFO:teuthology.orchestra.run.smithi078.stdout:(103/131): ceph-common-18.0.0-3908.g05a43c8f.el 1.3 MB/s | 22 MB 00:17 2023-10-10T17:31:18.394 INFO:teuthology.orchestra.run.smithi078.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 160 kB/s | 43 kB 00:00 2023-10-10T17:31:18.406 INFO:teuthology.orchestra.run.smithi157.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 943 kB/s | 220 kB 00:00 2023-10-10T17:31:18.412 INFO:teuthology.orchestra.run.smithi190.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 376 kB/s | 44 kB 00:00 2023-10-10T17:31:18.427 INFO:teuthology.orchestra.run.smithi078.stdout:(105/131): libselinux-devel-2.9-5.el8.x86_64.rp 443 kB/s | 200 kB 00:00 2023-10-10T17:31:18.432 INFO:teuthology.orchestra.run.smithi157.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 239 kB/s | 48 kB 00:00 2023-10-10T17:31:18.504 INFO:teuthology.orchestra.run.smithi190.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.3 MB/s | 476 kB 00:00 2023-10-10T17:31:18.536 INFO:teuthology.orchestra.run.smithi078.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 339 kB/s | 48 kB 00:00 2023-10-10T17:31:18.554 INFO:teuthology.orchestra.run.smithi190.stdout:(43/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.3 MB/s | 185 kB 00:00 2023-10-10T17:31:18.563 INFO:teuthology.orchestra.run.smithi078.stdout:(107/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.2 MB/s | 220 kB 00:00 2023-10-10T17:31:18.580 INFO:teuthology.orchestra.run.smithi190.stdout:(44/131): libunwind-1.3.1-3.el8.x86_64.rpm 160 kB/s | 75 kB 00:00 2023-10-10T17:31:18.613 INFO:teuthology.orchestra.run.smithi190.stdout:(45/131): python3-cachetools-3.1.1-4.el8.noarch 304 kB/s | 33 kB 00:00 2023-10-10T17:31:18.623 INFO:teuthology.orchestra.run.smithi157.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 198 kB/s | 43 kB 00:00 2023-10-10T17:31:18.657 INFO:teuthology.orchestra.run.smithi157.stdout:(109/131): libsepol-devel-2.9-3.el8.x86_64.rpm 306 kB/s | 87 kB 00:00 2023-10-10T17:31:18.678 INFO:teuthology.orchestra.run.smithi078.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 301 kB/s | 43 kB 00:00 2023-10-10T17:31:18.691 INFO:teuthology.orchestra.run.smithi157.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 422 kB/s | 109 kB 00:00 2023-10-10T17:31:18.704 INFO:teuthology.orchestra.run.smithi078.stdout:(109/131): libsepol-devel-2.9-3.el8.x86_64.rpm 315 kB/s | 87 kB 00:00 2023-10-10T17:31:18.716 INFO:teuthology.orchestra.run.smithi157.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.4 MB/s | 229 kB 00:00 2023-10-10T17:31:18.729 INFO:teuthology.orchestra.run.smithi078.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 652 kB/s | 109 kB 00:00 2023-10-10T17:31:18.738 INFO:teuthology.orchestra.run.smithi190.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 89 kB/s | 16 kB 00:00 2023-10-10T17:31:18.750 INFO:teuthology.orchestra.run.smithi157.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 5.9 MB/s | 560 kB 00:00 2023-10-10T17:31:18.754 INFO:teuthology.orchestra.run.smithi078.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.0 MB/s | 229 kB 00:00 2023-10-10T17:31:18.838 INFO:teuthology.orchestra.run.smithi078.stdout:(112/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.1 MB/s | 560 kB 00:00 2023-10-10T17:31:18.859 INFO:teuthology.orchestra.run.smithi157.stdout:(113/131): libaio-devel-0.3.112-1.el8.x86_64.rp 112 kB/s | 19 kB 00:00 2023-10-10T17:31:18.863 INFO:teuthology.orchestra.run.smithi078.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.4 MB/s | 155 kB 00:00 2023-10-10T17:31:18.880 INFO:teuthology.orchestra.run.smithi078.stdout:(114/131): libaio-devel-0.3.112-1.el8.x86_64.rp 124 kB/s | 19 kB 00:00 2023-10-10T17:31:18.892 INFO:teuthology.orchestra.run.smithi157.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 3.9 MB/s | 559 kB 00:00 2023-10-10T17:31:18.955 INFO:teuthology.orchestra.run.smithi078.stdout:(115/131): python3-cryptography-3.2.1-5.el8.x86 4.7 MB/s | 559 kB 00:00 2023-10-10T17:31:18.959 INFO:teuthology.orchestra.run.smithi157.stdout:(115/131): python3-oauthlib-2.1.0-1.el8.noarch. 640 kB/s | 155 kB 00:00 2023-10-10T17:31:18.989 INFO:teuthology.orchestra.run.smithi190.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.0 MB/s | 384 kB 00:00 2023-10-10T17:31:18.990 INFO:teuthology.orchestra.run.smithi078.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 142 kB/s | 18 kB 00:00 2023-10-10T17:31:19.001 INFO:teuthology.orchestra.run.smithi157.stdout:(116/131): libverto-devel-0.3.0-5.el8.x86_64.rp 126 kB/s | 18 kB 00:00 2023-10-10T17:31:19.031 INFO:teuthology.orchestra.run.smithi078.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.9 MB/s | 605 kB 00:00 2023-10-10T17:31:19.039 INFO:teuthology.orchestra.run.smithi190.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 376 kB/s | 173 kB 00:00 2023-10-10T17:31:19.060 INFO:teuthology.orchestra.run.smithi157.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 3.5 MB/s | 605 kB 00:00 2023-10-10T17:31:19.064 INFO:teuthology.orchestra.run.smithi190.stdout:(49/131): python3-google-auth-1.1.1-10.el8.noar 250 kB/s | 81 kB 00:00 2023-10-10T17:31:19.089 INFO:teuthology.orchestra.run.smithi190.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-10-10T17:31:19.140 INFO:teuthology.orchestra.run.smithi190.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 175 kB/s | 18 kB 00:00 2023-10-10T17:31:19.165 INFO:teuthology.orchestra.run.smithi078.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-10-10T17:31:19.177 INFO:teuthology.orchestra.run.smithi157.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 620 kB/s | 72 kB 00:00 2023-10-10T17:31:19.190 INFO:teuthology.orchestra.run.smithi078.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 456 kB/s | 72 kB 00:00 2023-10-10T17:31:19.190 INFO:teuthology.orchestra.run.smithi190.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 485 kB/s | 49 kB 00:00 2023-10-10T17:31:19.210 INFO:teuthology.orchestra.run.smithi157.stdout:(119/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 947 kB/s | 238 kB 00:00 2023-10-10T17:31:19.249 INFO:teuthology.orchestra.run.smithi190.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 541 kB/s | 59 kB 00:00 2023-10-10T17:31:19.265 INFO:teuthology.orchestra.run.smithi078.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 8.4 MB/s | 2.3 MB 00:00 2023-10-10T17:31:19.269 INFO:teuthology.orchestra.run.smithi157.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 8.7 MB/s | 2.3 MB 00:00 2023-10-10T17:31:19.291 INFO:teuthology.orchestra.run.smithi078.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 597 kB/s | 60 kB 00:00 2023-10-10T17:31:19.294 INFO:teuthology.orchestra.run.smithi157.stdout:(121/131): fuse-2.9.7-15.el8.x86_64.rpm 705 kB/s | 83 kB 00:00 2023-10-10T17:31:19.299 INFO:teuthology.orchestra.run.smithi190.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 556 kB/s | 60 kB 00:00 2023-10-10T17:31:19.316 INFO:teuthology.orchestra.run.smithi078.stdout:(122/131): fuse-2.9.7-15.el8.x86_64.rpm 549 kB/s | 83 kB 00:00 2023-10-10T17:31:19.341 INFO:teuthology.orchestra.run.smithi078.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.3 MB/s | 98 kB 00:00 2023-10-10T17:31:19.375 INFO:teuthology.orchestra.run.smithi078.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 689 kB/s | 58 kB 00:00 2023-10-10T17:31:19.377 INFO:teuthology.orchestra.run.smithi157.stdout:(122/131): elfutils-libelf-devel-0.186-1.el8.x8 358 kB/s | 60 kB 00:00 2023-10-10T17:31:19.399 INFO:teuthology.orchestra.run.smithi190.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 156 kB/s | 16 kB 00:00 2023-10-10T17:31:19.408 INFO:teuthology.orchestra.run.smithi078.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 419 kB/s | 39 kB 00:00 2023-10-10T17:31:19.411 INFO:teuthology.orchestra.run.smithi157.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 493 kB/s | 58 kB 00:00 2023-10-10T17:31:19.425 INFO:teuthology.orchestra.run.smithi190.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.6 MB/s | 283 kB 00:00 2023-10-10T17:31:19.450 INFO:teuthology.orchestra.run.smithi078.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.2 MB/s | 171 kB 00:00 2023-10-10T17:31:19.458 INFO:teuthology.orchestra.run.smithi190.stdout:(57/131): python3-kubernetes-11.0.0-6.el8.noarc 3.0 MB/s | 1.2 MB 00:00 2023-10-10T17:31:19.470 INFO:teuthology.orchestra.run.smithi157.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 419 kB/s | 39 kB 00:00 2023-10-10T17:31:19.495 INFO:teuthology.orchestra.run.smithi157.stdout:(125/131): libuuid-devel-2.32.1-35.el8.x86_64.r 434 kB/s | 98 kB 00:00 2023-10-10T17:31:19.500 INFO:teuthology.orchestra.run.smithi190.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 334 kB/s | 34 kB 00:00 2023-10-10T17:31:19.517 INFO:teuthology.orchestra.run.smithi078.stdout:(127/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.6 MB/s | 644 kB 00:00 2023-10-10T17:31:19.554 INFO:teuthology.orchestra.run.smithi157.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.0 MB/s | 171 kB 00:00 2023-10-10T17:31:19.559 INFO:teuthology.orchestra.run.smithi190.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.4 MB/s | 196 kB 00:00 2023-10-10T17:31:19.559 INFO:teuthology.orchestra.run.smithi078.stdout:(128/131): NetworkManager-initscripts-updown-1. 914 kB/s | 138 kB 00:00 2023-10-10T17:31:19.571 INFO:teuthology.orchestra.run.smithi157.stdout:(127/131): NetworkManager-initscripts-updown-1. 1.8 MB/s | 138 kB 00:00 2023-10-10T17:31:19.585 INFO:teuthology.orchestra.run.smithi190.stdout:(60/131): python3-rsa-4.9-2.el8.noarch.rpm 510 kB/s | 64 kB 00:00 2023-10-10T17:31:19.601 INFO:teuthology.orchestra.run.smithi190.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 186 kB/s | 19 kB 00:00 2023-10-10T17:31:19.604 INFO:teuthology.orchestra.run.smithi157.stdout:(128/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 3.3 MB/s | 644 kB 00:00 2023-10-10T17:31:19.660 INFO:teuthology.orchestra.run.smithi190.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 238 kB/s | 24 kB 00:00 2023-10-10T17:31:19.676 INFO:teuthology.orchestra.run.smithi078.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 6.7 MB/s | 798 kB 00:00 2023-10-10T17:31:19.685 INFO:teuthology.orchestra.run.smithi190.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 291 kB/s | 29 kB 00:00 2023-10-10T17:31:19.696 INFO:teuthology.orchestra.run.smithi157.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 8.5 MB/s | 798 kB 00:00 2023-10-10T17:31:19.702 INFO:teuthology.orchestra.run.smithi190.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 274 kB/s | 27 kB 00:00 2023-10-10T17:31:19.827 INFO:teuthology.orchestra.run.smithi190.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.7 MB/s | 251 kB 00:00 2023-10-10T17:31:19.852 INFO:teuthology.orchestra.run.smithi190.stdout:(66/131): python3-waitress-1.4.3-1.el8.noarch.r 1.3 MB/s | 255 kB 00:00 2023-10-10T17:31:19.878 INFO:teuthology.orchestra.run.smithi190.stdout:(67/131): python3-websocket-client-0.56.0-5.el8 349 kB/s | 61 kB 00:00 2023-10-10T17:31:19.936 INFO:teuthology.orchestra.run.smithi190.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 786 kB/s | 85 kB 00:00 2023-10-10T17:31:19.961 INFO:teuthology.orchestra.run.smithi190.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 211 kB/s | 23 kB 00:00 2023-10-10T17:31:19.987 INFO:teuthology.orchestra.run.smithi190.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-10-10T17:31:20.055 INFO:teuthology.orchestra.run.smithi157.stdout:(130/131): librbd1-18.0.0-3908.g05a43c8f.el8.x8 7.4 MB/s | 3.6 MB 00:00 2023-10-10T17:31:20.070 INFO:teuthology.orchestra.run.smithi190.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-10-10T17:31:20.112 INFO:teuthology.orchestra.run.smithi190.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-10-10T17:31:20.143 INFO:teuthology.orchestra.run.smithi078.stdout:(130/131): librados2-18.0.0-3908.g05a43c8f.el8. 5.3 MB/s | 3.7 MB 00:00 2023-10-10T17:31:20.171 INFO:teuthology.orchestra.run.smithi190.stdout:(73/131): thrift-0.13.0-2.el8.x86_64.rpm 5.9 MB/s | 1.7 MB 00:00 2023-10-10T17:31:20.196 INFO:teuthology.orchestra.run.smithi190.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.0 MB/s | 259 kB 00:00 2023-10-10T17:31:20.218 INFO:teuthology.orchestra.run.smithi078.stdout:(131/131): librbd1-18.0.0-3908.g05a43c8f.el8.x8 5.1 MB/s | 3.6 MB 00:00 2023-10-10T17:31:20.221 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-10-10T17:31:20.221 INFO:teuthology.orchestra.run.smithi078.stdout:Total 10 MB/s | 197 MB 00:18 2023-10-10T17:31:20.405 INFO:teuthology.orchestra.run.smithi190.stdout:(75/131): python3-scipy-1.0.0-21.module+el8.5.0 31 MB/s | 14 MB 00:00 2023-10-10T17:31:20.430 INFO:teuthology.orchestra.run.smithi190.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 170 kB/s | 39 kB 00:00 2023-10-10T17:31:20.455 INFO:teuthology.orchestra.run.smithi190.stdout:(77/131): socat-1.7.4.1-1.el8.x86_64.rpm 1.1 MB/s | 323 kB 00:00 2023-10-10T17:31:20.497 INFO:teuthology.orchestra.run.smithi190.stdout:(78/131): bison-3.0.4-10.el8.x86_64.rpm 7.3 MB/s | 688 kB 00:00 2023-10-10T17:31:20.548 INFO:teuthology.orchestra.run.smithi190.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 1.7 MB/s | 157 kB 00:00 2023-10-10T17:31:20.581 INFO:teuthology.orchestra.run.smithi190.stdout:(80/131): python3-jinja2-2.10.1-3.el8.noarch.rp 3.5 MB/s | 538 kB 00:00 2023-10-10T17:31:20.623 INFO:teuthology.orchestra.run.smithi190.stdout:(81/131): oniguruma-6.8.2-2.el8.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2023-10-10T17:31:20.673 INFO:teuthology.orchestra.run.smithi190.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 5.6 MB/s | 713 kB 00:00 2023-10-10T17:31:20.749 INFO:teuthology.orchestra.run.smithi190.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 212 kB/s | 27 kB 00:00 2023-10-10T17:31:20.791 INFO:teuthology.orchestra.run.smithi190.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 2.7 MB/s | 320 kB 00:00 2023-10-10T17:31:20.849 INFO:teuthology.orchestra.run.smithi190.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-10-10T17:31:20.924 INFO:teuthology.orchestra.run.smithi190.stdout:(86/131): python3-prettytable-0.7.2-14.el8.noar 250 kB/s | 44 kB 00:00 2023-10-10T17:31:21.050 INFO:teuthology.orchestra.run.smithi190.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 14 MB/s | 3.7 MB 00:00 2023-10-10T17:31:21.058 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-10-10T17:31:21.075 INFO:teuthology.orchestra.run.smithi190.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.5 MB/s | 225 kB 00:00 2023-10-10T17:31:21.134 INFO:teuthology.orchestra.run.smithi190.stdout:(89/131): python3-werkzeug-0.12.2-4.el8.noarch. 1.6 MB/s | 457 kB 00:00 2023-10-10T17:31:21.178 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-10-10T17:31:21.179 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-10-10T17:31:21.201 INFO:teuthology.orchestra.run.smithi190.stdout:(90/131): uuid-1.6.2-43.el8.x86_64.rpm 424 kB/s | 64 kB 00:00 2023-10-10T17:31:21.243 INFO:teuthology.orchestra.run.smithi190.stdout:(91/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 754 kB/s | 126 kB 00:00 2023-10-10T17:31:21.293 INFO:teuthology.orchestra.run.smithi190.stdout:(92/131): python3-jsonpointer-1.10-11.el8.noarc 126 kB/s | 20 kB 00:00 2023-10-10T17:31:21.360 INFO:teuthology.orchestra.run.smithi190.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 652 kB/s | 103 kB 00:00 2023-10-10T17:31:21.363 INFO:teuthology.orchestra.run.smithi157.stdout:(131/131): librados2-18.0.0-3908.g05a43c8f.el8. 2.0 MB/s | 3.7 MB 00:01 2023-10-10T17:31:21.367 INFO:teuthology.orchestra.run.smithi157.stdout:-------------------------------------------------------------------------------- 2023-10-10T17:31:21.367 INFO:teuthology.orchestra.run.smithi157.stdout:Total 10 MB/s | 197 MB 00:19 2023-10-10T17:31:21.394 INFO:teuthology.orchestra.run.smithi190.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2023-10-10T17:31:21.511 INFO:teuthology.orchestra.run.smithi190.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.3 MB/s | 198 kB 00:00 2023-10-10T17:31:21.577 INFO:teuthology.orchestra.run.smithi190.stdout:(96/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-10-10T17:31:21.619 INFO:teuthology.orchestra.run.smithi190.stdout:(97/131): jq-1.6-3.el8.x86_64.rpm 1.8 MB/s | 202 kB 00:00 2023-10-10T17:31:21.653 INFO:teuthology.orchestra.run.smithi190.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 4.6 MB/s | 354 kB 00:00 2023-10-10T17:31:21.745 INFO:teuthology.orchestra.run.smithi190.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-10-10T17:31:21.853 INFO:teuthology.orchestra.run.smithi190.stdout:(100/131): postgresql-contrib-10.21-2.module+el 3.9 MB/s | 808 kB 00:00 2023-10-10T17:31:21.945 INFO:teuthology.orchestra.run.smithi190.stdout:(101/131): postgresql-server-10.21-2.module+el8 16 MB/s | 5.1 MB 00:00 2023-10-10T17:31:21.979 INFO:teuthology.orchestra.run.smithi190.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 374 kB/s | 47 kB 00:00 2023-10-10T17:31:22.029 INFO:teuthology.orchestra.run.smithi190.stdout:(103/131): postgresql-10.21-2.module+el8.6.0+15 5.3 MB/s | 1.5 MB 00:00 2023-10-10T17:31:22.046 INFO:teuthology.orchestra.run.smithi190.stdout:(104/131): libselinux-devel-2.9-5.el8.x86_64.rp 1.9 MB/s | 200 kB 00:00 2023-10-10T17:31:22.121 INFO:teuthology.orchestra.run.smithi190.stdout:(105/131): python3-jwt-1.6.1-2.el8.noarch.rpm 299 kB/s | 43 kB 00:00 2023-10-10T17:31:22.189 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction check 2023-10-10T17:31:22.246 INFO:teuthology.orchestra.run.smithi190.stdout:(106/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2023-10-10T17:31:22.272 INFO:teuthology.orchestra.run.smithi190.stdout:(107/131): keyutils-libs-devel-1.5.10-9.el8.x86 213 kB/s | 48 kB 00:00 2023-10-10T17:31:22.277 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-10-10T17:31:22.281 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-10-10T17:31:22.289 INFO:teuthology.orchestra.run.smithi190.stdout:(108/131): libsepol-devel-2.9-3.el8.x86_64.rpm 520 kB/s | 87 kB 00:00 2023-10-10T17:31:22.309 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction check succeeded. 2023-10-10T17:31:22.310 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction test 2023-10-10T17:31:22.314 INFO:teuthology.orchestra.run.smithi190.stdout:(109/131): python3-requests-oauthlib-1.0.0-1.el 636 kB/s | 43 kB 00:00 2023-10-10T17:31:22.356 INFO:teuthology.orchestra.run.smithi190.stdout:(110/131): python3-pycparser-2.14-14.el8.noarch 1.3 MB/s | 109 kB 00:00 2023-10-10T17:31:22.381 INFO:teuthology.orchestra.run.smithi190.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.4 MB/s | 229 kB 00:00 2023-10-10T17:31:22.406 INFO:teuthology.orchestra.run.smithi190.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 374 kB/s | 19 kB 00:00 2023-10-10T17:31:22.440 INFO:teuthology.orchestra.run.smithi190.stdout:(113/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.4 MB/s | 560 kB 00:00 2023-10-10T17:31:22.499 INFO:teuthology.orchestra.run.smithi190.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.3 MB/s | 155 kB 00:00 2023-10-10T17:31:22.515 INFO:teuthology.orchestra.run.smithi190.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 238 kB/s | 18 kB 00:00 2023-10-10T17:31:22.549 INFO:teuthology.orchestra.run.smithi190.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 3.8 MB/s | 559 kB 00:00 2023-10-10T17:31:22.583 INFO:teuthology.orchestra.run.smithi190.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 7.1 MB/s | 605 kB 00:00 2023-10-10T17:31:22.608 INFO:teuthology.orchestra.run.smithi190.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.6 MB/s | 238 kB 00:00 2023-10-10T17:31:22.633 INFO:teuthology.orchestra.run.smithi190.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.4 MB/s | 72 kB 00:00 2023-10-10T17:31:22.700 INFO:teuthology.orchestra.run.smithi190.stdout:(120/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 15 MB/s | 2.3 MB 00:00 2023-10-10T17:31:22.768 INFO:teuthology.orchestra.run.smithi190.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 448 kB/s | 60 kB 00:00 2023-10-10T17:31:22.792 INFO:teuthology.orchestra.run.smithi190.stdout:(122/131): fuse-2.9.7-15.el8.x86_64.rpm 449 kB/s | 83 kB 00:00 2023-10-10T17:31:22.818 INFO:teuthology.orchestra.run.smithi190.stdout:(123/131): libuuid-devel-2.32.1-35.el8.x86_64.r 837 kB/s | 98 kB 00:00 2023-10-10T17:31:22.851 INFO:teuthology.orchestra.run.smithi190.stdout:(124/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 689 kB/s | 58 kB 00:00 2023-10-10T17:31:22.877 INFO:teuthology.orchestra.run.smithi190.stdout:(125/131): libcom_err-devel-1.45.6-4.el8.x86_64 460 kB/s | 39 kB 00:00 2023-10-10T17:31:22.960 INFO:teuthology.orchestra.run.smithi190.stdout:(126/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 4.4 MB/s | 644 kB 00:00 2023-10-10T17:31:22.994 INFO:teuthology.orchestra.run.smithi190.stdout:(127/131): NetworkManager-initscripts-updown-1. 1.2 MB/s | 138 kB 00:00 2023-10-10T17:31:23.019 INFO:teuthology.orchestra.run.smithi190.stdout:(128/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.0 MB/s | 171 kB 00:00 2023-10-10T17:31:23.170 INFO:teuthology.orchestra.run.smithi190.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 5.2 MB/s | 798 kB 00:00 2023-10-10T17:31:23.397 INFO:teuthology.orchestra.run.smithi157.stdout:Transaction test succeeded. 2023-10-10T17:31:23.401 INFO:teuthology.orchestra.run.smithi157.stdout:Running transaction 2023-10-10T17:31:23.886 INFO:teuthology.orchestra.run.smithi190.stdout:(130/131): librados2-18.0.0-3908.g05a43c8f.el8. 4.0 MB/s | 3.7 MB 00:00 2023-10-10T17:31:25.012 INFO:teuthology.orchestra.run.smithi190.stdout:(131/131): librbd1-18.0.0-3908.g05a43c8f.el8.x8 1.8 MB/s | 3.6 MB 00:02 2023-10-10T17:31:25.015 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2023-10-10T17:31:25.015 INFO:teuthology.orchestra.run.smithi190.stdout:Total 8.7 MB/s | 197 MB 00:22 2023-10-10T17:31:25.843 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2023-10-10T17:31:25.964 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2023-10-10T17:31:25.964 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2023-10-10T17:31:27.044 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2023-10-10T17:31:27.048 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2023-10-10T17:31:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-10-10T17:31:27.261 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-10T17:31:27.458 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-10T17:31:27.711 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-10T17:31:27.913 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-10T17:31:28.022 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-10T17:31:28.271 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-10T17:31:28.319 INFO:teuthology.orchestra.run.smithi157.stdout: Preparing : 1/1 2023-10-10T17:31:28.692 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-10T17:31:28.764 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-10T17:31:28.923 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-10T17:31:28.929 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:31:29.157 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-10T17:31:29.307 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-10T17:31:29.336 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:31:29.464 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-10T17:31:29.554 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:31:29.685 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:31:29.715 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-10T17:31:29.937 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:31:30.017 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:31:30.154 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-10T17:31:30.221 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-10T17:31:30.406 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:31:30.449 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-10T17:31:30.805 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:31:30.892 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-10T17:31:31.006 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:31:31.163 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:31:31.290 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-10T17:31:31.332 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:31:31.427 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:31:31.550 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-10T17:31:31.631 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-10T17:31:31.712 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-10T17:31:31.797 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-10T17:31:31.852 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-10T17:31:31.962 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-10T17:31:31.966 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2023-10-10T17:31:32.023 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-10T17:31:32.067 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-10T17:31:32.149 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-10-10T17:31:32.212 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-10T17:31:32.295 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-10-10T17:31:32.346 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-10T17:31:32.459 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-10T17:31:32.499 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-10-10T17:31:32.520 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-10T17:31:32.667 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-10T17:31:32.727 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-10T17:31:32.758 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-10-10T17:31:32.873 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-10T17:31:32.973 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-10T17:31:32.975 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-10T17:31:33.134 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-10T17:31:33.165 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-10T17:31:33.259 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-10T17:31:33.312 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-10T17:31:33.366 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-10-10T17:31:33.397 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-10T17:31:33.539 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-10T17:31:33.550 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:31:33.689 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-10T17:31:33.909 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-10T17:31:33.944 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:31:34.087 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-10T17:31:34.108 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:31:34.243 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:31:34.288 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-10T17:31:34.425 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:31:34.506 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-10T17:31:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:31:34.691 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-10T17:31:34.796 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-10T17:31:34.887 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-10-10T17:31:35.060 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-10T17:31:35.088 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-10T17:31:35.137 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-10-10T17:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-10T17:31:35.314 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-10T17:31:35.524 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-10-10T17:31:35.540 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-10T17:31:35.729 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-10-10T17:31:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-10T17:31:35.850 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-10T17:31:35.956 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 14/134 2023-10-10T17:31:35.976 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-10T17:31:36.010 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-10T17:31:36.092 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-10T17:31:36.169 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 15/134 2023-10-10T17:31:36.254 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-10T17:31:36.271 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-10T17:31:36.292 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-10T17:31:36.419 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-10-10T17:31:36.451 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 26/134 2023-10-10T17:31:36.498 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-10T17:31:36.575 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-10T17:31:36.590 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 27/134 2023-10-10T17:31:36.692 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-10T17:31:36.936 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-10T17:31:37.145 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-10T17:31:37.193 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-10-10T17:31:37.367 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-10T17:31:37.432 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-10T17:31:37.501 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 28/134 2023-10-10T17:31:37.612 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 26/134 2023-10-10T17:31:37.791 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 27/134 2023-10-10T17:31:37.877 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-10T17:31:37.967 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-10-10T17:31:38.125 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-10-10T17:31:38.164 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-10T17:31:38.457 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-10T17:31:38.661 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-10T17:31:38.705 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 28/134 2023-10-10T17:31:38.805 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-10-10T17:31:39.029 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-10T17:31:39.076 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-10-10T17:31:39.277 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-10-10T17:31:39.462 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-10-10T17:31:39.477 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-10T17:31:39.631 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-10-10T17:31:39.748 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-10T17:31:39.863 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-10-10T17:31:39.980 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-10T17:31:40.092 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-10-10T17:31:40.247 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 26/134 2023-10-10T17:31:40.368 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 27/134 2023-10-10T17:31:40.412 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-10T17:31:40.753 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-10T17:31:40.980 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-10T17:31:41.180 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-10T17:31:41.255 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 28/134 2023-10-10T17:31:41.539 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-10-10T17:31:41.677 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 36/134 2023-10-10T17:31:41.736 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-10T17:31:41.801 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-10-10T17:31:41.881 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 37/134 2023-10-10T17:31:42.051 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-10T17:31:42.066 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-10-10T17:31:42.097 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-10T17:31:42.254 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-10T17:31:42.257 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-10T17:31:42.334 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-10T17:31:42.447 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-10T17:31:42.524 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-10T17:31:42.614 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-10T17:31:42.752 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 36/134 2023-10-10T17:31:42.843 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-10T17:31:42.933 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 37/134 2023-10-10T17:31:43.093 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-10T17:31:43.116 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-10T17:31:43.289 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-10T17:31:43.482 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-10T17:31:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-10T17:31:43.666 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-10T17:31:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-10T17:31:43.887 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-10T17:31:44.012 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-10-10T17:31:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-10T17:31:44.110 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-10T17:31:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-10T17:31:44.375 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-10-10T17:31:44.520 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-10T17:31:44.540 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-10T17:31:44.561 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el8.noarch 34/134 2023-10-10T17:31:44.751 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-10-10T17:31:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-10T17:31:44.865 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-10T17:31:44.938 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 36/134 2023-10-10T17:31:45.047 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-10T17:31:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-10T17:31:45.102 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 37/134 2023-10-10T17:31:45.300 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-10T17:31:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-10T17:31:45.440 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-10-10T17:31:45.542 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-10T17:31:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-10T17:31:45.742 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-10-10T17:31:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-10T17:31:45.906 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-10T17:31:45.918 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-10-10T17:31:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-10T17:31:46.044 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-10-10T17:31:46.162 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-10T17:31:46.248 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-10-10T17:31:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-10T17:31:46.341 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-10T17:31:46.456 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-10-10T17:31:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-10T17:31:46.595 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-10T17:31:46.768 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-10T17:31:46.807 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-10-10T17:31:46.857 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-10T17:31:46.987 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-10T17:31:47.060 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-10-10T17:31:47.117 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-10T17:31:47.267 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-10-10T17:31:47.286 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-10T17:31:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-10T17:31:47.468 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-10-10T17:31:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-10T17:31:47.569 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-10T17:31:47.701 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-10T17:31:47.702 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-10-10T17:31:47.774 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-10T17:31:47.842 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-10T17:31:47.998 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-10-10T17:31:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-10T17:31:48.085 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-10T17:31:48.189 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-10-10T17:31:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-10T17:31:48.286 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-10T17:31:48.374 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-10-10T17:31:48.430 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-10T17:31:48.443 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-10T17:31:48.528 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-10T17:31:48.611 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-10-10T17:31:48.628 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-10T17:31:48.697 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-10T17:31:48.755 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-10T17:31:48.768 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-10-10T17:31:48.927 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-10-10T17:31:48.942 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-10T17:31:48.944 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-10T17:31:49.126 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-10T17:31:49.161 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-10T17:31:49.221 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-10-10T17:31:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-10T17:31:49.357 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-10T17:31:49.427 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-10T17:31:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-10-10T17:31:49.568 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-10T17:31:49.642 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-10T17:31:49.720 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-10-10T17:31:49.816 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-10T17:31:49.854 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-10T17:31:49.946 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-10-10T17:31:50.005 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-10T17:31:50.012 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-10T17:31:50.121 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-10-10T17:31:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-10T17:31:50.225 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-10T17:31:50.266 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-10-10T17:31:50.367 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-10T17:31:50.423 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-10-10T17:31:50.490 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-10-10T17:31:50.495 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-10T17:31:50.587 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-10T17:31:50.736 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-10T17:31:50.745 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-10-10T17:31:50.794 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-10T17:31:50.883 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-10T17:31:50.926 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-10-10T17:31:50.971 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-10T17:31:51.032 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-10T17:31:51.073 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-10T17:31:51.115 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-10T17:31:51.137 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-10-10T17:31:51.265 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-10T17:31:51.343 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-10-10T17:31:51.355 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-10T17:31:51.487 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-10T17:31:51.498 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-10-10T17:31:51.629 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-10T17:31:51.638 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-10-10T17:31:51.682 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-10T17:31:51.800 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-10-10T17:31:51.823 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-10T17:31:51.888 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-10T17:31:52.051 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-10T17:31:52.056 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-10-10T17:31:52.069 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-10T17:31:52.271 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-10-10T17:31:52.296 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-10T17:31:52.358 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-10T17:31:52.401 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-10-10T17:31:52.524 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-10T17:31:52.549 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-10-10T17:31:52.574 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-10T17:31:52.721 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-10-10T17:31:52.739 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-10T17:31:52.801 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-10T17:31:52.899 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-10-10T17:31:53.067 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-10T17:31:53.091 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-10-10T17:31:53.242 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-10-10T17:31:53.319 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-10T17:31:53.378 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-10-10T17:31:53.493 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-10T17:31:53.592 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-10-10T17:31:53.743 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-10T17:31:53.817 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-10-10T17:31:53.878 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-10T17:31:53.978 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-10-10T17:31:54.180 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-10T17:31:54.225 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-10-10T17:31:54.404 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-10-10T17:31:54.434 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-10T17:31:54.562 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-10-10T17:31:54.769 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-10T17:31:54.778 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-10-10T17:31:54.896 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-10T17:31:55.086 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-10T17:31:55.282 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-10T17:31:55.338 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-10T17:31:55.536 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-10T17:31:55.864 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-10-10T17:31:55.872 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-10T17:31:56.106 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-10T17:31:56.151 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-10-10T17:31:56.199 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-10T17:31:56.284 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-10T17:31:56.340 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-10T17:31:56.404 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-10T17:31:56.446 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-10-10T17:31:56.526 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 93/134 2023-10-10T17:31:56.748 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-10-10T17:31:57.006 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-10-10T17:31:57.184 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-10-10T17:31:57.186 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-10T17:31:57.393 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-10T17:31:57.454 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-10T17:31:57.644 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 93/134 2023-10-10T17:31:57.944 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-10-10T17:31:58.097 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-10T17:31:58.173 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 92/134 2023-10-10T17:31:58.359 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 93/134 2023-10-10T17:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:00.417 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:01.005 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:01.275 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-10T17:32:01.341 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-10T17:32:01.366 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:01.533 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-10T17:32:01.636 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:01.998 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:02.226 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:02.238 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:02.524 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-10T17:32:02.605 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-10T17:32:02.785 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-10T17:32:02.824 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 94/134 2023-10-10T17:32:03.059 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-10T17:32:03.123 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 95/134 2023-10-10T17:32:03.263 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-10T17:32:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-10T17:32:20.026 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2023-10-10T17:32:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2023-10-10T17:32:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-10-10T17:32:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /run 2023-10-10T17:32:20.027 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2023-10-10T17:32:20.028 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2023-10-10T17:32:20.028 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2023-10-10T17:32:20.028 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2023-10-10T17:32:20.028 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-10-10T17:32:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:32:20.398 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-10T17:32:21.483 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-10T17:32:21.483 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /sys 2023-10-10T17:32:21.483 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /proc 2023-10-10T17:32:21.484 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-10-10T17:32:21.484 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /run 2023-10-10T17:32:21.484 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /mnt 2023-10-10T17:32:21.485 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /var/tmp 2023-10-10T17:32:21.485 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /home 2023-10-10T17:32:21.485 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /tmp 2023-10-10T17:32:21.486 INFO:teuthology.orchestra.run.smithi157.stdout:skipping the directory /dev 2023-10-10T17:32:21.486 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:32:21.849 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-10T17:32:21.959 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-10T17:32:22.155 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-10T17:32:22.195 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 96/134 2023-10-10T17:32:22.196 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2023-10-10T17:32:22.196 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2023-10-10T17:32:22.196 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /dev 2023-10-10T17:32:22.196 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /run 2023-10-10T17:32:22.196 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2023-10-10T17:32:22.197 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2023-10-10T17:32:22.197 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2023-10-10T17:32:22.197 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2023-10-10T17:32:22.197 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /dev 2023-10-10T17:32:22.197 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:32:22.387 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-10T17:32:22.518 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-10T17:32:22.572 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-10T17:32:23.022 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-10T17:32:23.214 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-10T17:32:23.231 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-10T17:32:23.425 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-10T17:32:23.620 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-10T17:32:23.839 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-10T17:32:24.039 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-10T17:32:24.084 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 97/134 2023-10-10T17:32:24.264 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-10T17:32:24.483 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 98/134 2023-10-10T17:32:24.490 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-10T17:32:24.643 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-10T17:32:24.681 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-10T17:32:24.698 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-10T17:32:24.931 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-10T17:32:25.093 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 99/134 2023-10-10T17:32:25.149 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-10T17:32:25.234 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-10T17:32:25.251 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 100/134 2023-10-10T17:32:25.286 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 102/134 2023-10-10T17:32:25.511 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 103/134 2023-10-10T17:32:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-10T17:32:25.926 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-10T17:32:26.213 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-10T17:32:26.412 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-10T17:32:26.630 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-10T17:32:26.795 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 102/134 2023-10-10T17:32:26.909 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-10T17:32:26.995 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 103/134 2023-10-10T17:32:27.100 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 101/134 2023-10-10T17:32:27.171 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-10T17:32:27.213 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-10T17:32:27.215 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 102/134 2023-10-10T17:32:27.347 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-10T17:32:27.407 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 103/134 2023-10-10T17:32:27.425 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-10T17:32:27.436 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-10T17:32:27.584 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-10T17:32:27.702 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-10T17:32:27.798 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 104/134 2023-10-10T17:32:27.840 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 107/134 2023-10-10T17:32:28.055 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-10T17:32:28.081 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-10T17:32:28.116 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-10T17:32:28.280 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-10T17:32:28.292 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-10T17:32:28.663 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-10T17:32:28.872 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-10T17:32:28.962 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-10T17:32:29.017 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 105/134 2023-10-10T17:32:29.159 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-10T17:32:29.248 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 106/134 2023-10-10T17:32:29.347 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 107/134 2023-10-10T17:32:29.541 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 110/134 2023-10-10T17:32:29.557 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-10T17:32:29.584 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 107/134 2023-10-10T17:32:29.592 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-10T17:32:29.749 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-10T17:32:29.763 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-10T17:32:29.765 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-10T17:32:29.774 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-10T17:32:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-10T17:32:29.801 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 108/134 2023-10-10T17:32:29.938 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-10T17:32:29.967 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-10T17:32:29.978 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 109/134 2023-10-10T17:32:29.979 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-10T17:32:30.147 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-10T17:32:30.490 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-10T17:32:30.655 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-10T17:32:31.008 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 110/134 2023-10-10T17:32:31.196 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-10T17:32:31.221 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 110/134 2023-10-10T17:32:31.230 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-10T17:32:31.413 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-10T17:32:31.435 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-10T17:32:31.455 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-10T17:32:31.469 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 111/134 2023-10-10T17:32:31.605 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-10T17:32:31.617 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-10T17:32:31.658 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3908.g05a43 112/134 2023-10-10T17:32:31.792 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-10T17:32:31.949 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-10T17:32:32.136 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 113/134 2023-10-10T17:32:32.138 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-10T17:32:32.283 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-10T17:32:34.555 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-10T17:32:34.912 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 115/134 2023-10-10T17:32:35.087 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-10T17:32:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-10T17:32:36.063 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-10T17:32:36.198 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 114/134 2023-10-10T17:32:36.208 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-10T17:32:36.363 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 115/134 2023-10-10T17:32:36.450 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 115/134 2023-10-10T17:32:36.529 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-10T17:32:36.552 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-10T17:32:36.590 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-10-10T17:32:36.726 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-10T17:32:36.748 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 119/134 2023-10-10T17:32:36.784 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-10-10T17:32:36.949 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-10T17:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 121/134 2023-10-10T17:32:37.296 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-10T17:32:37.606 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-10T17:32:37.663 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-10-10T17:32:37.952 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-10T17:32:37.986 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-10-10T17:32:38.143 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 119/134 2023-10-10T17:32:38.173 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 119/134 2023-10-10T17:32:38.311 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-10T17:32:38.374 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-10T17:32:38.486 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 121/134 2023-10-10T17:32:38.541 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 121/134 2023-10-10T17:32:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-10T17:32:38.721 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-10T17:32:39.473 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:39.720 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:39.738 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:39.904 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 124/134 2023-10-10T17:32:40.228 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 125/134 2023-10-10T17:32:40.532 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 126/134 2023-10-10T17:32:40.824 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:40.863 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-10T17:32:41.003 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:41.020 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:41.030 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:41.065 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:41.083 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:41.249 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 124/134 2023-10-10T17:32:41.259 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-10T17:32:41.295 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:41.314 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-10-10T17:32:41.453 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-10T17:32:41.479 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 124/134 2023-10-10T17:32:41.506 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 125/134 2023-10-10T17:32:41.538 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-10T17:32:41.761 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 126/134 2023-10-10T17:32:41.762 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-10T17:32:41.774 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-10T17:32:41.774 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:41.794 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 125/134 2023-10-10T17:32:41.973 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:41.973 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-10T17:32:42.042 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-10T17:32:42.138 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 126/134 2023-10-10T17:32:42.190 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:42.198 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:42.216 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-10T17:32:42.393 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-10T17:32:42.443 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:42.488 INFO:teuthology.orchestra.run.smithi157.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-10-10T17:32:42.598 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-10T17:32:42.636 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:42.645 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:42.680 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-10T17:32:42.861 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-10-10T17:32:42.881 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-10T17:32:42.894 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-10T17:32:42.894 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:42.982 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 1/134 2023-10-10T17:32:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2/134 2023-10-10T17:32:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 3/134 2023-10-10T17:32:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 4/134 2023-10-10T17:32:42.983 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 5/134 2023-10-10T17:32:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:32:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:32:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:32:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-10T17:32:42.984 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 10/134 2023-10-10T17:32:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 11/134 2023-10-10T17:32:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 12/134 2023-10-10T17:32:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 13/134 2023-10-10T17:32:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 14/134 2023-10-10T17:32:42.985 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 15/134 2023-10-10T17:32:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 16/134 2023-10-10T17:32:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 17/134 2023-10-10T17:32:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 18/134 2023-10-10T17:32:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 19/134 2023-10-10T17:32:42.986 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 20/134 2023-10-10T17:32:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 21/134 2023-10-10T17:32:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 22/134 2023-10-10T17:32:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 23/134 2023-10-10T17:32:42.987 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 24/134 2023-10-10T17:32:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 25/134 2023-10-10T17:32:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 26/134 2023-10-10T17:32:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 27/134 2023-10-10T17:32:42.988 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 28/134 2023-10-10T17:32:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 29/134 2023-10-10T17:32:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 30/134 2023-10-10T17:32:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 31/134 2023-10-10T17:32:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 32/134 2023-10-10T17:32:42.989 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 33/134 2023-10-10T17:32:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 34/134 2023-10-10T17:32:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 35/134 2023-10-10T17:32:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 36/134 2023-10-10T17:32:42.990 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 37/134 2023-10-10T17:32:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 38/134 2023-10-10T17:32:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-10T17:32:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-10T17:32:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-10T17:32:42.991 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-10T17:32:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-10T17:32:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-10T17:32:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-10T17:32:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-10T17:32:42.992 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-10T17:32:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-10T17:32:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-10T17:32:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-10T17:32:42.993 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-10T17:32:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-10T17:32:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-10T17:32:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-10T17:32:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-10T17:32:42.994 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-10T17:32:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-10T17:32:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-10T17:32:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-10T17:32:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-10T17:32:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-10T17:32:42.995 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-10T17:32:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-10T17:32:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-10T17:32:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-10T17:32:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-10T17:32:42.996 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-10T17:32:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-10T17:32:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-10T17:32:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-10T17:32:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-10T17:32:42.997 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-10T17:32:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-10T17:32:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-10T17:32:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-10T17:32:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-10T17:32:42.998 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-10T17:32:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-10T17:32:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-10T17:32:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-10T17:32:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-10T17:32:42.999 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-10T17:32:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-10T17:32:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-10T17:32:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-10T17:32:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-10T17:32:43.000 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-10T17:32:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-10T17:32:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-10T17:32:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-10T17:32:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-10T17:32:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-10T17:32:43.001 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-10T17:32:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-10T17:32:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-10T17:32:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-10T17:32:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-10T17:32:43.002 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-10T17:32:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-10T17:32:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-10T17:32:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-10T17:32:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-10T17:32:43.003 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-10T17:32:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-10T17:32:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-10T17:32:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-10T17:32:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-10T17:32:43.004 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-10T17:32:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-10T17:32:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-10T17:32:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-10T17:32:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-10T17:32:43.005 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-10T17:32:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-10T17:32:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-10T17:32:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-10T17:32:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-10T17:32:43.006 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-10T17:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-10T17:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-10T17:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-10T17:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-10T17:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-10T17:32:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-10T17:32:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-10T17:32:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-10T17:32:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-10T17:32:43.008 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 129/134 2023-10-10T17:32:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-10T17:32:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 131/134 2023-10-10T17:32:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:43.009 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-10T17:32:43.027 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-10-10T17:32:43.076 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:43.076 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-10T17:32:43.114 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-10-10T17:32:43.278 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-10T17:32:43.344 INFO:teuthology.orchestra.run.smithi157.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-10-10T17:32:43.357 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-10-10T17:32:43.358 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:43.472 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:43.573 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:43.573 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-10T17:32:43.875 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-10-10T17:32:43.974 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:43.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 1/134 2023-10-10T17:32:43.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2/134 2023-10-10T17:32:43.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 3/134 2023-10-10T17:32:43.975 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 4/134 2023-10-10T17:32:43.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 5/134 2023-10-10T17:32:43.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:32:43.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:32:43.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:32:43.976 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-10T17:32:43.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 10/134 2023-10-10T17:32:43.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 11/134 2023-10-10T17:32:43.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 12/134 2023-10-10T17:32:43.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 13/134 2023-10-10T17:32:43.977 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 14/134 2023-10-10T17:32:43.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 15/134 2023-10-10T17:32:43.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 16/134 2023-10-10T17:32:43.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 17/134 2023-10-10T17:32:43.978 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 18/134 2023-10-10T17:32:43.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 19/134 2023-10-10T17:32:43.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 20/134 2023-10-10T17:32:43.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 21/134 2023-10-10T17:32:43.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 22/134 2023-10-10T17:32:43.979 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 23/134 2023-10-10T17:32:43.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 24/134 2023-10-10T17:32:43.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 25/134 2023-10-10T17:32:43.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 26/134 2023-10-10T17:32:43.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 27/134 2023-10-10T17:32:43.980 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 28/134 2023-10-10T17:32:43.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 29/134 2023-10-10T17:32:43.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 30/134 2023-10-10T17:32:43.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 31/134 2023-10-10T17:32:43.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 32/134 2023-10-10T17:32:43.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 33/134 2023-10-10T17:32:43.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 34/134 2023-10-10T17:32:43.991 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 35/134 2023-10-10T17:32:43.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 36/134 2023-10-10T17:32:43.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 37/134 2023-10-10T17:32:43.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 38/134 2023-10-10T17:32:43.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-10T17:32:43.992 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-10T17:32:43.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-10T17:32:43.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-10T17:32:43.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-10T17:32:43.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-10T17:32:43.993 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-10T17:32:43.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-10T17:32:43.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-10T17:32:43.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-10T17:32:43.994 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-10T17:32:43.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-10T17:32:43.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-10T17:32:43.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-10T17:32:43.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-10T17:32:43.995 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-10T17:32:43.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-10T17:32:43.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-10T17:32:43.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-10T17:32:43.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-10T17:32:43.996 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-10T17:32:43.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-10T17:32:43.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-10T17:32:43.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-10T17:32:43.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-10T17:32:43.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-10T17:32:43.997 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-10T17:32:43.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-10T17:32:43.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-10T17:32:43.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-10T17:32:43.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-10T17:32:43.998 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-10T17:32:43.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-10T17:32:43.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-10T17:32:43.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-10T17:32:43.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-10T17:32:43.999 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-10T17:32:44.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-10T17:32:44.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-10T17:32:44.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-10T17:32:44.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-10T17:32:44.000 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-10T17:32:44.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-10T17:32:44.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-10T17:32:44.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-10T17:32:44.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-10T17:32:44.001 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-10T17:32:44.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-10T17:32:44.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-10T17:32:44.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-10T17:32:44.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-10T17:32:44.002 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-10T17:32:44.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-10T17:32:44.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-10T17:32:44.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-10T17:32:44.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-10T17:32:44.003 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-10T17:32:44.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-10T17:32:44.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-10T17:32:44.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-10T17:32:44.004 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-10T17:32:44.005 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-10T17:32:44.005 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-10T17:32:44.005 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-10T17:32:44.005 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-10T17:32:44.005 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-10T17:32:44.006 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-10T17:32:44.006 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-10T17:32:44.006 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-10T17:32:44.006 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-10T17:32:44.006 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-10T17:32:44.007 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-10T17:32:44.007 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-10T17:32:44.007 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-10T17:32:44.007 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-10T17:32:44.007 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-10T17:32:44.008 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-10T17:32:44.008 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-10T17:32:44.008 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-10T17:32:44.008 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-10T17:32:44.008 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-10T17:32:44.009 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-10T17:32:44.009 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-10T17:32:44.009 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-10T17:32:44.009 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-10T17:32:44.009 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-10T17:32:44.010 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-10T17:32:44.010 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-10T17:32:44.010 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-10T17:32:44.010 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:44.010 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 129/134 2023-10-10T17:32:44.011 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-10T17:32:44.011 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 131/134 2023-10-10T17:32:44.011 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:44.011 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-10T17:32:44.118 INFO:teuthology.orchestra.run.smithi157.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:44.639 INFO:teuthology.orchestra.run.smithi157.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:44.639 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 1/134 2023-10-10T17:32:44.639 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2/134 2023-10-10T17:32:44.639 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 3/134 2023-10-10T17:32:44.640 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 4/134 2023-10-10T17:32:44.640 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3908.g05a43 5/134 2023-10-10T17:32:44.640 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 6/134 2023-10-10T17:32:44.640 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 7/134 2023-10-10T17:32:44.640 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 8/134 2023-10-10T17:32:44.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 9/134 2023-10-10T17:32:44.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 10/134 2023-10-10T17:32:44.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 11/134 2023-10-10T17:32:44.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 12/134 2023-10-10T17:32:44.641 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 13/134 2023-10-10T17:32:44.642 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_ 14/134 2023-10-10T17:32:44.642 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 15/134 2023-10-10T17:32:44.642 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 16/134 2023-10-10T17:32:44.642 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_6 17/134 2023-10-10T17:32:44.642 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86 18/134 2023-10-10T17:32:44.643 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 19/134 2023-10-10T17:32:44.643 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el 20/134 2023-10-10T17:32:44.643 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8. 21/134 2023-10-10T17:32:44.643 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_6 22/134 2023-10-10T17:32:44.643 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 23/134 2023-10-10T17:32:44.644 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 24/134 2023-10-10T17:32:44.644 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 25/134 2023-10-10T17:32:44.644 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 26/134 2023-10-10T17:32:44.644 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 27/134 2023-10-10T17:32:44.644 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 28/134 2023-10-10T17:32:44.645 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f. 29/134 2023-10-10T17:32:44.645 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noa 30/134 2023-10-10T17:32:44.645 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.n 31/134 2023-10-10T17:32:44.645 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a 32/134 2023-10-10T17:32:44.645 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el 33/134 2023-10-10T17:32:44.646 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 34/134 2023-10-10T17:32:44.646 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.e 35/134 2023-10-10T17:32:44.646 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 36/134 2023-10-10T17:32:44.646 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 37/134 2023-10-10T17:32:44.646 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 38/134 2023-10-10T17:32:44.647 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-10-10T17:32:44.647 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-10-10T17:32:44.647 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-10-10T17:32:44.647 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-10-10T17:32:44.647 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-10-10T17:32:44.648 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-10-10T17:32:44.648 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-10-10T17:32:44.648 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-10-10T17:32:44.648 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-10-10T17:32:44.648 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-10-10T17:32:44.649 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-10-10T17:32:44.649 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-10-10T17:32:44.649 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-10-10T17:32:44.649 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-10-10T17:32:44.649 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-10-10T17:32:44.650 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-10-10T17:32:44.650 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-10-10T17:32:44.650 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-10-10T17:32:44.650 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-10-10T17:32:44.650 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-10-10T17:32:44.650 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-10-10T17:32:44.651 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/134 2023-10-10T17:32:44.651 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-10-10T17:32:44.651 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-10-10T17:32:44.651 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-10-10T17:32:44.651 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-10-10T17:32:44.652 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-10-10T17:32:44.652 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-10-10T17:32:44.652 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-10-10T17:32:44.652 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-10-10T17:32:44.652 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-10-10T17:32:44.653 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-10-10T17:32:44.653 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-10-10T17:32:44.653 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-10-10T17:32:44.653 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-10-10T17:32:44.653 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-10-10T17:32:44.654 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-10-10T17:32:44.654 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-10-10T17:32:44.654 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-10-10T17:32:44.654 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-10-10T17:32:44.654 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-10-10T17:32:44.655 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-10-10T17:32:44.655 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-10-10T17:32:44.655 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-10-10T17:32:44.655 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-10-10T17:32:44.655 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-10-10T17:32:44.656 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-10-10T17:32:44.656 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-10-10T17:32:44.656 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-10-10T17:32:44.656 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-10-10T17:32:44.656 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-10-10T17:32:44.657 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-10-10T17:32:44.657 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-10-10T17:32:44.657 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-10-10T17:32:44.657 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-10-10T17:32:44.657 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-10-10T17:32:44.658 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-10-10T17:32:44.658 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-10-10T17:32:44.658 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-10-10T17:32:44.658 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-10-10T17:32:44.658 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-10-10T17:32:44.658 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-10-10T17:32:44.659 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-10-10T17:32:44.659 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-10-10T17:32:44.659 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-10-10T17:32:44.659 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-10-10T17:32:44.659 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-10-10T17:32:44.660 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-10-10T17:32:44.660 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-10-10T17:32:44.660 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-10-10T17:32:44.660 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-10-10T17:32:44.660 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-10-10T17:32:44.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-10-10T17:32:44.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-10-10T17:32:44.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-10-10T17:32:44.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-10-10T17:32:44.661 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-10-10T17:32:44.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-10-10T17:32:44.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-10-10T17:32:44.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-10-10T17:32:44.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-10-10T17:32:44.662 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-10-10T17:32:44.663 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-10-10T17:32:44.663 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-10-10T17:32:44.663 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-10-10T17:32:44.663 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-10-10T17:32:44.663 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-10-10T17:32:44.664 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-10-10T17:32:44.664 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-10-10T17:32:44.664 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-10-10T17:32:44.664 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 129/134 2023-10-10T17:32:44.664 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-10-10T17:32:44.664 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 131/134 2023-10-10T17:32:44.665 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-10-10T17:32:44.665 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-10-10T17:32:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:45.563 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:01:56 ago on Tue 10 Oct 2023 05:30:49 PM UTC. 2023-10-10T17:32:46.481 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:46.481 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:01:56 ago on Tue 10 Oct 2023 05:30:50 PM UTC. 2023-10-10T17:32:47.129 INFO:teuthology.orchestra.run.smithi157.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-10-10T17:32:47.130 INFO:teuthology.orchestra.run.smithi157.stdout:Last metadata expiration check: 0:01:57 ago on Tue 10 Oct 2023 05:30:50 PM UTC. 2023-10-10T17:32:48.422 INFO:teuthology.orchestra.run.smithi078.stdout:Installed products updated. 2023-10-10T17:32:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:32:48.636 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2023-10-10T17:32:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-10T17:32:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.637 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-10-10T17:32:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-10T17:32:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.1-7.el8.noarch 2023-10-10T17:32:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.0.4-10.el8.x86_64 2023-10-10T17:32:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-10T17:32:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.1-9.el8.x86_64 2023-10-10T17:32:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-10T17:32:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-10T17:32:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-3.el8.x86_64 2023-10-10T17:32:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-10T17:32:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-10T17:32:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-10T17:32:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-10T17:32:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-10T17:32:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-10T17:32:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: libpq-13.5-1.el8.x86_64 2023-10-10T17:32:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-10T17:32:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-10T17:32:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-10T17:32:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-10T17:32:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-10T17:32:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-10T17:32:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-10T17:32:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-10T17:32:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-10T17:32:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-10T17:32:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-10T17:32:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-10T17:32:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-10T17:32:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-10T17:32:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-10T17:32:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-10T17:32:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-10T17:32:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-10T17:32:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-10T17:32:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-10T17:32:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-10T17:32:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-10T17:32:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-10T17:32:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-10T17:32:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-10T17:32:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-10T17:32:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-10T17:32:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-10T17:32:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-10T17:32:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-10T17:32:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-10T17:32:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-10T17:32:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-10T17:32:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-10T17:32:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-10T17:32:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-10T17:32:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-10T17:32:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-10T17:32:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-10T17:32:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-10T17:32:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-10T17:32:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-10T17:32:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-10T17:32:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-10T17:32:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-2.6-1.el8.noarch 2023-10-10T17:32:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-10T17:32:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-10T17:32:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-10T17:32:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-10T17:32:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-10T17:32:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-10T17:32:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-10T17:32:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-10T17:32:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-10T17:32:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-10T17:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-10T17:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-10T17:32:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-10T17:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-10T17:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-10T17:32:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-10T17:32:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-10T17:32:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-10T17:32:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-10T17:32:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-10T17:32:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-10T17:32:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.666 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.666 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:48.666 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-10T17:32:48.667 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-10T17:32:48.667 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-10T17:32:48.667 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-10T17:32:48.668 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-10T17:32:48.669 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-10T17:32:48.669 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:32:48.670 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-10-10T17:32:48.826 DEBUG:teuthology.parallel:result is None 2023-10-10T17:32:50.613 INFO:teuthology.orchestra.run.smithi190.stdout:Installed products updated. 2023-10-10T17:32:50.824 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:32:50.824 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2023-10-10T17:32:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-10T17:32:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.825 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2023-10-10T17:32:50.825 INFO:teuthology.orchestra.run.smithi190.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-10T17:32:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: automake-1.16.1-7.el8.noarch 2023-10-10T17:32:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: bison-3.0.4-10.el8.x86_64 2023-10-10T17:32:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.826 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.827 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.828 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.829 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.830 INFO:teuthology.orchestra.run.smithi190.stdout: cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-10T17:32:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: flex-2.6.1-9.el8.x86_64 2023-10-10T17:32:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-10T17:32:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-10T17:32:50.831 INFO:teuthology.orchestra.run.smithi190.stdout: jq-1.6-3.el8.x86_64 2023-10-10T17:32:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-10T17:32:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-10T17:32:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-10T17:32:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.832 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-10T17:32:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-10T17:32:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-10T17:32:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: libpq-13.5-1.el8.x86_64 2023-10-10T17:32:50.833 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-10T17:32:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-10T17:32:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-10T17:32:50.834 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-10T17:32:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-10T17:32:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-10T17:32:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-10T17:32:50.835 INFO:teuthology.orchestra.run.smithi190.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-10T17:32:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-10T17:32:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-10T17:32:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-10T17:32:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-10T17:32:50.836 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-10T17:32:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-10T17:32:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-10T17:32:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-10T17:32:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-10T17:32:50.837 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-10T17:32:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-10T17:32:50.838 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-10T17:32:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-10T17:32:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-10T17:32:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.839 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-10T17:32:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-10T17:32:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-10T17:32:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-10T17:32:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-10T17:32:50.840 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-10T17:32:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-10T17:32:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-10T17:32:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-10T17:32:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-10T17:32:50.841 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-10T17:32:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-10T17:32:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-10T17:32:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-10T17:32:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-10T17:32:50.842 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-10T17:32:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-10T17:32:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-10T17:32:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-10T17:32:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-10T17:32:50.843 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-10T17:32:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-10T17:32:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-2.6-1.el8.noarch 2023-10-10T17:32:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-10T17:32:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-10T17:32:50.844 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-10T17:32:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-10T17:32:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-10T17:32:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-10T17:32:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.845 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-10T17:32:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-10T17:32:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-10T17:32:50.846 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-10T17:32:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-10T17:32:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-10T17:32:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-10T17:32:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-10T17:32:50.847 INFO:teuthology.orchestra.run.smithi190.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-10T17:32:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-10T17:32:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-10T17:32:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-10T17:32:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-10T17:32:50.848 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-10T17:32:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-10T17:32:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-10T17:32:50.849 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-10T17:32:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-10T17:32:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-10T17:32:50.850 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-10T17:32:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-10T17:32:50.851 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:32:50.851 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2023-10-10T17:32:51.014 DEBUG:teuthology.parallel:result is None 2023-10-10T17:32:51.944 INFO:teuthology.orchestra.run.smithi157.stdout:Installed products updated. 2023-10-10T17:32:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:32:52.160 INFO:teuthology.orchestra.run.smithi157.stdout:Upgraded: 2023-10-10T17:32:52.160 INFO:teuthology.orchestra.run.smithi157.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-10-10T17:32:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: librados2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: librbd1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.161 INFO:teuthology.orchestra.run.smithi157.stdout:Installed: 2023-10-10T17:32:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-10-10T17:32:52.161 INFO:teuthology.orchestra.run.smithi157.stdout: automake-1.16.1-7.el8.noarch 2023-10-10T17:32:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: bison-3.0.4-10.el8.x86_64 2023-10-10T17:32:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.162 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-grafana-dashboards-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-immutable-object-cache-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mds-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.163 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.164 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-rook-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mon-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-osd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.165 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-prometheus-alerts-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-radosgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-selinux-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-test-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-volume-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.166 INFO:teuthology.orchestra.run.smithi157.stdout: cephadm-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: cephfs-top-2:18.0.0-3908.g05a43c8f.el8.noarch 2023-10-10T17:32:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-10-10T17:32:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: flex-2.6.1-9.el8.x86_64 2023-10-10T17:32:52.167 INFO:teuthology.orchestra.run.smithi157.stdout: fuse-2.9.7-15.el8.x86_64 2023-10-10T17:32:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-10T17:32:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: jq-1.6-3.el8.x86_64 2023-10-10T17:32:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-10-10T17:32:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-10-10T17:32:52.168 INFO:teuthology.orchestra.run.smithi157.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-10T17:32:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: libcephsqlite-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-10-10T17:32:52.169 INFO:teuthology.orchestra.run.smithi157.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-10T17:32:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-10T17:32:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: libpq-13.5-1.el8.x86_64 2023-10-10T17:32:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-10T17:32:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-10T17:32:52.170 INFO:teuthology.orchestra.run.smithi157.stdout: librados-devel-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: libradosstriper1-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-10T17:32:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: librgw2-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-10T17:32:52.171 INFO:teuthology.orchestra.run.smithi157.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-10-10T17:32:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-10T17:32:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-10T17:32:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-10-10T17:32:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-10T17:32:52.172 INFO:teuthology.orchestra.run.smithi157.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-10T17:32:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-10T17:32:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-10T17:32:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-10T17:32:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-10-10T17:32:52.173 INFO:teuthology.orchestra.run.smithi157.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-10T17:32:52.174 INFO:teuthology.orchestra.run.smithi157.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-10T17:32:52.174 INFO:teuthology.orchestra.run.smithi157.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-10T17:32:52.174 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:52.174 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:52.174 INFO:teuthology.orchestra.run.smithi157.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-10-10T17:32:52.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-10T17:32:52.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-10T17:32:52.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-10T17:32:52.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-10T17:32:52.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-10T17:32:52.175 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.176 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.176 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cephfs-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.176 INFO:teuthology.orchestra.run.smithi157.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-10T17:32:52.176 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-10T17:32:52.176 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-10T17:32:52.177 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-10T17:32:52.177 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-10T17:32:52.177 INFO:teuthology.orchestra.run.smithi157.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-10T17:32:52.177 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-10T17:32:52.177 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-10T17:32:52.178 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-10T17:32:52.178 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-10T17:32:52.178 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-10T17:32:52.178 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-10T17:32:52.178 INFO:teuthology.orchestra.run.smithi157.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-10T17:32:52.179 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-10T17:32:52.179 INFO:teuthology.orchestra.run.smithi157.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-10T17:32:52.179 INFO:teuthology.orchestra.run.smithi157.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-10T17:32:52.179 INFO:teuthology.orchestra.run.smithi157.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-10T17:32:52.179 INFO:teuthology.orchestra.run.smithi157.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-10T17:32:52.179 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-10T17:32:52.180 INFO:teuthology.orchestra.run.smithi157.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-10T17:32:52.180 INFO:teuthology.orchestra.run.smithi157.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-10T17:32:52.180 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-10T17:32:52.180 INFO:teuthology.orchestra.run.smithi157.stdout: python3-portend-2.6-1.el8.noarch 2023-10-10T17:32:52.180 INFO:teuthology.orchestra.run.smithi157.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-10T17:32:52.181 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-10T17:32:52.181 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-10T17:32:52.181 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-10T17:32:52.181 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-10T17:32:52.181 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-10T17:32:52.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rados-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-10T17:32:52.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-10T17:32:52.182 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rgw-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.183 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-10T17:32:52.183 INFO:teuthology.orchestra.run.smithi157.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-10T17:32:52.183 INFO:teuthology.orchestra.run.smithi157.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-10T17:32:52.183 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-10T17:32:52.183 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-10T17:32:52.183 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-10T17:32:52.184 INFO:teuthology.orchestra.run.smithi157.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-10T17:32:52.184 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-10T17:32:52.184 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-10T17:32:52.184 INFO:teuthology.orchestra.run.smithi157.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-10T17:32:52.184 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-10T17:32:52.185 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-10T17:32:52.185 INFO:teuthology.orchestra.run.smithi157.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-10T17:32:52.185 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-fuse-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.185 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-mirror-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.185 INFO:teuthology.orchestra.run.smithi157.stdout: rbd-nbd-2:18.0.0-3908.g05a43c8f.el8.x86_64 2023-10-10T17:32:52.186 INFO:teuthology.orchestra.run.smithi157.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-10T17:32:52.186 INFO:teuthology.orchestra.run.smithi157.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-10T17:32:52.186 INFO:teuthology.orchestra.run.smithi157.stdout: uuid-1.6.2-43.el8.x86_64 2023-10-10T17:32:52.186 INFO:teuthology.orchestra.run.smithi157.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-10-10T17:32:52.186 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-10T17:32:52.187 INFO:teuthology.orchestra.run.smithi157.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-10-10T17:32:52.187 INFO:teuthology.orchestra.run.smithi157.stdout: 2023-10-10T17:32:52.187 INFO:teuthology.orchestra.run.smithi157.stdout:Complete! 2023-10-10T17:32:52.343 DEBUG:teuthology.parallel:result is None 2023-10-10T17:32:52.343 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:32:52.721 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-10T17:32:52.749 INFO:teuthology.orchestra.run.smithi078.stdout:18.0.0-3908.g05a43c8f.el8 2023-10-10T17:32:52.750 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3908.g05a43c8f.el8 2023-10-10T17:32:52.750 INFO:teuthology.task.install:The correct ceph version 18.0.0-3908.g05a43c8f is installed. 2023-10-10T17:32:52.751 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:32:52.924 DEBUG:teuthology.orchestra.run.smithi157:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-10T17:32:52.950 INFO:teuthology.orchestra.run.smithi157.stdout:18.0.0-3908.g05a43c8f.el8 2023-10-10T17:32:52.951 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3908.g05a43c8f.el8 2023-10-10T17:32:52.951 INFO:teuthology.task.install:The correct ceph version 18.0.0-3908.g05a43c8f is installed. 2023-10-10T17:32:52.952 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:32:53.342 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-10T17:32:53.370 INFO:teuthology.orchestra.run.smithi190.stdout:18.0.0-3908.g05a43c8f.el8 2023-10-10T17:32:53.371 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3908.g05a43c8f.el8 2023-10-10T17:32:53.371 INFO:teuthology.task.install:The correct ceph version 18.0.0-3908.g05a43c8f is installed. 2023-10-10T17:32:53.372 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-10T17:32:53.372 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:32:53.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-10T17:32:53.421 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:32:53.422 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-10T17:32:53.469 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:32:53.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-10T17:32:53.516 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-10T17:32:53.517 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:32:53.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2023-10-10T17:32:53.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-10T17:32:53.617 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:32:53.618 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2023-10-10T17:32:53.650 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-10T17:32:53.722 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:32:53.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2023-10-10T17:32:53.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-10T17:32:53.822 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-10T17:32:53.823 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:32:53.823 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-10T17:32:53.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-10T17:32:53.924 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:32:53.924 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-10T17:32:53.956 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-10T17:32:54.027 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:32:54.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-10T17:32:54.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-10T17:32:54.125 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-10T17:32:54.126 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:32:54.126 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2023-10-10T17:32:54.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-10T17:32:54.226 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:32:54.227 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2023-10-10T17:32:54.257 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-10T17:32:54.327 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:32:54.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2023-10-10T17:32:54.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-10T17:32:54.429 INFO:teuthology.run_tasks:Running task cephadm... 2023-10-10T17:32:54.809 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612'} 2023-10-10T17:32:54.810 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:32:54.810 INFO:tasks.cephadm:Cluster fsid is 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:32:54.811 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-10-10T17:32:54.816 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.b': '172.21.15.157', 'mon.c': '172.21.15.190'} 2023-10-10T17:32:54.816 INFO:tasks.cephadm:First mon is mon.a on smithi078 2023-10-10T17:32:54.816 INFO:tasks.cephadm:First mgr is x 2023-10-10T17:32:54.816 INFO:tasks.cephadm:Normalizing hostnames... 2023-10-10T17:32:54.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2023-10-10T17:32:54.850 DEBUG:teuthology.orchestra.run.smithi157:> sudo hostname $(hostname -s) 2023-10-10T17:32:54.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2023-10-10T17:32:54.917 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-10-10T17:32:54.917 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:32:55.298 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '05a43c8fb3988914c879a50970f9054a098e0612', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/72870/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-3908-g05a43c8f', 'node_name': '172.21.2.4+braggi04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-3908.g05a43c8f'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-09-28 20:32:16.258921', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-choffman-62793', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2023-10-10T17:32:55.722 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-choffman-62793, sha1 05a43c8fb3988914c879a50970f9054a098e0612 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=05a43c8fb3988914c879a50970f9054a098e0612 2023-10-10T17:32:55.723 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm 2023-10-10T17:32:55.723 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:56.824 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Oct 10 17:32 /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:56.824 DEBUG:teuthology.orchestra.run.smithi157:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:57.766 INFO:teuthology.orchestra.run.smithi157.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Oct 10 17:32 /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:57.766 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-choffman-62793/05a43c8fb3988914c879a50970f9054a098e0612/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:58.715 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-rw-r--. 1 ubuntu ubuntu 384133 Oct 10 17:32 /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:58.716 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:58.738 DEBUG:teuthology.orchestra.run.smithi157:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:58.759 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-10T17:32:58.802 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 on all hosts... 2023-10-10T17:32:58.803 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 pull 2023-10-10T17:32:58.806 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 pull 2023-10-10T17:32:58.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 pull 2023-10-10T17:33:03.361 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-10T17:33:03.546 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-10T17:33:03.768 INFO:teuthology.orchestra.run.smithi157.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-10T17:33:28.744 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-10-10T17:33:28.745 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-10T17:33:28.745 INFO:teuthology.orchestra.run.smithi157.stdout: "image_id": "2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b", 2023-10-10T17:33:28.745 INFO:teuthology.orchestra.run.smithi157.stdout: "repo_digests": [ 2023-10-10T17:33:28.745 INFO:teuthology.orchestra.run.smithi157.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e7b667c026dce026a4358e8fd0cc7b66586a86d2cb1765f49867fecd8b222e58" 2023-10-10T17:33:28.746 INFO:teuthology.orchestra.run.smithi157.stdout: ] 2023-10-10T17:33:28.746 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-10-10T17:33:33.833 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:33:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-10T17:33:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b", 2023-10-10T17:33:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2023-10-10T17:33:33.834 INFO:teuthology.orchestra.run.smithi078.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e7b667c026dce026a4358e8fd0cc7b66586a86d2cb1765f49867fecd8b222e58" 2023-10-10T17:33:33.835 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:33:33.835 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:33:34.360 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-10-10T17:33:34.360 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-10T17:33:34.361 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b", 2023-10-10T17:33:34.361 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2023-10-10T17:33:34.361 INFO:teuthology.orchestra.run.smithi190.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:e7b667c026dce026a4358e8fd0cc7b66586a86d2cb1765f49867fecd8b222e58" 2023-10-10T17:33:34.361 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2023-10-10T17:33:34.361 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-10-10T17:33:34.380 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2023-10-10T17:33:34.412 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph 2023-10-10T17:33:34.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2023-10-10T17:33:34.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2023-10-10T17:33:34.508 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /etc/ceph 2023-10-10T17:33:34.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2023-10-10T17:33:34.565 INFO:tasks.cephadm:Writing seed config... 2023-10-10T17:33:34.579 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-10-10T17:33:34.579 INFO:tasks.cephadm: override: [client] debug client = 20 2023-10-10T17:33:34.579 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-10-10T17:33:34.580 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-10-10T17:33:34.580 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-10-10T17:33:34.580 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-10-10T17:33:34.580 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-10-10T17:33:34.581 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-10-10T17:33:34.581 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-10-10T17:33:34.581 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-10-10T17:33:34.582 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-10-10T17:33:34.582 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-10-10T17:33:34.582 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-10-10T17:33:34.582 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-10-10T17:33:34.583 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-10-10T17:33:34.583 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-10-10T17:33:34.583 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-10-10T17:33:34.584 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2023-10-10T17:33:34.584 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-10-10T17:33:34.584 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-10-10T17:33:34.584 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-10-10T17:33:34.585 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-10-10T17:33:34.585 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-10-10T17:33:34.585 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-10-10T17:33:34.586 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-10-10T17:33:34.586 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-10-10T17:33:34.586 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-10-10T17:33:34.586 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-10-10T17:33:34.587 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-10-10T17:33:34.587 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-10-10T17:33:34.587 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2023-10-10T17:33:34.587 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-10-10T17:33:34.588 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-10-10T17:33:34.588 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-10-10T17:33:34.588 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-10-10T17:33:34.588 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-10-10T17:33:34.589 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-10-10T17:33:34.589 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-10-10T17:33:34.589 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-10-10T17:33:34.589 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-10-10T17:33:34.590 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-10-10T17:33:34.590 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-10-10T17:33:34.590 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-10-10T17:33:34.591 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-10-10T17:33:34.591 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-10-10T17:33:34.592 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:33:34.592 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-10-10T17:33:34.610 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0ba2616a-6793-11ee-8db6-212e2dc638e7 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-10-10T17:33:34.610 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a.service 2023-10-10T17:33:34.653 DEBUG:teuthology.orchestra.run.smithi078:mgr.x> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.x.service 2023-10-10T17:33:34.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:33:34.680 INFO:tasks.cephadm:Bootstrapping... 2023-10-10T17:33:34.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 -v bootstrap --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:33:34.706 INFO:journalctl@ceph.mgr.x.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:33:34.927 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-10-10T17:33:34.928 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612', '-v', 'bootstrap', '--fsid', '0ba2616a-6793-11ee-8db6-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.78', '--skip-admin-label'] 2023-10-10T17:33:35.078 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.1.1 2023-10-10T17:33:35.079 INFO:teuthology.orchestra.run.smithi078.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-10-10T17:33:35.079 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying podman|docker is present... 2023-10-10T17:33:35.220 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.1.1 2023-10-10T17:33:35.221 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying lvm2 is present... 2023-10-10T17:33:35.221 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying time synchronization is in place... 2023-10-10T17:33:35.227 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-10T17:33:35.228 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-10T17:33:35.233 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-10T17:33:35.233 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-10-10T17:33:35.239 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2023-10-10T17:33:35.245 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2023-10-10T17:33:35.246 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2023-10-10T17:33:35.246 INFO:teuthology.orchestra.run.smithi078.stdout:Repeating the final host check... 2023-10-10T17:33:35.387 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.1.1 2023-10-10T17:33:35.387 INFO:teuthology.orchestra.run.smithi078.stdout:podman (/bin/podman) version 4.1.1 is present 2023-10-10T17:33:35.387 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl is present 2023-10-10T17:33:35.388 INFO:teuthology.orchestra.run.smithi078.stdout:lvcreate is present 2023-10-10T17:33:35.393 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-10T17:33:35.393 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-10T17:33:35.399 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-10T17:33:35.399 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-10-10T17:33:35.404 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2023-10-10T17:33:35.410 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2023-10-10T17:33:35.410 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2023-10-10T17:33:35.411 INFO:teuthology.orchestra.run.smithi078.stdout:Host looks OK 2023-10-10T17:33:35.411 INFO:teuthology.orchestra.run.smithi078.stdout:Cluster fsid: 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:35.411 INFO:teuthology.orchestra.run.smithi078.stdout:Acquiring lock 140082519091688 on /run/cephadm/0ba2616a-6793-11ee-8db6-212e2dc638e7.lock 2023-10-10T17:33:35.411 INFO:teuthology.orchestra.run.smithi078.stdout:Lock 140082519091688 acquired on /run/cephadm/0ba2616a-6793-11ee-8db6-212e2dc638e7.lock 2023-10-10T17:33:35.412 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 3300 ... 2023-10-10T17:33:35.412 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 6789 ... 2023-10-10T17:33:35.412 INFO:teuthology.orchestra.run.smithi078.stdout:Base mon IP(s) is [172.21.15.78:3300, 172.21.15.78:6789], mon addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-10-10T17:33:35.415 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-10-10T17:33:35.415 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2023-10-10T17:33:35.417 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-10-10T17:33:35.417 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-10-10T17:33:35.418 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-10-10T17:33:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-10-10T17:33:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-10-10T17:33:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-10T17:33:35.420 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-10-10T17:33:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 fe80::d6a8:fe40:6759:a046/64 scope link noprefixroute 2023-10-10T17:33:35.421 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-10T17:33:35.422 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2023-10-10T17:33:35.422 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2023-10-10T17:33:35.422 INFO:teuthology.orchestra.run.smithi078.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-10-10T17:33:35.422 INFO:teuthology.orchestra.run.smithi078.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-10-10T17:33:35.423 INFO:teuthology.orchestra.run.smithi078.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-10T17:33:35.766 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b 2023-10-10T17:33:35.766 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612... 2023-10-10T17:33:35.767 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Getting image source signatures 2023-10-10T17:33:35.767 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:788df1f70699415e5b73051cdcd2e0fd42e95c6bdebf4a5149631a5f4ae456ad 2023-10-10T17:33:35.767 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:c16254b0810587cdd81d0647eccf1c3e4d200bf4eeba6596492e593ad063d4ee 2023-10-10T17:33:35.767 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying config sha256:2d174a053211967f7fee8451f23a750f1e106c9fe0321fd928e66358a2607a5b 2023-10-10T17:33:35.767 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Writing manifest to image destination 2023-10-10T17:33:35.768 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Storing signatures 2023-10-10T17:33:36.614 INFO:teuthology.orchestra.run.smithi078.stdout:ceph: stdout ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev) 2023-10-10T17:33:36.614 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph version: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev) 2023-10-10T17:33:36.614 INFO:teuthology.orchestra.run.smithi078.stdout:Extracting ceph user uid/gid from container image... 2023-10-10T17:33:37.368 INFO:teuthology.orchestra.run.smithi078.stdout:stat: stdout 167 167 2023-10-10T17:33:37.368 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial keys... 2023-10-10T17:33:38.117 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQDxiiVlu78fLBAAqfFU1fHozcnbP/6CwF5zog== 2023-10-10T17:33:38.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQDyiiVl9fGGHRAAN8m1R/UoXMFr1FuUpgxCww== 2023-10-10T17:33:39.518 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQDziiVlyA+iDBAAb2Hf2V3ujWJBXzGAo0CpZw== 2023-10-10T17:33:39.518 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial monmap... 2023-10-10T17:33:40.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-10T17:33:40.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-10-10T17:33:40.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:40.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-10T17:33:40.265 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-10T17:33:40.265 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = pacific 2023-10-10T17:33:40.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: set fsid to 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:40.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-10T17:33:40.266 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:33:40.266 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mon... 2023-10-10T17:33:41.112 INFO:teuthology.orchestra.run.smithi078.stdout:create mon.a on 2023-10-10T17:33:41.207 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-10-10T17:33:41.294 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-10-10T17:33:41.387 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7.target → /etc/systemd/system/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7.target. 2023-10-10T17:33:41.387 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7.target → /etc/systemd/system/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7.target. 2023-10-10T17:33:41.514 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a 2023-10-10T17:33:41.514 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a.service: Unit ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a.service not loaded. 2023-10-10T17:33:41.598 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7.target.wants/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@.service. 2023-10-10T17:33:41.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:41 smithi078 systemd[1]: Starting Ceph mon.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:33:42.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:41 smithi078 podman[98955]: 2023-10-10T17:33:42.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:42 smithi078 bash[98955]: f80f890d2257ca4d712757b28ece13f80cb926a1b6b13fcc2c45e9c22d0d2d48 2023-10-10T17:33:42.200 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-10T17:33:42.201 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-10-10T17:33:42.210 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-10T17:33:42.210 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-10-10T17:33:42.210 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-10-10T17:33:42.211 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2023-10-10T17:33:42.211 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon to start... 2023-10-10T17:33:42.211 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon... 2023-10-10T17:33:42.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:42 smithi078 ceph-mon[98995]: mkfs 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:42.450 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:42 smithi078 ceph-mon[98995]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-10T17:33:42.451 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:42 smithi078 systemd[1]: Started Ceph mon.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:33:43.263 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout cluster: 2023-10-10T17:33:43.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout id: 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:43.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-10-10T17:33:43.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:43.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout services: 2023-10-10T17:33:43.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.643498s) 2023-10-10T17:33:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-10-10T17:33:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-10-10T17:33:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout data: 2023-10-10T17:33:43.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-10-10T17:33:43.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-10-10T17:33:43.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-10-10T17:33:43.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pgs: 2023-10-10T17:33:43.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:43.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.693+0000 7f9ee2176700 1 Processor -- start 2023-10-10T17:33:43.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.693+0000 7f9ee2176700 1 -- start start 2023-10-10T17:33:43.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.693+0000 7f9ee2176700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:43.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.693+0000 7f9ee2176700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc079e00 con 0x7f9edc1064f0 2023-10-10T17:33:43.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9edb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:43.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9edb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40808/0 (socket says 172.21.15.78:40808) 2023-10-10T17:33:43.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9edb7fe700 1 -- 172.21.15.78:0/2589155106 learned_addr learned my addr 172.21.15.78:0/2589155106 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:43.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9edb7fe700 1 -- 172.21.15.78:0/2589155106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9edc108e10 con 0x7f9edc1064f0 2023-10-10T17:33:43.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9edb7fe700 1 --2- 172.21.15.78:0/2589155106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc1088d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00098a0 tx=0x7f9ed0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9447b5438e7e53a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:43.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9eda7fc700 1 -- 172.21.15.78:0/2589155106 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed0004160 con 0x7f9edc1064f0 2023-10-10T17:33:43.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9eda7fc700 1 -- 172.21.15.78:0/2589155106 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed00042c0 con 0x7f9edc1064f0 2023-10-10T17:33:43.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.694+0000 7f9eda7fc700 1 -- 172.21.15.78:0/2589155106 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed00045a0 con 0x7f9edc1064f0 2023-10-10T17:33:43.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 -- 172.21.15.78:0/2589155106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 msgr2=0x7f9edc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:43.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 --2- 172.21.15.78:0/2589155106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc1088d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00098a0 tx=0x7f9ed0009bb0 comp rx=0 tx=0).stop 2023-10-10T17:33:43.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 -- 172.21.15.78:0/2589155106 shutdown_connections 2023-10-10T17:33:43.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 --2- 172.21.15.78:0/2589155106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc1088d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:43.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 -- 172.21.15.78:0/2589155106 >> 172.21.15.78:0/2589155106 conn(0x7f9edc0ffed0 msgr2=0x7f9edc102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:43.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 -- 172.21.15.78:0/2589155106 shutdown_connections 2023-10-10T17:33:43.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 -- 172.21.15.78:0/2589155106 wait complete. 2023-10-10T17:33:43.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.695+0000 7f9ee2176700 1 Processor -- start 2023-10-10T17:33:43.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ee2176700 1 -- start start 2023-10-10T17:33:43.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ee2176700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:43.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ee2176700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9edc199470 con 0x7f9edc1064f0 2023-10-10T17:33:43.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9edb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:43.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9edb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc198f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40810/0 (socket says 172.21.15.78:40810) 2023-10-10T17:33:43.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9edb7fe700 1 -- 172.21.15.78:0/3889309745 learned_addr learned my addr 172.21.15.78:0/3889309745 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:43.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9edb7fe700 1 -- 172.21.15.78:0/3889309745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed0009550 con 0x7f9edc1064f0 2023-10-10T17:33:43.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9edb7fe700 1 --2- 172.21.15.78:0/3889309745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc198f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9edc079240 tx=0x7f9ed0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:43.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ed8ff9700 1 -- 172.21.15.78:0/3889309745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed0011630 con 0x7f9edc1064f0 2023-10-10T17:33:43.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ed8ff9700 1 -- 172.21.15.78:0/3889309745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed0011790 con 0x7f9edc1064f0 2023-10-10T17:33:43.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ed8ff9700 1 -- 172.21.15.78:0/3889309745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9ed0011a30 con 0x7f9edc1064f0 2023-10-10T17:33:43.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9edc1091f0 con 0x7f9edc1064f0 2023-10-10T17:33:43.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.696+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9edc109060 con 0x7f9edc1064f0 2023-10-10T17:33:43.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.697+0000 7f9ed8ff9700 1 -- 172.21.15.78:0/3889309745 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9ed0022070 con 0x7f9edc1064f0 2023-10-10T17:33:43.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.697+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9edc192dc0 con 0x7f9edc1064f0 2023-10-10T17:33:43.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.697+0000 7f9ed8ff9700 1 -- 172.21.15.78:0/3889309745 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9ed001c450 con 0x7f9edc1064f0 2023-10-10T17:33:43.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.699+0000 7f9ed8ff9700 1 -- 172.21.15.78:0/3889309745 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f9ed001b450 con 0x7f9edc1064f0 2023-10-10T17:33:43.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.770+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f9edc062120 con 0x7f9edc1064f0 2023-10-10T17:33:43.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.770+0000 7f9ed8ff9700 1 -- 172.21.15.78:0/3889309745 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f9ed001b5f0 con 0x7f9edc1064f0 2023-10-10T17:33:43.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.770+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 msgr2=0x7f9edc198f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:43.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.770+0000 7f9ee2176700 1 --2- 172.21.15.78:0/3889309745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc198f30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9edc079240 tx=0x7f9ed0004750 comp rx=0 tx=0).stop 2023-10-10T17:33:43.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.771+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 shutdown_connections 2023-10-10T17:33:43.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.771+0000 7f9ee2176700 1 --2- 172.21.15.78:0/3889309745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9edc1064f0 0x7f9edc198f30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:43.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.771+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 >> 172.21.15.78:0/3889309745 conn(0x7f9edc0ffed0 msgr2=0x7f9edc101870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:43.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.771+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 shutdown_connections 2023-10-10T17:33:43.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:42.771+0000 7f9ee2176700 1 -- 172.21.15.78:0/3889309745 wait complete. 2023-10-10T17:33:43.276 INFO:teuthology.orchestra.run.smithi078.stdout:mon is available 2023-10-10T17:33:43.276 INFO:teuthology.orchestra.run.smithi078.stdout:Assimilating anything we can from ceph.conf... 2023-10-10T17:33:43.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:43 smithi078 ceph-mon[98995]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-10T17:33:43.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:43 smithi078 ceph-mon[98995]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} 2023-10-10T17:33:43.474 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:43 smithi078 ceph-mon[98995]: fsmap 2023-10-10T17:33:43.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:43 smithi078 ceph-mon[98995]: osdmap e1: 0 total, 0 up, 0 in 2023-10-10T17:33:43.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:43 smithi078 ceph-mon[98995]: mgrmap e1: no daemons active 2023-10-10T17:33:43.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:43 smithi078 ceph-mon[98995]: from='client.? 172.21.15.78:0/3889309745' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-10T17:33:44.202 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:44.208 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2023-10-10T17:33:44.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:44.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-10-10T17:33:44.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-10T17:33:44.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-10T17:33:44.209 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-10T17:33:44.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-10T17:33:44.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:44.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-10T17:33:44.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-10T17:33:44.210 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:44.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2023-10-10T17:33:44.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-10-10T17:33:44.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-10-10T17:33:44.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-10-10T17:33:44.211 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-10T17:33:44.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-10T17:33:44.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.729+0000 7f51f2f16700 1 Processor -- start 2023-10-10T17:33:44.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.729+0000 7f51f2f16700 1 -- start start 2023-10-10T17:33:44.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.729+0000 7f51f2f16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec104730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:44.212 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.729+0000 7f51f2f16700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec079dc0 con 0x7f51ec104320 2023-10-10T17:33:44.213 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.729+0000 7f51f0cb2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec104730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:44.213 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.729+0000 7f51f0cb2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec104730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40812/0 (socket says 172.21.15.78:40812) 2023-10-10T17:33:44.213 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.729+0000 7f51f0cb2700 1 -- 172.21.15.78:0/352657869 learned_addr learned my addr 172.21.15.78:0/352657869 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:44.213 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f0cb2700 1 -- 172.21.15.78:0/352657869 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec079f00 con 0x7f51ec104320 2023-10-10T17:33:44.213 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f0cb2700 1 --2- 172.21.15.78:0/352657869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec104730 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f51e00098a0 tx=0x7f51e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca5d044c185062d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:44.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51eb7fe700 1 -- 172.21.15.78:0/352657869 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51e0004160 con 0x7f51ec104320 2023-10-10T17:33:44.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51eb7fe700 1 -- 172.21.15.78:0/352657869 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e00042c0 con 0x7f51ec104320 2023-10-10T17:33:44.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51eb7fe700 1 -- 172.21.15.78:0/352657869 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51e00045a0 con 0x7f51ec104320 2023-10-10T17:33:44.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f2f16700 1 -- 172.21.15.78:0/352657869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 msgr2=0x7f51ec104730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:44.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f2f16700 1 --2- 172.21.15.78:0/352657869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec104730 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f51e00098a0 tx=0x7f51e0009bb0 comp rx=0 tx=0).stop 2023-10-10T17:33:44.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f2f16700 1 -- 172.21.15.78:0/352657869 shutdown_connections 2023-10-10T17:33:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f2f16700 1 --2- 172.21.15.78:0/352657869 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec104730 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f2f16700 1 -- 172.21.15.78:0/352657869 >> 172.21.15.78:0/352657869 conn(0x7f51ec0ffed0 msgr2=0x7f51ec102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f2f16700 1 -- 172.21.15.78:0/352657869 shutdown_connections 2023-10-10T17:33:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.730+0000 7f51f2f16700 1 -- 172.21.15.78:0/352657869 wait complete. 2023-10-10T17:33:44.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.731+0000 7f51f2f16700 1 Processor -- start 2023-10-10T17:33:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.731+0000 7f51f2f16700 1 -- start start 2023-10-10T17:33:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.731+0000 7f51f2f16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec19b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.731+0000 7f51f2f16700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec19b6a0 con 0x7f51ec104320 2023-10-10T17:33:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.731+0000 7f51f0cb2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec19b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.731+0000 7f51f0cb2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec19b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40814/0 (socket says 172.21.15.78:40814) 2023-10-10T17:33:44.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.731+0000 7f51f0cb2700 1 -- 172.21.15.78:0/1996968509 learned_addr learned my addr 172.21.15.78:0/1996968509 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:44.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51f0cb2700 1 -- 172.21.15.78:0/1996968509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e0009550 con 0x7f51ec104320 2023-10-10T17:33:44.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51f0cb2700 1 --2- 172.21.15.78:0/1996968509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec19b160 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f51ec079200 tx=0x7f51e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:44.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51e0011630 con 0x7f51ec104320 2023-10-10T17:33:44.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e0011790 con 0x7f51ec104320 2023-10-10T17:33:44.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec104fe0 con 0x7f51ec104320 2023-10-10T17:33:44.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f51e0011a30 con 0x7f51ec104320 2023-10-10T17:33:44.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec104e50 con 0x7f51ec104320 2023-10-10T17:33:44.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f51e0022070 con 0x7f51ec104320 2023-10-10T17:33:44.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.732+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f51e001c450 con 0x7f51ec104320 2023-10-10T17:33:44.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.733+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec194db0 con 0x7f51ec104320 2023-10-10T17:33:44.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.735+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f51e001b450 con 0x7f51ec104320 2023-10-10T17:33:44.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.802+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f51ec062120 con 0x7f51ec104320 2023-10-10T17:33:44.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.805+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f51e001b5f0 con 0x7f51ec104320 2023-10-10T17:33:44.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.805+0000 7f51e9ffb700 1 -- 172.21.15.78:0/1996968509 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f51ec062120 con 0x7f51ec104320 2023-10-10T17:33:44.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.806+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 msgr2=0x7f51ec19b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:44.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.806+0000 7f51f2f16700 1 --2- 172.21.15.78:0/1996968509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec19b160 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f51ec079200 tx=0x7f51e0004750 comp rx=0 tx=0).stop 2023-10-10T17:33:44.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.806+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 shutdown_connections 2023-10-10T17:33:44.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.806+0000 7f51f2f16700 1 --2- 172.21.15.78:0/1996968509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f51ec104320 0x7f51ec19b160 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:44.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.806+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 >> 172.21.15.78:0/1996968509 conn(0x7f51ec0ffed0 msgr2=0x7f51ec191ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:44.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.806+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 shutdown_connections 2023-10-10T17:33:44.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:43.806+0000 7f51f2f16700 1 -- 172.21.15.78:0/1996968509 wait complete. 2023-10-10T17:33:44.221 INFO:teuthology.orchestra.run.smithi078.stdout:Generating new minimal ceph.conf... 2023-10-10T17:33:44.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:44 smithi078 ceph-mon[98995]: from='client.? 172.21.15.78:0/1996968509' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-10T17:33:44.415 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:44 smithi078 ceph-mon[98995]: from='client.? 172.21.15.78:0/1996968509' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-10-10T17:33:45.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:45 smithi078 ceph-mon[98995]: from='client.? 172.21.15.78:0/1204174986' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:33:46.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.684+0000 7f997dcd9700 1 Processor -- start 2023-10-10T17:33:46.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.684+0000 7f997dcd9700 1 -- start start 2023-10-10T17:33:46.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.684+0000 7f997dcd9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978108830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:46.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.684+0000 7f997dcd9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9978079e20 con 0x7f9978106450 2023-10-10T17:33:46.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978108830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:46.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978108830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40816/0 (socket says 172.21.15.78:40816) 2023-10-10T17:33:46.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99777fe700 1 -- 172.21.15.78:0/4267056246 learned_addr learned my addr 172.21.15.78:0/4267056246 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:46.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99777fe700 1 -- 172.21.15.78:0/4267056246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978108d70 con 0x7f9978106450 2023-10-10T17:33:46.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99777fe700 1 --2- 172.21.15.78:0/4267056246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978108830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f996c0098a0 tx=0x7f996c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a37cdcde2943e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:46.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99767fc700 1 -- 172.21.15.78:0/4267056246 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f996c004160 con 0x7f9978106450 2023-10-10T17:33:46.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99767fc700 1 -- 172.21.15.78:0/4267056246 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f996c0042c0 con 0x7f9978106450 2023-10-10T17:33:46.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.685+0000 7f99767fc700 1 -- 172.21.15.78:0/4267056246 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f996c011a00 con 0x7f9978106450 2023-10-10T17:33:46.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 -- 172.21.15.78:0/4267056246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 msgr2=0x7f9978108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:46.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 --2- 172.21.15.78:0/4267056246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978108830 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f996c0098a0 tx=0x7f996c009bb0 comp rx=0 tx=0).stop 2023-10-10T17:33:46.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 -- 172.21.15.78:0/4267056246 shutdown_connections 2023-10-10T17:33:46.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 --2- 172.21.15.78:0/4267056246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978108830 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:46.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 -- 172.21.15.78:0/4267056246 >> 172.21.15.78:0/4267056246 conn(0x7f99780ffe70 msgr2=0x7f9978102280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:46.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 -- 172.21.15.78:0/4267056246 shutdown_connections 2023-10-10T17:33:46.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 -- 172.21.15.78:0/4267056246 wait complete. 2023-10-10T17:33:46.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.686+0000 7f997dcd9700 1 Processor -- start 2023-10-10T17:33:46.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f997dcd9700 1 -- start start 2023-10-10T17:33:46.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f997dcd9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:46.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f997dcd9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9978199c50 con 0x7f9978106450 2023-10-10T17:33:46.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f99777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:46.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f99777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40818/0 (socket says 172.21.15.78:40818) 2023-10-10T17:33:46.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f99777fe700 1 -- 172.21.15.78:0/1204174986 learned_addr learned my addr 172.21.15.78:0/1204174986 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:46.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f99777fe700 1 -- 172.21.15.78:0/1204174986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f996c009550 con 0x7f9978106450 2023-10-10T17:33:46.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f99777fe700 1 --2- 172.21.15.78:0/1204174986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978199710 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f996c009fb0 tx=0x7f996c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:46.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f9974ff9700 1 -- 172.21.15.78:0/1204174986 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f996c003930 con 0x7f9978106450 2023-10-10T17:33:46.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f9974ff9700 1 -- 172.21.15.78:0/1204174986 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f996c003a90 con 0x7f9978106450 2023-10-10T17:33:46.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f9974ff9700 1 -- 172.21.15.78:0/1204174986 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f996c019610 con 0x7f9978106450 2023-10-10T17:33:46.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.687+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9978109ca0 con 0x7f9978106450 2023-10-10T17:33:46.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.688+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9978109b10 con 0x7f9978106450 2023-10-10T17:33:46.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.688+0000 7f9974ff9700 1 -- 172.21.15.78:0/1204174986 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f996c019bd0 con 0x7f9978106450 2023-10-10T17:33:46.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.688+0000 7f9974ff9700 1 -- 172.21.15.78:0/1204174986 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f996c0243f0 con 0x7f9978106450 2023-10-10T17:33:46.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.689+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9978062120 con 0x7f9978106450 2023-10-10T17:33:46.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.691+0000 7f9974ff9700 1 -- 172.21.15.78:0/1204174986 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f996c003c00 con 0x7f9978106450 2023-10-10T17:33:46.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.757+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f997819a190 con 0x7f9978106450 2023-10-10T17:33:46.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.758+0000 7f9974ff9700 1 -- 172.21.15.78:0/1204174986 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f996c01a9d0 con 0x7f9978106450 2023-10-10T17:33:46.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.758+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 msgr2=0x7f9978199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:46.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.758+0000 7f997dcd9700 1 --2- 172.21.15.78:0/1204174986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978199710 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f996c009fb0 tx=0x7f996c011ad0 comp rx=0 tx=0).stop 2023-10-10T17:33:46.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.758+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 shutdown_connections 2023-10-10T17:33:46.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.758+0000 7f997dcd9700 1 --2- 172.21.15.78:0/1204174986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9978106450 0x7f9978199710 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:46.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.758+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 >> 172.21.15.78:0/1204174986 conn(0x7f99780ffe70 msgr2=0x7f99781014f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:46.664 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.759+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 shutdown_connections 2023-10-10T17:33:46.664 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:44.759+0000 7f997dcd9700 1 -- 172.21.15.78:0/1204174986 wait complete. 2023-10-10T17:33:46.664 INFO:teuthology.orchestra.run.smithi078.stdout:Restarting the monitor... 2023-10-10T17:33:47.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:46 smithi078 systemd[1]: Stopping Ceph mon.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:33:47.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:47 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-a[98971]: 2023-10-10T17:33:47.251+0000 7fec8b352700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:33:47.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:47 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-a[98971]: 2023-10-10T17:33:47.251+0000 7fec8b352700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-10T17:33:47.966 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:47 smithi078 bash[99328]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-a 2023-10-10T17:33:48.247 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:47 smithi078 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a.service: Succeeded. 2023-10-10T17:33:48.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:47 smithi078 systemd[1]: Stopped Ceph mon.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:33:48.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:47 smithi078 systemd[1]: Starting Ceph mon.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:33:48.518 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 podman[99454]: 2023-10-10T17:33:48.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: set uid:gid to 167:167 (ceph:ceph) 2023-10-10T17:33:48.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev), process ceph-mon, pid 7 2023-10-10T17:33:48.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: pidfile_write: ignore empty --pid-file 2023-10-10T17:33:48.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: load: jerasure load: lrc 2023-10-10T17:33:48.519 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: RocksDB version: 7.9.2 2023-10-10T17:33:48.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Git sha 0 2023-10-10T17:33:48.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Compile date 2023-09-28 19:19:12 2023-10-10T17:33:48.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: DB SUMMARY 2023-10-10T17:33:48.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: DB Session ID: CXQ291CS249F2G2JCUMZ 2023-10-10T17:33:48.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: CURRENT file: CURRENT 2023-10-10T17:33:48.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: IDENTITY file: IDENTITY 2023-10-10T17:33:48.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-10-10T17:33:48.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-10-10T17:33:48.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99218 ; 2023-10-10T17:33:48.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.error_if_exists: 0 2023-10-10T17:33:48.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.create_if_missing: 0 2023-10-10T17:33:48.522 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.paranoid_checks: 1 2023-10-10T17:33:48.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-10T17:33:48.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-10T17:33:48.523 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-10T17:33:48.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.env: 0x5578d7a4f720 2023-10-10T17:33:48.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.fs: PosixFileSystem 2023-10-10T17:33:48.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.info_log: 0x5578d95c93c0 2023-10-10T17:33:48.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_file_opening_threads: 16 2023-10-10T17:33:48.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.statistics: (nil) 2023-10-10T17:33:48.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.use_fsync: 0 2023-10-10T17:33:48.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_log_file_size: 0 2023-10-10T17:33:48.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-10T17:33:48.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-10T17:33:48.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.keep_log_file_num: 1000 2023-10-10T17:33:48.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.recycle_log_file_num: 0 2023-10-10T17:33:48.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.allow_fallocate: 1 2023-10-10T17:33:48.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.allow_mmap_reads: 0 2023-10-10T17:33:48.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.allow_mmap_writes: 0 2023-10-10T17:33:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.use_direct_reads: 0 2023-10-10T17:33:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-10T17:33:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.create_missing_column_families: 0 2023-10-10T17:33:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.db_log_dir: 2023-10-10T17:33:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.wal_dir: 2023-10-10T17:33:48.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-10T17:33:48.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-10T17:33:48.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-10T17:33:48.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-10T17:33:48.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-10T17:33:48.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-10T17:33:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.advise_random_on_open: 1 2023-10-10T17:33:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.db_write_buffer_size: 0 2023-10-10T17:33:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.write_buffer_manager: 0x5578d883a3c0 2023-10-10T17:33:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-10T17:33:48.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-10T17:33:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-10T17:33:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.rate_limiter: (nil) 2023-10-10T17:33:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-10T17:33:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.wal_recovery_mode: 2 2023-10-10T17:33:48.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.enable_thread_tracking: 0 2023-10-10T17:33:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.enable_pipelined_write: 0 2023-10-10T17:33:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.unordered_write: 0 2023-10-10T17:33:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-10T17:33:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-10T17:33:48.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-10T17:33:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-10T17:33:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.row_cache: None 2023-10-10T17:33:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.wal_filter: None 2023-10-10T17:33:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-10T17:33:48.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.allow_ingest_behind: 0 2023-10-10T17:33:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.two_write_queues: 0 2023-10-10T17:33:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.manual_wal_flush: 0 2023-10-10T17:33:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.wal_compression: 0 2023-10-10T17:33:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.atomic_flush: 0 2023-10-10T17:33:48.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-10T17:33:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-10T17:33:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-10T17:33:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.log_readahead_size: 0 2023-10-10T17:33:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-10T17:33:48.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.best_efforts_recovery: 0 2023-10-10T17:33:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-10T17:33:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-10T17:33:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.allow_data_in_errors: 0 2023-10-10T17:33:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.db_host_id: __hostname__ 2023-10-10T17:33:48.535 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-10T17:33:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_background_jobs: 2 2023-10-10T17:33:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_background_compactions: -1 2023-10-10T17:33:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_subcompactions: 1 2023-10-10T17:33:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-10T17:33:48.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-10T17:33:48.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-10T17:33:48.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_total_wal_size: 0 2023-10-10T17:33:48.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-10T17:33:48.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-10T17:33:48.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-10T17:33:48.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-10T17:33:48.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_open_files: -1 2023-10-10T17:33:48.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bytes_per_sync: 0 2023-10-10T17:33:48.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-10T17:33:48.538 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-10T17:33:48.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_readahead_size: 0 2023-10-10T17:33:48.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_background_flushes: -1 2023-10-10T17:33:48.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Compression algorithms supported: 2023-10-10T17:33:48.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-10T17:33:48.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kZSTD supported: 0 2023-10-10T17:33:48.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kXpressCompression supported: 0 2023-10-10T17:33:48.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kLZ4HCCompression supported: 1 2023-10-10T17:33:48.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kZlibCompression supported: 1 2023-10-10T17:33:48.540 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kSnappyCompression supported: 1 2023-10-10T17:33:48.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kLZ4Compression supported: 1 2023-10-10T17:33:48.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: kBZip2Compression supported: 0 2023-10-10T17:33:48.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-10T17:33:48.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-10T17:33:48.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-10-10T17:33:48.541 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-10T17:33:48.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-10T17:33:48.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.merge_operator: 2023-10-10T17:33:48.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_filter: None 2023-10-10T17:33:48.542 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_filter_factory: None 2023-10-10T17:33:48.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.sst_partitioner_factory: None 2023-10-10T17:33:48.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-10T17:33:48.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-10T17:33:48.543 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5578d95c94e0) 2023-10-10T17:33:48.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2023-10-10T17:33:48.543 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-10T17:33:48.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-10T17:33:48.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2023-10-10T17:33:48.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2023-10-10T17:33:48.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2023-10-10T17:33:48.544 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2023-10-10T17:33:48.545 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-10T17:33:48.545 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2023-10-10T17:33:48.545 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2023-10-10T17:33:48.545 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x5578d88db1f0 2023-10-10T17:33:48.545 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2023-10-10T17:33:48.546 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2023-10-10T17:33:48.546 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2023-10-10T17:33:48.546 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2023-10-10T17:33:48.546 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2023-10-10T17:33:48.546 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2023-10-10T17:33:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2023-10-10T17:33:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2023-10-10T17:33:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2023-10-10T17:33:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2023-10-10T17:33:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2023-10-10T17:33:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2023-10-10T17:33:48.548 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2023-10-10T17:33:48.548 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2023-10-10T17:33:48.548 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2023-10-10T17:33:48.548 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2023-10-10T17:33:48.549 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2023-10-10T17:33:48.549 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2023-10-10T17:33:48.549 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2023-10-10T17:33:48.549 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2023-10-10T17:33:48.549 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2023-10-10T17:33:48.549 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2023-10-10T17:33:48.550 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2023-10-10T17:33:48.550 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2023-10-10T17:33:48.550 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2023-10-10T17:33:48.550 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2023-10-10T17:33:48.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.write_buffer_size: 33554432 2023-10-10T17:33:48.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_write_buffer_number: 2 2023-10-10T17:33:48.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression: NoCompression 2023-10-10T17:33:48.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression: Disabled 2023-10-10T17:33:48.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.prefix_extractor: nullptr 2023-10-10T17:33:48.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-10T17:33:48.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.num_levels: 7 2023-10-10T17:33:48.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-10T17:33:48.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-10T17:33:48.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-10T17:33:48.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-10T17:33:48.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-10T17:33:48.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-10T17:33:48.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-10T17:33:48.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-10T17:33:48.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-10T17:33:48.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-10T17:33:48.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-10T17:33:48.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-10T17:33:48.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-10T17:33:48.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.level: 32767 2023-10-10T17:33:48.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.strategy: 0 2023-10-10T17:33:48.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-10T17:33:48.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-10T17:33:48.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-10T17:33:48.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-10T17:33:48.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.enabled: false 2023-10-10T17:33:48.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-10T17:33:48.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-10T17:33:48.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-10T17:33:48.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-10T17:33:48.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.target_file_size_base: 67108864 2023-10-10T17:33:48.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-10T17:33:48.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-10T17:33:48.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-10T17:33:48.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-10T17:33:48.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-10T17:33:48.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-10T17:33:48.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-10T17:33:48.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-10T17:33:48.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-10T17:33:48.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-10T17:33:48.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-10T17:33:48.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-10T17:33:48.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-10T17:33:48.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-10T17:33:48.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.arena_block_size: 1048576 2023-10-10T17:33:48.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-10T17:33:48.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-10T17:33:48.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.disable_auto_compactions: 0 2023-10-10T17:33:48.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-10T17:33:48.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-10T17:33:48.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-10T17:33:48.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-10T17:33:48.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-10T17:33:48.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-10T17:33:48.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-10T17:33:48.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-10T17:33:48.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-10T17:33:48.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-10T17:33:48.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.table_properties_collectors: 2023-10-10T17:33:48.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.inplace_update_support: 0 2023-10-10T17:33:48.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-10T17:33:48.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-10T17:33:48.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-10T17:33:48.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-10T17:33:48.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.bloom_locality: 0 2023-10-10T17:33:48.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.max_successive_merges: 0 2023-10-10T17:33:48.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-10T17:33:48.566 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.paranoid_file_checks: 0 2023-10-10T17:33:48.566 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.force_consistency_checks: 1 2023-10-10T17:33:48.566 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.report_bg_io_stats: 0 2023-10-10T17:33:48.566 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.ttl: 2592000 2023-10-10T17:33:48.566 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-10T17:33:48.567 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-10T17:33:48.567 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-10T17:33:48.569 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.enable_blob_files: false 2023-10-10T17:33:48.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.min_blob_size: 0 2023-10-10T17:33:48.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.blob_file_size: 268435456 2023-10-10T17:33:48.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-10T17:33:48.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-10T17:33:48.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-10T17:33:48.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-10T17:33:48.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-10T17:33:48.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.blob_file_starting_level: 0 2023-10-10T17:33:48.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-10T17:33:48.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-10-10T17:33:48.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-10-10T17:33:48.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a9564ec0-1376-4ea5-a719-5b3c7da57223 2023-10-10T17:33:48.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959228446549, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-10-10T17:33:48.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-10-10T17:33:48.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959228447554, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94227, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 323, "table_properties": {"data_size": 92226, "index_size": 283, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13674, "raw_average_key_size": 49, "raw_value_size": 84754, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 274, "num_filter_entries": 274, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1696959228, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a9564ec0-1376-4ea5-a719-5b3c7da57223", "db_session_id": "CXQ291CS249F2G2JCUMZ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-10-10T17:33:48.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959228447639, "job": 1, "event": "recovery_finished"} 2023-10-10T17:33:48.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-10-10T17:33:48.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-10T17:33:48.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5578d8978000 2023-10-10T17:33:48.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: DB pointer 0x5578d8964000 2023-10-10T17:33:48.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-10T17:33:48.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-10T17:33:48.575 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2023-10-10T17:33:48.575 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-10T17:33:48.575 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-10T17:33:48.575 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-10T17:33:48.575 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-10T17:33:48.576 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-10T17:33:48.576 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-10T17:33:48.576 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-10T17:33:48.576 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-10-10T17:33:48.576 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2023-10-10T17:33:48.577 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-10T17:33:48.577 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-10T17:33:48.577 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 2/0 93.85 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:33:48.577 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 93.85 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:33:48.577 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:33:48.577 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-10-10T17:33:48.578 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2023-10-10T17:33:48.578 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-10T17:33:48.578 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-10T17:33:48.578 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:33:48.578 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-10-10T17:33:48.579 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-10T17:33:48.579 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-10-10T17:33:48.579 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-10T17:33:48.579 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-10T17:33:48.579 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-10T17:33:48.580 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-10T17:33:48.580 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-10T17:33:48.580 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-10T17:33:48.580 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 25.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-10T17:33:48.580 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 25.87 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-10T17:33:48.581 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-10T17:33:48.581 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x5578d88db1f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-10-10T17:33:48.581 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-10-10T17:33:48.581 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-10-10T17:33:48.581 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-10T17:33:48.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:48.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???) e1 preinit fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:33:48.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???).mds e1 new map 2023-10-10T17:33:48.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???).mds e1 print_map 2023-10-10T17:33:48.582 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2023-10-10T17:33:48.582 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-10T17:33:48.583 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-10T17:33:48.583 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2023-10-10T17:33:48.583 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-10-10T17:33:48.583 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2023-10-10T17:33:48.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-10T17:33:48.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:33:48.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:33:48.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:33:48.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-10-10T17:33:48.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expand map: {default=false} 2023-10-10T17:33:48.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta from 'false' to 'false' 2023-10-10T17:33:48.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expanded map: {default=false} 2023-10-10T17:33:48.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expand map: {default=info} 2023-10-10T17:33:48.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta from 'info' to 'info' 2023-10-10T17:33:48.585 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expanded map: {default=info} 2023-10-10T17:33:48.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expand map: {default=daemon} 2023-10-10T17:33:48.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-10T17:33:48.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expanded map: {default=daemon} 2023-10-10T17:33:48.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expand map: {default=debug} 2023-10-10T17:33:48.586 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta from 'debug' to 'debug' 2023-10-10T17:33:48.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: expand_channel_meta expanded map: {default=debug} 2023-10-10T17:33:48.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 bash[99454]: d45cef894d1af66293c48eb8e28deea367d0250a4b302ee5194ed7bf18619fc6 2023-10-10T17:33:48.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-10T17:33:48.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} 2023-10-10T17:33:48.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: fsmap 2023-10-10T17:33:48.587 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: osdmap e1: 0 total, 0 up, 0 in 2023-10-10T17:33:48.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 ceph-mon[99494]: mgrmap e1: no daemons active 2023-10-10T17:33:48.588 INFO:teuthology.orchestra.run.smithi078.stdout:Setting mon public_network to 172.21.0.0/20 2023-10-10T17:33:48.769 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:48 smithi078 systemd[1]: Started Ceph mon.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:33:49.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.996+0000 7f0debdd2700 1 Processor -- start 2023-10-10T17:33:49.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.996+0000 7f0debdd2700 1 -- start start 2023-10-10T17:33:49.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.996+0000 7f0debdd2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de4104710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:49.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.996+0000 7f0debdd2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de4079de0 con 0x7f0de4104300 2023-10-10T17:33:49.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de9b6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de4104710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:49.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de9b6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de4104710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40820/0 (socket says 172.21.15.78:40820) 2023-10-10T17:33:49.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de9b6e700 1 -- 172.21.15.78:0/3954180311 learned_addr learned my addr 172.21.15.78:0/3954180311 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:49.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de9b6e700 1 -- 172.21.15.78:0/3954180311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de4104c50 con 0x7f0de4104300 2023-10-10T17:33:49.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de9b6e700 1 --2- 172.21.15.78:0/3954180311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de4104710 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40098a0 tx=0x7f0dd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfffa75a128d25eb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:49.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de8b6c700 1 -- 172.21.15.78:0/3954180311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dd4004160 con 0x7f0de4104300 2023-10-10T17:33:49.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de8b6c700 1 -- 172.21.15.78:0/3954180311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dd40042c0 con 0x7f0de4104300 2023-10-10T17:33:49.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.997+0000 7f0de8b6c700 1 -- 172.21.15.78:0/3954180311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dd4011a00 con 0x7f0de4104300 2023-10-10T17:33:49.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.998+0000 7f0debdd2700 1 -- 172.21.15.78:0/3954180311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 msgr2=0x7f0de4104710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:49.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.998+0000 7f0debdd2700 1 --2- 172.21.15.78:0/3954180311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de4104710 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0dd40098a0 tx=0x7f0dd4009bb0 comp rx=0 tx=0).stop 2023-10-10T17:33:49.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.998+0000 7f0debdd2700 1 -- 172.21.15.78:0/3954180311 shutdown_connections 2023-10-10T17:33:49.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.998+0000 7f0debdd2700 1 --2- 172.21.15.78:0/3954180311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de4104710 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:49.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.998+0000 7f0debdd2700 1 -- 172.21.15.78:0/3954180311 >> 172.21.15.78:0/3954180311 conn(0x7f0de40ffed0 msgr2=0x7f0de4102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:49.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.998+0000 7f0debdd2700 1 -- 172.21.15.78:0/3954180311 shutdown_connections 2023-10-10T17:33:49.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.998+0000 7f0debdd2700 1 -- 172.21.15.78:0/3954180311 wait complete. 2023-10-10T17:33:49.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0debdd2700 1 Processor -- start 2023-10-10T17:33:49.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0debdd2700 1 -- start start 2023-10-10T17:33:49.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0debdd2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:49.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0debdd2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de419bee0 con 0x7f0de4104300 2023-10-10T17:33:49.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0de9b6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:49.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0de9b6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40822/0 (socket says 172.21.15.78:40822) 2023-10-10T17:33:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0de9b6e700 1 -- 172.21.15.78:0/3566159704 learned_addr learned my addr 172.21.15.78:0/3566159704 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0de9b6e700 1 -- 172.21.15.78:0/3566159704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4009550 con 0x7f0de4104300 2023-10-10T17:33:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:48.999+0000 7f0de9b6e700 1 --2- 172.21.15.78:0/3566159704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de419b9a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009fb0 tx=0x7f0dd4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.000+0000 7f0ddaffd700 1 -- 172.21.15.78:0/3566159704 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dd4003930 con 0x7f0de4104300 2023-10-10T17:33:49.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.000+0000 7f0ddaffd700 1 -- 172.21.15.78:0/3566159704 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0dd4003a90 con 0x7f0de4104300 2023-10-10T17:33:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.000+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de4105c50 con 0x7f0de4104300 2023-10-10T17:33:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.000+0000 7f0ddaffd700 1 -- 172.21.15.78:0/3566159704 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0dd4019610 con 0x7f0de4104300 2023-10-10T17:33:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.000+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de4105ac0 con 0x7f0de4104300 2023-10-10T17:33:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.000+0000 7f0ddaffd700 1 -- 172.21.15.78:0/3566159704 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0dd4019ba0 con 0x7f0de4104300 2023-10-10T17:33:49.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.000+0000 7f0ddaffd700 1 -- 172.21.15.78:0/3566159704 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0dd40243f0 con 0x7f0de4104300 2023-10-10T17:33:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.001+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de4195250 con 0x7f0de4104300 2023-10-10T17:33:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.003+0000 7f0ddaffd700 1 -- 172.21.15.78:0/3566159704 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0dd4003c00 con 0x7f0de4104300 2023-10-10T17:33:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.068+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f0de402cfe0 con 0x7f0de4104300 2023-10-10T17:33:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.070+0000 7f0ddaffd700 1 -- 172.21.15.78:0/3566159704 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0dd401a9d0 con 0x7f0de4104300 2023-10-10T17:33:49.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.071+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 msgr2=0x7f0de419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.071+0000 7f0debdd2700 1 --2- 172.21.15.78:0/3566159704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de419b9a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009fb0 tx=0x7f0dd4011ad0 comp rx=0 tx=0).stop 2023-10-10T17:33:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.071+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 shutdown_connections 2023-10-10T17:33:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.071+0000 7f0debdd2700 1 --2- 172.21.15.78:0/3566159704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0de4104300 0x7f0de419b9a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.071+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 >> 172.21.15.78:0/3566159704 conn(0x7f0de40ffed0 msgr2=0x7f0de418fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:49.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.071+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 shutdown_connections 2023-10-10T17:33:49.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:49.071+0000 7f0debdd2700 1 -- 172.21.15.78:0/3566159704 wait complete. 2023-10-10T17:33:49.410 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote config to /etc/ceph/ceph.conf 2023-10-10T17:33:49.410 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:33:49.410 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mgr... 2023-10-10T17:33:49.410 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 9283 ... 2023-10-10T17:33:49.411 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 8765 ... 2023-10-10T17:33:49.527 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.x 2023-10-10T17:33:49.527 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.x.service: Unit ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.x.service not loaded. 2023-10-10T17:33:49.615 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7.target.wants/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@.service. 2023-10-10T17:33:49.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:49 smithi078 systemd[1]: Starting Ceph mgr.x for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:33:50.208 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:50 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3566159704' entity='client.admin' 2023-10-10T17:33:50.208 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:49 smithi078 podman[99721]: 2023-10-10T17:33:50.317 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-10T17:33:50.317 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-10-10T17:33:50.324 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-10T17:33:50.324 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-10-10T17:33:50.324 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-10-10T17:33:50.325 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2023-10-10T17:33:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-10T17:33:50.333 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-10-10T17:33:50.340 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-10T17:33:50.340 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-10-10T17:33:50.341 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-10-10T17:33:50.341 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-10-10T17:33:50.341 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr to start... 2023-10-10T17:33:50.341 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr... 2023-10-10T17:33:50.502 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:50 smithi078 bash[99721]: 49476fedb0b9175de6483eccb042ddcdca064fff29d51a532b4a01a3df0a67bf 2023-10-10T17:33:50.503 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:50 smithi078 systemd[1]: Started Ceph mgr.x for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:33:50.503 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:50.324+0000 7f5ba7e5d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:33:50.503 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:50.412+0000 7f5ba7e5d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:33:50.503 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:50.501+0000 7f5ba7e5d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:33:50.772 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:50.597+0000 7f5ba7e5d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:33:51.071 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:50.771+0000 7f5ba7e5d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:33:51.327 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:51 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/997625668' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-10T17:33:51.328 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:51.106+0000 7f5ba7e5d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:33:51.328 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:51.197+0000 7f5ba7e5d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:33:51.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:51.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:33:51.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:33:51.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-10-10T17:33:51.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-10T17:33:51.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-10T17:33:51.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-10T17:33:51.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:51.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-10T17:33:51.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-10T17:33:51.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-10-10T17:33:51.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:51.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-10T17:33:51.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-10-10T17:33:51.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:51.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-10-10T17:33:51.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-10T17:33:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-10T17:33:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-10T17:33:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:51.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-10T17:33:51.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:51.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-10T17:33:51.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-10T17:33:51.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-10T17:33:51.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-10T17:33:51.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-10T17:33:51.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-10T17:33:51.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:51.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-10T17:33:51.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-10T17:33:51.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-10T17:33:51.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-10T17:33:51.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-10T17:33:51.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-10T17:33:51.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-10T17:33:51.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-10T17:33:51.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-10T17:33:51.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:51.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-10T17:33:51.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:51.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-10T17:33:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-10T17:33:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-10T17:33:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-10T17:33:51.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-10T17:33:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-10T17:33:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-10T17:33:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-10T17:33:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-10-10T17:33:51.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:33:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-10T17:33:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:51.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-10-10T17:33:42.140583+0000", 2023-10-10T17:33:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:33:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-10T17:33:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:33:51.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.889+0000 7f7162d1c700 1 Processor -- start 2023-10-10T17:33:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.889+0000 7f7162d1c700 1 -- start start 2023-10-10T17:33:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.890+0000 7f7162d1c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.890+0000 7f7162d1c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c105300 con 0x7f715c104920 2023-10-10T17:33:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.890+0000 7f7160ab8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:51.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.890+0000 7f7160ab8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40828/0 (socket says 172.21.15.78:40828) 2023-10-10T17:33:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.890+0000 7f7160ab8700 1 -- 172.21.15.78:0/2573353453 learned_addr learned my addr 172.21.15.78:0/2573353453 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.890+0000 7f7160ab8700 1 -- 172.21.15.78:0/2573353453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c105b20 con 0x7f715c104920 2023-10-10T17:33:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.890+0000 7f7160ab8700 1 --2- 172.21.15.78:0/2573353453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c104d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71500098a0 tx=0x7f7150009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4762e66cbe8170af server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f715b7fe700 1 -- 172.21.15.78:0/2573353453 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7150004160 con 0x7f715c104920 2023-10-10T17:33:51.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f715b7fe700 1 -- 172.21.15.78:0/2573353453 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71500042c0 con 0x7f715c104920 2023-10-10T17:33:51.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f715b7fe700 1 -- 172.21.15.78:0/2573353453 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7150011a00 con 0x7f715c104920 2023-10-10T17:33:51.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f7162d1c700 1 -- 172.21.15.78:0/2573353453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 msgr2=0x7f715c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:51.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f7162d1c700 1 --2- 172.21.15.78:0/2573353453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c104d30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f71500098a0 tx=0x7f7150009bb0 comp rx=0 tx=0).stop 2023-10-10T17:33:51.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f7162d1c700 1 -- 172.21.15.78:0/2573353453 shutdown_connections 2023-10-10T17:33:51.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f7162d1c700 1 --2- 172.21.15.78:0/2573353453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c104d30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:51.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f7162d1c700 1 -- 172.21.15.78:0/2573353453 >> 172.21.15.78:0/2573353453 conn(0x7f715c0ffed0 msgr2=0x7f715c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:51.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f7162d1c700 1 -- 172.21.15.78:0/2573353453 shutdown_connections 2023-10-10T17:33:51.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.891+0000 7f7162d1c700 1 -- 172.21.15.78:0/2573353453 wait complete. 2023-10-10T17:33:51.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7162d1c700 1 Processor -- start 2023-10-10T17:33:51.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7162d1c700 1 -- start start 2023-10-10T17:33:51.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7162d1c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:51.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7162d1c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c19c6b0 con 0x7f715c104920 2023-10-10T17:33:51.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7160ab8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:51.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7160ab8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c079ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40830/0 (socket says 172.21.15.78:40830) 2023-10-10T17:33:51.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7160ab8700 1 -- 172.21.15.78:0/997625668 learned_addr learned my addr 172.21.15.78:0/997625668 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:51.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.892+0000 7f7160ab8700 1 -- 172.21.15.78:0/997625668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7150009550 con 0x7f715c104920 2023-10-10T17:33:51.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7160ab8700 1 --2- 172.21.15.78:0/997625668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c079ae0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7150006af0 tx=0x7f7150011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:51.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7159ffb700 1 -- 172.21.15.78:0/997625668 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7150003a40 con 0x7f715c104920 2023-10-10T17:33:51.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715c108d50 con 0x7f715c104920 2023-10-10T17:33:51.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7159ffb700 1 -- 172.21.15.78:0/997625668 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7150003ba0 con 0x7f715c104920 2023-10-10T17:33:51.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7159ffb700 1 -- 172.21.15.78:0/997625668 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f715001a700 con 0x7f715c104920 2023-10-10T17:33:51.424 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f715c108ee0 con 0x7f715c104920 2023-10-10T17:33:51.425 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7159ffb700 1 -- 172.21.15.78:0/997625668 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f715001ac80 con 0x7f715c104920 2023-10-10T17:33:51.425 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.893+0000 7f7159ffb700 1 -- 172.21.15.78:0/997625668 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7150019700 con 0x7f715c104920 2023-10-10T17:33:51.425 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.894+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715c106990 con 0x7f715c104920 2023-10-10T17:33:51.425 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.896+0000 7f7159ffb700 1 -- 172.21.15.78:0/997625668 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f715c106990 con 0x7f715c104920 2023-10-10T17:33:51.426 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.968+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f715c19cf50 con 0x7f715c104920 2023-10-10T17:33:51.426 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.968+0000 7f7159ffb700 1 -- 172.21.15.78:0/997625668 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f7150024ad0 con 0x7f715c104920 2023-10-10T17:33:51.426 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.969+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 msgr2=0x7f715c079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.969+0000 7f7162d1c700 1 --2- 172.21.15.78:0/997625668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c079ae0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7150006af0 tx=0x7f7150011ad0 comp rx=0 tx=0).stop 2023-10-10T17:33:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.969+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 shutdown_connections 2023-10-10T17:33:51.427 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.969+0000 7f7162d1c700 1 --2- 172.21.15.78:0/997625668 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f715c104920 0x7f715c079ae0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.969+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 >> 172.21.15.78:0/997625668 conn(0x7f715c0ffed0 msgr2=0x7f715c101570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.970+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 shutdown_connections 2023-10-10T17:33:51.428 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:50.970+0000 7f7162d1c700 1 -- 172.21.15.78:0/997625668 wait complete. 2023-10-10T17:33:51.429 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (1/15)... 2023-10-10T17:33:51.627 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:51.327+0000 7f5ba7e5d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:33:51.628 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:51.424+0000 7f5ba7e5d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:33:52.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:51.839+0000 7f5ba7e5d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:33:52.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:52.408+0000 7f5ba7e5d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:33:52.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:52.574+0000 7f5ba7e5d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:33:53.596 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:53.326+0000 7f5ba7e5d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:33:53.597 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:53.419+0000 7f5ba7e5d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:33:53.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:53.594+0000 7f5ba7e5d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:33:54.270 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:53.885+0000 7f5ba7e5d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:33:54.271 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:54.120+0000 7f5ba7e5d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:33:54.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:54.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:33:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:33:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-10-10T17:33:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-10T17:33:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-10T17:33:54.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-10T17:33:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-10T17:33:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-10T17:33:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-10-10T17:33:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:54.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-10T17:33:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-10-10T17:33:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-10-10T17:33:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-10T17:33:54.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:54.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-10T17:33:54.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-10T17:33:54.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:54.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-10T17:33:54.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:54.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-10T17:33:54.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-10T17:33:54.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-10T17:33:54.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-10T17:33:54.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-10T17:33:54.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-10T17:33:54.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:54.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-10T17:33:54.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-10T17:33:54.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-10T17:33:54.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-10T17:33:54.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-10T17:33:54.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-10T17:33:54.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-10T17:33:54.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-10T17:33:54.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-10T17:33:54.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:54.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-10T17:33:54.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:54.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-10T17:33:54.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-10T17:33:54.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:54.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-10T17:33:54.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-10T17:33:54.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-10T17:33:54.359 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-10T17:33:54.359 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-10T17:33:54.359 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-10T17:33:54.359 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-10-10T17:33:54.359 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:54.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:33:54.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:54.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-10T17:33:54.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:54.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-10-10T17:33:42.140583+0000", 2023-10-10T17:33:54.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:33:54.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:54.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-10T17:33:54.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:33:54.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.886+0000 7f45bcaa6700 1 Processor -- start 2023-10-10T17:33:54.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.886+0000 7f45bcaa6700 1 -- start start 2023-10-10T17:33:54.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45bcaa6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8108870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:54.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45bcaa6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b8079e60 con 0x7f45b8106490 2023-10-10T17:33:54.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8108870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:54.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8108870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40832/0 (socket says 172.21.15.78:40832) 2023-10-10T17:33:54.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b659c700 1 -- 172.21.15.78:0/2024509569 learned_addr learned my addr 172.21.15.78:0/2024509569 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:54.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b659c700 1 -- 172.21.15.78:0/2024509569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b8108db0 con 0x7f45b8106490 2023-10-10T17:33:54.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b659c700 1 --2- 172.21.15.78:0/2024509569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8108870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f45a00098a0 tx=0x7f45a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c989aca835293fc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:54.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b559a700 1 -- 172.21.15.78:0/2024509569 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45a0004160 con 0x7f45b8106490 2023-10-10T17:33:54.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b559a700 1 -- 172.21.15.78:0/2024509569 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45a00042c0 con 0x7f45b8106490 2023-10-10T17:33:54.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.887+0000 7f45b559a700 1 -- 172.21.15.78:0/2024509569 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45a0011a00 con 0x7f45b8106490 2023-10-10T17:33:54.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.888+0000 7f45bcaa6700 1 -- 172.21.15.78:0/2024509569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 msgr2=0x7f45b8108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:54.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.888+0000 7f45bcaa6700 1 --2- 172.21.15.78:0/2024509569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8108870 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f45a00098a0 tx=0x7f45a0009bb0 comp rx=0 tx=0).stop 2023-10-10T17:33:54.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.888+0000 7f45bcaa6700 1 -- 172.21.15.78:0/2024509569 shutdown_connections 2023-10-10T17:33:54.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.888+0000 7f45bcaa6700 1 --2- 172.21.15.78:0/2024509569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8108870 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:54.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.888+0000 7f45bcaa6700 1 -- 172.21.15.78:0/2024509569 >> 172.21.15.78:0/2024509569 conn(0x7f45b80ffed0 msgr2=0x7f45b8102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:54.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.888+0000 7f45bcaa6700 1 -- 172.21.15.78:0/2024509569 shutdown_connections 2023-10-10T17:33:54.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.888+0000 7f45bcaa6700 1 -- 172.21.15.78:0/2024509569 wait complete. 2023-10-10T17:33:54.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.889+0000 7f45bcaa6700 1 Processor -- start 2023-10-10T17:33:54.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.889+0000 7f45bcaa6700 1 -- start start 2023-10-10T17:33:54.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.889+0000 7f45bcaa6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:54.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45bcaa6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b8199ca0 con 0x7f45b8106490 2023-10-10T17:33:54.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45b659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:54.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45b659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40834/0 (socket says 172.21.15.78:40834) 2023-10-10T17:33:54.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45b659c700 1 -- 172.21.15.78:0/4089575282 learned_addr learned my addr 172.21.15.78:0/4089575282 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:54.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45b659c700 1 -- 172.21.15.78:0/4089575282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a0009550 con 0x7f45b8106490 2023-10-10T17:33:54.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45b659c700 1 --2- 172.21.15.78:0/4089575282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8199760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f45a0006af0 tx=0x7f45a0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:54.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45af7fe700 1 -- 172.21.15.78:0/4089575282 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45a0003a40 con 0x7f45b8106490 2023-10-10T17:33:54.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b8109d40 con 0x7f45b8106490 2023-10-10T17:33:54.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45af7fe700 1 -- 172.21.15.78:0/4089575282 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45a0003ba0 con 0x7f45b8106490 2023-10-10T17:33:54.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45af7fe700 1 -- 172.21.15.78:0/4089575282 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45a001a6e0 con 0x7f45b8106490 2023-10-10T17:33:54.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.890+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b8109bb0 con 0x7f45b8106490 2023-10-10T17:33:54.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.891+0000 7f45af7fe700 1 -- 172.21.15.78:0/4089575282 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f45a001ac50 con 0x7f45b8106490 2023-10-10T17:33:54.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.891+0000 7f45af7fe700 1 -- 172.21.15.78:0/4089575282 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f45a0019700 con 0x7f45b8106490 2023-10-10T17:33:54.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.891+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45b8062120 con 0x7f45b8106490 2023-10-10T17:33:54.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.895+0000 7f45af7fe700 1 -- 172.21.15.78:0/4089575282 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f45a002de30 con 0x7f45b8106490 2023-10-10T17:33:54.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.966+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f45b819a1e0 con 0x7f45b8106490 2023-10-10T17:33:54.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.967+0000 7f45af7fe700 1 -- 172.21.15.78:0/4089575282 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f45a0024a60 con 0x7f45b8106490 2023-10-10T17:33:54.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.968+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 msgr2=0x7f45b8199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:54.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.968+0000 7f45bcaa6700 1 --2- 172.21.15.78:0/4089575282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8199760 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f45a0006af0 tx=0x7f45a0011ad0 comp rx=0 tx=0).stop 2023-10-10T17:33:54.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.968+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 shutdown_connections 2023-10-10T17:33:54.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.968+0000 7f45bcaa6700 1 --2- 172.21.15.78:0/4089575282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45b8106490 0x7f45b8199760 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:54.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.968+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 >> 172.21.15.78:0/4089575282 conn(0x7f45b80ffed0 msgr2=0x7f45b8100cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:54.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.968+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 shutdown_connections 2023-10-10T17:33:54.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:53.968+0000 7f45bcaa6700 1 -- 172.21.15.78:0/4089575282 wait complete. 2023-10-10T17:33:54.370 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (2/15)... 2023-10-10T17:33:54.532 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:54 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4089575282' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-10T17:33:54.533 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:54.269+0000 7f5ba7e5d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:33:54.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:54.530+0000 7f5ba7e5d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:33:55.536 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:55.219+0000 7f5ba7e5d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:33:55.537 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:55.347+0000 7f5ba7e5d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:33:55.860 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:55.534+0000 7f5ba7e5d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:33:55.861 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:55.646+0000 7f5ba7e5d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:33:56.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:55.859+0000 7f5ba7e5d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:33:56.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:33:56.726+0000 7f5ba7e5d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:33:56.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: Activating manager daemon x 2023-10-10T17:33:56.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: mgrmap e2: x(active, starting, since 0.00329956s) 2023-10-10T17:33:56.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-10T17:33:56.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-10T17:33:56.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-10T17:33:56.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:33:56.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-10T17:33:56.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: Manager daemon x is now available 2023-10-10T17:33:56.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-10T17:33:56.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-10T17:33:56.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' 2023-10-10T17:33:56.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' 2023-10-10T17:33:56.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:56 smithi078 ceph-mon[99494]: from='mgr.14100 172.21.15.78:0/322401361' entity='mgr.x' 2023-10-10T17:33:57.288 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:33:57.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:33:57.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:33:57.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-10-10T17:33:57.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-10T17:33:57.289 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-10T17:33:57.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-10T17:33:57.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:57.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-10T17:33:57.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-10T17:33:57.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-10-10T17:33:57.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:57.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-10T17:33:57.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-10-10T17:33:57.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:57.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-10-10T17:33:57.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-10T17:33:57.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:57.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-10T17:33:57.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-10T17:33:57.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:57.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-10T17:33:57.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:57.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-10T17:33:57.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-10T17:33:57.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-10T17:33:57.300 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-10T17:33:57.300 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-10T17:33:57.300 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-10T17:33:57.301 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:57.301 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-10T17:33:57.301 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-10T17:33:57.301 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-10T17:33:57.301 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-10T17:33:57.302 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-10T17:33:57.302 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-10T17:33:57.302 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-10T17:33:57.302 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-10T17:33:57.302 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-10T17:33:57.303 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:57.303 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-10T17:33:57.303 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:57.303 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-10T17:33:57.303 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-10T17:33:57.304 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:57.304 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-10T17:33:57.304 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-10T17:33:57.304 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-10T17:33:57.304 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-10T17:33:57.304 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-10T17:33:57.305 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-10T17:33:57.305 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-10-10T17:33:57.305 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:33:57.305 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:33:57.305 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:57.306 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-10T17:33:57.306 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:33:57.306 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-10-10T17:33:42.140583+0000", 2023-10-10T17:33:57.306 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:33:57.306 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:33:57.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-10T17:33:57.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:33:57.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.852+0000 7f026a320700 1 Processor -- start 2023-10-10T17:33:57.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.852+0000 7f026a320700 1 -- start start 2023-10-10T17:33:57.307 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.852+0000 7f026a320700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f02641088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:57.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.852+0000 7f026a320700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264079e20 con 0x7f02641064d0 2023-10-10T17:33:57.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.852+0000 7f0263fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f02641088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:57.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.852+0000 7f0263fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f02641088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40850/0 (socket says 172.21.15.78:40850) 2023-10-10T17:33:57.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.852+0000 7f0263fff700 1 -- 172.21.15.78:0/2209280525 learned_addr learned my addr 172.21.15.78:0/2209280525 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:57.308 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.853+0000 7f0263fff700 1 -- 172.21.15.78:0/2209280525 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264108df0 con 0x7f02641064d0 2023-10-10T17:33:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.853+0000 7f0263fff700 1 --2- 172.21.15.78:0/2209280525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f02641088b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02540098a0 tx=0x7f0254009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5940697fc2d1b81 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.853+0000 7f0262ffd700 1 -- 172.21.15.78:0/2209280525 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0254004160 con 0x7f02641064d0 2023-10-10T17:33:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.853+0000 7f0262ffd700 1 -- 172.21.15.78:0/2209280525 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02540042c0 con 0x7f02641064d0 2023-10-10T17:33:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.853+0000 7f0262ffd700 1 -- 172.21.15.78:0/2209280525 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0254011a00 con 0x7f02641064d0 2023-10-10T17:33:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.853+0000 7f026a320700 1 -- 172.21.15.78:0/2209280525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 msgr2=0x7f02641088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:57.309 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 --2- 172.21.15.78:0/2209280525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f02641088b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f02540098a0 tx=0x7f0254009bb0 comp rx=0 tx=0).stop 2023-10-10T17:33:57.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 -- 172.21.15.78:0/2209280525 shutdown_connections 2023-10-10T17:33:57.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 --2- 172.21.15.78:0/2209280525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f02641088b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:57.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 -- 172.21.15.78:0/2209280525 >> 172.21.15.78:0/2209280525 conn(0x7f02640ffed0 msgr2=0x7f0264102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:57.310 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 -- 172.21.15.78:0/2209280525 shutdown_connections 2023-10-10T17:33:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 -- 172.21.15.78:0/2209280525 wait complete. 2023-10-10T17:33:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 Processor -- start 2023-10-10T17:33:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.854+0000 7f026a320700 1 -- start start 2023-10-10T17:33:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f026a320700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f0264199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:33:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f026a320700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0264199c50 con 0x7f02641064d0 2023-10-10T17:33:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f0263fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f0264199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:33:57.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f0263fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f0264199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40852/0 (socket says 172.21.15.78:40852) 2023-10-10T17:33:57.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f0263fff700 1 -- 172.21.15.78:0/1337023116 learned_addr learned my addr 172.21.15.78:0/1337023116 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:33:57.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f0263fff700 1 -- 172.21.15.78:0/1337023116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0254009550 con 0x7f02641064d0 2023-10-10T17:33:57.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f0263fff700 1 --2- 172.21.15.78:0/1337023116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f0264199710 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0254009fb0 tx=0x7f0254011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:33:57.312 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.855+0000 7f02617fa700 1 -- 172.21.15.78:0/1337023116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0254003930 con 0x7f02641064d0 2023-10-10T17:33:57.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.856+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0264109d20 con 0x7f02641064d0 2023-10-10T17:33:57.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.856+0000 7f02617fa700 1 -- 172.21.15.78:0/1337023116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0254003a90 con 0x7f02641064d0 2023-10-10T17:33:57.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.856+0000 7f02617fa700 1 -- 172.21.15.78:0/1337023116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0254019610 con 0x7f02641064d0 2023-10-10T17:33:57.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.856+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0264109b90 con 0x7f02641064d0 2023-10-10T17:33:57.313 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.856+0000 7f02617fa700 1 -- 172.21.15.78:0/1337023116 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 2) v1 ==== 45239+0+0 (secure 0 0 0) 0x7f02540198d0 con 0x7f02641064d0 2023-10-10T17:33:57.314 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.856+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0264062120 con 0x7f02641064d0 2023-10-10T17:33:57.314 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.857+0000 7f02617fa700 1 -- 172.21.15.78:0/1337023116 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f025404c100 con 0x7f02641064d0 2023-10-10T17:33:57.314 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.859+0000 7f02617fa700 1 -- 172.21.15.78:0/1337023116 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0254018d20 con 0x7f02641064d0 2023-10-10T17:33:57.314 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.929+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f026419a3e0 con 0x7f02641064d0 2023-10-10T17:33:57.314 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.930+0000 7f02617fa700 1 -- 172.21.15.78:0/1337023116 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f0254020cb0 con 0x7f02641064d0 2023-10-10T17:33:57.314 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.930+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 msgr2=0x7f0264199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:33:57.315 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.930+0000 7f026a320700 1 --2- 172.21.15.78:0/1337023116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f0264199710 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0254009fb0 tx=0x7f0254011ad0 comp rx=0 tx=0).stop 2023-10-10T17:33:57.315 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.930+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 shutdown_connections 2023-10-10T17:33:57.315 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.930+0000 7f026a320700 1 --2- 172.21.15.78:0/1337023116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02641064d0 0x7f0264199710 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:33:57.315 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.930+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 >> 172.21.15.78:0/1337023116 conn(0x7f02640ffed0 msgr2=0x7f0264101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:33:57.315 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.931+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 shutdown_connections 2023-10-10T17:33:57.316 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:33:56.931+0000 7f026a320700 1 -- 172.21.15.78:0/1337023116 wait complete. 2023-10-10T17:33:57.316 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (3/15)... 2023-10-10T17:33:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:57 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1337023116' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-10T17:33:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:33:57 smithi078 ceph-mon[99494]: mgrmap e3: x(active, since 1.00651s) 2023-10-10T17:34:00.720 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:00 smithi078 ceph-mon[99494]: mgrmap e4: x(active, since 2s) 2023-10-10T17:34:02.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:01 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1418563951' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-10T17:34:04.261 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:34:04.262 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:34:04.262 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:34:04.262 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-10-10T17:34:04.262 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-10T17:34:04.263 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-10T17:34:04.263 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-10T17:34:04.263 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:34:04.263 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-10T17:34:04.263 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-10T17:34:04.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-10-10T17:34:04.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:34:04.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-10T17:34:04.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-10-10T17:34:04.264 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:34:04.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2023-10-10T17:34:04.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-10T17:34:04.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:34:04.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-10T17:34:04.265 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-10T17:34:04.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:34:04.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-10T17:34:04.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:34:04.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-10T17:34:04.266 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-10T17:34:04.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-10T17:34:04.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-10T17:34:04.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-10T17:34:04.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-10T17:34:04.267 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:34:04.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-10T17:34:04.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-10T17:34:04.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-10T17:34:04.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-10T17:34:04.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-10T17:34:04.268 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-10T17:34:04.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-10T17:34:04.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-10T17:34:04.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-10T17:34:04.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:34:04.269 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-10T17:34:04.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:34:04.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-10T17:34:04.270 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-10T17:34:04.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:34:04.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-10T17:34:04.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-10T17:34:04.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-10T17:34:04.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-10T17:34:04.271 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-10T17:34:04.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-10T17:34:04.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-10-10T17:34:04.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-10-10T17:34:04.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:34:04.272 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:34:04.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-10T17:34:04.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-10T17:34:04.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-10-10T17:33:42.140583+0000", 2023-10-10T17:34:04.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-10T17:34:04.273 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-10-10T17:34:04.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-10T17:34:04.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:34:04.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.443+0000 7fb7c2a59700 1 Processor -- start 2023-10-10T17:34:04.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.443+0000 7fb7c2a59700 1 -- start start 2023-10-10T17:34:04.274 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.443+0000 7fb7c2a59700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:04.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.443+0000 7fb7c2a59700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc079e00 con 0x7fb7bc1064f0 2023-10-10T17:34:04.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:04.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40864/0 (socket says 172.21.15.78:40864) 2023-10-10T17:34:04.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7bbfff700 1 -- 172.21.15.78:0/1136092485 learned_addr learned my addr 172.21.15.78:0/1136092485 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:04.275 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7bbfff700 1 -- 172.21.15.78:0/1136092485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc108e10 con 0x7fb7bc1064f0 2023-10-10T17:34:04.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7bbfff700 1 --2- 172.21.15.78:0/1136092485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc1088d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00098a0 tx=0x7fb7b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cbe33380367641 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:04.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7baffd700 1 -- 172.21.15.78:0/1136092485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7b0004160 con 0x7fb7bc1064f0 2023-10-10T17:34:04.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7baffd700 1 -- 172.21.15.78:0/1136092485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7b00042c0 con 0x7fb7bc1064f0 2023-10-10T17:34:04.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.444+0000 7fb7baffd700 1 -- 172.21.15.78:0/1136092485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7b0011a00 con 0x7fb7bc1064f0 2023-10-10T17:34:04.276 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1136092485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 msgr2=0x7fb7bc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:04.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 --2- 172.21.15.78:0/1136092485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc1088d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb7b00098a0 tx=0x7fb7b0009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:04.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1136092485 shutdown_connections 2023-10-10T17:34:04.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 --2- 172.21.15.78:0/1136092485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc1088d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:04.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1136092485 >> 172.21.15.78:0/1136092485 conn(0x7fb7bc0ffed0 msgr2=0x7fb7bc102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:04.277 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1136092485 shutdown_connections 2023-10-10T17:34:04.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1136092485 wait complete. 2023-10-10T17:34:04.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 Processor -- start 2023-10-10T17:34:04.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.445+0000 7fb7c2a59700 1 -- start start 2023-10-10T17:34:04.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7c2a59700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc19b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:04.278 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7c2a59700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc19bed0 con 0x7fb7bc1064f0 2023-10-10T17:34:04.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc19b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:04.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc19b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40866/0 (socket says 172.21.15.78:40866) 2023-10-10T17:34:04.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7bbfff700 1 -- 172.21.15.78:0/1418563951 learned_addr learned my addr 172.21.15.78:0/1418563951 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:04.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7bbfff700 1 -- 172.21.15.78:0/1418563951 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b0009550 con 0x7fb7bc1064f0 2023-10-10T17:34:04.279 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7bbfff700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc19b990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009fb0 tx=0x7fb7b0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:04.280 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7b97fa700 1 -- 172.21.15.78:0/1418563951 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7b0003930 con 0x7fb7bc1064f0 2023-10-10T17:34:04.280 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7b97fa700 1 -- 172.21.15.78:0/1418563951 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7b0003a90 con 0x7fb7bc1064f0 2023-10-10T17:34:04.280 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc109d40 con 0x7fb7bc1064f0 2023-10-10T17:34:04.280 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7b97fa700 1 -- 172.21.15.78:0/1418563951 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb7b0019610 con 0x7fb7bc1064f0 2023-10-10T17:34:04.280 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.446+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc109bb0 con 0x7fb7bc1064f0 2023-10-10T17:34:04.281 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.447+0000 7fb7b97fa700 1 -- 172.21.15.78:0/1418563951 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fb7b0019880 con 0x7fb7bc1064f0 2023-10-10T17:34:04.281 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.447+0000 7fb7b97fa700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fb7a40386c0 0x7fb7a403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:04.281 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.447+0000 7fb7b97fa700 1 -- 172.21.15.78:0/1418563951 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb7b004c460 con 0x7fb7bc1064f0 2023-10-10T17:34:04.281 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.447+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc062120 con 0x7fb7bc1064f0 2023-10-10T17:34:04.281 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.448+0000 7fb7bb7fe700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fb7a40386c0 0x7fb7a403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:04.282 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.448+0000 7fb7bb7fe700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fb7a40386c0 0x7fb7a403ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac006fd0 tx=0x7fb7ac006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:04.282 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.452+0000 7fb7b97fa700 1 -- 172.21.15.78:0/1418563951 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb7b0019b30 con 0x7fb7bc1064f0 2023-10-10T17:34:04.282 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.719+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb7bc19f520 con 0x7fb7bc1064f0 2023-10-10T17:34:04.282 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.719+0000 7fb7b97fa700 1 -- 172.21.15.78:0/1418563951 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fb7b0018350 con 0x7fb7bc1064f0 2023-10-10T17:34:04.282 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fb7a40386c0 msgr2=0x7fb7a403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:04.283 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fb7a40386c0 0x7fb7a403ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac006fd0 tx=0x7fb7ac006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:04.283 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 msgr2=0x7fb7bc19b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:04.283 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc19b990 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb7b0009fb0 tx=0x7fb7b0011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:04.283 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 shutdown_connections 2023-10-10T17:34:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fb7a40386c0 0x7fb7a403ab70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 --2- 172.21.15.78:0/1418563951 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7bc1064f0 0x7fb7bc19b990 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 >> 172.21.15.78:0/1418563951 conn(0x7fb7bc0ffed0 msgr2=0x7fb7bc101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 shutdown_connections 2023-10-10T17:34:04.284 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:01.721+0000 7fb7c2a59700 1 -- 172.21.15.78:0/1418563951 wait complete. 2023-10-10T17:34:04.285 INFO:teuthology.orchestra.run.smithi078.stdout:mgr is available 2023-10-10T17:34:06.280 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:05 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1878532269' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-10T17:34:06.306 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:34:06.318 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2023-10-10T17:34:06.318 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:34:06.318 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-10T17:34:06.319 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-10T17:34:06.319 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-10T17:34:06.319 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-10T17:34:06.319 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:34:06.319 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-10T17:34:06.320 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-10T17:34:06.320 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-10-10T17:34:06.320 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2023-10-10T17:34:06.320 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-10-10T17:34:06.320 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-10-10T17:34:06.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-10-10T17:34:06.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-10T17:34:06.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-10T17:34:06.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.741+0000 7fa3677f6700 1 Processor -- start 2023-10-10T17:34:06.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa3677f6700 1 -- start start 2023-10-10T17:34:06.321 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa3677f6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa3601046d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:06.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa3677f6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa360079e20 con 0x7fa3601042c0 2023-10-10T17:34:06.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa365592700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa3601046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:06.322 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa365592700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa3601046d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40870/0 (socket says 172.21.15.78:40870) 2023-10-10T17:34:06.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa365592700 1 -- 172.21.15.78:0/2302757400 learned_addr learned my addr 172.21.15.78:0/2302757400 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:06.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa365592700 1 -- 172.21.15.78:0/2302757400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa360104c10 con 0x7fa3601042c0 2023-10-10T17:34:06.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa365592700 1 --2- 172.21.15.78:0/2302757400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa3601046d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa3500098a0 tx=0x7fa350009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3095664281715b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:06.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.742+0000 7fa357fff700 1 -- 172.21.15.78:0/2302757400 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa350004160 con 0x7fa3601042c0 2023-10-10T17:34:06.323 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa357fff700 1 -- 172.21.15.78:0/2302757400 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3500042c0 con 0x7fa3601042c0 2023-10-10T17:34:06.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa357fff700 1 -- 172.21.15.78:0/2302757400 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa350011a00 con 0x7fa3601042c0 2023-10-10T17:34:06.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa3677f6700 1 -- 172.21.15.78:0/2302757400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 msgr2=0x7fa3601046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:06.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa3677f6700 1 --2- 172.21.15.78:0/2302757400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa3601046d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa3500098a0 tx=0x7fa350009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:06.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa3677f6700 1 -- 172.21.15.78:0/2302757400 shutdown_connections 2023-10-10T17:34:06.324 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa3677f6700 1 --2- 172.21.15.78:0/2302757400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa3601046d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:06.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa3677f6700 1 -- 172.21.15.78:0/2302757400 >> 172.21.15.78:0/2302757400 conn(0x7fa3600ffed0 msgr2=0x7fa360102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:06.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa3677f6700 1 -- 172.21.15.78:0/2302757400 shutdown_connections 2023-10-10T17:34:06.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.743+0000 7fa3677f6700 1 -- 172.21.15.78:0/2302757400 wait complete. 2023-10-10T17:34:06.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.744+0000 7fa3677f6700 1 Processor -- start 2023-10-10T17:34:06.325 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.744+0000 7fa3677f6700 1 -- start start 2023-10-10T17:34:06.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.744+0000 7fa3677f6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa36019b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:06.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.744+0000 7fa3677f6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36019be80 con 0x7fa3601042c0 2023-10-10T17:34:06.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.744+0000 7fa365592700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa36019b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:06.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.744+0000 7fa365592700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa36019b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40872/0 (socket says 172.21.15.78:40872) 2023-10-10T17:34:06.326 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.744+0000 7fa365592700 1 -- 172.21.15.78:0/1878532269 learned_addr learned my addr 172.21.15.78:0/1878532269 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:06.327 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.745+0000 7fa365592700 1 -- 172.21.15.78:0/1878532269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa350009550 con 0x7fa3601042c0 2023-10-10T17:34:06.327 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.745+0000 7fa365592700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa36019b940 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa350009fb0 tx=0x7fa350011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:06.327 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.745+0000 7fa3567fc700 1 -- 172.21.15.78:0/1878532269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa350003930 con 0x7fa3601042c0 2023-10-10T17:34:06.327 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.745+0000 7fa3567fc700 1 -- 172.21.15.78:0/1878532269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa350003a90 con 0x7fa3601042c0 2023-10-10T17:34:06.327 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.745+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa360105c10 con 0x7fa3601042c0 2023-10-10T17:34:06.328 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.745+0000 7fa3567fc700 1 -- 172.21.15.78:0/1878532269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa350019610 con 0x7fa3601042c0 2023-10-10T17:34:06.328 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.745+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa360105a80 con 0x7fa3601042c0 2023-10-10T17:34:06.328 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.746+0000 7fa3567fc700 1 -- 172.21.15.78:0/1878532269 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fa35001a430 con 0x7fa3601042c0 2023-10-10T17:34:06.328 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.746+0000 7fa3567fc700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fa34c038670 0x7fa34c03ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:06.328 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.746+0000 7fa3567fc700 1 -- 172.21.15.78:0/1878532269 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa35004c470 con 0x7fa3601042c0 2023-10-10T17:34:06.329 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.746+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa360195260 con 0x7fa3601042c0 2023-10-10T17:34:06.329 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.746+0000 7fa364d91700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fa34c038670 0x7fa34c03ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:06.329 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.746+0000 7fa364d91700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fa34c038670 0x7fa34c03ab20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006fd0 tx=0x7fa35c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:06.329 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.751+0000 7fa3567fc700 1 -- 172.21.15.78:0/1878532269 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa350003c00 con 0x7fa3601042c0 2023-10-10T17:34:06.329 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.935+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa36019c470 con 0x7fa3601042c0 2023-10-10T17:34:06.330 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.936+0000 7fa3567fc700 1 -- 172.21.15.78:0/1878532269 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fa350023b80 con 0x7fa3601042c0 2023-10-10T17:34:06.330 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fa34c038670 msgr2=0x7fa34c03ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:06.330 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fa34c038670 0x7fa34c03ab20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa35c006fd0 tx=0x7fa35c006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:06.330 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 msgr2=0x7fa36019b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:06.330 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa36019b940 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa350009fb0 tx=0x7fa350011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:06.331 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 shutdown_connections 2023-10-10T17:34:06.331 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fa34c038670 0x7fa34c03ab20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:06.331 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 --2- 172.21.15.78:0/1878532269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3601042c0 0x7fa36019b940 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:06.331 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 >> 172.21.15.78:0/1878532269 conn(0x7fa3600ffed0 msgr2=0x7fa36018fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:06.331 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 shutdown_connections 2023-10-10T17:34:06.332 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:05.938+0000 7fa3677f6700 1 -- 172.21.15.78:0/1878532269 wait complete. 2023-10-10T17:34:06.332 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling cephadm module... 2023-10-10T17:34:07.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1351381713' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-10-10T17:34:07.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: ignoring --setuser ceph since I am not root 2023-10-10T17:34:07.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: ignoring --setgroup ceph since I am not root 2023-10-10T17:34:07.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:07.252+0000 7f395f4e8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:34:07.380 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:07.332+0000 7f395f4e8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:34:07.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.791+0000 7fc158cfd700 1 Processor -- start 2023-10-10T17:34:07.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.791+0000 7fc158cfd700 1 -- start start 2023-10-10T17:34:07.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.791+0000 7fc158cfd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc1541046f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:07.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.791+0000 7fc158cfd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc154079e00 con 0x7fc1541042e0 2023-10-10T17:34:07.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.791+0000 7fc15259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc1541046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:07.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.792+0000 7fc15259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc1541046f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40876/0 (socket says 172.21.15.78:40876) 2023-10-10T17:34:07.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.792+0000 7fc15259c700 1 -- 172.21.15.78:0/1054458866 learned_addr learned my addr 172.21.15.78:0/1054458866 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:07.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.792+0000 7fc15259c700 1 -- 172.21.15.78:0/1054458866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc154104c30 con 0x7fc1541042e0 2023-10-10T17:34:07.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.792+0000 7fc15259c700 1 --2- 172.21.15.78:0/1054458866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc1541046f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc1480098a0 tx=0x7fc148009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ee21962606c7bd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:07.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.792+0000 7fc15159a700 1 -- 172.21.15.78:0/1054458866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc148004160 con 0x7fc1541042e0 2023-10-10T17:34:07.465 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.792+0000 7fc15159a700 1 -- 172.21.15.78:0/1054458866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1480042c0 con 0x7fc1541042e0 2023-10-10T17:34:07.465 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.792+0000 7fc15159a700 1 -- 172.21.15.78:0/1054458866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc148011a00 con 0x7fc1541042e0 2023-10-10T17:34:07.465 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 -- 172.21.15.78:0/1054458866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 msgr2=0x7fc1541046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:07.466 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 --2- 172.21.15.78:0/1054458866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc1541046f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc1480098a0 tx=0x7fc148009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:07.466 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 -- 172.21.15.78:0/1054458866 shutdown_connections 2023-10-10T17:34:07.467 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 --2- 172.21.15.78:0/1054458866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc1541046f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:07.467 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 -- 172.21.15.78:0/1054458866 >> 172.21.15.78:0/1054458866 conn(0x7fc1540ffed0 msgr2=0x7fc154102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:07.467 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 -- 172.21.15.78:0/1054458866 shutdown_connections 2023-10-10T17:34:07.468 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 -- 172.21.15.78:0/1054458866 wait complete. 2023-10-10T17:34:07.468 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.793+0000 7fc158cfd700 1 Processor -- start 2023-10-10T17:34:07.468 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc158cfd700 1 -- start start 2023-10-10T17:34:07.469 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc158cfd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc15419b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:07.469 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc158cfd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15419bee0 con 0x7fc1541042e0 2023-10-10T17:34:07.469 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc15259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc15419b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:07.470 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc15259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc15419b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40878/0 (socket says 172.21.15.78:40878) 2023-10-10T17:34:07.470 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc15259c700 1 -- 172.21.15.78:0/1351381713 learned_addr learned my addr 172.21.15.78:0/1351381713 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:07.471 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc15259c700 1 -- 172.21.15.78:0/1351381713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009550 con 0x7fc1541042e0 2023-10-10T17:34:07.471 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc15259c700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc15419b9a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc148009fb0 tx=0x7fc148011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:07.471 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc148003930 con 0x7fc1541042e0 2023-10-10T17:34:07.472 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc148003a90 con 0x7fc1541042e0 2023-10-10T17:34:07.472 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc154105c30 con 0x7fc1541042e0 2023-10-10T17:34:07.472 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc148019610 con 0x7fc1541042e0 2023-10-10T17:34:07.472 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.794+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc154105aa0 con 0x7fc1541042e0 2023-10-10T17:34:07.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.795+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7fc14801a9d0 con 0x7fc1541042e0 2023-10-10T17:34:07.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.795+0000 7fc1437fe700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fc13c0386c0 0x7fc13c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:07.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.795+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc14804c3f0 con 0x7fc1541042e0 2023-10-10T17:34:07.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.796+0000 7fc151d9b700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fc13c0386c0 0x7fc13c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:07.473 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.796+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc154105dc0 con 0x7fc1541042e0 2023-10-10T17:34:07.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.796+0000 7fc151d9b700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fc13c0386c0 0x7fc13c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc144006fd0 tx=0x7fc144006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:07.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:06.801+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc148019880 con 0x7fc1541042e0 2023-10-10T17:34:07.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.039+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fc15402cfe0 con 0x7fc1541042e0 2023-10-10T17:34:07.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.092+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc148019c00 con 0x7fc1541042e0 2023-10-10T17:34:07.474 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.093+0000 7fc1437fe700 1 -- 172.21.15.78:0/1351381713 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fc148003c00 con 0x7fc1541042e0 2023-10-10T17:34:07.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fc13c0386c0 msgr2=0x7fc13c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:07.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fc13c0386c0 0x7fc13c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc144006fd0 tx=0x7fc144006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:07.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 msgr2=0x7fc15419b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:07.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc15419b9a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc148009fb0 tx=0x7fc148011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:07.475 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 shutdown_connections 2023-10-10T17:34:07.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7fc13c0386c0 0x7fc13c03ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:07.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 --2- 172.21.15.78:0/1351381713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1541042e0 0x7fc15419b9a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:07.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 >> 172.21.15.78:0/1351381713 conn(0x7fc1540ffed0 msgr2=0x7fc15418fea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:07.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 shutdown_connections 2023-10-10T17:34:07.476 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.095+0000 7fc158cfd700 1 -- 172.21.15.78:0/1351381713 wait complete. 2023-10-10T17:34:07.679 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:07.422+0000 7f395f4e8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:34:07.679 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:07.528+0000 7f395f4e8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:34:08.029 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:07.707+0000 7f395f4e8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:34:08.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:08 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1351381713' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-10-10T17:34:08.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:08 smithi078 ceph-mon[99494]: mgrmap e5: x(active, since 10s) 2023-10-10T17:34:08.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:08 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:08.025+0000 7f395f4e8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:34:08.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:08 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:08.117+0000 7f395f4e8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:34:08.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:08 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:08.250+0000 7f395f4e8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:34:08.358 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:08 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:08.355+0000 7f395f4e8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:34:08.611 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:34:08.612 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-10-10T17:34:08.612 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-10T17:34:08.612 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-10T17:34:08.612 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-10T17:34:08.612 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:34:08.613 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f4981349700 1 Processor -- start 2023-10-10T17:34:08.613 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f4981349700 1 -- start start 2023-10-10T17:34:08.613 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f4981349700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c106310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:08.613 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f4981349700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c1068e0 con 0x7f497c105f00 2023-10-10T17:34:08.613 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f497affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:08.613 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f497affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c106310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40892/0 (socket says 172.21.15.78:40892) 2023-10-10T17:34:08.614 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f497affd700 1 -- 172.21.15.78:0/761855205 learned_addr learned my addr 172.21.15.78:0/761855205 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:08.614 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.974+0000 7f497affd700 1 -- 172.21.15.78:0/761855205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c104b10 con 0x7f497c105f00 2023-10-10T17:34:08.614 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f497affd700 1 --2- 172.21.15.78:0/761855205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c106310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098a0 tx=0x7f496c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8550fa89f99f5e2f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:08.614 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4979ffb700 1 -- 172.21.15.78:0/761855205 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f496c004160 con 0x7f497c105f00 2023-10-10T17:34:08.615 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4979ffb700 1 -- 172.21.15.78:0/761855205 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f496c0042c0 con 0x7f497c105f00 2023-10-10T17:34:08.615 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4979ffb700 1 -- 172.21.15.78:0/761855205 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f496c011a00 con 0x7f497c105f00 2023-10-10T17:34:08.615 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4981349700 1 -- 172.21.15.78:0/761855205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 msgr2=0x7f497c106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:08.615 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4981349700 1 --2- 172.21.15.78:0/761855205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c106310 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f496c0098a0 tx=0x7f496c009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:08.615 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4981349700 1 -- 172.21.15.78:0/761855205 shutdown_connections 2023-10-10T17:34:08.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4981349700 1 --2- 172.21.15.78:0/761855205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c106310 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:08.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4981349700 1 -- 172.21.15.78:0/761855205 >> 172.21.15.78:0/761855205 conn(0x7f497c0ff650 msgr2=0x7f497c101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:08.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4981349700 1 -- 172.21.15.78:0/761855205 shutdown_connections 2023-10-10T17:34:08.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.975+0000 7f4981349700 1 -- 172.21.15.78:0/761855205 wait complete. 2023-10-10T17:34:08.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.976+0000 7f4981349700 1 Processor -- start 2023-10-10T17:34:08.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.976+0000 7f4981349700 1 -- start start 2023-10-10T17:34:08.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.976+0000 7f4981349700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:08.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.976+0000 7f4981349700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497c198da0 con 0x7f497c105f00 2023-10-10T17:34:08.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f497affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:08.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f497affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c198860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40894/0 (socket says 172.21.15.78:40894) 2023-10-10T17:34:08.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f497affd700 1 -- 172.21.15.78:0/3950382481 learned_addr learned my addr 172.21.15.78:0/3950382481 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:08.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f497affd700 1 -- 172.21.15.78:0/3950382481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c009550 con 0x7f497c105f00 2023-10-10T17:34:08.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f497affd700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c198860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f496c009fb0 tx=0x7f496c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:08.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f4963fff700 1 -- 172.21.15.78:0/3950382481 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f496c003930 con 0x7f497c105f00 2023-10-10T17:34:08.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f4963fff700 1 -- 172.21.15.78:0/3950382481 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f496c003a90 con 0x7f497c105f00 2023-10-10T17:34:08.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f497c108d50 con 0x7f497c105f00 2023-10-10T17:34:08.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f4963fff700 1 -- 172.21.15.78:0/3950382481 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f496c0195e0 con 0x7f497c105f00 2023-10-10T17:34:08.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.977+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f497c108ee0 con 0x7f497c105f00 2023-10-10T17:34:08.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.978+0000 7f4963fff700 1 -- 172.21.15.78:0/3950382481 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f496c019900 con 0x7f497c105f00 2023-10-10T17:34:08.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.978+0000 7f4963fff700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 0x7f496403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:08.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.978+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f497c105980 con 0x7f497c105f00 2023-10-10T17:34:08.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.978+0000 7f497a7fc700 1 -- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 msgr2=0x7f496403ab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3346226185 2023-10-10T17:34:08.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.978+0000 7f4963fff700 1 -- 172.21.15.78:0/3950382481 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f496c04d490 con 0x7f497c105f00 2023-10-10T17:34:08.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.978+0000 7f497a7fc700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 0x7f496403ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:34:08.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:07.983+0000 7f4963fff700 1 -- 172.21.15.78:0/3950382481 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f497c105980 con 0x7f497c105f00 2023-10-10T17:34:08.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.179+0000 7f497a7fc700 1 -- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 msgr2=0x7f496403ab70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3346226185 2023-10-10T17:34:08.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.179+0000 7f497a7fc700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 0x7f496403ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-10T17:34:08.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.235+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f497c105980 con 0x7f497c105f00 2023-10-10T17:34:08.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.236+0000 7f4963fff700 1 -- 172.21.15.78:0/3950382481 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f497c105980 con 0x7f497c105f00 2023-10-10T17:34:08.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 msgr2=0x7f496403ab70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:34:08.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 0x7f496403ab70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:08.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 msgr2=0x7f497c198860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:08.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c198860 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f496c009fb0 tx=0x7f496c011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:08.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 shutdown_connections 2023-10-10T17:34:08.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f49640386c0 0x7f496403ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:08.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 --2- 172.21.15.78:0/3950382481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f497c105f00 0x7f497c198860 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:08.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 >> 172.21.15.78:0/3950382481 conn(0x7f497c0ff650 msgr2=0x7f497c100ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:08.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 shutdown_connections 2023-10-10T17:34:08.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:08.238+0000 7f4981349700 1 -- 172.21.15.78:0/3950382481 wait complete. 2023-10-10T17:34:08.624 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2023-10-10T17:34:08.624 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 5... 2023-10-10T17:34:09.099 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:08 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:08.797+0000 7f395f4e8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:34:09.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:09 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3950382481' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-10T17:34:09.877 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:09 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:09.378+0000 7f395f4e8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:34:09.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:09 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:09.546+0000 7f395f4e8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:34:10.564 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:10 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:10.303+0000 7f395f4e8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:34:10.565 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:10 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:10.393+0000 7f395f4e8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:34:11.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:10 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:10.563+0000 7f395f4e8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:34:11.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:10 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:10.837+0000 7f395f4e8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:34:11.129 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:11 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:11.060+0000 7f395f4e8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:34:11.466 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:11 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:11.206+0000 7f395f4e8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:34:11.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:11 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:11.464+0000 7f395f4e8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:34:12.477 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:12 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:12.158+0000 7f395f4e8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:34:12.477 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:12 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:12.287+0000 7f395f4e8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:34:12.801 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:12 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:12.475+0000 7f395f4e8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:34:12.801 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:12 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:12.587+0000 7f395f4e8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:34:14.127 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:12 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:12.799+0000 7f395f4e8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:34:14.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:13 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:13.645+0000 7f395f4e8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:34:14.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: Active manager daemon x restarted 2023-10-10T17:34:14.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: Activating manager daemon x 2023-10-10T17:34:14.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: osdmap e2: 0 total, 0 up, 0 in 2023-10-10T17:34:14.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: mgrmap e6: x(active, starting, since 0.00453528s) 2023-10-10T17:34:14.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:34:14.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-10T17:34:14.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-10T17:34:14.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-10T17:34:14.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-10T17:34:14.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: Manager daemon x is now available 2023-10-10T17:34:14.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: Found migration_current of "None". Setting to last migration. 2023-10-10T17:34:14.824 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:14.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:14.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:14.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:14.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:14.825 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-10T17:34:14.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-10T17:34:14.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:14.826 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:14 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:15.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:34:15.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-10-10T17:34:15.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-10T17:34:15.075 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:34:15.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.080+0000 7f5845b97700 1 Processor -- start 2023-10-10T17:34:15.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f5845b97700 1 -- start start 2023-10-10T17:34:15.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f5845b97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:15.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f5845b97700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5840105300 con 0x7f5840104920 2023-10-10T17:34:15.076 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:15.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40902/0 (socket says 172.21.15.78:40902) 2023-10-10T17:34:15.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f583f7fe700 1 -- 172.21.15.78:0/2133893813 learned_addr learned my addr 172.21.15.78:0/2133893813 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:15.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f583f7fe700 1 -- 172.21.15.78:0/2133893813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5840105b20 con 0x7f5840104920 2023-10-10T17:34:15.077 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.081+0000 7f583f7fe700 1 --2- 172.21.15.78:0/2133893813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840104d30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58300098a0 tx=0x7f5830009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3caef32f11045bc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:15.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f583e7fc700 1 -- 172.21.15.78:0/2133893813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830004160 con 0x7f5840104920 2023-10-10T17:34:15.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f583e7fc700 1 -- 172.21.15.78:0/2133893813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58300042c0 con 0x7f5840104920 2023-10-10T17:34:15.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f583e7fc700 1 -- 172.21.15.78:0/2133893813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830011a00 con 0x7f5840104920 2023-10-10T17:34:15.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f5845b97700 1 -- 172.21.15.78:0/2133893813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 msgr2=0x7f5840104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:15.078 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f5845b97700 1 --2- 172.21.15.78:0/2133893813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840104d30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58300098a0 tx=0x7f5830009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f5845b97700 1 -- 172.21.15.78:0/2133893813 shutdown_connections 2023-10-10T17:34:15.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f5845b97700 1 --2- 172.21.15.78:0/2133893813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840104d30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f5845b97700 1 -- 172.21.15.78:0/2133893813 >> 172.21.15.78:0/2133893813 conn(0x7f58400ffed0 msgr2=0x7f5840102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:15.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f5845b97700 1 -- 172.21.15.78:0/2133893813 shutdown_connections 2023-10-10T17:34:15.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.082+0000 7f5845b97700 1 -- 172.21.15.78:0/2133893813 wait complete. 2023-10-10T17:34:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.083+0000 7f5845b97700 1 Processor -- start 2023-10-10T17:34:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.083+0000 7f5845b97700 1 -- start start 2023-10-10T17:34:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.083+0000 7f5845b97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.083+0000 7f5845b97700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584019c680 con 0x7f5840104920 2023-10-10T17:34:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.083+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:15.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.083+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840079ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40904/0 (socket says 172.21.15.78:40904) 2023-10-10T17:34:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.083+0000 7f583f7fe700 1 -- 172.21.15.78:0/3796263942 learned_addr learned my addr 172.21.15.78:0/3796263942 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.084+0000 7f583f7fe700 1 -- 172.21.15.78:0/3796263942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5830009550 con 0x7f5840104920 2023-10-10T17:34:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.084+0000 7f583f7fe700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840079ae0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5830009fb0 tx=0x7f5830011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.084+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5830003930 con 0x7f5840104920 2023-10-10T17:34:15.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.084+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5830003a90 con 0x7f5840104920 2023-10-10T17:34:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.084+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5840108d50 con 0x7f5840104920 2023-10-10T17:34:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.084+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f58300195e0 con 0x7f5840104920 2023-10-10T17:34:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.084+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5840108ee0 con 0x7f5840104920 2023-10-10T17:34:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.085+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5830019850 con 0x7f5840104920 2023-10-10T17:34:15.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.085+0000 7f583cff9700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.085+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 --> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f582803b2d0 con 0x7f5828038710 2023-10-10T17:34:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.085+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f583004c450 con 0x7f5840104920 2023-10-10T17:34:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.085+0000 7f583effd700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 msgr2=0x7f582803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3346226185 2023-10-10T17:34:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.085+0000 7f583effd700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:34:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.286+0000 7f583effd700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 msgr2=0x7f582803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3346226185 2023-10-10T17:34:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.286+0000 7f583effd700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-10T17:34:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.687+0000 7f583effd700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 msgr2=0x7f582803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3346226185 2023-10-10T17:34:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:09.687+0000 7f583effd700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-10T17:34:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:10.488+0000 7f583effd700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 msgr2=0x7f582803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3346226185 2023-10-10T17:34:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:10.488+0000 7f583effd700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-10T17:34:15.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:12.089+0000 7f583effd700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 msgr2=0x7f582803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3346226185 2023-10-10T17:34:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:12.089+0000 7f583effd700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-10T17:34:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:13.661+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7f5830019d30 con 0x7f5840104920 2023-10-10T17:34:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:13.661+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 msgr2=0x7f582803abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:34:15.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:13.661+0000 7f583cff9700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.662+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f583004dd70 con 0x7f5840104920 2023-10-10T17:34:15.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.662+0000 7f583cff9700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f582803bfe0 0x7f582803e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:15.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.662+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f582803b2d0 con 0x7f582803bfe0 2023-10-10T17:34:15.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.662+0000 7f583effd700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f582803bfe0 0x7f582803e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:15.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.663+0000 7f583effd700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f582803bfe0 0x7f582803e3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5834003a10 tx=0x7f58340092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:15.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.664+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f582803b2d0 con 0x7f582803bfe0 2023-10-10T17:34:15.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5840079680 con 0x7f582803bfe0 2023-10-10T17:34:15.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f583cff9700 1 -- 172.21.15.78:0/3796263942 <== mgr.14120 v2:172.21.15.78:6800/1662641434 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5840079680 con 0x7f582803bfe0 2023-10-10T17:34:15.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f582803bfe0 msgr2=0x7f582803e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:15.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f582803bfe0 0x7f582803e3c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5834003a10 tx=0x7f58340092b0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 msgr2=0x7f5840079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:15.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840079ae0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5830009fb0 tx=0x7f5830011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 shutdown_connections 2023-10-10T17:34:15.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f582803bfe0 0x7f582803e3c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5840104920 0x7f5840079ae0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 --2- 172.21.15.78:0/3796263942 >> [v2:172.21.15.78:6800/3346226185,v1:172.21.15.78:6801/3346226185] conn(0x7f5828038710 0x7f582803abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:15.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 >> 172.21.15.78:0/3796263942 conn(0x7f58400ffed0 msgr2=0x7f5840101540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:15.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 shutdown_connections 2023-10-10T17:34:15.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:14.670+0000 7f5845b97700 1 -- 172.21.15.78:0/3796263942 wait complete. 2023-10-10T17:34:15.090 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 5 is available 2023-10-10T17:34:15.091 INFO:teuthology.orchestra.run.smithi078.stdout:Setting orchestrator backend to cephadm... 2023-10-10T17:34:15.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:14] ENGINE Bus STARTING 2023-10-10T17:34:15.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:14] ENGINE Serving on https://172.21.15.78:7150 2023-10-10T17:34:15.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:14] ENGINE Serving on http://172.21.15.78:8765 2023-10-10T17:34:15.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:14] ENGINE Bus STARTED 2023-10-10T17:34:15.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:15.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-10T17:34:15.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: mgrmap e7: x(active, since 1.00594s) 2023-10-10T17:34:15.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:15 smithi078 ceph-mon[99494]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-10T17:34:16.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.557+0000 7f41cc58c700 1 Processor -- start 2023-10-10T17:34:16.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41cc58c700 1 -- start start 2023-10-10T17:34:16.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41cc58c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4108870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:16.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41cc58c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c4079e60 con 0x7f41c4106490 2023-10-10T17:34:16.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41ca328700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4108870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:16.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41ca328700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4108870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40956/0 (socket says 172.21.15.78:40956) 2023-10-10T17:34:16.222 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41ca328700 1 -- 172.21.15.78:0/4075879308 learned_addr learned my addr 172.21.15.78:0/4075879308 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:16.223 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41ca328700 1 -- 172.21.15.78:0/4075879308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c4108db0 con 0x7f41c4106490 2023-10-10T17:34:16.223 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.558+0000 7f41ca328700 1 --2- 172.21.15.78:0/4075879308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4108870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f41b40098a0 tx=0x7f41b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=968b99faf4285985 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:16.223 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41c9326700 1 -- 172.21.15.78:0/4075879308 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41b4004160 con 0x7f41c4106490 2023-10-10T17:34:16.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41c9326700 1 -- 172.21.15.78:0/4075879308 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41b40042c0 con 0x7f41c4106490 2023-10-10T17:34:16.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41c9326700 1 -- 172.21.15.78:0/4075879308 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41b4011a00 con 0x7f41c4106490 2023-10-10T17:34:16.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41cc58c700 1 -- 172.21.15.78:0/4075879308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 msgr2=0x7f41c4108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:16.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41cc58c700 1 --2- 172.21.15.78:0/4075879308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4108870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f41b40098a0 tx=0x7f41b4009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:16.224 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41cc58c700 1 -- 172.21.15.78:0/4075879308 shutdown_connections 2023-10-10T17:34:16.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41cc58c700 1 --2- 172.21.15.78:0/4075879308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4108870 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:16.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41cc58c700 1 -- 172.21.15.78:0/4075879308 >> 172.21.15.78:0/4075879308 conn(0x7f41c40ffed0 msgr2=0x7f41c4102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:16.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41cc58c700 1 -- 172.21.15.78:0/4075879308 shutdown_connections 2023-10-10T17:34:16.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.559+0000 7f41cc58c700 1 -- 172.21.15.78:0/4075879308 wait complete. 2023-10-10T17:34:16.225 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41cc58c700 1 Processor -- start 2023-10-10T17:34:16.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41cc58c700 1 -- start start 2023-10-10T17:34:16.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41cc58c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4199770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:16.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41cc58c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c4199cb0 con 0x7f41c4106490 2023-10-10T17:34:16.226 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41ca328700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4199770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:16.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41ca328700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4199770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40958/0 (socket says 172.21.15.78:40958) 2023-10-10T17:34:16.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41ca328700 1 -- 172.21.15.78:0/3135695106 learned_addr learned my addr 172.21.15.78:0/3135695106 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:16.227 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.560+0000 7f41ca328700 1 -- 172.21.15.78:0/3135695106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b4009550 con 0x7f41c4106490 2023-10-10T17:34:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.561+0000 7f41ca328700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4199770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009fb0 tx=0x7f41b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:16.229 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.561+0000 7f41bb7fe700 1 -- 172.21.15.78:0/3135695106 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41b4003930 con 0x7f41c4106490 2023-10-10T17:34:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.561+0000 7f41bb7fe700 1 -- 172.21.15.78:0/3135695106 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f41b4003a90 con 0x7f41c4106490 2023-10-10T17:34:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.561+0000 7f41bb7fe700 1 -- 172.21.15.78:0/3135695106 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41b4019610 con 0x7f41c4106490 2023-10-10T17:34:16.230 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.561+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c4109d40 con 0x7f41c4106490 2023-10-10T17:34:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.561+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c4109bb0 con 0x7f41c4106490 2023-10-10T17:34:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.562+0000 7f41bb7fe700 1 -- 172.21.15.78:0/3135695106 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f41b401a9d0 con 0x7f41c4106490 2023-10-10T17:34:16.231 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.562+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c4062120 con 0x7f41c4106490 2023-10-10T17:34:16.232 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.562+0000 7f41bb7fe700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f41b00385f0 0x7f41b003aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:16.232 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.562+0000 7f41bb7fe700 1 -- 172.21.15.78:0/3135695106 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f41b404c5e0 con 0x7f41c4106490 2023-10-10T17:34:16.232 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.562+0000 7f41c9b27700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f41b00385f0 0x7f41b003aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:16.233 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.562+0000 7f41c9b27700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f41b00385f0 0x7f41b003aaa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f41c0006fd0 tx=0x7f41c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:16.233 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.567+0000 7f41bb7fe700 1 -- 172.21.15.78:0/3135695106 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f41b4003c00 con 0x7f41c4106490 2023-10-10T17:34:16.233 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.768+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c4102190 con 0x7f41b00385f0 2023-10-10T17:34:16.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.775+0000 7f41bb7fe700 1 -- 172.21.15.78:0/3135695106 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f41c4102190 con 0x7f41b00385f0 2023-10-10T17:34:16.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.777+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f41b00385f0 msgr2=0x7f41b003aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:16.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.777+0000 7f41cc58c700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f41b00385f0 0x7f41b003aaa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f41c0006fd0 tx=0x7f41c0006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:16.234 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 msgr2=0x7f41c4199770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:16.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4199770 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f41b4009fb0 tx=0x7f41b4011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:16.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 shutdown_connections 2023-10-10T17:34:16.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f41b00385f0 0x7f41b003aaa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:16.235 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 --2- 172.21.15.78:0/3135695106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c4106490 0x7f41c4199770 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:16.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 >> 172.21.15.78:0/3135695106 conn(0x7f41c40ffed0 msgr2=0x7f41c4101530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:16.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 shutdown_connections 2023-10-10T17:34:16.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:15.778+0000 7f41cc58c700 1 -- 172.21.15.78:0/3135695106 wait complete. 2023-10-10T17:34:17.063 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:16 smithi078 ceph-mon[99494]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:17.063 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:16 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:17.063 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:16 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:17.064 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:16 smithi078 ceph-mon[99494]: mgrmap e8: x(active, since 2s) 2023-10-10T17:34:17.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout value unchanged 2023-10-10T17:34:17.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff7592ac700 1 Processor -- start 2023-10-10T17:34:17.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff7592ac700 1 -- start start 2023-10-10T17:34:17.360 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff7592ac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754104120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:17.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff7592ac700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7541046f0 con 0x7ff754103d10 2023-10-10T17:34:17.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff752ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:17.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff752ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754104120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40962/0 (socket says 172.21.15.78:40962) 2023-10-10T17:34:17.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff752ffd700 1 -- 172.21.15.78:0/2739420181 learned_addr learned my addr 172.21.15.78:0/2739420181 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:17.361 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.703+0000 7ff752ffd700 1 -- 172.21.15.78:0/2739420181 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff754106ca0 con 0x7ff754103d10 2023-10-10T17:34:17.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff752ffd700 1 --2- 172.21.15.78:0/2739420181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754104120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7440098a0 tx=0x7ff744009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310e45d2fbdbdd9c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:17.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff751ffb700 1 -- 172.21.15.78:0/2739420181 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff744004160 con 0x7ff754103d10 2023-10-10T17:34:17.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff751ffb700 1 -- 172.21.15.78:0/2739420181 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff7440042c0 con 0x7ff754103d10 2023-10-10T17:34:17.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff751ffb700 1 -- 172.21.15.78:0/2739420181 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff744011a00 con 0x7ff754103d10 2023-10-10T17:34:17.362 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff7592ac700 1 -- 172.21.15.78:0/2739420181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 msgr2=0x7ff754104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:17.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff7592ac700 1 --2- 172.21.15.78:0/2739420181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754104120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7440098a0 tx=0x7ff744009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:17.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff7592ac700 1 -- 172.21.15.78:0/2739420181 shutdown_connections 2023-10-10T17:34:17.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff7592ac700 1 --2- 172.21.15.78:0/2739420181 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754104120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:17.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff7592ac700 1 -- 172.21.15.78:0/2739420181 >> 172.21.15.78:0/2739420181 conn(0x7ff7540ff650 msgr2=0x7ff754101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:17.363 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff7592ac700 1 -- 172.21.15.78:0/2739420181 shutdown_connections 2023-10-10T17:34:17.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.704+0000 7ff7592ac700 1 -- 172.21.15.78:0/2739420181 wait complete. 2023-10-10T17:34:17.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.705+0000 7ff7592ac700 1 Processor -- start 2023-10-10T17:34:17.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.705+0000 7ff7592ac700 1 -- start start 2023-10-10T17:34:17.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.705+0000 7ff7592ac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:17.364 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.705+0000 7ff7592ac700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff754106b40 con 0x7ff754103d10 2023-10-10T17:34:17.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.705+0000 7ff752ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:17.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.705+0000 7ff752ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40964/0 (socket says 172.21.15.78:40964) 2023-10-10T17:34:17.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.705+0000 7ff752ffd700 1 -- 172.21.15.78:0/483753493 learned_addr learned my addr 172.21.15.78:0/483753493 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:17.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.706+0000 7ff752ffd700 1 -- 172.21.15.78:0/483753493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff744009550 con 0x7ff754103d10 2023-10-10T17:34:17.365 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.706+0000 7ff752ffd700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754106600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff744009fb0 tx=0x7ff744011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:17.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.706+0000 7ff73bfff700 1 -- 172.21.15.78:0/483753493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff744003930 con 0x7ff754103d10 2023-10-10T17:34:17.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.706+0000 7ff73bfff700 1 -- 172.21.15.78:0/483753493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff744003a90 con 0x7ff754103d10 2023-10-10T17:34:17.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.706+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff754109ed0 con 0x7ff754103d10 2023-10-10T17:34:17.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.706+0000 7ff73bfff700 1 -- 172.21.15.78:0/483753493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7440195e0 con 0x7ff754103d10 2023-10-10T17:34:17.366 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.706+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75410a060 con 0x7ff754103d10 2023-10-10T17:34:17.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.707+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff754107b10 con 0x7ff754103d10 2023-10-10T17:34:17.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.707+0000 7ff73bfff700 1 -- 172.21.15.78:0/483753493 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff74401a9d0 con 0x7ff754103d10 2023-10-10T17:34:17.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.707+0000 7ff73bfff700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff73c0386c0 0x7ff73c03ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:17.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.707+0000 7ff73bfff700 1 -- 172.21.15.78:0/483753493 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff74404c7b0 con 0x7ff754103d10 2023-10-10T17:34:17.367 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.707+0000 7ff7527fc700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff73c0386c0 0x7ff73c03ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:17.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.708+0000 7ff7527fc700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff73c0386c0 0x7ff73c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff748006fd0 tx=0x7ff748006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:17.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.712+0000 7ff73bfff700 1 -- 172.21.15.78:0/483753493 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff754107b10 con 0x7ff754103d10 2023-10-10T17:34:17.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.900+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff754107b10 con 0x7ff73c0386c0 2023-10-10T17:34:17.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.900+0000 7ff73bfff700 1 -- 172.21.15.78:0/483753493 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff754107b10 con 0x7ff73c0386c0 2023-10-10T17:34:17.368 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff73c0386c0 msgr2=0x7ff73c03ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:17.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff73c0386c0 0x7ff73c03ab70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff748006fd0 tx=0x7ff748006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:17.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 msgr2=0x7ff754106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:17.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754106600 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff744009fb0 tx=0x7ff744011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:17.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 shutdown_connections 2023-10-10T17:34:17.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff73c0386c0 0x7ff73c03ab70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:17.369 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 --2- 172.21.15.78:0/483753493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff754103d10 0x7ff754106600 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:17.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 >> 172.21.15.78:0/483753493 conn(0x7ff7540ff650 msgr2=0x7ff754100410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:17.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 shutdown_connections 2023-10-10T17:34:17.370 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:16.902+0000 7ff7592ac700 1 -- 172.21.15.78:0/483753493 wait complete. 2023-10-10T17:34:17.370 INFO:teuthology.orchestra.run.smithi078.stdout:Generating ssh key... 2023-10-10T17:34:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:17 smithi078 ceph-mon[99494]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:18.378 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: Generating public/private rsa key pair. 2023-10-10T17:34:18.378 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: Your identification has been saved in /tmp/tmp4lvoh0os/key. 2023-10-10T17:34:18.378 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: Your public key has been saved in /tmp/tmp4lvoh0os/key.pub. 2023-10-10T17:34:18.378 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: The key fingerprint is: 2023-10-10T17:34:18.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: SHA256:eN6th5DJ6S0emKY7mMHbFtIAG1Wo6fkvD7LGJKJ4C4E ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:34:18.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: The key's randomart image is: 2023-10-10T17:34:18.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: +---[RSA 3072]----+ 2023-10-10T17:34:18.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: | ..o. | 2023-10-10T17:34:18.379 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: |o . | 2023-10-10T17:34:18.380 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: | * | 2023-10-10T17:34:18.380 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: |= . . | 2023-10-10T17:34:18.380 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: |E..o ..S+ | 2023-10-10T17:34:18.380 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: |o=+ o =*. . | 2023-10-10T17:34:18.380 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: |O..O .+.oo... | 2023-10-10T17:34:18.380 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: |o+Bo=o o.o.. | 2023-10-10T17:34:18.381 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: |.o.o*= ..... | 2023-10-10T17:34:18.381 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: +----[SHA256]-----+ 2023-10-10T17:34:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.803+0000 7f45afa6e700 1 Processor -- start 2023-10-10T17:34:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.803+0000 7f45afa6e700 1 -- start start 2023-10-10T17:34:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.803+0000 7f45afa6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a8105e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:18.502 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.803+0000 7f45afa6e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a8106400 con 0x7f45a8105a20 2023-10-10T17:34:18.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.803+0000 7f45ad80a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a8105e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:18.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.803+0000 7f45ad80a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a8105e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40968/0 (socket says 172.21.15.78:40968) 2023-10-10T17:34:18.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.804+0000 7f45ad80a700 1 -- 172.21.15.78:0/2920111221 learned_addr learned my addr 172.21.15.78:0/2920111221 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:18.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.804+0000 7f45ad80a700 1 -- 172.21.15.78:0/2920111221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a8106c20 con 0x7f45a8105a20 2023-10-10T17:34:18.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.804+0000 7f45ad80a700 1 --2- 172.21.15.78:0/2920111221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a8105e30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f45980098a0 tx=0x7f4598009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b93a68786e6de33 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:18.503 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.804+0000 7f45ac808700 1 -- 172.21.15.78:0/2920111221 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4598004160 con 0x7f45a8105a20 2023-10-10T17:34:18.504 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.804+0000 7f45ac808700 1 -- 172.21.15.78:0/2920111221 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45980042c0 con 0x7f45a8105a20 2023-10-10T17:34:18.504 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.804+0000 7f45ac808700 1 -- 172.21.15.78:0/2920111221 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4598011a00 con 0x7f45a8105a20 2023-10-10T17:34:18.504 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 -- 172.21.15.78:0/2920111221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 msgr2=0x7f45a8105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:18.504 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 --2- 172.21.15.78:0/2920111221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a8105e30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f45980098a0 tx=0x7f4598009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:18.504 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 -- 172.21.15.78:0/2920111221 shutdown_connections 2023-10-10T17:34:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 --2- 172.21.15.78:0/2920111221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a8105e30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 -- 172.21.15.78:0/2920111221 >> 172.21.15.78:0/2920111221 conn(0x7f45a8100fd0 msgr2=0x7f45a8103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 -- 172.21.15.78:0/2920111221 shutdown_connections 2023-10-10T17:34:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 -- 172.21.15.78:0/2920111221 wait complete. 2023-10-10T17:34:18.505 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 Processor -- start 2023-10-10T17:34:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 -- start start 2023-10-10T17:34:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a807bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.805+0000 7f45afa6e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a807c140 con 0x7f45a8105a20 2023-10-10T17:34:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f45ad80a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a807bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:18.506 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f45ad80a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a807bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40970/0 (socket says 172.21.15.78:40970) 2023-10-10T17:34:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f45ad80a700 1 -- 172.21.15.78:0/3047441835 learned_addr learned my addr 172.21.15.78:0/3047441835 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f45ad80a700 1 -- 172.21.15.78:0/3047441835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4598009550 con 0x7f45a8105a20 2023-10-10T17:34:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f45ad80a700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a807bc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4598009fb0 tx=0x7f4598011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f459effd700 1 -- 172.21.15.78:0/3047441835 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4598003930 con 0x7f45a8105a20 2023-10-10T17:34:18.507 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a8109e50 con 0x7f45a8105a20 2023-10-10T17:34:18.508 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f459effd700 1 -- 172.21.15.78:0/3047441835 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4598003a90 con 0x7f45a8105a20 2023-10-10T17:34:18.508 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f459effd700 1 -- 172.21.15.78:0/3047441835 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45980195e0 con 0x7f45a8105a20 2023-10-10T17:34:18.508 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.806+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a8109fe0 con 0x7f45a8105a20 2023-10-10T17:34:18.508 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.807+0000 7f459effd700 1 -- 172.21.15.78:0/3047441835 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f45980198a0 con 0x7f45a8105a20 2023-10-10T17:34:18.508 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.807+0000 7f459effd700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f45940386c0 0x7f459403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:18.508 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.807+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a8107a90 con 0x7f45a8105a20 2023-10-10T17:34:18.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.807+0000 7f459effd700 1 -- 172.21.15.78:0/3047441835 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f459804c890 con 0x7f45a8105a20 2023-10-10T17:34:18.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.807+0000 7f45ad009700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f45940386c0 0x7f459403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:18.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.808+0000 7f45ad009700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f45940386c0 0x7f459403ab70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f45a4006fd0 tx=0x7f45a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:18.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.813+0000 7f459effd700 1 -- 172.21.15.78:0/3047441835 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f45a8107a90 con 0x7f45a8105a20 2023-10-10T17:34:18.509 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:17.999+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f45a8107a90 con 0x7f45940386c0 2023-10-10T17:34:18.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.098+0000 7f459effd700 1 -- 172.21.15.78:0/3047441835 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f45a8107a90 con 0x7f45940386c0 2023-10-10T17:34:18.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f45940386c0 msgr2=0x7f459403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:18.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f45940386c0 0x7f459403ab70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f45a4006fd0 tx=0x7f45a4006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:18.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 msgr2=0x7f45a807bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:18.510 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a807bc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4598009fb0 tx=0x7f4598011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:18.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 shutdown_connections 2023-10-10T17:34:18.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f45940386c0 0x7f459403ab70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:18.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 --2- 172.21.15.78:0/3047441835 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45a8105a20 0x7f45a807bc00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:18.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 >> 172.21.15.78:0/3047441835 conn(0x7f45a8100fd0 msgr2=0x7f45a8102640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:18.511 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 shutdown_connections 2023-10-10T17:34:18.512 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.101+0000 7f45afa6e700 1 -- 172.21.15.78:0/3047441835 wait complete. 2023-10-10T17:34:19.363 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:19 smithi078 ceph-mon[99494]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:19.363 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:19 smithi078 ceph-mon[99494]: Generating ssh key... 2023-10-10T17:34:19.363 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:19 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:19.364 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:19 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:19.644 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC04uixS8LRvZPoJRTPLqrYaxTz4Og5+Hot0IdV0HknwJocjymwDfusEJKovPA7q2v0y7KQYHQMrISXIvVxMsuYqraO25km3OuBbd4riwCqMgQ2aaLIq/WmXWVMDK0hGzVQKkvMtjIQjmeTRD7VX8umD4qTe4O62JyXl3mWUQWW9P66YBLf7KWyLwKh1j2q3sqXsrEOvOk8zIsrbuVQ0ubn34gCc9AkH+NLWhv+NI2pKXMJxkrDThafFg9L07Lze4FYWYIhSTtiDWrk5aH+8Ag2feqA+CjPg5x6MX0LdgLV4262YaJtsx5gCDy7zWe5zTEjgQmSrvHABZUk3a0afSISuR/BLiEGjSoWh0dWBs4hx1o6ajX1EeunC7hNVvTxBsjnLQG9uSDYdF3EP2l3lkC42r2SD0wchsebJ5VmMWNcOPg1w+6wehavXnN4Svrc7UPcmpvQlJb3s2+szWgPXRYb3Wtw8h48T4GlpinMAjdEdqICbqL5VyqYvl/IzyeixJE= ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:34:19.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.997+0000 7f6dfb40a700 1 Processor -- start 2023-10-10T17:34:19.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.997+0000 7f6dfb40a700 1 -- start start 2023-10-10T17:34:19.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.997+0000 7f6dfb40a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df4106300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:19.645 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.997+0000 7f6dfb40a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df41068d0 con 0x7f6df4105ef0 2023-10-10T17:34:19.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.997+0000 7f6df91a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df4106300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:19.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.997+0000 7f6df91a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df4106300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40974/0 (socket says 172.21.15.78:40974) 2023-10-10T17:34:19.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.997+0000 7f6df91a6700 1 -- 172.21.15.78:0/300793452 learned_addr learned my addr 172.21.15.78:0/300793452 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:19.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6df91a6700 1 -- 172.21.15.78:0/300793452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df4106a10 con 0x7f6df4105ef0 2023-10-10T17:34:19.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6df91a6700 1 --2- 172.21.15.78:0/300793452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df4106300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6df00098a0 tx=0x7f6df0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a58a8fa3c190e6c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:19.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6debfff700 1 -- 172.21.15.78:0/300793452 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df0004160 con 0x7f6df4105ef0 2023-10-10T17:34:19.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6debfff700 1 -- 172.21.15.78:0/300793452 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6df00042c0 con 0x7f6df4105ef0 2023-10-10T17:34:19.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6debfff700 1 -- 172.21.15.78:0/300793452 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df0011a00 con 0x7f6df4105ef0 2023-10-10T17:34:19.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6dfb40a700 1 -- 172.21.15.78:0/300793452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 msgr2=0x7f6df4106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:19.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6dfb40a700 1 --2- 172.21.15.78:0/300793452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df4106300 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6df00098a0 tx=0x7f6df0009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:19.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6dfb40a700 1 -- 172.21.15.78:0/300793452 shutdown_connections 2023-10-10T17:34:19.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6dfb40a700 1 --2- 172.21.15.78:0/300793452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df4106300 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:19.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.998+0000 7f6dfb40a700 1 -- 172.21.15.78:0/300793452 >> 172.21.15.78:0/300793452 conn(0x7f6df40ff650 msgr2=0x7f6df4101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:19.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6dfb40a700 1 -- 172.21.15.78:0/300793452 shutdown_connections 2023-10-10T17:34:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6dfb40a700 1 -- 172.21.15.78:0/300793452 wait complete. 2023-10-10T17:34:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6dfb40a700 1 Processor -- start 2023-10-10T17:34:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6dfb40a700 1 -- start start 2023-10-10T17:34:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6dfb40a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df419b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:19.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6dfb40a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df419bef0 con 0x7f6df4105ef0 2023-10-10T17:34:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6df91a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df419b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6df91a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df419b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40976/0 (socket says 172.21.15.78:40976) 2023-10-10T17:34:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:18.999+0000 7f6df91a6700 1 -- 172.21.15.78:0/2031090673 learned_addr learned my addr 172.21.15.78:0/2031090673 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.000+0000 7f6df91a6700 1 -- 172.21.15.78:0/2031090673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0009550 con 0x7f6df4105ef0 2023-10-10T17:34:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.000+0000 7f6df91a6700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df419b9b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009fb0 tx=0x7f6df0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.000+0000 7f6dea7fc700 1 -- 172.21.15.78:0/2031090673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df0003930 con 0x7f6df4105ef0 2023-10-10T17:34:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.000+0000 7f6dea7fc700 1 -- 172.21.15.78:0/2031090673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6df0003a90 con 0x7f6df4105ef0 2023-10-10T17:34:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.000+0000 7f6dea7fc700 1 -- 172.21.15.78:0/2031090673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6df0019610 con 0x7f6df4105ef0 2023-10-10T17:34:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.000+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df4104820 con 0x7f6df4105ef0 2023-10-10T17:34:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.000+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df4104690 con 0x7f6df4105ef0 2023-10-10T17:34:19.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.001+0000 7f6dea7fc700 1 -- 172.21.15.78:0/2031090673 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f6df001a430 con 0x7f6df4105ef0 2023-10-10T17:34:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.001+0000 7f6dea7fc700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6de0038710 0x7f6de003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.001+0000 7f6dea7fc700 1 -- 172.21.15.78:0/2031090673 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6df004c7e0 con 0x7f6df4105ef0 2023-10-10T17:34:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.001+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df41955e0 con 0x7f6df4105ef0 2023-10-10T17:34:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.001+0000 7f6df89a5700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6de0038710 0x7f6de003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:19.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.001+0000 7f6df89a5700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6de0038710 0x7f6de003abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6de4006fd0 tx=0x7f6de4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.006+0000 7f6dea7fc700 1 -- 172.21.15.78:0/2031090673 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6df0003c00 con 0x7f6df4105ef0 2023-10-10T17:34:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.196+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6df419c670 con 0x7f6de0038710 2023-10-10T17:34:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.196+0000 7f6dea7fc700 1 -- 172.21.15.78:0/2031090673 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f6df419c670 con 0x7f6de0038710 2023-10-10T17:34:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6de0038710 msgr2=0x7f6de003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:19.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6de0038710 0x7f6de003abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6de4006fd0 tx=0x7f6de4006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:19.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 msgr2=0x7f6df419b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:19.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df419b9b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009fb0 tx=0x7f6df0011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:19.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 shutdown_connections 2023-10-10T17:34:19.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6de0038710 0x7f6de003abc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:19.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 --2- 172.21.15.78:0/2031090673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df4105ef0 0x7f6df419b9b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 >> 172.21.15.78:0/2031090673 conn(0x7f6df40ff650 msgr2=0x7f6df4192720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 shutdown_connections 2023-10-10T17:34:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:19.198+0000 7f6dfb40a700 1 -- 172.21.15.78:0/2031090673 wait complete. 2023-10-10T17:34:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-10-10T17:34:19.655 INFO:teuthology.orchestra.run.smithi078.stdout:Adding key to root@localhost authorized_keys... 2023-10-10T17:34:19.656 INFO:teuthology.orchestra.run.smithi078.stdout:Adding host smithi078... 2023-10-10T17:34:20.539 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:20 smithi078 ceph-mon[99494]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:21.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:21 smithi078 ceph-mon[99494]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:21.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:21 smithi078 ceph-mon[99494]: Deploying cephadm binary to smithi078 2023-10-10T17:34:22.795 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Added host 'smithi078' with addr '172.21.15.78' 2023-10-10T17:34:22.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.100+0000 7fd46425e700 1 Processor -- start 2023-10-10T17:34:22.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.100+0000 7fd46425e700 1 -- start start 2023-10-10T17:34:22.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.100+0000 7fd46425e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:22.796 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.100+0000 7fd46425e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c079e00 con 0x7fd45c1064f0 2023-10-10T17:34:22.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.100+0000 7fd461ffa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:22.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.100+0000 7fd461ffa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40980/0 (socket says 172.21.15.78:40980) 2023-10-10T17:34:22.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.100+0000 7fd461ffa700 1 -- 172.21.15.78:0/4117492763 learned_addr learned my addr 172.21.15.78:0/4117492763 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:22.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.101+0000 7fd461ffa700 1 -- 172.21.15.78:0/4117492763 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c108e10 con 0x7fd45c1064f0 2023-10-10T17:34:22.797 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.101+0000 7fd461ffa700 1 --2- 172.21.15.78:0/4117492763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1088d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd44c0098a0 tx=0x7fd44c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950fd8d3b28b8dba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:22.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.101+0000 7fd460ff8700 1 -- 172.21.15.78:0/4117492763 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd44c004160 con 0x7fd45c1064f0 2023-10-10T17:34:22.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.101+0000 7fd460ff8700 1 -- 172.21.15.78:0/4117492763 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd44c0042c0 con 0x7fd45c1064f0 2023-10-10T17:34:22.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.101+0000 7fd460ff8700 1 -- 172.21.15.78:0/4117492763 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd44c011a00 con 0x7fd45c1064f0 2023-10-10T17:34:22.798 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 -- 172.21.15.78:0/4117492763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 msgr2=0x7fd45c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:22.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 --2- 172.21.15.78:0/4117492763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1088d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd44c0098a0 tx=0x7fd44c009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:22.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 -- 172.21.15.78:0/4117492763 shutdown_connections 2023-10-10T17:34:22.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 --2- 172.21.15.78:0/4117492763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1088d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:22.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 -- 172.21.15.78:0/4117492763 >> 172.21.15.78:0/4117492763 conn(0x7fd45c0ffed0 msgr2=0x7fd45c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:22.799 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 -- 172.21.15.78:0/4117492763 shutdown_connections 2023-10-10T17:34:22.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 -- 172.21.15.78:0/4117492763 wait complete. 2023-10-10T17:34:22.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 Processor -- start 2023-10-10T17:34:22.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 -- start start 2023-10-10T17:34:22.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:22.800 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.102+0000 7fd46425e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c199cf0 con 0x7fd45c1064f0 2023-10-10T17:34:22.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd461ffa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:22.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd461ffa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:40982/0 (socket says 172.21.15.78:40982) 2023-10-10T17:34:22.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd461ffa700 1 -- 172.21.15.78:0/2767463585 learned_addr learned my addr 172.21.15.78:0/2767463585 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:22.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd461ffa700 1 -- 172.21.15.78:0/2767463585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c009550 con 0x7fd45c1064f0 2023-10-10T17:34:22.801 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd461ffa700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1997b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009fb0 tx=0x7fd44c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:22.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd452ffd700 1 -- 172.21.15.78:0/2767463585 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd44c003930 con 0x7fd45c1064f0 2023-10-10T17:34:22.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd452ffd700 1 -- 172.21.15.78:0/2767463585 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd44c003a90 con 0x7fd45c1064f0 2023-10-10T17:34:22.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45c109d40 con 0x7fd45c1064f0 2023-10-10T17:34:22.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd452ffd700 1 -- 172.21.15.78:0/2767463585 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd44c019610 con 0x7fd45c1064f0 2023-10-10T17:34:22.802 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.103+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45c109bb0 con 0x7fd45c1064f0 2023-10-10T17:34:22.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.104+0000 7fd452ffd700 1 -- 172.21.15.78:0/2767463585 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd44c019880 con 0x7fd45c1064f0 2023-10-10T17:34:22.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.104+0000 7fd452ffd700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fd448038710 0x7fd44803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:22.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.104+0000 7fd452ffd700 1 -- 172.21.15.78:0/2767463585 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd44c04c9d0 con 0x7fd45c1064f0 2023-10-10T17:34:22.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.104+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45c062120 con 0x7fd45c1064f0 2023-10-10T17:34:22.803 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.104+0000 7fd4617f9700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fd448038710 0x7fd44803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:22.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.105+0000 7fd4617f9700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fd448038710 0x7fd44803abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd458006fd0 tx=0x7fd458006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:22.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.109+0000 7fd452ffd700 1 -- 172.21.15.78:0/2767463585 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd44c019b80 con 0x7fd45c1064f0 2023-10-10T17:34:22.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:20.301+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}) v1 -- 0x7fd45c1021e0 con 0x7fd448038710 2023-10-10T17:34:22.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.356+0000 7fd452ffd700 1 -- 172.21.15.78:0/2767463585 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fd45c1021e0 con 0x7fd448038710 2023-10-10T17:34:22.804 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.358+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fd448038710 msgr2=0x7fd44803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:22.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.358+0000 7fd46425e700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fd448038710 0x7fd44803abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fd458006fd0 tx=0x7fd458006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:22.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 msgr2=0x7fd45c1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:22.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1997b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009fb0 tx=0x7fd44c011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:22.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 shutdown_connections 2023-10-10T17:34:22.805 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fd448038710 0x7fd44803abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:22.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 --2- 172.21.15.78:0/2767463585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd45c1064f0 0x7fd45c1997b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:22.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 >> 172.21.15.78:0/2767463585 conn(0x7fd45c0ffed0 msgr2=0x7fd45c1015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:22.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 shutdown_connections 2023-10-10T17:34:22.806 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:22.359+0000 7fd46425e700 1 -- 172.21.15.78:0/2767463585 wait complete. 2023-10-10T17:34:22.806 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mon service... 2023-10-10T17:34:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:23 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:23 smithi078 ceph-mon[99494]: Added host smithi078 2023-10-10T17:34:23.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:23 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:24.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-10-10T17:34:24.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.316+0000 7f6cdf3eb700 1 Processor -- start 2023-10-10T17:34:24.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.317+0000 7f6cdf3eb700 1 -- start start 2023-10-10T17:34:24.244 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.317+0000 7f6cdf3eb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8105e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.317+0000 7f6cdf3eb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8106390 con 0x7f6cd8105a40 2023-10-10T17:34:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.317+0000 7f6cdd187700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8105e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.317+0000 7f6cdd187700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8105e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41016/0 (socket says 172.21.15.78:41016) 2023-10-10T17:34:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.317+0000 7f6cdd187700 1 -- 172.21.15.78:0/477435713 learned_addr learned my addr 172.21.15.78:0/477435713 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.318+0000 7f6cdd187700 1 -- 172.21.15.78:0/477435713 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd81064d0 con 0x7f6cd8105a40 2023-10-10T17:34:24.246 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.318+0000 7f6cdd187700 1 --2- 172.21.15.78:0/477435713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8105e50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40098a0 tx=0x7f6cd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a7c0087bc399cf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:24.246 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.318+0000 7f6ccffff700 1 -- 172.21.15.78:0/477435713 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cd4004160 con 0x7f6cd8105a40 2023-10-10T17:34:24.246 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.318+0000 7f6ccffff700 1 -- 172.21.15.78:0/477435713 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cd40042c0 con 0x7f6cd8105a40 2023-10-10T17:34:24.246 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.318+0000 7f6ccffff700 1 -- 172.21.15.78:0/477435713 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cd4011a00 con 0x7f6cd8105a40 2023-10-10T17:34:24.246 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.319+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/477435713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 msgr2=0x7f6cd8105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:24.247 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.319+0000 7f6cdf3eb700 1 --2- 172.21.15.78:0/477435713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8105e50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cd40098a0 tx=0x7f6cd4009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:24.247 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.319+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/477435713 shutdown_connections 2023-10-10T17:34:24.247 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.319+0000 7f6cdf3eb700 1 --2- 172.21.15.78:0/477435713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8105e50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:24.247 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.319+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/477435713 >> 172.21.15.78:0/477435713 conn(0x7f6cd81010d0 msgr2=0x7f6cd81034e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:24.247 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.319+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/477435713 shutdown_connections 2023-10-10T17:34:24.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.319+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/477435713 wait complete. 2023-10-10T17:34:24.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdf3eb700 1 Processor -- start 2023-10-10T17:34:24.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdf3eb700 1 -- start start 2023-10-10T17:34:24.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdf3eb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:24.248 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdf3eb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd8199cd0 con 0x7f6cd8105a40 2023-10-10T17:34:24.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdd187700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:24.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdd187700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41018/0 (socket says 172.21.15.78:41018) 2023-10-10T17:34:24.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdd187700 1 -- 172.21.15.78:0/365729153 learned_addr learned my addr 172.21.15.78:0/365729153 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:24.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdd187700 1 -- 172.21.15.78:0/365729153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd4009550 con 0x7f6cd8105a40 2023-10-10T17:34:24.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.320+0000 7f6cdd187700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8199790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4006af0 tx=0x7f6cd4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:24.249 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.321+0000 7f6cce7fc700 1 -- 172.21.15.78:0/365729153 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cd4003a40 con 0x7f6cd8105a40 2023-10-10T17:34:24.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.321+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd8107a90 con 0x7f6cd8105a40 2023-10-10T17:34:24.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.321+0000 7f6cce7fc700 1 -- 172.21.15.78:0/365729153 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6cd4003ba0 con 0x7f6cd8105a40 2023-10-10T17:34:24.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.321+0000 7f6cce7fc700 1 -- 172.21.15.78:0/365729153 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6cd401a6e0 con 0x7f6cd8105a40 2023-10-10T17:34:24.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.321+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd8107900 con 0x7f6cd8105a40 2023-10-10T17:34:24.250 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.322+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd8062120 con 0x7f6cd8105a40 2023-10-10T17:34:24.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.322+0000 7f6cce7fc700 1 -- 172.21.15.78:0/365729153 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f6cd401a950 con 0x7f6cd8105a40 2023-10-10T17:34:24.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.322+0000 7f6cce7fc700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6cc4038710 0x7f6cc403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:24.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.322+0000 7f6cce7fc700 1 -- 172.21.15.78:0/365729153 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6cd404c710 con 0x7f6cd8105a40 2023-10-10T17:34:24.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.322+0000 7f6cdc986700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6cc4038710 0x7f6cc403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:24.251 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.322+0000 7f6cdc986700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6cc4038710 0x7f6cc403abc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8006fd0 tx=0x7f6cc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.327+0000 7f6cce7fc700 1 -- 172.21.15.78:0/365729153 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6cd401ac00 con 0x7f6cd8105a40 2023-10-10T17:34:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.538+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6cd8102570 con 0x7f6cc4038710 2023-10-10T17:34:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.543+0000 7f6cce7fc700 1 -- 172.21.15.78:0/365729153 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6cd8102570 con 0x7f6cc4038710 2023-10-10T17:34:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6cc4038710 msgr2=0x7f6cc403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6cc4038710 0x7f6cc403abc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6cc8006fd0 tx=0x7f6cc8006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:24.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 msgr2=0x7f6cd8199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:24.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8199790 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4006af0 tx=0x7f6cd4011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:24.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 shutdown_connections 2023-10-10T17:34:24.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f6cc4038710 0x7f6cc403abc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:24.253 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 --2- 172.21.15.78:0/365729153 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6cd8105a40 0x7f6cd8199790 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:24.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 >> 172.21.15.78:0/365729153 conn(0x7f6cd81010d0 msgr2=0x7f6cd8101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:24.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 shutdown_connections 2023-10-10T17:34:24.254 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:23.545+0000 7f6cdf3eb700 1 -- 172.21.15.78:0/365729153 wait complete. 2023-10-10T17:34:24.254 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mgr service... 2023-10-10T17:34:24.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:24 smithi078 ceph-mon[99494]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:24.845 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:24 smithi078 ceph-mon[99494]: Saving service mon spec with placement count:5 2023-10-10T17:34:24.845 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:24 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:25.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-10-10T17:34:25.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.807+0000 7fc5d3b4a700 1 Processor -- start 2023-10-10T17:34:25.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.807+0000 7fc5d3b4a700 1 -- start start 2023-10-10T17:34:25.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d3b4a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc104120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:25.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d3b4a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc1046f0 con 0x7fc5cc103d10 2023-10-10T17:34:25.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d18e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:25.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d18e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc104120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41022/0 (socket says 172.21.15.78:41022) 2023-10-10T17:34:25.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d18e6700 1 -- 172.21.15.78:0/3342780306 learned_addr learned my addr 172.21.15.78:0/3342780306 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:25.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d18e6700 1 -- 172.21.15.78:0/3342780306 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5cc106ca0 con 0x7fc5cc103d10 2023-10-10T17:34:25.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d18e6700 1 --2- 172.21.15.78:0/3342780306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc104120 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0098a0 tx=0x7fc5bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786e34f334f5acaa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:25.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d08e4700 1 -- 172.21.15.78:0/3342780306 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5bc004160 con 0x7fc5cc103d10 2023-10-10T17:34:25.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d08e4700 1 -- 172.21.15.78:0/3342780306 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5bc0042c0 con 0x7fc5cc103d10 2023-10-10T17:34:25.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.808+0000 7fc5d08e4700 1 -- 172.21.15.78:0/3342780306 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5bc011a00 con 0x7fc5cc103d10 2023-10-10T17:34:25.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.809+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/3342780306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 msgr2=0x7fc5cc104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:25.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.809+0000 7fc5d3b4a700 1 --2- 172.21.15.78:0/3342780306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc104120 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc0098a0 tx=0x7fc5bc009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:25.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.809+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/3342780306 shutdown_connections 2023-10-10T17:34:25.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.809+0000 7fc5d3b4a700 1 --2- 172.21.15.78:0/3342780306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc104120 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:25.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.809+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/3342780306 >> 172.21.15.78:0/3342780306 conn(0x7fc5cc0ff650 msgr2=0x7fc5cc101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:25.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.809+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/3342780306 shutdown_connections 2023-10-10T17:34:25.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.809+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/3342780306 wait complete. 2023-10-10T17:34:25.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d3b4a700 1 Processor -- start 2023-10-10T17:34:25.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d3b4a700 1 -- start start 2023-10-10T17:34:25.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d3b4a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:25.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d3b4a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5cc199e20 con 0x7fc5cc103d10 2023-10-10T17:34:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d18e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d18e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc1998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41024/0 (socket says 172.21.15.78:41024) 2023-10-10T17:34:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d18e6700 1 -- 172.21.15.78:0/1832027992 learned_addr learned my addr 172.21.15.78:0/1832027992 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.810+0000 7fc5d18e6700 1 -- 172.21.15.78:0/1832027992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc009550 con 0x7fc5cc103d10 2023-10-10T17:34:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.811+0000 7fc5d18e6700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc1998e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006af0 tx=0x7fc5bc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.811+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5bc003a40 con 0x7fc5cc103d10 2023-10-10T17:34:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.811+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5cc109ed0 con 0x7fc5cc103d10 2023-10-10T17:34:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.811+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc5bc003ba0 con 0x7fc5cc103d10 2023-10-10T17:34:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.811+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc5bc01a700 con 0x7fc5cc103d10 2023-10-10T17:34:25.395 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.811+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5cc10a060 con 0x7fc5cc103d10 2023-10-10T17:34:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.812+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc5bc024430 con 0x7fc5cc103d10 2023-10-10T17:34:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.812+0000 7fc5c2ffd700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fc5b8038710 0x7fc5b803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.812+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc107b10 con 0x7fc5cc103d10 2023-10-10T17:34:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.812+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc5bc04c6e0 con 0x7fc5cc103d10 2023-10-10T17:34:25.396 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.812+0000 7fc5d10e5700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fc5b8038710 0x7fc5b803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:25.397 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.812+0000 7fc5d10e5700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fc5b8038710 0x7fc5b803abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8006fd0 tx=0x7fc5c8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:25.397 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.816+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc5cc107b10 con 0x7fc5cc103d10 2023-10-10T17:34:25.397 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:24.924+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mon.0 v2:172.21.15.78:3300/0 7 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5bc04c0f0 con 0x7fc5cc103d10 2023-10-10T17:34:25.397 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.030+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc5cc107b10 con 0x7fc5b8038710 2023-10-10T17:34:25.397 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.035+0000 7fc5c2ffd700 1 -- 172.21.15.78:0/1832027992 <== mgr.14120 v2:172.21.15.78:6800/1662641434 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc5cc107b10 con 0x7fc5b8038710 2023-10-10T17:34:25.397 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fc5b8038710 msgr2=0x7fc5b803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:25.398 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fc5b8038710 0x7fc5b803abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8006fd0 tx=0x7fc5c8006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:25.398 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 msgr2=0x7fc5cc1998e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:25.398 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc1998e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc5bc006af0 tx=0x7fc5bc011ad0 comp rx=0 tx=0).stop 2023-10-10T17:34:25.398 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 shutdown_connections 2023-10-10T17:34:25.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fc5b8038710 0x7fc5b803abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:25.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 --2- 172.21.15.78:0/1832027992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5cc103d10 0x7fc5cc1998e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:25.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 >> 172.21.15.78:0/1832027992 conn(0x7fc5cc0ff650 msgr2=0x7fc5cc100410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:25.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 shutdown_connections 2023-10-10T17:34:25.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.037+0000 7fc5d3b4a700 1 -- 172.21.15.78:0/1832027992 wait complete. 2023-10-10T17:34:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:25 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:25 smithi078 ceph-mon[99494]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:25 smithi078 ceph-mon[99494]: Saving service mgr spec with placement count:2 2023-10-10T17:34:26.286 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:25 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:26.287 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:25 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:26.533 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.923+0000 7fe28fd6d700 1 Processor -- start 2023-10-10T17:34:26.533 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28fd6d700 1 -- start start 2023-10-10T17:34:26.534 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28fd6d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe2881088d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:26.534 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28fd6d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe288079e00 con 0x7fe2881064f0 2023-10-10T17:34:26.534 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28db09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe2881088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:26.534 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28db09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe2881088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41028/0 (socket says 172.21.15.78:41028) 2023-10-10T17:34:26.535 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28db09700 1 -- 172.21.15.78:0/3463348207 learned_addr learned my addr 172.21.15.78:0/3463348207 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:26.535 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28db09700 1 -- 172.21.15.78:0/3463348207 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe288108e10 con 0x7fe2881064f0 2023-10-10T17:34:26.535 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.924+0000 7fe28db09700 1 --2- 172.21.15.78:0/3463348207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe2881088d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2840098a0 tx=0x7fe284009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c24b810cccefaf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:26.535 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28cb07700 1 -- 172.21.15.78:0/3463348207 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe284004160 con 0x7fe2881064f0 2023-10-10T17:34:26.535 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28cb07700 1 -- 172.21.15.78:0/3463348207 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe2840042c0 con 0x7fe2881064f0 2023-10-10T17:34:26.536 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28cb07700 1 -- 172.21.15.78:0/3463348207 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2840119b0 con 0x7fe2881064f0 2023-10-10T17:34:26.536 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28fd6d700 1 -- 172.21.15.78:0/3463348207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 msgr2=0x7fe2881088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:26.536 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28fd6d700 1 --2- 172.21.15.78:0/3463348207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe2881088d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe2840098a0 tx=0x7fe284009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:26.536 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28fd6d700 1 -- 172.21.15.78:0/3463348207 shutdown_connections 2023-10-10T17:34:26.536 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28fd6d700 1 --2- 172.21.15.78:0/3463348207 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe2881088d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:26.537 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28fd6d700 1 -- 172.21.15.78:0/3463348207 >> 172.21.15.78:0/3463348207 conn(0x7fe2880ffed0 msgr2=0x7fe288102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:26.537 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28fd6d700 1 -- 172.21.15.78:0/3463348207 shutdown_connections 2023-10-10T17:34:26.537 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.925+0000 7fe28fd6d700 1 -- 172.21.15.78:0/3463348207 wait complete. 2023-10-10T17:34:26.537 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.926+0000 7fe28fd6d700 1 Processor -- start 2023-10-10T17:34:26.538 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.926+0000 7fe28fd6d700 1 -- start start 2023-10-10T17:34:26.539 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.926+0000 7fe28fd6d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe288199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:26.539 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.926+0000 7fe28fd6d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe288199cc0 con 0x7fe2881064f0 2023-10-10T17:34:26.539 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe28db09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe288199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:26.539 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe28db09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe288199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41030/0 (socket says 172.21.15.78:41030) 2023-10-10T17:34:26.540 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe28db09700 1 -- 172.21.15.78:0/623186223 learned_addr learned my addr 172.21.15.78:0/623186223 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:26.540 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe28db09700 1 -- 172.21.15.78:0/623186223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe284009550 con 0x7fe2881064f0 2023-10-10T17:34:26.540 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe28db09700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe288199780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe284011980 tx=0x7fe284011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:26.540 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe27effd700 1 -- 172.21.15.78:0/623186223 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe284003a60 con 0x7fe2881064f0 2023-10-10T17:34:26.540 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe27effd700 1 -- 172.21.15.78:0/623186223 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe284003bc0 con 0x7fe2881064f0 2023-10-10T17:34:26.541 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe288109d80 con 0x7fe2881064f0 2023-10-10T17:34:26.541 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe27effd700 1 -- 172.21.15.78:0/623186223 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe28401a670 con 0x7fe2881064f0 2023-10-10T17:34:26.541 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.927+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe288109bf0 con 0x7fe2881064f0 2023-10-10T17:34:26.541 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.928+0000 7fe27effd700 1 -- 172.21.15.78:0/623186223 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fe28401a8e0 con 0x7fe2881064f0 2023-10-10T17:34:26.541 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.928+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe288109f10 con 0x7fe2881064f0 2023-10-10T17:34:26.542 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.928+0000 7fe27effd700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fe274038760 0x7fe27403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:26.542 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.928+0000 7fe27effd700 1 -- 172.21.15.78:0/623186223 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe28404c760 con 0x7fe2881064f0 2023-10-10T17:34:26.542 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.928+0000 7fe28d308700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fe274038760 0x7fe27403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:26.542 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.929+0000 7fe28d308700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fe274038760 0x7fe27403ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe278006fd0 tx=0x7fe278006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:26.542 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:25.933+0000 7fe27effd700 1 -- 172.21.15.78:0/623186223 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe28401ab90 con 0x7fe2881064f0 2023-10-10T17:34:26.543 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.122+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fe28819d390 con 0x7fe2881064f0 2023-10-10T17:34:26.543 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.126+0000 7fe27effd700 1 -- 172.21.15.78:0/623186223 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fe284018350 con 0x7fe2881064f0 2023-10-10T17:34:26.543 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fe274038760 msgr2=0x7fe27403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:26.543 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fe274038760 0x7fe27403ac10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe278006fd0 tx=0x7fe278006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:26.543 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 msgr2=0x7fe288199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:26.544 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe288199780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe284011980 tx=0x7fe284011a60 comp rx=0 tx=0).stop 2023-10-10T17:34:26.544 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 shutdown_connections 2023-10-10T17:34:26.544 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7fe274038760 0x7fe27403ac10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:26.544 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 --2- 172.21.15.78:0/623186223 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe2881064f0 0x7fe288199780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:26.544 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 >> 172.21.15.78:0/623186223 conn(0x7fe2880ffed0 msgr2=0x7fe288100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:26.544 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 shutdown_connections 2023-10-10T17:34:26.545 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:26.128+0000 7fe28fd6d700 1 -- 172.21.15.78:0/623186223 wait complete. 2023-10-10T17:34:27.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:27 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/623186223' entity='client.admin' 2023-10-10T17:34:27.779 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.220+0000 7f8b1286a700 1 Processor -- start 2023-10-10T17:34:27.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.220+0000 7f8b1286a700 1 -- start start 2023-10-10T17:34:27.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b1286a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c07a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:27.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b1286a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c07a810 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b0bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c07a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:27.780 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b0bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c07a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41034/0 (socket says 172.21.15.78:41034) 2023-10-10T17:34:27.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b0bfff700 1 -- 172.21.15.78:0/2474340041 learned_addr learned my addr 172.21.15.78:0/2474340041 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:27.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b0bfff700 1 -- 172.21.15.78:0/2474340041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c07a950 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b0bfff700 1 --2- 172.21.15.78:0/2474340041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c07a2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0098a0 tx=0x7f8afc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd81bac8ca183c64 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:27.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.221+0000 7f8b0affd700 1 -- 172.21.15.78:0/2474340041 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8afc004160 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.781 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.222+0000 7f8b0affd700 1 -- 172.21.15.78:0/2474340041 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8afc0042c0 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.222+0000 7f8b0affd700 1 -- 172.21.15.78:0/2474340041 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8afc0119b0 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.222+0000 7f8b1286a700 1 -- 172.21.15.78:0/2474340041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 msgr2=0x7f8b0c07a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:27.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.222+0000 7f8b1286a700 1 --2- 172.21.15.78:0/2474340041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c07a2d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0098a0 tx=0x7f8afc009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:27.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.223+0000 7f8b1286a700 1 -- 172.21.15.78:0/2474340041 shutdown_connections 2023-10-10T17:34:27.782 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.223+0000 7f8b1286a700 1 --2- 172.21.15.78:0/2474340041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c07a2d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:27.783 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.223+0000 7f8b1286a700 1 -- 172.21.15.78:0/2474340041 >> 172.21.15.78:0/2474340041 conn(0x7f8b0c101170 msgr2=0x7f8b0c103580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:27.783 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.223+0000 7f8b1286a700 1 -- 172.21.15.78:0/2474340041 shutdown_connections 2023-10-10T17:34:27.783 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.223+0000 7f8b1286a700 1 -- 172.21.15.78:0/2474340041 wait complete. 2023-10-10T17:34:27.784 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.223+0000 7f8b1286a700 1 Processor -- start 2023-10-10T17:34:27.784 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.223+0000 7f8b1286a700 1 -- start start 2023-10-10T17:34:27.784 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b1286a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:27.784 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b1286a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c19bfc0 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.784 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b0bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:27.785 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b0bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c19ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41036/0 (socket says 172.21.15.78:41036) 2023-10-10T17:34:27.785 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b0bfff700 1 -- 172.21.15.78:0/2885839004 learned_addr learned my addr 172.21.15.78:0/2885839004 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:27.785 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b0bfff700 1 -- 172.21.15.78:0/2885839004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc009550 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.785 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b0bfff700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c19ba80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8afc003e60 tx=0x7f8afc011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:27.785 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b097fa700 1 -- 172.21.15.78:0/2885839004 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8afc0039e0 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.785 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b097fa700 1 -- 172.21.15.78:0/2885839004 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8afc003b40 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.786 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b097fa700 1 -- 172.21.15.78:0/2885839004 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8afc0196c0 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.786 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.224+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0c10bfc0 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.786 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.225+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0c10be30 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.786 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.225+0000 7f8b097fa700 1 -- 172.21.15.78:0/2885839004 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f8afc019980 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.786 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.225+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c10c150 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.787 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.226+0000 7f8b097fa700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f8af40386c0 0x7f8af403ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:27.787 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.226+0000 7f8b097fa700 1 -- 172.21.15.78:0/2885839004 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8afc04c940 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.787 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.226+0000 7f8b0b7fe700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f8af40386c0 0x7f8af403ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:27.787 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.226+0000 7f8b0b7fe700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f8af40386c0 0x7f8af403ab70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8b00006fd0 tx=0x7f8b00006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:27.788 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.230+0000 7f8b097fa700 1 -- 172.21.15.78:0/2885839004 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8afc018d20 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.788 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.423+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f8b0c062120 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.788 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.427+0000 7f8b097fa700 1 -- 172.21.15.78:0/2885839004 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f8afc018b40 con 0x7f8b0c07bdd0 2023-10-10T17:34:27.788 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f8af40386c0 msgr2=0x7f8af403ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:27.788 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f8af40386c0 0x7f8af403ab70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8b00006fd0 tx=0x7f8b00006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:27.788 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 msgr2=0x7f8b0c19ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:27.789 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c19ba80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8afc003e60 tx=0x7f8afc011b70 comp rx=0 tx=0).stop 2023-10-10T17:34:27.789 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 shutdown_connections 2023-10-10T17:34:27.789 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f8af40386c0 0x7f8af403ab70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:27.789 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 --2- 172.21.15.78:0/2885839004 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b0c07bdd0 0x7f8b0c19ba80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:27.789 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 >> 172.21.15.78:0/2885839004 conn(0x7f8b0c101170 msgr2=0x7f8b0c102830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:27.790 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 shutdown_connections 2023-10-10T17:34:27.790 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:27.429+0000 7f8b1286a700 1 -- 172.21.15.78:0/2885839004 wait complete. 2023-10-10T17:34:27.790 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling the dashboard module... 2023-10-10T17:34:28.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:28 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:28.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2885839004' entity='client.admin' 2023-10-10T17:34:29.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2883363497' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-10-10T17:34:29.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-mon[99494]: from='mgr.14120 172.21.15.78:0/2233568301' entity='mgr.x' 2023-10-10T17:34:29.739 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: ignoring --setuser ceph since I am not root 2023-10-10T17:34:29.739 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: ignoring --setgroup ceph since I am not root 2023-10-10T17:34:29.739 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:29.657+0000 7fe42cf60200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:34:29.926 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.267+0000 7f42467bf700 1 Processor -- start 2023-10-10T17:34:29.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.267+0000 7f42467bf700 1 -- start start 2023-10-10T17:34:29.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.268+0000 7f42467bf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f4240106310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:29.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.268+0000 7f42467bf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42401068e0 con 0x7f4240105f00 2023-10-10T17:34:29.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.268+0000 7f423ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f4240106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:29.927 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.268+0000 7f423ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f4240106310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41040/0 (socket says 172.21.15.78:41040) 2023-10-10T17:34:29.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.268+0000 7f423ffff700 1 -- 172.21.15.78:0/1307370419 learned_addr learned my addr 172.21.15.78:0/1307370419 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:29.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.268+0000 7f423ffff700 1 -- 172.21.15.78:0/1307370419 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4240106a20 con 0x7f4240105f00 2023-10-10T17:34:29.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.268+0000 7f423ffff700 1 --2- 172.21.15.78:0/1307370419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f4240106310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f423000bb40 tx=0x7f423000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d11b019543a15cb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:29.928 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.269+0000 7f423effd700 1 -- 172.21.15.78:0/1307370419 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f423000dbe0 con 0x7f4240105f00 2023-10-10T17:34:29.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.269+0000 7f423effd700 1 -- 172.21.15.78:0/1307370419 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42300044f0 con 0x7f4240105f00 2023-10-10T17:34:29.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.269+0000 7f423effd700 1 -- 172.21.15.78:0/1307370419 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4230003e60 con 0x7f4240105f00 2023-10-10T17:34:29.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.269+0000 7f42467bf700 1 -- 172.21.15.78:0/1307370419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 msgr2=0x7f4240106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:29.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.269+0000 7f42467bf700 1 --2- 172.21.15.78:0/1307370419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f4240106310 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f423000bb40 tx=0x7f423000be50 comp rx=0 tx=0).stop 2023-10-10T17:34:29.929 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.269+0000 7f42467bf700 1 -- 172.21.15.78:0/1307370419 shutdown_connections 2023-10-10T17:34:29.930 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.270+0000 7f42467bf700 1 --2- 172.21.15.78:0/1307370419 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f4240106310 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:29.930 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.270+0000 7f42467bf700 1 -- 172.21.15.78:0/1307370419 >> 172.21.15.78:0/1307370419 conn(0x7f42400ff650 msgr2=0x7f4240101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:29.930 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.270+0000 7f42467bf700 1 -- 172.21.15.78:0/1307370419 shutdown_connections 2023-10-10T17:34:29.930 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.270+0000 7f42467bf700 1 -- 172.21.15.78:0/1307370419 wait complete. 2023-10-10T17:34:29.931 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.270+0000 7f42467bf700 1 Processor -- start 2023-10-10T17:34:29.931 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.270+0000 7f42467bf700 1 -- start start 2023-10-10T17:34:29.931 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f42467bf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f42401975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:29.932 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f42467bf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4240197b00 con 0x7f4240105f00 2023-10-10T17:34:29.932 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f423ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f42401975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:29.932 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f423ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f42401975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41042/0 (socket says 172.21.15.78:41042) 2023-10-10T17:34:29.932 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f423ffff700 1 -- 172.21.15.78:0/2883363497 learned_addr learned my addr 172.21.15.78:0/2883363497 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:29.932 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f423ffff700 1 -- 172.21.15.78:0/2883363497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423000b7f0 con 0x7f4240105f00 2023-10-10T17:34:29.933 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f423ffff700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f42401975c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42300046e0 tx=0x7f4230004020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:29.933 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.271+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f423001e070 con 0x7f4240105f00 2023-10-10T17:34:29.933 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.272+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4240103aa0 con 0x7f4240105f00 2023-10-10T17:34:29.933 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.272+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4230010c20 con 0x7f4240105f00 2023-10-10T17:34:29.933 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.272+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4230022b60 con 0x7f4240105f00 2023-10-10T17:34:29.934 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.272+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42401058d0 con 0x7f4240105f00 2023-10-10T17:34:29.934 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.273+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f4230022410 con 0x7f4240105f00 2023-10-10T17:34:29.934 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.273+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4240103c30 con 0x7f4240105f00 2023-10-10T17:34:29.934 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.273+0000 7f423d7fa700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f42280386c0 0x7f422803ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:29.935 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.273+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f423004ef00 con 0x7f4240105f00 2023-10-10T17:34:29.935 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.273+0000 7f423f7fe700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f42280386c0 0x7f422803ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:29.935 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.274+0000 7f423f7fe700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f42280386c0 0x7f422803ab70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4234006fd0 tx=0x7f4234006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:29.935 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.278+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4230021590 con 0x7f4240105f00 2023-10-10T17:34:29.935 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:28.543+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f424002cfe0 con 0x7f4240105f00 2023-10-10T17:34:29.936 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.468+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f423002c440 con 0x7f4240105f00 2023-10-10T17:34:29.936 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.470+0000 7f423d7fa700 1 -- 172.21.15.78:0/2883363497 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f423000ad70 con 0x7f4240105f00 2023-10-10T17:34:29.936 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f42280386c0 msgr2=0x7f422803ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:29.936 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f42280386c0 0x7f422803ab70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4234006fd0 tx=0x7f4234006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:29.936 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 msgr2=0x7f42401975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:29.937 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f42401975c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42300046e0 tx=0x7f4230004020 comp rx=0 tx=0).stop 2023-10-10T17:34:29.937 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 shutdown_connections 2023-10-10T17:34:29.937 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f42280386c0 0x7f422803ab70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:29.937 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 --2- 172.21.15.78:0/2883363497 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4240105f00 0x7f42401975c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:29.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 >> 172.21.15.78:0/2883363497 conn(0x7f42400ff650 msgr2=0x7f424018e390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:29.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 shutdown_connections 2023-10-10T17:34:29.938 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:29.473+0000 7f42467bf700 1 -- 172.21.15.78:0/2883363497 wait complete. 2023-10-10T17:34:30.119 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:29.737+0000 7fe42cf60200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:34:30.120 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:29.828+0000 7fe42cf60200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:34:30.120 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:29.931+0000 7fe42cf60200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:34:30.378 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:30.118+0000 7fe42cf60200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:34:30.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:30 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2883363497' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-10-10T17:34:30.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:30 smithi078 ceph-mon[99494]: mgrmap e9: x(active, since 15s) 2023-10-10T17:34:30.767 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:30.438+0000 7fe42cf60200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:34:30.767 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:30.532+0000 7fe42cf60200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:34:30.768 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:30.669+0000 7fe42cf60200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:34:30.768 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:30.766+0000 7fe42cf60200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:34:31.186 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:34:31.186 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-10-10T17:34:31.186 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-10T17:34:31.187 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-10T17:34:31.187 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-10T17:34:31.187 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:34:31.187 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4de504700 1 Processor -- start 2023-10-10T17:34:31.187 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4de504700 1 -- start start 2023-10-10T17:34:31.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4de504700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d8106310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:31.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4de504700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d81068e0 con 0x7ff4d8105f00 2023-10-10T17:34:31.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4d7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d8106310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:31.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4d7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d8106310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41056/0 (socket says 172.21.15.78:41056) 2023-10-10T17:34:31.188 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4d7fff700 1 -- 172.21.15.78:0/3676813870 learned_addr learned my addr 172.21.15.78:0/3676813870 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:31.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.555+0000 7ff4d7fff700 1 -- 172.21.15.78:0/3676813870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8104b10 con 0x7ff4d8105f00 2023-10-10T17:34:31.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4d7fff700 1 --2- 172.21.15.78:0/3676813870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d8106310 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80098a0 tx=0x7ff4c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab6e515b01a928b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:31.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4d6ffd700 1 -- 172.21.15.78:0/3676813870 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4c8004160 con 0x7ff4d8105f00 2023-10-10T17:34:31.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4d6ffd700 1 -- 172.21.15.78:0/3676813870 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4c80042c0 con 0x7ff4d8105f00 2023-10-10T17:34:31.189 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4d6ffd700 1 -- 172.21.15.78:0/3676813870 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4c80119b0 con 0x7ff4d8105f00 2023-10-10T17:34:31.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4de504700 1 -- 172.21.15.78:0/3676813870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 msgr2=0x7ff4d8106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:31.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4de504700 1 --2- 172.21.15.78:0/3676813870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d8106310 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff4c80098a0 tx=0x7ff4c8009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:31.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4de504700 1 -- 172.21.15.78:0/3676813870 shutdown_connections 2023-10-10T17:34:31.190 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4de504700 1 --2- 172.21.15.78:0/3676813870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d8106310 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:31.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.556+0000 7ff4de504700 1 -- 172.21.15.78:0/3676813870 >> 172.21.15.78:0/3676813870 conn(0x7ff4d80ff650 msgr2=0x7ff4d8101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:31.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.557+0000 7ff4de504700 1 -- 172.21.15.78:0/3676813870 shutdown_connections 2023-10-10T17:34:31.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.557+0000 7ff4de504700 1 -- 172.21.15.78:0/3676813870 wait complete. 2023-10-10T17:34:31.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.557+0000 7ff4de504700 1 Processor -- start 2023-10-10T17:34:31.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.557+0000 7ff4de504700 1 -- start start 2023-10-10T17:34:31.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4de504700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d81045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:31.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4de504700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d819c4c0 con 0x7ff4d8105f00 2023-10-10T17:34:31.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d81045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:31.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d81045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41058/0 (socket says 172.21.15.78:41058) 2023-10-10T17:34:31.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d7fff700 1 -- 172.21.15.78:0/3919701292 learned_addr learned my addr 172.21.15.78:0/3919701292 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:31.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d7fff700 1 -- 172.21.15.78:0/3919701292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c8009550 con 0x7ff4d8105f00 2023-10-10T17:34:31.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d7fff700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d81045c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8003e60 tx=0x7ff4c8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:31.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d57fa700 1 -- 172.21.15.78:0/3919701292 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4c8003970 con 0x7ff4d8105f00 2023-10-10T17:34:31.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d57fa700 1 -- 172.21.15.78:0/3919701292 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4c8003ad0 con 0x7ff4d8105f00 2023-10-10T17:34:31.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d8108ee0 con 0x7ff4d8105f00 2023-10-10T17:34:31.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4d57fa700 1 -- 172.21.15.78:0/3919701292 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4c80195e0 con 0x7ff4d8105f00 2023-10-10T17:34:31.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.558+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d8108d50 con 0x7ff4d8105f00 2023-10-10T17:34:31.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.560+0000 7ff4d57fa700 1 -- 172.21.15.78:0/3919701292 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7ff4c8019850 con 0x7ff4d8105f00 2023-10-10T17:34:31.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.560+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d8109070 con 0x7ff4d8105f00 2023-10-10T17:34:31.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.560+0000 7ff4d57fa700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 0x7ff4c003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.560+0000 7ff4d57fa700 1 -- 172.21.15.78:0/3919701292 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff4c804c870 con 0x7ff4d8105f00 2023-10-10T17:34:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.560+0000 7ff4d77fe700 1 -- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 msgr2=0x7ff4c003ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1662641434 2023-10-10T17:34:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.560+0000 7ff4d77fe700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 0x7ff4c003ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:34:31.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.565+0000 7ff4d57fa700 1 -- 172.21.15.78:0/3919701292 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff4c802b340 con 0x7ff4d8105f00 2023-10-10T17:34:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.761+0000 7ff4d77fe700 1 -- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 msgr2=0x7ff4c003ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1662641434 2023-10-10T17:34:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.761+0000 7ff4d77fe700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 0x7ff4c003ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-10T17:34:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.816+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff4d819d3f0 con 0x7ff4d8105f00 2023-10-10T17:34:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.816+0000 7ff4d57fa700 1 -- 172.21.15.78:0/3919701292 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff4c8018b40 con 0x7ff4d8105f00 2023-10-10T17:34:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 msgr2=0x7ff4c003ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:34:31.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 0x7ff4c003ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:31.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 msgr2=0x7ff4d81045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:31.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d81045c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8003e60 tx=0x7ff4c8011aa0 comp rx=0 tx=0).stop 2023-10-10T17:34:31.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 shutdown_connections 2023-10-10T17:34:31.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7ff4c0038760 0x7ff4c003ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:31.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 --2- 172.21.15.78:0/3919701292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4d8105f00 0x7ff4d81045c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:31.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 >> 172.21.15.78:0/3919701292 conn(0x7ff4d80ff650 msgr2=0x7ff4d8100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:31.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.818+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 shutdown_connections 2023-10-10T17:34:31.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:30.819+0000 7ff4de504700 1 -- 172.21.15.78:0/3919701292 wait complete. 2023-10-10T17:34:31.198 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2023-10-10T17:34:31.199 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 9... 2023-10-10T17:34:31.468 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:31.209+0000 7fe42cf60200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:34:31.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:31 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3919701292' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-10T17:34:32.127 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:31.801+0000 7fe42cf60200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:34:32.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:31.973+0000 7fe42cf60200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:34:32.994 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:32.732+0000 7fe42cf60200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:34:32.994 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:32.823+0000 7fe42cf60200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:34:33.270 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:32.993+0000 7fe42cf60200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:34:33.627 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:33.269+0000 7fe42cf60200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:34:33.628 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:33.494+0000 7fe42cf60200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:34:33.902 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:33.641+0000 7fe42cf60200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:34:34.377 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:33.901+0000 7fe42cf60200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:34:34.877 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:34.597+0000 7fe42cf60200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:34:34.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:34.730+0000 7fe42cf60200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:34:35.252 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:34.922+0000 7fe42cf60200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:34:35.253 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:35.035+0000 7fe42cf60200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:34:36.628 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:35.251+0000 7fe42cf60200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:34:36.628 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:34:36.130+0000 7fe42cf60200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:34:36.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: Active manager daemon x restarted 2023-10-10T17:34:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: Activating manager daemon x 2023-10-10T17:34:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: osdmap e3: 0 total, 0 up, 0 in 2023-10-10T17:34:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: mgrmap e10: x(active, starting, since 0.00320328s) 2023-10-10T17:34:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:34:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-10T17:34:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-10T17:34:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-10T17:34:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-10T17:34:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: Manager daemon x is now available 2023-10-10T17:34:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:36.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:37.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-10T17:34:37.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-10T17:34:37.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:37 smithi078 ceph-mon[99494]: mgrmap e11: x(active, since 1.00589s) 2023-10-10T17:34:38.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:38 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:36] ENGINE Bus STARTING 2023-10-10T17:34:38.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:38 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:36] ENGINE Serving on http://172.21.15.78:8765 2023-10-10T17:34:38.294 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:38 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:36] ENGINE Serving on https://172.21.15.78:7150 2023-10-10T17:34:38.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:38 smithi078 ceph-mon[99494]: [10/Oct/2023:17:34:36] ENGINE Bus STARTED 2023-10-10T17:34:38.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:38 smithi078 ceph-mon[99494]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-10T17:34:38.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:38 smithi078 ceph-mon[99494]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-10T17:34:38.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-10-10T17:34:38.646 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-10-10T17:34:38.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-10T17:34:38.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-10-10T17:34:38.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.826+0000 7f4876196700 1 Processor -- start 2023-10-10T17:34:38.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.826+0000 7f4876196700 1 -- start start 2023-10-10T17:34:38.647 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.826+0000 7f4876196700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:38.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.826+0000 7f4876196700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870079e20 con 0x7f48701064d0 2023-10-10T17:34:38.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.826+0000 7f486f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:38.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.826+0000 7f486f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41066/0 (socket says 172.21.15.78:41066) 2023-10-10T17:34:38.648 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.826+0000 7f486f7fe700 1 -- 172.21.15.78:0/1572179302 learned_addr learned my addr 172.21.15.78:0/1572179302 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:38.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.827+0000 7f486f7fe700 1 -- 172.21.15.78:0/1572179302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870108df0 con 0x7f48701064d0 2023-10-10T17:34:38.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.827+0000 7f486f7fe700 1 --2- 172.21.15.78:0/1572179302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701088b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f48600098a0 tx=0x7f4860009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11e5aa15b29ec1e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:38.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.827+0000 7f486e7fc700 1 -- 172.21.15.78:0/1572179302 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4860004160 con 0x7f48701064d0 2023-10-10T17:34:38.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.827+0000 7f486e7fc700 1 -- 172.21.15.78:0/1572179302 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48600042c0 con 0x7f48701064d0 2023-10-10T17:34:38.649 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.827+0000 7f486e7fc700 1 -- 172.21.15.78:0/1572179302 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48600119b0 con 0x7f48701064d0 2023-10-10T17:34:38.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 -- 172.21.15.78:0/1572179302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 msgr2=0x7f48701088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:38.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 --2- 172.21.15.78:0/1572179302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701088b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f48600098a0 tx=0x7f4860009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:38.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 -- 172.21.15.78:0/1572179302 shutdown_connections 2023-10-10T17:34:38.650 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 --2- 172.21.15.78:0/1572179302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701088b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:38.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 -- 172.21.15.78:0/1572179302 >> 172.21.15.78:0/1572179302 conn(0x7f48700ffed0 msgr2=0x7f4870102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:38.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 -- 172.21.15.78:0/1572179302 shutdown_connections 2023-10-10T17:34:38.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 -- 172.21.15.78:0/1572179302 wait complete. 2023-10-10T17:34:38.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.828+0000 7f4876196700 1 Processor -- start 2023-10-10T17:34:38.651 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.829+0000 7f4876196700 1 -- start start 2023-10-10T17:34:38.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.829+0000 7f4876196700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:38.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.829+0000 7f4876196700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4870199ce0 con 0x7f48701064d0 2023-10-10T17:34:38.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.829+0000 7f486f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:38.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.829+0000 7f486f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41068/0 (socket says 172.21.15.78:41068) 2023-10-10T17:34:38.652 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.829+0000 7f486f7fe700 1 -- 172.21.15.78:0/3703150673 learned_addr learned my addr 172.21.15.78:0/3703150673 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:38.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.829+0000 7f486f7fe700 1 -- 172.21.15.78:0/3703150673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860009550 con 0x7f48701064d0 2023-10-10T17:34:38.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.830+0000 7f486f7fe700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701997a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4860011980 tx=0x7f4860011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:38.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.830+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4860003a60 con 0x7f48701064d0 2023-10-10T17:34:38.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.830+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4870109d60 con 0x7f48701064d0 2023-10-10T17:34:38.653 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.830+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4860003bc0 con 0x7f48701064d0 2023-10-10T17:34:38.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.830+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f486001a670 con 0x7f48701064d0 2023-10-10T17:34:38.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.830+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4870109bd0 con 0x7f48701064d0 2023-10-10T17:34:38.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.831+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f4860024a00 con 0x7f48701064d0 2023-10-10T17:34:38.654 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.831+0000 7f486cff9700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:38.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.831+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 --> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485803b320 con 0x7f4858038760 2023-10-10T17:34:38.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.831+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f486004c8b0 con 0x7f48701064d0 2023-10-10T17:34:38.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.831+0000 7f486effd700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 msgr2=0x7f485803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1662641434 2023-10-10T17:34:38.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:31.831+0000 7f486effd700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:34:38.655 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:32.032+0000 7f486effd700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 msgr2=0x7f485803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1662641434 2023-10-10T17:34:38.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:32.032+0000 7f486effd700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-10T17:34:38.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:32.433+0000 7f486effd700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 msgr2=0x7f485803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1662641434 2023-10-10T17:34:38.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:32.433+0000 7f486effd700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-10T17:34:38.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:33.234+0000 7f486effd700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 msgr2=0x7f485803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1662641434 2023-10-10T17:34:38.656 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:33.234+0000 7f486effd700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-10T17:34:38.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:34.835+0000 7f486effd700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 msgr2=0x7f485803ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/1662641434 2023-10-10T17:34:38.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:34.835+0000 7f486effd700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-10T17:34:38.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:36.145+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7f4860024620 con 0x7f48701064d0 2023-10-10T17:34:38.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:36.145+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 msgr2=0x7f485803ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:34:38.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:36.145+0000 7f486cff9700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:38.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.146+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7f486004df40 con 0x7f48701064d0 2023-10-10T17:34:38.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.147+0000 7f486cff9700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f485803bff0 0x7f485803e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:38.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.147+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f485803b320 con 0x7f485803bff0 2023-10-10T17:34:38.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.147+0000 7f486effd700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f485803bff0 0x7f485803e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:38.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.147+0000 7f486effd700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f485803bff0 0x7f485803e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4864003a10 tx=0x7f48640092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:38.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.148+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f485803b320 con 0x7f485803bff0 2023-10-10T17:34:38.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.155+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f487019aae0 con 0x7f485803bff0 2023-10-10T17:34:38.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f486cff9700 1 -- 172.21.15.78:0/3703150673 <== mgr.14152 v2:172.21.15.78:6800/321866266 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f487019aae0 con 0x7f485803bff0 2023-10-10T17:34:38.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f485803bff0 msgr2=0x7f485803e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:38.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f485803bff0 0x7f485803e3d0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4864003a10 tx=0x7f48640092b0 comp rx=0 tx=0).stop 2023-10-10T17:34:38.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 msgr2=0x7f48701997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:38.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701997a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4860011980 tx=0x7f4860011a60 comp rx=0 tx=0).stop 2023-10-10T17:34:38.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 shutdown_connections 2023-10-10T17:34:38.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f485803bff0 0x7f485803e3d0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:38.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48701064d0 0x7f48701997a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:38.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 --2- 172.21.15.78:0/3703150673 >> [v2:172.21.15.78:6800/1662641434,v1:172.21.15.78:6801/1662641434] conn(0x7f4858038760 0x7f485803ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:38.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 >> 172.21.15.78:0/3703150673 conn(0x7f48700ffed0 msgr2=0x7f4870101570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:38.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 shutdown_connections 2023-10-10T17:34:38.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:37.156+0000 7f4876196700 1 -- 172.21.15.78:0/3703150673 wait complete. 2023-10-10T17:34:38.663 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 9 is available 2023-10-10T17:34:38.663 INFO:teuthology.orchestra.run.smithi078.stdout:Generating a dashboard self-signed certificate... 2023-10-10T17:34:39.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:39 smithi078 ceph-mon[99494]: mgrmap e12: x(active, since 2s) 2023-10-10T17:34:39.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:39.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-10-10T17:34:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.165+0000 7f6e641e3700 1 Processor -- start 2023-10-10T17:34:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.165+0000 7f6e641e3700 1 -- start start 2023-10-10T17:34:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.165+0000 7f6e641e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.165+0000 7f6e641e3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c079e20 con 0x7f6e5c1064d0 2023-10-10T17:34:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.165+0000 7f6e61f7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.165+0000 7f6e61f7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41122/0 (socket says 172.21.15.78:41122) 2023-10-10T17:34:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.165+0000 7f6e61f7f700 1 -- 172.21.15.78:0/1787014336 learned_addr learned my addr 172.21.15.78:0/1787014336 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.166+0000 7f6e61f7f700 1 -- 172.21.15.78:0/1787014336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c108df0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.166+0000 7f6e61f7f700 1 --2- 172.21.15.78:0/1787014336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1088b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0098a0 tx=0x7f6e4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd241b3f6664d22e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:39.896 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.166+0000 7f6e60f7d700 1 -- 172.21.15.78:0/1787014336 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e4c004160 con 0x7f6e5c1064d0 2023-10-10T17:34:39.896 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.166+0000 7f6e60f7d700 1 -- 172.21.15.78:0/1787014336 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4c0042c0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.896 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.166+0000 7f6e60f7d700 1 -- 172.21.15.78:0/1787014336 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e4c0119b0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.896 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.166+0000 7f6e641e3700 1 -- 172.21.15.78:0/1787014336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 msgr2=0x7f6e5c1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:39.896 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.166+0000 7f6e641e3700 1 --2- 172.21.15.78:0/1787014336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1088b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c0098a0 tx=0x7f6e4c009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:39.897 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 -- 172.21.15.78:0/1787014336 shutdown_connections 2023-10-10T17:34:39.897 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 --2- 172.21.15.78:0/1787014336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1088b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:39.897 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 -- 172.21.15.78:0/1787014336 >> 172.21.15.78:0/1787014336 conn(0x7f6e5c0ffed0 msgr2=0x7f6e5c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:39.897 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 -- 172.21.15.78:0/1787014336 shutdown_connections 2023-10-10T17:34:39.898 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 -- 172.21.15.78:0/1787014336 wait complete. 2023-10-10T17:34:39.898 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 Processor -- start 2023-10-10T17:34:39.898 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 -- start start 2023-10-10T17:34:39.898 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:39.898 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e641e3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c199ce0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.899 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e61f7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:39.899 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e61f7f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41124/0 (socket says 172.21.15.78:41124) 2023-10-10T17:34:39.899 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.167+0000 7f6e61f7f700 1 -- 172.21.15.78:0/1784958585 learned_addr learned my addr 172.21.15.78:0/1784958585 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:39.899 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.168+0000 7f6e61f7f700 1 -- 172.21.15.78:0/1784958585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4c009550 con 0x7f6e5c1064d0 2023-10-10T17:34:39.899 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.168+0000 7f6e61f7f700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1997a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c011980 tx=0x7f6e4c011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:39.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.168+0000 7f6e52ffd700 1 -- 172.21.15.78:0/1784958585 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e4c003a60 con 0x7f6e5c1064d0 2023-10-10T17:34:39.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.168+0000 7f6e52ffd700 1 -- 172.21.15.78:0/1784958585 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e4c003bc0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.168+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c109d60 con 0x7f6e5c1064d0 2023-10-10T17:34:39.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.168+0000 7f6e52ffd700 1 -- 172.21.15.78:0/1784958585 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e4c01a670 con 0x7f6e5c1064d0 2023-10-10T17:34:39.900 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.168+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c109bd0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.169+0000 7f6e52ffd700 1 -- 172.21.15.78:0/1784958585 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f6e4c01a8e0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.169+0000 7f6e52ffd700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e480387b0 0x7f6e4803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:39.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.169+0000 7f6e52ffd700 1 -- 172.21.15.78:0/1784958585 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6e4c04c8c0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.169+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c109ef0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.901 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.169+0000 7f6e6177e700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e480387b0 0x7f6e4803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:39.902 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.169+0000 7f6e6177e700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e480387b0 0x7f6e4803ac60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006fd0 tx=0x7f6e58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:39.902 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.174+0000 7f6e52ffd700 1 -- 172.21.15.78:0/1784958585 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6e4c01abe0 con 0x7f6e5c1064d0 2023-10-10T17:34:39.902 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.375+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e5c102190 con 0x7f6e480387b0 2023-10-10T17:34:39.902 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.451+0000 7f6e52ffd700 1 -- 172.21.15.78:0/1784958585 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f6e5c102190 con 0x7f6e480387b0 2023-10-10T17:34:39.902 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e480387b0 msgr2=0x7f6e4803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:39.903 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e480387b0 0x7f6e4803ac60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6e58006fd0 tx=0x7f6e58006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:39.903 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 msgr2=0x7f6e5c1997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:39.903 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1997a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c011980 tx=0x7f6e4c011a60 comp rx=0 tx=0).stop 2023-10-10T17:34:39.903 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 shutdown_connections 2023-10-10T17:34:39.903 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e480387b0 0x7f6e4803ac60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:39.904 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 --2- 172.21.15.78:0/1784958585 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6e5c1064d0 0x7f6e5c1997a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:39.904 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 >> 172.21.15.78:0/1784958585 conn(0x7f6e5c0ffed0 msgr2=0x7f6e5c101570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:39.904 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 shutdown_connections 2023-10-10T17:34:39.904 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:39.454+0000 7f6e641e3700 1 -- 172.21.15.78:0/1784958585 wait complete. 2023-10-10T17:34:39.904 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial admin user... 2023-10-10T17:34:40.844 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:40 smithi078 ceph-mon[99494]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:40.845 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:40.845 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:40.845 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:41.448 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$HFNoaN.mQtbr/VKtFQYgfeMfNGCZdxWelHwJUDwuNdhrIpBZkYAfK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1696959280, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-10-10T17:34:41.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.370+0000 7f9ff6d6b700 1 Processor -- start 2023-10-10T17:34:41.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.370+0000 7f9ff6d6b700 1 -- start start 2023-10-10T17:34:41.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9ff6d6b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff01088b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:41.449 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9ff6d6b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff0079e20 con 0x7f9ff01064d0 2023-10-10T17:34:41.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9ff4b07700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff01088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:41.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9ff4b07700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff01088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41128/0 (socket says 172.21.15.78:41128) 2023-10-10T17:34:41.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9ff4b07700 1 -- 172.21.15.78:0/676882988 learned_addr learned my addr 172.21.15.78:0/676882988 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:41.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9ff4b07700 1 -- 172.21.15.78:0/676882988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff0108df0 con 0x7f9ff01064d0 2023-10-10T17:34:41.450 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9ff4b07700 1 --2- 172.21.15.78:0/676882988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff01088b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40098a0 tx=0x7f9fe4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1e22d8308aeeab4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:41.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9fef7fe700 1 -- 172.21.15.78:0/676882988 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fe4004160 con 0x7f9ff01064d0 2023-10-10T17:34:41.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9fef7fe700 1 -- 172.21.15.78:0/676882988 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fe40042c0 con 0x7f9ff01064d0 2023-10-10T17:34:41.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.371+0000 7f9fef7fe700 1 -- 172.21.15.78:0/676882988 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fe40119b0 con 0x7f9ff01064d0 2023-10-10T17:34:41.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.372+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/676882988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 msgr2=0x7f9ff01088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:41.451 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.372+0000 7f9ff6d6b700 1 --2- 172.21.15.78:0/676882988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff01088b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9fe40098a0 tx=0x7f9fe4009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:41.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.372+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/676882988 shutdown_connections 2023-10-10T17:34:41.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.372+0000 7f9ff6d6b700 1 --2- 172.21.15.78:0/676882988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff01088b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:41.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.372+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/676882988 >> 172.21.15.78:0/676882988 conn(0x7f9ff00ffed0 msgr2=0x7f9ff0102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:41.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.372+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/676882988 shutdown_connections 2023-10-10T17:34:41.452 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.372+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/676882988 wait complete. 2023-10-10T17:34:41.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff6d6b700 1 Processor -- start 2023-10-10T17:34:41.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff6d6b700 1 -- start start 2023-10-10T17:34:41.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff6d6b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff0199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:41.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff6d6b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff0199cd0 con 0x7f9ff01064d0 2023-10-10T17:34:41.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff4b07700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff0199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:41.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff4b07700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff0199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41130/0 (socket says 172.21.15.78:41130) 2023-10-10T17:34:41.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff4b07700 1 -- 172.21.15.78:0/3704121890 learned_addr learned my addr 172.21.15.78:0/3704121890 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:41.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff4b07700 1 -- 172.21.15.78:0/3704121890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe4009550 con 0x7f9ff01064d0 2023-10-10T17:34:41.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff4b07700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff0199790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4003e60 tx=0x7f9fe4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:41.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9fedffb700 1 -- 172.21.15.78:0/3704121890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fe4003970 con 0x7f9ff01064d0 2023-10-10T17:34:41.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9fedffb700 1 -- 172.21.15.78:0/3704121890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fe4003ad0 con 0x7f9ff01064d0 2023-10-10T17:34:41.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff0109d60 con 0x7f9ff01064d0 2023-10-10T17:34:41.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.373+0000 7f9fedffb700 1 -- 172.21.15.78:0/3704121890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9fe40195e0 con 0x7f9ff01064d0 2023-10-10T17:34:41.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.374+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff0109bd0 con 0x7f9ff01064d0 2023-10-10T17:34:41.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.374+0000 7f9fedffb700 1 -- 172.21.15.78:0/3704121890 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f9fe4019850 con 0x7f9ff01064d0 2023-10-10T17:34:41.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.375+0000 7f9fedffb700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fe0038760 0x7f9fe003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:41.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.375+0000 7f9fedffb700 1 -- 172.21.15.78:0/3704121890 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9fe404c960 con 0x7f9ff01064d0 2023-10-10T17:34:41.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.375+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff0109ef0 con 0x7f9ff01064d0 2023-10-10T17:34:41.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.375+0000 7f9feffff700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fe0038760 0x7f9fe003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:41.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.375+0000 7f9feffff700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fe0038760 0x7f9fe003ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc006fd0 tx=0x7f9fdc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:41.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.379+0000 7f9fedffb700 1 -- 172.21.15.78:0/3704121890 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9fe4003c40 con 0x7f9ff01064d0 2023-10-10T17:34:41.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.597+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9ff0061f40 con 0x7f9fe0038760 2023-10-10T17:34:41.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.822+0000 7f9fedffb700 1 -- 172.21.15.78:0/3704121890 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f9ff0061f40 con 0x7f9fe0038760 2023-10-10T17:34:41.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fe0038760 msgr2=0x7f9fe003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:41.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fe0038760 0x7f9fe003ac10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc006fd0 tx=0x7f9fdc006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:41.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 msgr2=0x7f9ff0199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:41.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff0199790 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4003e60 tx=0x7f9fe4011aa0 comp rx=0 tx=0).stop 2023-10-10T17:34:41.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 shutdown_connections 2023-10-10T17:34:41.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fe0038760 0x7f9fe003ac10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:41.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 --2- 172.21.15.78:0/3704121890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ff01064d0 0x7f9ff0199790 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:41.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 >> 172.21.15.78:0/3704121890 conn(0x7f9ff00ffed0 msgr2=0x7f9ff0100cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:41.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 shutdown_connections 2023-10-10T17:34:41.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:40.824+0000 7f9ff6d6b700 1 -- 172.21.15.78:0/3704121890 wait complete. 2023-10-10T17:34:41.459 INFO:teuthology.orchestra.run.smithi078.stdout:Fetching dashboard port number... 2023-10-10T17:34:41.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:41 smithi078 ceph-mon[99494]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:41.914 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:42.817 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 8443 2023-10-10T17:34:42.817 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.132+0000 7f10dd45f700 1 Processor -- start 2023-10-10T17:34:42.817 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.132+0000 7f10dd45f700 1 -- start start 2023-10-10T17:34:42.818 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.132+0000 7f10dd45f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8104920 0x7f10d8104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:42.818 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.132+0000 7f10dd45f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8105300 con 0x7f10d8104920 2023-10-10T17:34:42.818 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8104920 0x7f10d8104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:42.818 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8104920 0x7f10d8104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41138/0 (socket says 172.21.15.78:41138) 2023-10-10T17:34:42.819 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d6ffd700 1 -- 172.21.15.78:0/982913240 learned_addr learned my addr 172.21.15.78:0/982913240 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:42.819 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d6ffd700 1 -- 172.21.15.78:0/982913240 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d8105b20 con 0x7f10d8104920 2023-10-10T17:34:42.819 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d6ffd700 1 --2- 172.21.15.78:0/982913240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8104920 0x7f10d8104d30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098a0 tx=0x7f10c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90e246e9abed4a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:42.819 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d5ffb700 1 -- 172.21.15.78:0/982913240 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10c0004160 con 0x7f10d8104920 2023-10-10T17:34:42.819 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d5ffb700 1 -- 172.21.15.78:0/982913240 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10c00042c0 con 0x7f10d8104920 2023-10-10T17:34:42.820 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.133+0000 7f10d5ffb700 1 -- 172.21.15.78:0/982913240 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10c00119b0 con 0x7f10d8104920 2023-10-10T17:34:42.820 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.134+0000 7f10dd45f700 1 -- 172.21.15.78:0/982913240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8104920 msgr2=0x7f10d8104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:42.820 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.134+0000 7f10dd45f700 1 --2- 172.21.15.78:0/982913240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8104920 0x7f10d8104d30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f10c00098a0 tx=0x7f10c0009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:42.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.134+0000 7f10dd45f700 1 -- 172.21.15.78:0/982913240 shutdown_connections 2023-10-10T17:34:42.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.134+0000 7f10dd45f700 1 --2- 172.21.15.78:0/982913240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8104920 0x7f10d8104d30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:42.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.134+0000 7f10dd45f700 1 -- 172.21.15.78:0/982913240 >> 172.21.15.78:0/982913240 conn(0x7f10d80ffed0 msgr2=0x7f10d8102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:42.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.134+0000 7f10dd45f700 1 -- 172.21.15.78:0/982913240 shutdown_connections 2023-10-10T17:34:42.821 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.134+0000 7f10dd45f700 1 -- 172.21.15.78:0/982913240 wait complete. 2023-10-10T17:34:42.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10dd45f700 1 Processor -- start 2023-10-10T17:34:42.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10dd45f700 1 -- start start 2023-10-10T17:34:42.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10dd45f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8079aa0 0x7f10d819c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:42.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10dd45f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d8079eb0 con 0x7f10d8079aa0 2023-10-10T17:34:42.822 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10d6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8079aa0 0x7f10d819c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:42.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10d6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8079aa0 0x7f10d819c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41140/0 (socket says 172.21.15.78:41140) 2023-10-10T17:34:42.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10d6ffd700 1 -- 172.21.15.78:0/1383749784 learned_addr learned my addr 172.21.15.78:0/1383749784 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:42.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10d6ffd700 1 -- 172.21.15.78:0/1383749784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c0009550 con 0x7f10d8079aa0 2023-10-10T17:34:42.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.135+0000 7f10d6ffd700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8079aa0 0x7f10d819c740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f10c0006af0 tx=0x7f10c0012fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:42.823 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.136+0000 7f10cffff700 1 -- 172.21.15.78:0/1383749784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10c00037d0 con 0x7f10d8079aa0 2023-10-10T17:34:42.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.136+0000 7f10cffff700 1 -- 172.21.15.78:0/1383749784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10c001f070 con 0x7f10d8079aa0 2023-10-10T17:34:42.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.136+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d8108ee0 con 0x7f10d8079aa0 2023-10-10T17:34:42.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.136+0000 7f10cffff700 1 -- 172.21.15.78:0/1383749784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f10c0026d80 con 0x7f10d8079aa0 2023-10-10T17:34:42.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.136+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d8108d50 con 0x7f10d8079aa0 2023-10-10T17:34:42.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.137+0000 7f10cffff700 1 -- 172.21.15.78:0/1383749784 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f10c001a440 con 0x7f10d8079aa0 2023-10-10T17:34:42.824 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.137+0000 7f10cffff700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10c4038760 0x7f10c403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:42.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.137+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d8109070 con 0x7f10d8079aa0 2023-10-10T17:34:42.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.137+0000 7f10cffff700 1 -- 172.21.15.78:0/1383749784 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f10c0021070 con 0x7f10d8079aa0 2023-10-10T17:34:42.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.137+0000 7f10d67fc700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10c4038760 0x7f10c403ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:42.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.137+0000 7f10d67fc700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10c4038760 0x7f10c403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f10c8006fd0 tx=0x7f10c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:42.825 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.142+0000 7f10cffff700 1 -- 172.21.15.78:0/1383749784 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f10c00253e0 con 0x7f10d8079aa0 2023-10-10T17:34:42.826 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.327+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f10d819d380 con 0x7f10d8079aa0 2023-10-10T17:34:42.826 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.328+0000 7f10cffff700 1 -- 172.21.15.78:0/1383749784 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f10c001c040 con 0x7f10d8079aa0 2023-10-10T17:34:42.826 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10c4038760 msgr2=0x7f10c403ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:42.826 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10c4038760 0x7f10c403ac10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f10c8006fd0 tx=0x7f10c8006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:42.826 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8079aa0 msgr2=0x7f10d819c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8079aa0 0x7f10d819c740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f10c0006af0 tx=0x7f10c0012fd0 comp rx=0 tx=0).stop 2023-10-10T17:34:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 shutdown_connections 2023-10-10T17:34:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10c4038760 0x7f10c403ac10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 --2- 172.21.15.78:0/1383749784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10d8079aa0 0x7f10d819c740 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:42.827 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 >> 172.21.15.78:0/1383749784 conn(0x7f10d80ffed0 msgr2=0x7f10d8101620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 shutdown_connections 2023-10-10T17:34:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:42.330+0000 7f10dd45f700 1 -- 172.21.15.78:0/1383749784 wait complete. 2023-10-10T17:34:42.828 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-10T17:34:42.829 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-10-10T17:34:42.835 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-10T17:34:42.835 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-10-10T17:34:42.836 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-10-10T17:34:42.836 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-10-10T17:34:42.837 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph Dashboard is now available at: 2023-10-10T17:34:42.837 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:42.837 INFO:teuthology.orchestra.run.smithi078.stdout: URL: https://smithi078:8443/ 2023-10-10T17:34:42.837 INFO:teuthology.orchestra.run.smithi078.stdout: User: admin 2023-10-10T17:34:42.838 INFO:teuthology.orchestra.run.smithi078.stdout: Password: aqshuc1kvx 2023-10-10T17:34:42.838 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:42.838 INFO:teuthology.orchestra.run.smithi078.stdout:Saving cluster configuration to /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config directory 2023-10-10T17:34:42.839 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling autotune for osd_memory_target 2023-10-10T17:34:43.076 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1383749784' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-10-10T17:34:43.885 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.299+0000 7f7392ce5700 1 Processor -- start 2023-10-10T17:34:43.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.299+0000 7f7392ce5700 1 -- start start 2023-10-10T17:34:43.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.299+0000 7f7392ce5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:43.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.299+0000 7f7392ce5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c105300 con 0x7f738c104920 2023-10-10T17:34:43.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.299+0000 7f7390a81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:43.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.299+0000 7f7390a81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41144/0 (socket says 172.21.15.78:41144) 2023-10-10T17:34:43.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.299+0000 7f7390a81700 1 -- 172.21.15.78:0/2126383266 learned_addr learned my addr 172.21.15.78:0/2126383266 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:43.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f7390a81700 1 -- 172.21.15.78:0/2126383266 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738c105b20 con 0x7f738c104920 2023-10-10T17:34:43.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f7390a81700 1 --2- 172.21.15.78:0/2126383266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c104d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73800098a0 tx=0x7f7380009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a1fd271b61ef57 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:43.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f738b7fe700 1 -- 172.21.15.78:0/2126383266 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7380004160 con 0x7f738c104920 2023-10-10T17:34:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f738b7fe700 1 -- 172.21.15.78:0/2126383266 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73800042c0 con 0x7f738c104920 2023-10-10T17:34:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f738b7fe700 1 -- 172.21.15.78:0/2126383266 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73800119b0 con 0x7f738c104920 2023-10-10T17:34:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f7392ce5700 1 -- 172.21.15.78:0/2126383266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 msgr2=0x7f738c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f7392ce5700 1 --2- 172.21.15.78:0/2126383266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c104d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f73800098a0 tx=0x7f7380009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:43.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f7392ce5700 1 -- 172.21.15.78:0/2126383266 shutdown_connections 2023-10-10T17:34:43.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f7392ce5700 1 --2- 172.21.15.78:0/2126383266 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c104d30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:43.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.300+0000 7f7392ce5700 1 -- 172.21.15.78:0/2126383266 >> 172.21.15.78:0/2126383266 conn(0x7f738c0ffed0 msgr2=0x7f738c102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:43.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7392ce5700 1 -- 172.21.15.78:0/2126383266 shutdown_connections 2023-10-10T17:34:43.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7392ce5700 1 -- 172.21.15.78:0/2126383266 wait complete. 2023-10-10T17:34:43.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7392ce5700 1 Processor -- start 2023-10-10T17:34:43.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7392ce5700 1 -- start start 2023-10-10T17:34:43.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7392ce5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c18fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:43.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7392ce5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738c1900d0 con 0x7f738c104920 2023-10-10T17:34:43.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7390a81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c18fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:43.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7390a81700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c18fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41146/0 (socket says 172.21.15.78:41146) 2023-10-10T17:34:43.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.301+0000 7f7390a81700 1 -- 172.21.15.78:0/2101556290 learned_addr learned my addr 172.21.15.78:0/2101556290 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:43.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.302+0000 7f7390a81700 1 -- 172.21.15.78:0/2101556290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7380009550 con 0x7f738c104920 2023-10-10T17:34:43.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.302+0000 7f7390a81700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c18fb90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7380000c00 tx=0x7f7380011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:43.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.302+0000 7f7389ffb700 1 -- 172.21.15.78:0/2101556290 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73800039f0 con 0x7f738c104920 2023-10-10T17:34:43.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.302+0000 7f7389ffb700 1 -- 172.21.15.78:0/2101556290 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7380003b50 con 0x7f738c104920 2023-10-10T17:34:43.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.302+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738c108ee0 con 0x7f738c104920 2023-10-10T17:34:43.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.302+0000 7f7389ffb700 1 -- 172.21.15.78:0/2101556290 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f738001a780 con 0x7f738c104920 2023-10-10T17:34:43.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.302+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738c108d50 con 0x7f738c104920 2023-10-10T17:34:43.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.303+0000 7f7389ffb700 1 -- 172.21.15.78:0/2101556290 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f738001a9f0 con 0x7f738c104920 2023-10-10T17:34:43.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.303+0000 7f7389ffb700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f737c038850 0x7f737c03ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:43.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.303+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f738c062190 con 0x7f738c104920 2023-10-10T17:34:43.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.303+0000 7f7389ffb700 1 -- 172.21.15.78:0/2101556290 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f738004c8f0 con 0x7f738c104920 2023-10-10T17:34:43.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.303+0000 7f738bfff700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f737c038850 0x7f737c03ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:43.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.304+0000 7f738bfff700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f737c038850 0x7f737c03ad00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7378006fd0 tx=0x7f7378006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:43.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.308+0000 7f7389ffb700 1 -- 172.21.15.78:0/2101556290 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f738c062190 con 0x7f738c104920 2023-10-10T17:34:43.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.494+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f738c18ed90 con 0x7f738c104920 2023-10-10T17:34:43.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.495+0000 7f7389ffb700 1 -- 172.21.15.78:0/2101556290 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f73800237b0 con 0x7f738c104920 2023-10-10T17:34:43.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.496+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f737c038850 msgr2=0x7f737c03ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:43.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.496+0000 7f7392ce5700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f737c038850 0x7f737c03ad00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7378006fd0 tx=0x7f7378006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:43.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 msgr2=0x7f738c18fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:43.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c18fb90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7380000c00 tx=0x7f7380011a30 comp rx=0 tx=0).stop 2023-10-10T17:34:43.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 shutdown_connections 2023-10-10T17:34:43.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f737c038850 0x7f737c03ad00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:43.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 --2- 172.21.15.78:0/2101556290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738c104920 0x7f738c18fb90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:43.895 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 >> 172.21.15.78:0/2101556290 conn(0x7f738c0ffed0 msgr2=0x7f738c079830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:43.896 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 shutdown_connections 2023-10-10T17:34:43.896 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:43.497+0000 7f7392ce5700 1 -- 172.21.15.78:0/2101556290 wait complete. 2023-10-10T17:34:44.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:43 smithi078 ceph-mon[99494]: mgrmap e13: x(active, since 6s) 2023-10-10T17:34:45.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.389+0000 7f4e6ac85700 1 Processor -- start 2023-10-10T17:34:45.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.389+0000 7f4e6ac85700 1 -- start start 2023-10-10T17:34:45.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.389+0000 7f4e6ac85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64104d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:45.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.389+0000 7f4e6ac85700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e64105300 con 0x7f4e64104920 2023-10-10T17:34:45.128 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.389+0000 7f4e68a21700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64104d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:45.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.389+0000 7f4e68a21700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64104d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41150/0 (socket says 172.21.15.78:41150) 2023-10-10T17:34:45.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.389+0000 7f4e68a21700 1 -- 172.21.15.78:0/3703266205 learned_addr learned my addr 172.21.15.78:0/3703266205 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:45.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.390+0000 7f4e68a21700 1 -- 172.21.15.78:0/3703266205 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e64105b20 con 0x7f4e64104920 2023-10-10T17:34:45.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.390+0000 7f4e68a21700 1 --2- 172.21.15.78:0/3703266205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64104d30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4e580098a0 tx=0x7f4e58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da01cd6070a12f81 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:45.129 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.390+0000 7f4e637fe700 1 -- 172.21.15.78:0/3703266205 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e58004160 con 0x7f4e64104920 2023-10-10T17:34:45.130 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.390+0000 7f4e637fe700 1 -- 172.21.15.78:0/3703266205 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e580042c0 con 0x7f4e64104920 2023-10-10T17:34:45.130 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.390+0000 7f4e637fe700 1 -- 172.21.15.78:0/3703266205 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e580119b0 con 0x7f4e64104920 2023-10-10T17:34:45.130 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.390+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3703266205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 msgr2=0x7f4e64104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:45.130 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.390+0000 7f4e6ac85700 1 --2- 172.21.15.78:0/3703266205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64104d30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4e580098a0 tx=0x7f4e58009bb0 comp rx=0 tx=0).stop 2023-10-10T17:34:45.131 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3703266205 shutdown_connections 2023-10-10T17:34:45.131 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 --2- 172.21.15.78:0/3703266205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64104d30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:45.131 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3703266205 >> 172.21.15.78:0/3703266205 conn(0x7f4e640ffed0 msgr2=0x7f4e64102300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:45.131 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3703266205 shutdown_connections 2023-10-10T17:34:45.131 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3703266205 wait complete. 2023-10-10T17:34:45.132 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 Processor -- start 2023-10-10T17:34:45.132 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 -- start start 2023-10-10T17:34:45.132 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64079ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:45.132 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e6ac85700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e6419c500 con 0x7f4e64104920 2023-10-10T17:34:45.132 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e68a21700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64079ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:45.132 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e68a21700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64079ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41152/0 (socket says 172.21.15.78:41152) 2023-10-10T17:34:45.133 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.391+0000 7f4e68a21700 1 -- 172.21.15.78:0/3954368277 learned_addr learned my addr 172.21.15.78:0/3954368277 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:45.133 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.392+0000 7f4e68a21700 1 -- 172.21.15.78:0/3954368277 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e58009550 con 0x7f4e64104920 2023-10-10T17:34:45.133 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.392+0000 7f4e68a21700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64079ae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e58003e60 tx=0x7f4e58011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:45.133 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.392+0000 7f4e61ffb700 1 -- 172.21.15.78:0/3954368277 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e58003970 con 0x7f4e64104920 2023-10-10T17:34:45.133 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.392+0000 7f4e61ffb700 1 -- 172.21.15.78:0/3954368277 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4e58003ad0 con 0x7f4e64104920 2023-10-10T17:34:45.134 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.392+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e64108ee0 con 0x7f4e64104920 2023-10-10T17:34:45.134 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.392+0000 7f4e61ffb700 1 -- 172.21.15.78:0/3954368277 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4e580195e0 con 0x7f4e64104920 2023-10-10T17:34:45.134 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.392+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e64108d50 con 0x7f4e64104920 2023-10-10T17:34:45.134 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.393+0000 7f4e61ffb700 1 -- 172.21.15.78:0/3954368277 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4e58019850 con 0x7f4e64104920 2023-10-10T17:34:45.134 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.393+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e64109070 con 0x7f4e64104920 2023-10-10T17:34:45.135 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.393+0000 7f4e61ffb700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4e54038800 0x7f4e5403acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:45.135 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.393+0000 7f4e61ffb700 1 -- 172.21.15.78:0/3954368277 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4e5804cb60 con 0x7f4e64104920 2023-10-10T17:34:45.135 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.393+0000 7f4e63fff700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4e54038800 0x7f4e5403acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:45.135 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.394+0000 7f4e63fff700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4e54038800 0x7f4e5403acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4e50006fd0 tx=0x7f4e50006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:45.135 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.398+0000 7f4e61ffb700 1 -- 172.21.15.78:0/3954368277 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4e580187c0 con 0x7f4e64104920 2023-10-10T17:34:45.136 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.688+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4e6419cda0 con 0x7f4e64104920 2023-10-10T17:34:45.136 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.690+0000 7f4e61ffb700 1 -- 172.21.15.78:0/3954368277 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4e580187c0 con 0x7f4e64104920 2023-10-10T17:34:45.136 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4e54038800 msgr2=0x7f4e5403acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:45.136 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4e54038800 0x7f4e5403acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4e50006fd0 tx=0x7f4e50006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:45.136 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 msgr2=0x7f4e64079ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:45.136 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64079ae0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e58003e60 tx=0x7f4e58011aa0 comp rx=0 tx=0).stop 2023-10-10T17:34:45.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 shutdown_connections 2023-10-10T17:34:45.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4e54038800 0x7f4e5403acb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:45.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 --2- 172.21.15.78:0/3954368277 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4e64104920 0x7f4e64079ae0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:45.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.692+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 >> 172.21.15.78:0/3954368277 conn(0x7f4e640ffed0 msgr2=0x7f4e64101580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:45.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.693+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 shutdown_connections 2023-10-10T17:34:45.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-10-10T17:34:44.693+0000 7f4e6ac85700 1 -- 172.21.15.78:0/3954368277 wait complete. 2023-10-10T17:34:45.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-10-10T17:34:45.138 INFO:teuthology.orchestra.run.smithi078.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-10-10T17:34:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:34:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.139 INFO:teuthology.orchestra.run.smithi078.stdout:Or, if you are only running a single cluster on this host: 2023-10-10T17:34:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-10-10T17:34:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.139 INFO:teuthology.orchestra.run.smithi078.stdout:Please consider enabling telemetry to help improve Ceph: 2023-10-10T17:34:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: ceph telemetry on 2023-10-10T17:34:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.140 INFO:teuthology.orchestra.run.smithi078.stdout:For more information see: 2023-10-10T17:34:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-10-10T17:34:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:45.141 INFO:teuthology.orchestra.run.smithi078.stdout:Bootstrap complete. 2023-10-10T17:34:45.162 INFO:tasks.cephadm:Fetching config... 2023-10-10T17:34:45.162 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:34:45.163 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-10-10T17:34:45.180 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-10-10T17:34:45.180 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:34:45.180 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-10-10T17:34:45.239 INFO:tasks.cephadm:Fetching mon keyring... 2023-10-10T17:34:45.239 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:34:45.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-10-10T17:34:45.306 INFO:tasks.cephadm:Fetching pub ssh key... 2023-10-10T17:34:45.306 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:34:45.307 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-10-10T17:34:45.363 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-10-10T17:34:45.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC04uixS8LRvZPoJRTPLqrYaxTz4Og5+Hot0IdV0HknwJocjymwDfusEJKovPA7q2v0y7KQYHQMrISXIvVxMsuYqraO25km3OuBbd4riwCqMgQ2aaLIq/WmXWVMDK0hGzVQKkvMtjIQjmeTRD7VX8umD4qTe4O62JyXl3mWUQWW9P66YBLf7KWyLwKh1j2q3sqXsrEOvOk8zIsrbuVQ0ubn34gCc9AkH+NLWhv+NI2pKXMJxkrDThafFg9L07Lze4FYWYIhSTtiDWrk5aH+8Ag2feqA+CjPg5x6MX0LdgLV4262YaJtsx5gCDy7zWe5zTEjgQmSrvHABZUk3a0afSISuR/BLiEGjSoWh0dWBs4hx1o6ajX1EeunC7hNVvTxBsjnLQG9uSDYdF3EP2l3lkC42r2SD0wchsebJ5VmMWNcOPg1w+6wehavXnN4Svrc7UPcmpvQlJb3s2+szWgPXRYb3Wtw8h48T4GlpinMAjdEdqICbqL5VyqYvl/IzyeixJE= ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-10T17:34:45.443 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC04uixS8LRvZPoJRTPLqrYaxTz4Og5+Hot0IdV0HknwJocjymwDfusEJKovPA7q2v0y7KQYHQMrISXIvVxMsuYqraO25km3OuBbd4riwCqMgQ2aaLIq/WmXWVMDK0hGzVQKkvMtjIQjmeTRD7VX8umD4qTe4O62JyXl3mWUQWW9P66YBLf7KWyLwKh1j2q3sqXsrEOvOk8zIsrbuVQ0ubn34gCc9AkH+NLWhv+NI2pKXMJxkrDThafFg9L07Lze4FYWYIhSTtiDWrk5aH+8Ag2feqA+CjPg5x6MX0LdgLV4262YaJtsx5gCDy7zWe5zTEjgQmSrvHABZUk3a0afSISuR/BLiEGjSoWh0dWBs4hx1o6ajX1EeunC7hNVvTxBsjnLQG9uSDYdF3EP2l3lkC42r2SD0wchsebJ5VmMWNcOPg1w+6wehavXnN4Svrc7UPcmpvQlJb3s2+szWgPXRYb3Wtw8h48T4GlpinMAjdEdqICbqL5VyqYvl/IzyeixJE= ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:34:45.455 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC04uixS8LRvZPoJRTPLqrYaxTz4Og5+Hot0IdV0HknwJocjymwDfusEJKovPA7q2v0y7KQYHQMrISXIvVxMsuYqraO25km3OuBbd4riwCqMgQ2aaLIq/WmXWVMDK0hGzVQKkvMtjIQjmeTRD7VX8umD4qTe4O62JyXl3mWUQWW9P66YBLf7KWyLwKh1j2q3sqXsrEOvOk8zIsrbuVQ0ubn34gCc9AkH+NLWhv+NI2pKXMJxkrDThafFg9L07Lze4FYWYIhSTtiDWrk5aH+8Ag2feqA+CjPg5x6MX0LdgLV4262YaJtsx5gCDy7zWe5zTEjgQmSrvHABZUk3a0afSISuR/BLiEGjSoWh0dWBs4hx1o6ajX1EeunC7hNVvTxBsjnLQG9uSDYdF3EP2l3lkC42r2SD0wchsebJ5VmMWNcOPg1w+6wehavXnN4Svrc7UPcmpvQlJb3s2+szWgPXRYb3Wtw8h48T4GlpinMAjdEdqICbqL5VyqYvl/IzyeixJE= ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-10T17:34:45.502 INFO:teuthology.orchestra.run.smithi157.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC04uixS8LRvZPoJRTPLqrYaxTz4Og5+Hot0IdV0HknwJocjymwDfusEJKovPA7q2v0y7KQYHQMrISXIvVxMsuYqraO25km3OuBbd4riwCqMgQ2aaLIq/WmXWVMDK0hGzVQKkvMtjIQjmeTRD7VX8umD4qTe4O62JyXl3mWUQWW9P66YBLf7KWyLwKh1j2q3sqXsrEOvOk8zIsrbuVQ0ubn34gCc9AkH+NLWhv+NI2pKXMJxkrDThafFg9L07Lze4FYWYIhSTtiDWrk5aH+8Ag2feqA+CjPg5x6MX0LdgLV4262YaJtsx5gCDy7zWe5zTEjgQmSrvHABZUk3a0afSISuR/BLiEGjSoWh0dWBs4hx1o6ajX1EeunC7hNVvTxBsjnLQG9uSDYdF3EP2l3lkC42r2SD0wchsebJ5VmMWNcOPg1w+6wehavXnN4Svrc7UPcmpvQlJb3s2+szWgPXRYb3Wtw8h48T4GlpinMAjdEdqICbqL5VyqYvl/IzyeixJE= ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:34:45.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC04uixS8LRvZPoJRTPLqrYaxTz4Og5+Hot0IdV0HknwJocjymwDfusEJKovPA7q2v0y7KQYHQMrISXIvVxMsuYqraO25km3OuBbd4riwCqMgQ2aaLIq/WmXWVMDK0hGzVQKkvMtjIQjmeTRD7VX8umD4qTe4O62JyXl3mWUQWW9P66YBLf7KWyLwKh1j2q3sqXsrEOvOk8zIsrbuVQ0ubn34gCc9AkH+NLWhv+NI2pKXMJxkrDThafFg9L07Lze4FYWYIhSTtiDWrk5aH+8Ag2feqA+CjPg5x6MX0LdgLV4262YaJtsx5gCDy7zWe5zTEjgQmSrvHABZUk3a0afSISuR/BLiEGjSoWh0dWBs4hx1o6ajX1EeunC7hNVvTxBsjnLQG9uSDYdF3EP2l3lkC42r2SD0wchsebJ5VmMWNcOPg1w+6wehavXnN4Svrc7UPcmpvQlJb3s2+szWgPXRYb3Wtw8h48T4GlpinMAjdEdqICbqL5VyqYvl/IzyeixJE= ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-10T17:34:45.563 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC04uixS8LRvZPoJRTPLqrYaxTz4Og5+Hot0IdV0HknwJocjymwDfusEJKovPA7q2v0y7KQYHQMrISXIvVxMsuYqraO25km3OuBbd4riwCqMgQ2aaLIq/WmXWVMDK0hGzVQKkvMtjIQjmeTRD7VX8umD4qTe4O62JyXl3mWUQWW9P66YBLf7KWyLwKh1j2q3sqXsrEOvOk8zIsrbuVQ0ubn34gCc9AkH+NLWhv+NI2pKXMJxkrDThafFg9L07Lze4FYWYIhSTtiDWrk5aH+8Ag2feqA+CjPg5x6MX0LdgLV4262YaJtsx5gCDy7zWe5zTEjgQmSrvHABZUk3a0afSISuR/BLiEGjSoWh0dWBs4hx1o6ajX1EeunC7hNVvTxBsjnLQG9uSDYdF3EP2l3lkC42r2SD0wchsebJ5VmMWNcOPg1w+6wehavXnN4Svrc7UPcmpvQlJb3s2+szWgPXRYb3Wtw8h48T4GlpinMAjdEdqICbqL5VyqYvl/IzyeixJE= ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:34:45.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-10-10T17:34:46.010 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:45 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3954368277' entity='client.admin' 2023-10-10T17:34:46.423 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:34:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:47.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:34:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:47.786 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:48.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.158+0000 7faad44b8700 1 -- 172.21.15.78:0/3186618876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 msgr2=0x7faacc100130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:48.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.158+0000 7faad44b8700 1 --2- 172.21.15.78:0/3186618876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc100130 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7faac8009910 tx=0x7faac8009c20 comp rx=0 tx=0).stop 2023-10-10T17:34:48.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 -- 172.21.15.78:0/3186618876 shutdown_connections 2023-10-10T17:34:48.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 --2- 172.21.15.78:0/3186618876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc100130 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:48.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 -- 172.21.15.78:0/3186618876 >> 172.21.15.78:0/3186618876 conn(0x7faacc0fb2d0 msgr2=0x7faacc0fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:48.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 -- 172.21.15.78:0/3186618876 shutdown_connections 2023-10-10T17:34:48.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 -- 172.21.15.78:0/3186618876 wait complete. 2023-10-10T17:34:48.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 Processor -- start 2023-10-10T17:34:48.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 -- start start 2023-10-10T17:34:48.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:48.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad44b8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faacc195340 con 0x7faacc0ffd20 2023-10-10T17:34:48.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad2254700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:48.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.159+0000 7faad2254700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41158/0 (socket says 172.21.15.78:41158) 2023-10-10T17:34:48.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faad2254700 1 -- 172.21.15.78:0/528149276 learned_addr learned my addr 172.21.15.78:0/528149276 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faad2254700 1 -- 172.21.15.78:0/528149276 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac80095f0 con 0x7faacc0ffd20 2023-10-10T17:34:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faad2254700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc194e00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faac8006010 tx=0x7faac80057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faac37fe700 1 -- 172.21.15.78:0/528149276 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faac801d070 con 0x7faacc0ffd20 2023-10-10T17:34:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faac37fe700 1 -- 172.21.15.78:0/528149276 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faac80044e0 con 0x7faacc0ffd20 2023-10-10T17:34:48.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faacc105bf0 con 0x7faacc0ffd20 2023-10-10T17:34:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faac37fe700 1 -- 172.21.15.78:0/528149276 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faac80039b0 con 0x7faacc0ffd20 2023-10-10T17:34:48.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.160+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faacc105a60 con 0x7faacc0ffd20 2023-10-10T17:34:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.161+0000 7faac37fe700 1 -- 172.21.15.78:0/528149276 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7faac8003c70 con 0x7faacc0ffd20 2023-10-10T17:34:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.161+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faacc105d80 con 0x7faacc0ffd20 2023-10-10T17:34:48.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.161+0000 7faac37fe700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faab80388c0 0x7faab803ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.161+0000 7faac37fe700 1 -- 172.21.15.78:0/528149276 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faac804ce80 con 0x7faacc0ffd20 2023-10-10T17:34:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.161+0000 7faad1a53700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faab80388c0 0x7faab803ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.161+0000 7faad1a53700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faab80388c0 0x7faab803ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faabc006fd0 tx=0x7faabc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:48.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.166+0000 7faac37fe700 1 -- 172.21.15.78:0/528149276 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faac8050030 con 0x7faacc0ffd20 2023-10-10T17:34:48.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.354+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7faacc195930 con 0x7faacc0ffd20 2023-10-10T17:34:48.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.358+0000 7faac37fe700 1 -- 172.21.15.78:0/528149276 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7faac802a950 con 0x7faacc0ffd20 2023-10-10T17:34:48.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faab80388c0 msgr2=0x7faab803ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:48.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faab80388c0 0x7faab803ad70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7faabc006fd0 tx=0x7faabc006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:48.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 msgr2=0x7faacc194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:48.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc194e00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7faac8006010 tx=0x7faac80057c0 comp rx=0 tx=0).stop 2023-10-10T17:34:48.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 shutdown_connections 2023-10-10T17:34:48.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faab80388c0 0x7faab803ad70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:48.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 --2- 172.21.15.78:0/528149276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faacc0ffd20 0x7faacc194e00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:48.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 >> 172.21.15.78:0/528149276 conn(0x7faacc0fb2d0 msgr2=0x7faacc0fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:48.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 shutdown_connections 2023-10-10T17:34:48.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:48.360+0000 7faad44b8700 1 -- 172.21.15.78:0/528149276 wait complete. 2023-10-10T17:34:48.790 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-10-10T17:34:48.791 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-10-10T17:34:49.431 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:34:49.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:49 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/528149276' entity='client.admin' 2023-10-10T17:34:49.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:49.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 -- 172.21.15.78:0/3741838702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 msgr2=0x7f7398100130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 --2- 172.21.15.78:0/3741838702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398100130 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7388009910 tx=0x7f7388009c20 comp rx=0 tx=0).stop 2023-10-10T17:34:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 -- 172.21.15.78:0/3741838702 shutdown_connections 2023-10-10T17:34:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 --2- 172.21.15.78:0/3741838702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398100130 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 -- 172.21.15.78:0/3741838702 >> 172.21.15.78:0/3741838702 conn(0x7f73980fb2d0 msgr2=0x7f73980fd700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 -- 172.21.15.78:0/3741838702 shutdown_connections 2023-10-10T17:34:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 -- 172.21.15.78:0/3741838702 wait complete. 2023-10-10T17:34:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.742+0000 7f739f799700 1 Processor -- start 2023-10-10T17:34:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739f799700 1 -- start start 2023-10-10T17:34:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739f799700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739f799700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7398195340 con 0x7f73980ffd20 2023-10-10T17:34:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739d535700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739d535700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41164/0 (socket says 172.21.15.78:41164) 2023-10-10T17:34:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739d535700 1 -- 172.21.15.78:0/2156360562 learned_addr learned my addr 172.21.15.78:0/2156360562 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739d535700 1 -- 172.21.15.78:0/2156360562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880095f0 con 0x7f73980ffd20 2023-10-10T17:34:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739d535700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398194e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7388006010 tx=0x7f73880057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f738e7fc700 1 -- 172.21.15.78:0/2156360562 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f738801d070 con 0x7f73980ffd20 2023-10-10T17:34:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f738e7fc700 1 -- 172.21.15.78:0/2156360562 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73880044e0 con 0x7f73980ffd20 2023-10-10T17:34:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7398105bf0 con 0x7f73980ffd20 2023-10-10T17:34:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f738e7fc700 1 -- 172.21.15.78:0/2156360562 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73880039b0 con 0x7f73980ffd20 2023-10-10T17:34:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.743+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7398105a60 con 0x7f73980ffd20 2023-10-10T17:34:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.744+0000 7f738e7fc700 1 -- 172.21.15.78:0/2156360562 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f7388003c20 con 0x7f73980ffd20 2023-10-10T17:34:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.744+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7398105d80 con 0x7f73980ffd20 2023-10-10T17:34:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.744+0000 7f738e7fc700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73840388c0 0x7f738403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.744+0000 7f738e7fc700 1 -- 172.21.15.78:0/2156360562 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f738804cb10 con 0x7f73980ffd20 2023-10-10T17:34:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.745+0000 7f739cd34700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73840388c0 0x7f738403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.745+0000 7f739cd34700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73840388c0 0x7f738403ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7394006fd0 tx=0x7f7394006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:50.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.749+0000 7f738e7fc700 1 -- 172.21.15.78:0/2156360562 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f738802a950 con 0x7f73980ffd20 2023-10-10T17:34:50.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.942+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7398060f30 con 0x7f73840388c0 2023-10-10T17:34:50.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.946+0000 7f738e7fc700 1 -- 172.21.15.78:0/2156360562 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7398060f30 con 0x7f73840388c0 2023-10-10T17:34:50.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73840388c0 msgr2=0x7f738403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:50.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73840388c0 0x7f738403ad70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7394006fd0 tx=0x7f7394006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:50.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 msgr2=0x7f7398194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:50.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398194e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7388006010 tx=0x7f73880057c0 comp rx=0 tx=0).stop 2023-10-10T17:34:50.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 shutdown_connections 2023-10-10T17:34:50.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73840388c0 0x7f738403ad70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:50.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 --2- 172.21.15.78:0/2156360562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73980ffd20 0x7f7398194e00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:50.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 >> 172.21.15.78:0/2156360562 conn(0x7f73980fb2d0 msgr2=0x7f73980fbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:50.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 shutdown_connections 2023-10-10T17:34:50.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:50.948+0000 7f739f799700 1 -- 172.21.15.78:0/2156360562 wait complete. 2023-10-10T17:34:51.388 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi157 2023-10-10T17:34:51.389 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:34:51.390 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/etc/ceph/ceph.conf 2023-10-10T17:34:51.412 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:34:51.412 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:34:51.472 INFO:tasks.cephadm:Adding host smithi157 to orchestrator... 2023-10-10T17:34:51.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch host add smithi157 2023-10-10T17:34:52.137 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:34:52.324 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:52.324 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:52.324 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:52.324 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:34:52.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:34:52.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:34:52.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: Updating smithi078:/etc/ceph/ceph.conf 2023-10-10T17:34:52.325 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:34:52.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:34:52.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:52.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:52.326 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.365+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/4222888906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 msgr2=0x7f6d080ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.365+0000 7f6d0fbdf700 1 --2- 172.21.15.78:0/4222888906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d080ff2d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009910 tx=0x7f6d04009c20 comp rx=0 tx=0).stop 2023-10-10T17:34:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.365+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/4222888906 shutdown_connections 2023-10-10T17:34:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.366+0000 7f6d0fbdf700 1 --2- 172.21.15.78:0/4222888906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d080ff2d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.366+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/4222888906 >> 172.21.15.78:0/4222888906 conn(0x7f6d08074930 msgr2=0x7f6d08074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.366+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/4222888906 shutdown_connections 2023-10-10T17:34:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.366+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/4222888906 wait complete. 2023-10-10T17:34:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.366+0000 7f6d0fbdf700 1 Processor -- start 2023-10-10T17:34:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.366+0000 7f6d0fbdf700 1 -- start start 2023-10-10T17:34:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0fbdf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d081940c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0fbdf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d08194600 con 0x7f6d080fef00 2023-10-10T17:34:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0d97b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d081940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0d97b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d081940c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41170/0 (socket says 172.21.15.78:41170) 2023-10-10T17:34:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0d97b700 1 -- 172.21.15.78:0/263548060 learned_addr learned my addr 172.21.15.78:0/263548060 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0d97b700 1 -- 172.21.15.78:0/263548060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040095f0 con 0x7f6d080fef00 2023-10-10T17:34:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0d97b700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d081940c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d04004f40 tx=0x7f6d04005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6cfeffd700 1 -- 172.21.15.78:0/263548060 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d0401d070 con 0x7f6d080fef00 2023-10-10T17:34:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6cfeffd700 1 -- 172.21.15.78:0/263548060 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d04004370 con 0x7f6d080fef00 2023-10-10T17:34:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d08109090 con 0x7f6d080fef00 2023-10-10T17:34:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6cfeffd700 1 -- 172.21.15.78:0/263548060 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6d040039b0 con 0x7f6d080fef00 2023-10-10T17:34:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.367+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d08108f00 con 0x7f6d080fef00 2023-10-10T17:34:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.368+0000 7f6cfeffd700 1 -- 172.21.15.78:0/263548060 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6d04003c20 con 0x7f6d080fef00 2023-10-10T17:34:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.368+0000 7f6cfeffd700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6cf40388c0 0x7f6cf403ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.368+0000 7f6cfeffd700 1 -- 172.21.15.78:0/263548060 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6d0404c510 con 0x7f6d080fef00 2023-10-10T17:34:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.368+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d08109220 con 0x7f6d080fef00 2023-10-10T17:34:53.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.368+0000 7f6d0d17a700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6cf40388c0 0x7f6cf403ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:53.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.369+0000 7f6d0d17a700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6cf40388c0 0x7f6cf403ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:53.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.373+0000 7f6cfeffd700 1 -- 172.21.15.78:0/263548060 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6d04026020 con 0x7f6d080fef00 2023-10-10T17:34:53.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:52 smithi078 ceph-mon[99494]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:34:53.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:53.563+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d08106fa0 con 0x7f6cf40388c0 2023-10-10T17:34:54.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:54 smithi078 ceph-mon[99494]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi157", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:34:54.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:54 smithi078 ceph-mon[99494]: Deploying cephadm binary to smithi157 2023-10-10T17:34:55.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.658+0000 7f6cfeffd700 1 -- 172.21.15.78:0/263548060 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6d08106fa0 con 0x7f6cf40388c0 2023-10-10T17:34:55.660 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi157' with addr '172.21.15.157' 2023-10-10T17:34:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6cf40388c0 msgr2=0x7f6cf403ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6cf40388c0 0x7f6cf403ad70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6cf8006fd0 tx=0x7f6cf8006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 msgr2=0x7f6d081940c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d081940c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6d04004f40 tx=0x7f6d04005020 comp rx=0 tx=0).stop 2023-10-10T17:34:55.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 shutdown_connections 2023-10-10T17:34:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6cf40388c0 0x7f6cf403ad70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 --2- 172.21.15.78:0/263548060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d080fef00 0x7f6d081940c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 >> 172.21.15.78:0/263548060 conn(0x7f6d08074930 msgr2=0x7f6d08103090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 shutdown_connections 2023-10-10T17:34:55.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:55.660+0000 7f6d0fbdf700 1 -- 172.21.15.78:0/263548060 wait complete. 2023-10-10T17:34:56.084 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch host ls --format=json 2023-10-10T17:34:56.734 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:34:56.938 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:56 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:56.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:56 smithi078 ceph-mon[99494]: Added host smithi157 2023-10-10T17:34:56.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:56 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:34:56.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:56 smithi078 ceph-mon[99494]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:34:56.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:56 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.971+0000 7fab6e580700 1 -- 172.21.15.78:0/1583207039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 msgr2=0x7fab681011c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.971+0000 7fab6e580700 1 --2- 172.21.15.78:0/1583207039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681011c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fab58009910 tx=0x7fab58009c20 comp rx=0 tx=0).stop 2023-10-10T17:34:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.971+0000 7fab6e580700 1 -- 172.21.15.78:0/1583207039 shutdown_connections 2023-10-10T17:34:57.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.971+0000 7fab6e580700 1 --2- 172.21.15.78:0/1583207039 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681011c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.971+0000 7fab6e580700 1 -- 172.21.15.78:0/1583207039 >> 172.21.15.78:0/1583207039 conn(0x7fab68074930 msgr2=0x7fab68074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.971+0000 7fab6e580700 1 -- 172.21.15.78:0/1583207039 shutdown_connections 2023-10-10T17:34:57.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.971+0000 7fab6e580700 1 -- 172.21.15.78:0/1583207039 wait complete. 2023-10-10T17:34:57.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.972+0000 7fab6e580700 1 Processor -- start 2023-10-10T17:34:57.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.972+0000 7fab6e580700 1 -- start start 2023-10-10T17:34:57.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.972+0000 7fab6e580700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:57.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.972+0000 7fab6e580700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab68195700 con 0x7fab68100df0 2023-10-10T17:34:57.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.972+0000 7fab67fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:57.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.972+0000 7fab67fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41196/0 (socket says 172.21.15.78:41196) 2023-10-10T17:34:57.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.972+0000 7fab67fff700 1 -- 172.21.15.78:0/2626215813 learned_addr learned my addr 172.21.15.78:0/2626215813 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:34:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.973+0000 7fab67fff700 1 -- 172.21.15.78:0/2626215813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab580095f0 con 0x7fab68100df0 2023-10-10T17:34:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.973+0000 7fab67fff700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681951c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab58005850 tx=0x7fab58004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.973+0000 7fab657fa700 1 -- 172.21.15.78:0/2626215813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab5801d070 con 0x7fab68100df0 2023-10-10T17:34:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.973+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab6810a190 con 0x7fab68100df0 2023-10-10T17:34:57.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.973+0000 7fab657fa700 1 -- 172.21.15.78:0/2626215813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab58004370 con 0x7fab68100df0 2023-10-10T17:34:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.973+0000 7fab657fa700 1 -- 172.21.15.78:0/2626215813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab58003ab0 con 0x7fab68100df0 2023-10-10T17:34:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.973+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab6810a000 con 0x7fab68100df0 2023-10-10T17:34:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.974+0000 7fab657fa700 1 -- 172.21.15.78:0/2626215813 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fab5800f460 con 0x7fab68100df0 2023-10-10T17:34:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.974+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab6810a320 con 0x7fab68100df0 2023-10-10T17:34:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.974+0000 7fab657fa700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fab50038870 0x7fab5003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:34:57.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.974+0000 7fab657fa700 1 -- 172.21.15.78:0/2626215813 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab5804c6e0 con 0x7fab68100df0 2023-10-10T17:34:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.974+0000 7fab677fe700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fab50038870 0x7fab5003ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:34:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.974+0000 7fab677fe700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fab50038870 0x7fab5003ad20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab5c006fd0 tx=0x7fab5c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:34:57.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:57.979+0000 7fab657fa700 1 -- 172.21.15.78:0/2626215813 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fab5800fd20 con 0x7fab68100df0 2023-10-10T17:34:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:57 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:34:58.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.169+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab681073e0 con 0x7fab50038870 2023-10-10T17:34:58.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.170+0000 7fab657fa700 1 -- 172.21.15.78:0/2626215813 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fab681073e0 con 0x7fab50038870 2023-10-10T17:34:58.172 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:34:58.173 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.157", "hostname": "smithi157", "labels": [], "status": ""}] 2023-10-10T17:34:58.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fab50038870 msgr2=0x7fab5003ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:58.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fab50038870 0x7fab5003ad20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fab5c006fd0 tx=0x7fab5c006e40 comp rx=0 tx=0).stop 2023-10-10T17:34:58.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 msgr2=0x7fab681951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:34:58.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681951c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fab58005850 tx=0x7fab58004ff0 comp rx=0 tx=0).stop 2023-10-10T17:34:58.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 shutdown_connections 2023-10-10T17:34:58.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fab50038870 0x7fab5003ad20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:58.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 --2- 172.21.15.78:0/2626215813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab68100df0 0x7fab681951c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:34:58.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 >> 172.21.15.78:0/2626215813 conn(0x7fab68074930 msgr2=0x7fab68104610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:34:58.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 shutdown_connections 2023-10-10T17:34:58.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:34:58.172+0000 7fab6e580700 1 -- 172.21.15.78:0/2626215813 wait complete. 2023-10-10T17:34:58.590 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2023-10-10T17:34:58.591 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:34:58.591 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2023-10-10T17:34:58.612 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:34:58.612 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:34:58.671 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2023-10-10T17:34:58.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch host add smithi190 2023-10-10T17:34:59.310 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:34:59.497 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:59 smithi078 ceph-mon[99494]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:34:59.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:59 smithi078 ceph-mon[99494]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:34:59.524 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:34:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:00.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.509+0000 7fe82c312700 1 -- 172.21.15.78:0/3304431848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 msgr2=0x7fe8240ff2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:00.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.509+0000 7fe82c312700 1 --2- 172.21.15.78:0/3304431848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8240ff2d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe81c009910 tx=0x7fe81c009c20 comp rx=0 tx=0).stop 2023-10-10T17:35:00.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.509+0000 7fe82c312700 1 -- 172.21.15.78:0/3304431848 shutdown_connections 2023-10-10T17:35:00.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.509+0000 7fe82c312700 1 --2- 172.21.15.78:0/3304431848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8240ff2d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:00.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.509+0000 7fe82c312700 1 -- 172.21.15.78:0/3304431848 >> 172.21.15.78:0/3304431848 conn(0x7fe824074930 msgr2=0x7fe824074d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:00.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.509+0000 7fe82c312700 1 -- 172.21.15.78:0/3304431848 shutdown_connections 2023-10-10T17:35:00.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.509+0000 7fe82c312700 1 -- 172.21.15.78:0/3304431848 wait complete. 2023-10-10T17:35:00.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82c312700 1 Processor -- start 2023-10-10T17:35:00.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82c312700 1 -- start start 2023-10-10T17:35:00.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82c312700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8241940c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:00.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82c312700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe824194600 con 0x7fe8240fef00 2023-10-10T17:35:00.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82a0ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8241940c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:00.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82a0ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8241940c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41202/0 (socket says 172.21.15.78:41202) 2023-10-10T17:35:00.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82a0ae700 1 -- 172.21.15.78:0/3465203616 learned_addr learned my addr 172.21.15.78:0/3465203616 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:35:00.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82a0ae700 1 -- 172.21.15.78:0/3465203616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81c0095f0 con 0x7fe8240fef00 2023-10-10T17:35:00.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.510+0000 7fe82a0ae700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8241940c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe81c004f40 tx=0x7fe81c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:00.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.511+0000 7fe8237fe700 1 -- 172.21.15.78:0/3465203616 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe81c01d070 con 0x7fe8240fef00 2023-10-10T17:35:00.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.511+0000 7fe8237fe700 1 -- 172.21.15.78:0/3465203616 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe81c004370 con 0x7fe8240fef00 2023-10-10T17:35:00.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.511+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe824109090 con 0x7fe8240fef00 2023-10-10T17:35:00.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.511+0000 7fe8237fe700 1 -- 172.21.15.78:0/3465203616 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe81c0039b0 con 0x7fe8240fef00 2023-10-10T17:35:00.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.511+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe824108f00 con 0x7fe8240fef00 2023-10-10T17:35:00.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.511+0000 7fe8237fe700 1 -- 172.21.15.78:0/3465203616 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fe81c003c20 con 0x7fe8240fef00 2023-10-10T17:35:00.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.512+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe824109220 con 0x7fe8240fef00 2023-10-10T17:35:00.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.512+0000 7fe8237fe700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe80c0388c0 0x7fe80c03ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:00.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.512+0000 7fe8237fe700 1 -- 172.21.15.78:0/3465203616 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe81c04c4f0 con 0x7fe8240fef00 2023-10-10T17:35:00.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.512+0000 7fe8298ad700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe80c0388c0 0x7fe80c03ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:00.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.512+0000 7fe8298ad700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe80c0388c0 0x7fe80c03ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe818006fd0 tx=0x7fe818006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:00.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.516+0000 7fe8237fe700 1 -- 172.21.15.78:0/3465203616 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe81c00f460 con 0x7fe8240fef00 2023-10-10T17:35:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:00.710+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}) v1 -- 0x7fe824106ff0 con 0x7fe80c0388c0 2023-10-10T17:35:01.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:01.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:01.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:01.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:01.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:01.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi157", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:35:01.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:01.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:02.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:02 smithi078 ceph-mon[99494]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:02.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:02 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:02.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:02 smithi078 ceph-mon[99494]: Updating smithi157:/etc/ceph/ceph.conf 2023-10-10T17:35:02.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:02 smithi078 ceph-mon[99494]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:02.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:02 smithi078 ceph-mon[99494]: Deploying cephadm binary to smithi190 2023-10-10T17:35:02.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:02 smithi078 ceph-mon[99494]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:35:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.809+0000 7fe8237fe700 1 -- 172.21.15.78:0/3465203616 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe824106ff0 con 0x7fe80c0388c0 2023-10-10T17:35:02.812 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi190' with addr '172.21.15.190' 2023-10-10T17:35:02.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe80c0388c0 msgr2=0x7fe80c03ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe80c0388c0 0x7fe80c03ad70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe818006fd0 tx=0x7fe818006e40 comp rx=0 tx=0).stop 2023-10-10T17:35:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 msgr2=0x7fe8241940c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8241940c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe81c004f40 tx=0x7fe81c005020 comp rx=0 tx=0).stop 2023-10-10T17:35:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 shutdown_connections 2023-10-10T17:35:02.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe80c0388c0 0x7fe80c03ad70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 --2- 172.21.15.78:0/3465203616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8240fef00 0x7fe8241940c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 >> 172.21.15.78:0/3465203616 conn(0x7fe824074930 msgr2=0x7fe824103090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 shutdown_connections 2023-10-10T17:35:02.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:02.812+0000 7fe82c312700 1 -- 172.21.15.78:0/3465203616 wait complete. 2023-10-10T17:35:03.269 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch host ls --format=json 2023-10-10T17:35:03.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:03 smithi078 ceph-mon[99494]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:35:03.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:03 smithi078 ceph-mon[99494]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:03.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:03.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:03.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:03.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:03.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:03.910 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:35:04.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:04 smithi078 ceph-mon[99494]: Added host smithi190 2023-10-10T17:35:04.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:04 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:04.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:04 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:05.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.228+0000 7f8900f27700 1 -- 172.21.15.78:0/2480372049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 msgr2=0x7f88fc0ff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:05.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.228+0000 7f8900f27700 1 --2- 172.21.15.78:0/2480372049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc0ff350 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009910 tx=0x7f88f0009c20 comp rx=0 tx=0).stop 2023-10-10T17:35:05.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.228+0000 7f8900f27700 1 -- 172.21.15.78:0/2480372049 shutdown_connections 2023-10-10T17:35:05.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.228+0000 7f8900f27700 1 --2- 172.21.15.78:0/2480372049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc0ff350 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:05.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.228+0000 7f8900f27700 1 -- 172.21.15.78:0/2480372049 >> 172.21.15.78:0/2480372049 conn(0x7f88fc0fa880 msgr2=0x7f88fc0fcc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.228+0000 7f8900f27700 1 -- 172.21.15.78:0/2480372049 shutdown_connections 2023-10-10T17:35:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.228+0000 7f8900f27700 1 -- 172.21.15.78:0/2480372049 wait complete. 2023-10-10T17:35:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f8900f27700 1 Processor -- start 2023-10-10T17:35:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f8900f27700 1 -- start start 2023-10-10T17:35:05.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f8900f27700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f8900f27700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc102910 con 0x7f88fc0fef80 2023-10-10T17:35:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f88fa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f88fa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41228/0 (socket says 172.21.15.78:41228) 2023-10-10T17:35:05.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f88fa59c700 1 -- 172.21.15.78:0/3730155700 learned_addr learned my addr 172.21.15.78:0/3730155700 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:35:05.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.229+0000 7f88fa59c700 1 -- 172.21.15.78:0/3730155700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00095f0 con 0x7f88fc0fef80 2023-10-10T17:35:05.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.230+0000 7f88fa59c700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc100990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004f40 tx=0x7f88f0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:05.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.230+0000 7f88eb7fe700 1 -- 172.21.15.78:0/3730155700 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88f001d070 con 0x7f88fc0fef80 2023-10-10T17:35:05.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.230+0000 7f88eb7fe700 1 -- 172.21.15.78:0/3730155700 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f88f0004370 con 0x7f88fc0fef80 2023-10-10T17:35:05.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.230+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc109160 con 0x7f88fc0fef80 2023-10-10T17:35:05.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.230+0000 7f88eb7fe700 1 -- 172.21.15.78:0/3730155700 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88f00039b0 con 0x7f88fc0fef80 2023-10-10T17:35:05.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.230+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc108fd0 con 0x7f88fc0fef80 2023-10-10T17:35:05.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.231+0000 7f88eb7fe700 1 -- 172.21.15.78:0/3730155700 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f88f0003c70 con 0x7f88fc0fef80 2023-10-10T17:35:05.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.231+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc1092f0 con 0x7f88fc0fef80 2023-10-10T17:35:05.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.231+0000 7f88eb7fe700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f88e4038870 0x7f88e403ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:05.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.231+0000 7f88eb7fe700 1 -- 172.21.15.78:0/3730155700 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f88f004c7b0 con 0x7f88fc0fef80 2023-10-10T17:35:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.231+0000 7f88f9d9b700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f88e4038870 0x7f88e403ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.231+0000 7f88f9d9b700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f88e4038870 0x7f88e403ad20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f88ec006fd0 tx=0x7f88ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:05.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.236+0000 7f88eb7fe700 1 -- 172.21.15.78:0/3730155700 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f88f0029770 con 0x7f88fc0fef80 2023-10-10T17:35:05.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.430+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88fc101700 con 0x7f88e4038870 2023-10-10T17:35:05.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.431+0000 7f88eb7fe700 1 -- 172.21.15.78:0/3730155700 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f88fc101700 con 0x7f88e4038870 2023-10-10T17:35:05.433 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:35:05.433 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.157", "hostname": "smithi157", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2023-10-10T17:35:05.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.433+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f88e4038870 msgr2=0x7f88e403ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:05.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.433+0000 7f8900f27700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f88e4038870 0x7f88e403ad20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f88ec006fd0 tx=0x7f88ec006e40 comp rx=0 tx=0).stop 2023-10-10T17:35:05.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.433+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 msgr2=0x7f88fc100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:05.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.433+0000 7f8900f27700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc100990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f88f0004f40 tx=0x7f88f0005020 comp rx=0 tx=0).stop 2023-10-10T17:35:05.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.434+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 shutdown_connections 2023-10-10T17:35:05.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.434+0000 7f8900f27700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f88e4038870 0x7f88e403ad20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:05.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.434+0000 7f8900f27700 1 --2- 172.21.15.78:0/3730155700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f88fc0fef80 0x7f88fc100990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:05.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.434+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 >> 172.21.15.78:0/3730155700 conn(0x7f88fc0fa880 msgr2=0x7f88fc068580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:05.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.434+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 shutdown_connections 2023-10-10T17:35:05.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:05.434+0000 7f8900f27700 1 -- 172.21.15.78:0/3730155700 wait complete. 2023-10-10T17:35:05.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:05 smithi078 ceph-mon[99494]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:05.965 INFO:tasks.cephadm:Setting crush tunables to default 2023-10-10T17:35:05.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd crush tunables default 2023-10-10T17:35:06.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:06.354 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:06 smithi078 ceph-mon[99494]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:35:06.606 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:35:07.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:07 smithi078 ceph-mon[99494]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.809+0000 7fe611830700 1 -- 172.21.15.78:0/1513362981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 msgr2=0x7fe60c100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.809+0000 7fe611830700 1 --2- 172.21.15.78:0/1513362981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c100250 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009910 tx=0x7fe5fc009c20 comp rx=0 tx=0).stop 2023-10-10T17:35:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.809+0000 7fe611830700 1 -- 172.21.15.78:0/1513362981 shutdown_connections 2023-10-10T17:35:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.809+0000 7fe611830700 1 --2- 172.21.15.78:0/1513362981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c100250 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.809+0000 7fe611830700 1 -- 172.21.15.78:0/1513362981 >> 172.21.15.78:0/1513362981 conn(0x7fe60c0763d0 msgr2=0x7fe60c0767d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.809+0000 7fe611830700 1 -- 172.21.15.78:0/1513362981 shutdown_connections 2023-10-10T17:35:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.809+0000 7fe611830700 1 -- 172.21.15.78:0/1513362981 wait complete. 2023-10-10T17:35:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.810+0000 7fe611830700 1 Processor -- start 2023-10-10T17:35:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.810+0000 7fe611830700 1 -- start start 2023-10-10T17:35:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.810+0000 7fe611830700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c194050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.810+0000 7fe611830700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c194590 con 0x7fe60c0ffe80 2023-10-10T17:35:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.810+0000 7fe60affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c194050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.810+0000 7fe60affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c194050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41234/0 (socket says 172.21.15.78:41234) 2023-10-10T17:35:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.810+0000 7fe60affd700 1 -- 172.21.15.78:0/1160213321 learned_addr learned my addr 172.21.15.78:0/1160213321 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:35:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.811+0000 7fe60affd700 1 -- 172.21.15.78:0/1160213321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0095f0 con 0x7fe60c0ffe80 2023-10-10T17:35:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.811+0000 7fe60affd700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c194050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc004f40 tx=0x7fe5fc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.811+0000 7fe61082e700 1 -- 172.21.15.78:0/1160213321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5fc01d070 con 0x7fe60c0ffe80 2023-10-10T17:35:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.811+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c109110 con 0x7fe60c0ffe80 2023-10-10T17:35:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.811+0000 7fe61082e700 1 -- 172.21.15.78:0/1160213321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5fc004370 con 0x7fe60c0ffe80 2023-10-10T17:35:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.811+0000 7fe61082e700 1 -- 172.21.15.78:0/1160213321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5fc0039b0 con 0x7fe60c0ffe80 2023-10-10T17:35:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.811+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c108f80 con 0x7fe60c0ffe80 2023-10-10T17:35:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.812+0000 7fe61082e700 1 -- 172.21.15.78:0/1160213321 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fe5fc003c70 con 0x7fe60c0ffe80 2023-10-10T17:35:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.812+0000 7fe61082e700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe5f4038870 0x7fe5f403ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.812+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60c1092a0 con 0x7fe60c0ffe80 2023-10-10T17:35:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.812+0000 7fe61082e700 1 -- 172.21.15.78:0/1160213321 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe5fc04c7a0 con 0x7fe60c0ffe80 2023-10-10T17:35:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.812+0000 7fe60a7fc700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe5f4038870 0x7fe5f403ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.812+0000 7fe60a7fc700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe5f4038870 0x7fe5f403ad20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe600006fd0 tx=0x7fe600006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:07.817+0000 7fe61082e700 1 -- 172.21.15.78:0/1160213321 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe5fc026020 con 0x7fe60c0ffe80 2023-10-10T17:35:08.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.008+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe60c194b50 con 0x7fe60c0ffe80 2023-10-10T17:35:08.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.358+0000 7fe61082e700 1 -- 172.21.15.78:0/1160213321 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe5fc02b440 con 0x7fe60c0ffe80 2023-10-10T17:35:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe5f4038870 msgr2=0x7fe5f403ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:08.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe5f4038870 0x7fe5f403ad20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe600006fd0 tx=0x7fe600006e40 comp rx=0 tx=0).stop 2023-10-10T17:35:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 msgr2=0x7fe60c194050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c194050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc004f40 tx=0x7fe5fc005020 comp rx=0 tx=0).stop 2023-10-10T17:35:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 shutdown_connections 2023-10-10T17:35:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe5f4038870 0x7fe5f403ad20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:08.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 --2- 172.21.15.78:0/1160213321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffe80 0x7fe60c194050 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 >> 172.21.15.78:0/1160213321 conn(0x7fe60c0763d0 msgr2=0x7fe60c103730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 shutdown_connections 2023-10-10T17:35:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:08.360+0000 7fe611830700 1 -- 172.21.15.78:0/1160213321 wait complete. 2023-10-10T17:35:08.364 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2023-10-10T17:35:08.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1160213321' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-10T17:35:08.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:08.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:08.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:08.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:08.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:35:08.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:08.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:09.061 INFO:tasks.cephadm:Adding mon.a on smithi078 2023-10-10T17:35:09.061 INFO:tasks.cephadm:Adding mon.b on smithi157 2023-10-10T17:35:09.062 INFO:tasks.cephadm:Adding mon.c on smithi190 2023-10-10T17:35:09.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c' 2023-10-10T17:35:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:35:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:09.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1160213321' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-10T17:35:09.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:09.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:35:09.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:09.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:09.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:09 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:09.723 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:10.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:10 smithi078 ceph-mon[99494]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:35:10.923 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.921+0000 7f8cc9e23700 1 -- 172.21.15.190:0/140658010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 msgr2=0x7f8cc4100360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:10.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.921+0000 7f8cc9e23700 1 --2- 172.21.15.190:0/140658010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc4100360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4009910 tx=0x7f8cb4009c20 comp rx=0 tx=0).stop 2023-10-10T17:35:10.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.921+0000 7f8cc9e23700 1 -- 172.21.15.190:0/140658010 shutdown_connections 2023-10-10T17:35:10.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.921+0000 7f8cc9e23700 1 --2- 172.21.15.190:0/140658010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc4100360 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:10.924 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.921+0000 7f8cc9e23700 1 -- 172.21.15.190:0/140658010 >> 172.21.15.190:0/140658010 conn(0x7f8cc40fb850 msgr2=0x7f8cc40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:10.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.921+0000 7f8cc9e23700 1 -- 172.21.15.190:0/140658010 shutdown_connections 2023-10-10T17:35:10.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.921+0000 7f8cc9e23700 1 -- 172.21.15.190:0/140658010 wait complete. 2023-10-10T17:35:10.925 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.922+0000 7f8cc9e23700 1 Processor -- start 2023-10-10T17:35:10.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.922+0000 7f8cc9e23700 1 -- start start 2023-10-10T17:35:10.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.922+0000 7f8cc9e23700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc41951c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:10.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.922+0000 7f8cc9e23700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4195700 con 0x7f8cc40fff90 2023-10-10T17:35:10.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.922+0000 7f8cc37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc41951c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:10.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.922+0000 7f8cc37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc41951c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.190:46656/0 (socket says 172.21.15.190:46656) 2023-10-10T17:35:10.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.922+0000 7f8cc37fe700 1 -- 172.21.15.190:0/2597933139 learned_addr learned my addr 172.21.15.190:0/2597933139 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:35:10.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.923+0000 7f8cc37fe700 1 -- 172.21.15.190:0/2597933139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb40095f0 con 0x7f8cc40fff90 2023-10-10T17:35:10.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.923+0000 7f8cc37fe700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc41951c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4004f40 tx=0x7f8cb4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:10.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.923+0000 7f8cc0ff9700 1 -- 172.21.15.190:0/2597933139 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8cb401d070 con 0x7f8cc40fff90 2023-10-10T17:35:10.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.923+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc410a170 con 0x7f8cc40fff90 2023-10-10T17:35:10.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.923+0000 7f8cc0ff9700 1 -- 172.21.15.190:0/2597933139 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8cb4004370 con 0x7f8cc40fff90 2023-10-10T17:35:10.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.923+0000 7f8cc0ff9700 1 -- 172.21.15.190:0/2597933139 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8cb40039b0 con 0x7f8cc40fff90 2023-10-10T17:35:10.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.923+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc4109fe0 con 0x7f8cc40fff90 2023-10-10T17:35:10.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.924+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc410a300 con 0x7f8cc40fff90 2023-10-10T17:35:10.929 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.924+0000 7f8cc0ff9700 1 -- 172.21.15.190:0/2597933139 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8cb4003c70 con 0x7f8cc40fff90 2023-10-10T17:35:10.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.924+0000 7f8cc0ff9700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8cac038870 0x7f8cac03ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:10.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.924+0000 7f8cc0ff9700 1 -- 172.21.15.190:0/2597933139 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8cb404c960 con 0x7f8cc40fff90 2023-10-10T17:35:10.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.925+0000 7f8cc2ffd700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8cac038870 0x7f8cac03ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:10.930 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.925+0000 7f8cc2ffd700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8cac038870 0x7f8cac03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8006fd0 tx=0x7f8cb8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:10.931 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:10.930+0000 7f8cc0ff9700 1 -- 172.21.15.190:0/2597933139 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8cb4029360 con 0x7f8cc40fff90 2023-10-10T17:35:11.127 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.125+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cc4108190 con 0x7f8cac038870 2023-10-10T17:35:11.133 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.132+0000 7f8cc0ff9700 1 -- 172.21.15.190:0/2597933139 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f8cc4108190 con 0x7f8cac038870 2023-10-10T17:35:11.134 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2023-10-10T17:35:11.136 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.133+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8cac038870 msgr2=0x7f8cac03ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:11.136 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8cac038870 0x7f8cac03ad20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8006fd0 tx=0x7f8cb8006e40 comp rx=0 tx=0).stop 2023-10-10T17:35:11.136 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 msgr2=0x7f8cc41951c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:11.136 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc41951c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8cb4004f40 tx=0x7f8cb4005020 comp rx=0 tx=0).stop 2023-10-10T17:35:11.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 shutdown_connections 2023-10-10T17:35:11.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8cac038870 0x7f8cac03ad20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:11.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 --2- 172.21.15.190:0/2597933139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8cc40fff90 0x7f8cc41951c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:11.137 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 >> 172.21.15.190:0/2597933139 conn(0x7f8cc40fb850 msgr2=0x7f8cc4104170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:11.138 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 shutdown_connections 2023-10-10T17:35:11.138 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:11.134+0000 7f8cc9e23700 1 -- 172.21.15.190:0/2597933139 wait complete. 2023-10-10T17:35:11.601 DEBUG:teuthology.orchestra.run.smithi157:mon.b> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.b.service 2023-10-10T17:35:11.604 DEBUG:teuthology.orchestra.run.smithi190:mon.c> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.c.service 2023-10-10T17:35:11.606 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-10T17:35:11.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-10T17:35:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:11.636 INFO:journalctl@ceph.mon.c.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:35:11.639 INFO:journalctl@ceph.mon.b.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:35:12.475 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:12 smithi078 ceph-mon[99494]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:12 smithi078 ceph-mon[99494]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c;count:3 2023-10-10T17:35:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:12 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:12 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:12 smithi078 ceph-mon[99494]: Deploying daemon mon.c on smithi190 2023-10-10T17:35:13.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:13 smithi078 ceph-mon[99494]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:14.139 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.135+0000 7f576ab7e700 1 -- 172.21.15.190:0/1757878216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 msgr2=0x7f57640feec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:14.139 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.135+0000 7f576ab7e700 1 --2- 172.21.15.190:0/1757878216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57640feec0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5758009910 tx=0x7f5758009c20 comp rx=0 tx=0).stop 2023-10-10T17:35:14.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.135+0000 7f576ab7e700 1 -- 172.21.15.190:0/1757878216 shutdown_connections 2023-10-10T17:35:14.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.135+0000 7f576ab7e700 1 --2- 172.21.15.190:0/1757878216 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57640feec0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:14.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.135+0000 7f576ab7e700 1 -- 172.21.15.190:0/1757878216 >> 172.21.15.190:0/1757878216 conn(0x7f57640fa760 msgr2=0x7f57640fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:14.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.135+0000 7f576ab7e700 1 -- 172.21.15.190:0/1757878216 shutdown_connections 2023-10-10T17:35:14.140 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.135+0000 7f576ab7e700 1 -- 172.21.15.190:0/1757878216 wait complete. 2023-10-10T17:35:14.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.136+0000 7f576ab7e700 1 Processor -- start 2023-10-10T17:35:14.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.136+0000 7f576ab7e700 1 -- start start 2023-10-10T17:35:14.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.136+0000 7f576ab7e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57641973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:14.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.136+0000 7f576ab7e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57641978f0 con 0x7f57640feaf0 2023-10-10T17:35:14.141 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.136+0000 7f576891a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57641973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:14.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.136+0000 7f576891a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57641973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.190:46662/0 (socket says 172.21.15.190:46662) 2023-10-10T17:35:14.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.136+0000 7f576891a700 1 -- 172.21.15.190:0/3250204155 learned_addr learned my addr 172.21.15.190:0/3250204155 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:35:14.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.137+0000 7f576891a700 1 -- 172.21.15.190:0/3250204155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580095f0 con 0x7f57640feaf0 2023-10-10T17:35:14.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.137+0000 7f576891a700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57641973b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5758004f40 tx=0x7f5758005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:14.142 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.137+0000 7f5761ffb700 1 -- 172.21.15.190:0/3250204155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f575801d070 con 0x7f57640feaf0 2023-10-10T17:35:14.143 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.137+0000 7f5761ffb700 1 -- 172.21.15.190:0/3250204155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5758004370 con 0x7f57640feaf0 2023-10-10T17:35:14.143 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.137+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f576410c360 con 0x7f57640feaf0 2023-10-10T17:35:14.143 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.137+0000 7f5761ffb700 1 -- 172.21.15.190:0/3250204155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57580039b0 con 0x7f57640feaf0 2023-10-10T17:35:14.144 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.137+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f576410c1d0 con 0x7f57640feaf0 2023-10-10T17:35:14.144 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.138+0000 7f5761ffb700 1 -- 172.21.15.190:0/3250204155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f5758003c70 con 0x7f57640feaf0 2023-10-10T17:35:14.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.138+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f576410c4f0 con 0x7f57640feaf0 2023-10-10T17:35:14.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.138+0000 7f5761ffb700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5754038870 0x7f575403ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:14.145 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.138+0000 7f5761ffb700 1 -- 172.21.15.190:0/3250204155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f575804c620 con 0x7f57640feaf0 2023-10-10T17:35:14.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.139+0000 7f5763fff700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5754038870 0x7f575403ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:14.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.139+0000 7f5763fff700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5754038870 0x7f575403ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5750006fd0 tx=0x7f5750006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:14.146 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.144+0000 7f5761ffb700 1 -- 172.21.15.190:0/3250204155 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5758029360 con 0x7f57640feaf0 2023-10-10T17:35:14.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:13 smithi190 systemd[1]: Starting Ceph mon.c for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:35:14.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.423+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5764062120 con 0x7f57640feaf0 2023-10-10T17:35:14.425 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.423+0000 7f5761ffb700 1 -- 172.21.15.190:0/3250204155 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f5758026030 con 0x7f57640feaf0 2023-10-10T17:35:14.426 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:35:14.426 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","modified":"2023-10-10T17:33:39.867990Z","created":"2023-10-10T17:33:39.867990Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-10T17:35:14.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5754038870 msgr2=0x7f575403ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:14.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5754038870 0x7f575403ad20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5750006fd0 tx=0x7f5750006e40 comp rx=0 tx=0).stop 2023-10-10T17:35:14.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 msgr2=0x7f57641973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:14.428 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57641973b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5758004f40 tx=0x7f5758005020 comp rx=0 tx=0).stop 2023-10-10T17:35:14.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 shutdown_connections 2023-10-10T17:35:14.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5754038870 0x7f575403ad20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:14.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 --2- 172.21.15.190:0/3250204155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57640feaf0 0x7f57641973b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:14.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 >> 172.21.15.190:0/3250204155 conn(0x7f57640fa760 msgr2=0x7f57640fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:14.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 shutdown_connections 2023-10-10T17:35:14.430 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:14.426+0000 7f576ab7e700 1 -- 172.21.15.190:0/3250204155 wait complete. 2023-10-10T17:35:14.430 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2023-10-10T17:35:14.561 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 podman[101159]: 2023-10-10T17:35:14.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:14 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/3250204155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:14.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 bash[101159]: 8cb7d5cc38e1d71eff97e628c8f4c011953f418856a43cb5e5fbbe2c075f742f 2023-10-10T17:35:14.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: set uid:gid to 167:167 (ceph:ceph) 2023-10-10T17:35:14.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev), process ceph-mon, pid 7 2023-10-10T17:35:14.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: pidfile_write: ignore empty --pid-file 2023-10-10T17:35:14.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: load: jerasure load: lrc 2023-10-10T17:35:14.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: RocksDB version: 7.9.2 2023-10-10T17:35:14.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Git sha 0 2023-10-10T17:35:14.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Compile date 2023-09-28 19:19:12 2023-10-10T17:35:14.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: DB SUMMARY 2023-10-10T17:35:14.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: DB Session ID: OEK17DFCXHXXJ0T579SK 2023-10-10T17:35:14.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: CURRENT file: CURRENT 2023-10-10T17:35:14.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: IDENTITY file: IDENTITY 2023-10-10T17:35:14.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-10-10T17:35:14.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-10-10T17:35:14.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-10-10T17:35:14.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.error_if_exists: 0 2023-10-10T17:35:14.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.create_if_missing: 0 2023-10-10T17:35:14.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.paranoid_checks: 1 2023-10-10T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-10T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-10T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-10T17:35:14.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.env: 0x5617bb7a1720 2023-10-10T17:35:14.922 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.fs: PosixFileSystem 2023-10-10T17:35:14.922 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.info_log: 0x5617be3c13c0 2023-10-10T17:35:14.922 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_file_opening_threads: 16 2023-10-10T17:35:14.922 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.statistics: (nil) 2023-10-10T17:35:14.923 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.use_fsync: 0 2023-10-10T17:35:14.923 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_log_file_size: 0 2023-10-10T17:35:14.923 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-10T17:35:14.923 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-10T17:35:14.923 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.keep_log_file_num: 1000 2023-10-10T17:35:14.923 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.recycle_log_file_num: 0 2023-10-10T17:35:14.924 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.allow_fallocate: 1 2023-10-10T17:35:14.924 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.allow_mmap_reads: 0 2023-10-10T17:35:14.924 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.allow_mmap_writes: 0 2023-10-10T17:35:14.924 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.use_direct_reads: 0 2023-10-10T17:35:14.924 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-10T17:35:14.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.create_missing_column_families: 0 2023-10-10T17:35:14.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.db_log_dir: 2023-10-10T17:35:14.925 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.wal_dir: 2023-10-10T17:35:14.927 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-10T17:35:14.927 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-10T17:35:14.927 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-10T17:35:14.927 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-10T17:35:14.928 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-10T17:35:14.928 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-10T17:35:14.928 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.advise_random_on_open: 1 2023-10-10T17:35:14.928 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.db_write_buffer_size: 0 2023-10-10T17:35:14.928 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.write_buffer_manager: 0x5617bd6323c0 2023-10-10T17:35:14.929 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-10T17:35:14.929 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-10T17:35:14.930 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-10T17:35:14.930 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.rate_limiter: (nil) 2023-10-10T17:35:14.930 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-10T17:35:14.930 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.wal_recovery_mode: 2 2023-10-10T17:35:14.931 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.enable_thread_tracking: 0 2023-10-10T17:35:14.931 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.enable_pipelined_write: 0 2023-10-10T17:35:14.931 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.unordered_write: 0 2023-10-10T17:35:14.931 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-10T17:35:14.932 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-10T17:35:14.932 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-10T17:35:14.932 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-10T17:35:14.932 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.row_cache: None 2023-10-10T17:35:14.933 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.wal_filter: None 2023-10-10T17:35:14.933 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-10T17:35:14.933 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.allow_ingest_behind: 0 2023-10-10T17:35:14.933 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.two_write_queues: 0 2023-10-10T17:35:14.933 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.manual_wal_flush: 0 2023-10-10T17:35:14.933 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.wal_compression: 0 2023-10-10T17:35:14.934 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.atomic_flush: 0 2023-10-10T17:35:14.934 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-10T17:35:14.934 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-10T17:35:14.934 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-10T17:35:14.934 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.log_readahead_size: 0 2023-10-10T17:35:14.935 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-10T17:35:14.935 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.best_efforts_recovery: 0 2023-10-10T17:35:14.935 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-10T17:35:14.935 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-10T17:35:14.935 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.allow_data_in_errors: 0 2023-10-10T17:35:14.936 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.db_host_id: __hostname__ 2023-10-10T17:35:14.936 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-10T17:35:14.936 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_background_jobs: 2 2023-10-10T17:35:14.936 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_background_compactions: -1 2023-10-10T17:35:14.936 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_subcompactions: 1 2023-10-10T17:35:14.937 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-10T17:35:14.937 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-10T17:35:14.937 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-10T17:35:14.937 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_total_wal_size: 0 2023-10-10T17:35:14.937 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-10T17:35:14.937 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-10T17:35:14.938 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-10T17:35:14.938 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-10T17:35:14.938 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_open_files: -1 2023-10-10T17:35:14.938 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bytes_per_sync: 0 2023-10-10T17:35:14.938 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-10T17:35:14.939 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-10T17:35:14.939 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_readahead_size: 0 2023-10-10T17:35:14.939 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_background_flushes: -1 2023-10-10T17:35:14.939 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Compression algorithms supported: 2023-10-10T17:35:14.939 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-10T17:35:14.940 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kZSTD supported: 0 2023-10-10T17:35:14.940 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kXpressCompression supported: 0 2023-10-10T17:35:14.940 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kLZ4HCCompression supported: 1 2023-10-10T17:35:14.940 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kZlibCompression supported: 1 2023-10-10T17:35:14.940 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kSnappyCompression supported: 1 2023-10-10T17:35:14.941 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kLZ4Compression supported: 1 2023-10-10T17:35:14.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: kBZip2Compression supported: 0 2023-10-10T17:35:14.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-10T17:35:14.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-10T17:35:14.942 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-10-10T17:35:14.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-10T17:35:14.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-10T17:35:14.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.merge_operator: 2023-10-10T17:35:14.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_filter: None 2023-10-10T17:35:14.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_filter_factory: None 2023-10-10T17:35:14.943 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.sst_partitioner_factory: None 2023-10-10T17:35:14.944 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-10T17:35:14.944 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-10T17:35:14.944 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5617be3c14e0) 2023-10-10T17:35:14.944 INFO:journalctl@ceph.mon.c.smithi190.stdout: cache_index_and_filter_blocks: 1 2023-10-10T17:35:14.944 INFO:journalctl@ceph.mon.c.smithi190.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-10T17:35:14.945 INFO:journalctl@ceph.mon.c.smithi190.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-10T17:35:14.945 INFO:journalctl@ceph.mon.c.smithi190.stdout: pin_top_level_index_and_filter: 1 2023-10-10T17:35:14.945 INFO:journalctl@ceph.mon.c.smithi190.stdout: index_type: 0 2023-10-10T17:35:14.945 INFO:journalctl@ceph.mon.c.smithi190.stdout: data_block_index_type: 0 2023-10-10T17:35:14.945 INFO:journalctl@ceph.mon.c.smithi190.stdout: index_shortening: 1 2023-10-10T17:35:14.946 INFO:journalctl@ceph.mon.c.smithi190.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-10T17:35:14.946 INFO:journalctl@ceph.mon.c.smithi190.stdout: checksum: 4 2023-10-10T17:35:14.946 INFO:journalctl@ceph.mon.c.smithi190.stdout: no_block_cache: 0 2023-10-10T17:35:14.946 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_cache: 0x5617bd6d31f0 2023-10-10T17:35:14.946 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_cache_name: BinnedLRUCache 2023-10-10T17:35:14.947 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_cache_options: 2023-10-10T17:35:14.947 INFO:journalctl@ceph.mon.c.smithi190.stdout: capacity : 536870912 2023-10-10T17:35:14.947 INFO:journalctl@ceph.mon.c.smithi190.stdout: num_shard_bits : 4 2023-10-10T17:35:14.947 INFO:journalctl@ceph.mon.c.smithi190.stdout: strict_capacity_limit : 0 2023-10-10T17:35:14.947 INFO:journalctl@ceph.mon.c.smithi190.stdout: high_pri_pool_ratio: 0.000 2023-10-10T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_cache_compressed: (nil) 2023-10-10T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi190.stdout: persistent_cache: (nil) 2023-10-10T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_size: 4096 2023-10-10T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_size_deviation: 10 2023-10-10T17:35:14.948 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_restart_interval: 16 2023-10-10T17:35:14.949 INFO:journalctl@ceph.mon.c.smithi190.stdout: index_block_restart_interval: 1 2023-10-10T17:35:14.949 INFO:journalctl@ceph.mon.c.smithi190.stdout: metadata_block_size: 4096 2023-10-10T17:35:14.949 INFO:journalctl@ceph.mon.c.smithi190.stdout: partition_filters: 0 2023-10-10T17:35:14.949 INFO:journalctl@ceph.mon.c.smithi190.stdout: use_delta_encoding: 1 2023-10-10T17:35:14.949 INFO:journalctl@ceph.mon.c.smithi190.stdout: filter_policy: bloomfilter 2023-10-10T17:35:14.949 INFO:journalctl@ceph.mon.c.smithi190.stdout: whole_key_filtering: 1 2023-10-10T17:35:14.950 INFO:journalctl@ceph.mon.c.smithi190.stdout: verify_compression: 0 2023-10-10T17:35:14.950 INFO:journalctl@ceph.mon.c.smithi190.stdout: read_amp_bytes_per_bit: 0 2023-10-10T17:35:14.950 INFO:journalctl@ceph.mon.c.smithi190.stdout: format_version: 5 2023-10-10T17:35:14.950 INFO:journalctl@ceph.mon.c.smithi190.stdout: enable_index_compression: 1 2023-10-10T17:35:14.951 INFO:journalctl@ceph.mon.c.smithi190.stdout: block_align: 0 2023-10-10T17:35:14.951 INFO:journalctl@ceph.mon.c.smithi190.stdout: max_auto_readahead_size: 262144 2023-10-10T17:35:14.951 INFO:journalctl@ceph.mon.c.smithi190.stdout: prepopulate_block_cache: 0 2023-10-10T17:35:14.951 INFO:journalctl@ceph.mon.c.smithi190.stdout: initial_auto_readahead_size: 8192 2023-10-10T17:35:14.951 INFO:journalctl@ceph.mon.c.smithi190.stdout: num_file_reads_for_auto_readahead: 2 2023-10-10T17:35:14.952 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.write_buffer_size: 33554432 2023-10-10T17:35:14.952 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_write_buffer_number: 2 2023-10-10T17:35:14.952 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression: NoCompression 2023-10-10T17:35:14.952 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression: Disabled 2023-10-10T17:35:14.952 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.prefix_extractor: nullptr 2023-10-10T17:35:14.953 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-10T17:35:14.953 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.num_levels: 7 2023-10-10T17:35:14.953 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-10T17:35:14.953 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-10T17:35:14.953 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-10T17:35:14.954 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-10T17:35:14.954 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-10T17:35:14.954 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-10T17:35:14.954 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-10T17:35:14.954 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-10T17:35:14.955 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-10T17:35:14.955 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-10T17:35:14.955 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-10T17:35:14.955 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-10T17:35:14.955 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-10T17:35:14.956 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.level: 32767 2023-10-10T17:35:14.956 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.strategy: 0 2023-10-10T17:35:14.956 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-10T17:35:14.956 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-10T17:35:14.956 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-10T17:35:14.957 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-10T17:35:14.957 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.enabled: false 2023-10-10T17:35:14.957 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-10T17:35:14.957 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-10T17:35:14.957 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-10T17:35:14.958 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-10T17:35:14.958 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.target_file_size_base: 67108864 2023-10-10T17:35:14.958 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-10T17:35:14.958 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-10T17:35:14.959 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-10T17:35:14.959 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-10T17:35:14.959 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-10T17:35:14.959 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-10T17:35:14.959 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-10T17:35:14.960 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-10T17:35:14.960 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-10T17:35:14.960 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-10T17:35:14.960 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-10T17:35:14.961 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-10T17:35:14.961 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-10T17:35:14.961 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-10T17:35:14.961 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.arena_block_size: 1048576 2023-10-10T17:35:14.961 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-10T17:35:14.962 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-10T17:35:14.962 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.disable_auto_compactions: 0 2023-10-10T17:35:14.962 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-10T17:35:14.962 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-10T17:35:14.962 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-10T17:35:14.963 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-10T17:35:14.963 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-10T17:35:14.963 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-10T17:35:14.963 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-10T17:35:14.963 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-10T17:35:14.963 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-10T17:35:14.964 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-10T17:35:14.964 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.table_properties_collectors: 2023-10-10T17:35:14.964 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.inplace_update_support: 0 2023-10-10T17:35:14.964 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-10T17:35:14.964 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-10T17:35:14.965 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-10T17:35:14.965 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-10T17:35:14.965 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.bloom_locality: 0 2023-10-10T17:35:14.965 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.max_successive_merges: 0 2023-10-10T17:35:14.965 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-10T17:35:14.966 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.paranoid_file_checks: 0 2023-10-10T17:35:14.966 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.force_consistency_checks: 1 2023-10-10T17:35:14.966 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.report_bg_io_stats: 0 2023-10-10T17:35:14.966 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.ttl: 2592000 2023-10-10T17:35:14.966 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-10T17:35:14.967 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-10T17:35:14.967 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-10T17:35:14.967 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.enable_blob_files: false 2023-10-10T17:35:14.967 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.min_blob_size: 0 2023-10-10T17:35:14.967 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.blob_file_size: 268435456 2023-10-10T17:35:14.967 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-10T17:35:14.968 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-10T17:35:14.968 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-10T17:35:14.968 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-10T17:35:14.968 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-10T17:35:14.968 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.blob_file_starting_level: 0 2023-10-10T17:35:14.969 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-10T17:35:14.969 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-10-10T17:35:14.969 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-10-10T17:35:14.969 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2792740f-3eb8-4a9d-8836-19be2efa20f9 2023-10-10T17:35:14.969 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959314568269, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-10-10T17:35:14.970 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-10-10T17:35:14.970 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959314568732, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1696959314, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2792740f-3eb8-4a9d-8836-19be2efa20f9", "db_session_id": "OEK17DFCXHXXJ0T579SK", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-10-10T17:35:14.970 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959314568822, "job": 1, "event": "recovery_finished"} 2023-10-10T17:35:14.970 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-10-10T17:35:14.970 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 systemd[1]: Started Ceph mon.c for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:35:14.971 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-10T17:35:14.971 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5617bd770000 2023-10-10T17:35:14.971 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: DB pointer 0x5617bd75c000 2023-10-10T17:35:14.971 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-10T17:35:14.971 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-10T17:35:14.971 INFO:journalctl@ceph.mon.c.smithi190.stdout: ** DB Stats ** 2023-10-10T17:35:14.972 INFO:journalctl@ceph.mon.c.smithi190.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-10T17:35:14.972 INFO:journalctl@ceph.mon.c.smithi190.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-10T17:35:14.972 INFO:journalctl@ceph.mon.c.smithi190.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-10T17:35:14.972 INFO:journalctl@ceph.mon.c.smithi190.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-10T17:35:14.972 INFO:journalctl@ceph.mon.c.smithi190.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-10T17:35:14.973 INFO:journalctl@ceph.mon.c.smithi190.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-10T17:35:14.973 INFO:journalctl@ceph.mon.c.smithi190.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-10T17:35:14.973 INFO:journalctl@ceph.mon.c.smithi190.stdout: 2023-10-10T17:35:14.973 INFO:journalctl@ceph.mon.c.smithi190.stdout: ** Compaction Stats [default] ** 2023-10-10T17:35:14.973 INFO:journalctl@ceph.mon.c.smithi190.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-10T17:35:14.974 INFO:journalctl@ceph.mon.c.smithi190.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-10T17:35:14.974 INFO:journalctl@ceph.mon.c.smithi190.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-10T17:35:14.974 INFO:journalctl@ceph.mon.c.smithi190.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-10T17:35:14.974 INFO:journalctl@ceph.mon.c.smithi190.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-10T17:35:14.974 INFO:journalctl@ceph.mon.c.smithi190.stdout: 2023-10-10T17:35:14.975 INFO:journalctl@ceph.mon.c.smithi190.stdout: ** Compaction Stats [default] ** 2023-10-10T17:35:14.975 INFO:journalctl@ceph.mon.c.smithi190.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-10T17:35:14.975 INFO:journalctl@ceph.mon.c.smithi190.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-10T17:35:14.975 INFO:journalctl@ceph.mon.c.smithi190.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-10T17:35:14.975 INFO:journalctl@ceph.mon.c.smithi190.stdout: 2023-10-10T17:35:14.976 INFO:journalctl@ceph.mon.c.smithi190.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-10T17:35:14.976 INFO:journalctl@ceph.mon.c.smithi190.stdout: 2023-10-10T17:35:14.976 INFO:journalctl@ceph.mon.c.smithi190.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-10T17:35:14.976 INFO:journalctl@ceph.mon.c.smithi190.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-10T17:35:14.976 INFO:journalctl@ceph.mon.c.smithi190.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-10T17:35:14.977 INFO:journalctl@ceph.mon.c.smithi190.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-10T17:35:14.977 INFO:journalctl@ceph.mon.c.smithi190.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-10T17:35:14.977 INFO:journalctl@ceph.mon.c.smithi190.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-10T17:35:14.977 INFO:journalctl@ceph.mon.c.smithi190.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-10T17:35:14.977 INFO:journalctl@ceph.mon.c.smithi190.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-10T17:35:14.977 INFO:journalctl@ceph.mon.c.smithi190.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-10T17:35:14.978 INFO:journalctl@ceph.mon.c.smithi190.stdout: Block cache BinnedLRUCache@0x5617bd6d31f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2023-10-10T17:35:14.978 INFO:journalctl@ceph.mon.c.smithi190.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-10-10T17:35:14.978 INFO:journalctl@ceph.mon.c.smithi190.stdout: 2023-10-10T17:35:14.978 INFO:journalctl@ceph.mon.c.smithi190.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-10T17:35:14.979 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-10-10T17:35:14.979 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: using public_addr v2:172.21.15.190:0/0 -> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2023-10-10T17:35:14.979 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: starting mon.c rank -1 at public addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] at bind addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:35:14.979 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(???) e0 preinit fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:35:14.979 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).mds e1 new map 2023-10-10T17:35:14.979 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).mds e1 print_map 2023-10-10T17:35:14.980 INFO:journalctl@ceph.mon.c.smithi190.stdout: e1 2023-10-10T17:35:14.980 INFO:journalctl@ceph.mon.c.smithi190.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-10T17:35:14.980 INFO:journalctl@ceph.mon.c.smithi190.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-10T17:35:14.980 INFO:journalctl@ceph.mon.c.smithi190.stdout: legacy client fscid: -1 2023-10-10T17:35:14.980 INFO:journalctl@ceph.mon.c.smithi190.stdout: 2023-10-10T17:35:14.981 INFO:journalctl@ceph.mon.c.smithi190.stdout: No filesystems configured 2023-10-10T17:35:14.981 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-10-10T17:35:14.981 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-10-10T17:35:14.981 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-10-10T17:35:14.981 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-10-10T17:35:14.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-10-10T17:35:14.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-10-10T17:35:14.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-10T17:35:14.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:35:14.982 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:35:14.983 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:35:14.983 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:35:14.983 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:14.983 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.983 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.983 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:14.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Added host smithi190 2023-10-10T17:35:14.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.984 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.985 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:14.985 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.985 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:35:14.985 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:14.985 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1160213321' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-10T17:35:14.986 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.986 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.986 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.986 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.986 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:35:14.987 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:14.987 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:14.987 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:14.987 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:35:14.987 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:14.987 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:14.988 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1160213321' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-10T17:35:14.988 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:14.988 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:35:14.988 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.988 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.989 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.989 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:35:14.989 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:14.989 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.989 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:14.990 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:14.990 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:14.990 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:14.990 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:14.990 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:14.991 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:14.991 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c;count:3 2023-10-10T17:35:14.991 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:14.991 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:14.991 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: Deploying daemon mon.c on smithi190 2023-10-10T17:35:14.992 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:14.992 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/3250204155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:14.992 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-10-10T17:35:14.992 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-10-10T17:35:14.992 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expand map: {default=false} 2023-10-10T17:35:14.993 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta from 'false' to 'false' 2023-10-10T17:35:14.993 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expanded map: {default=false} 2023-10-10T17:35:14.993 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expand map: {default=info} 2023-10-10T17:35:14.993 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta from 'info' to 'info' 2023-10-10T17:35:14.993 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expanded map: {default=info} 2023-10-10T17:35:14.994 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expand map: {default=daemon} 2023-10-10T17:35:14.994 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-10T17:35:14.994 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expanded map: {default=daemon} 2023-10-10T17:35:14.994 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expand map: {default=debug} 2023-10-10T17:35:14.994 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta from 'debug' to 'debug' 2023-10-10T17:35:14.994 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:14 smithi190 ceph-mon[101220]: expand_channel_meta expanded map: {default=debug} 2023-10-10T17:35:16.086 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-10T17:35:16.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-10T17:35:16.737 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:35:20.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:35:20.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: mon.a calling monitor election 2023-10-10T17:35:20.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:20.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: mon.c calling monitor election 2023-10-10T17:35:20.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:20.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-10T17:35:20.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2023-10-10T17:35:20.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: fsmap 2023-10-10T17:35:20.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:20.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: mgrmap e13: x(active, since 43s) 2023-10-10T17:35:20.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: overall HEALTH_OK 2023-10-10T17:35:20.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:20.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:20.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:20.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:20.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:20.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:35:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: mon.a calling monitor election 2023-10-10T17:35:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:20.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: mon.c calling monitor election 2023-10-10T17:35:20.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:20.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:20.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-10T17:35:20.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2023-10-10T17:35:20.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: fsmap 2023-10-10T17:35:20.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:20.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: mgrmap e13: x(active, since 43s) 2023-10-10T17:35:20.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: overall HEALTH_OK 2023-10-10T17:35:20.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:20.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:20.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:20.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:20.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:20.940 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.939+0000 7f4627c06700 1 -- 172.21.15.190:0/3112416610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 msgr2=0x7f460c007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:20.940 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.939+0000 7f4627c06700 1 --2- 172.21.15.190:0/3112416610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 0x7f460c007c50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f46200fec90 tx=0x7f461000feb0 comp rx=0 tx=0).stop 2023-10-10T17:35:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.939+0000 7f4627c06700 1 -- 172.21.15.190:0/3112416610 shutdown_connections 2023-10-10T17:35:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.939+0000 7f4627c06700 1 --2- 172.21.15.190:0/3112416610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 0x7f460c007c50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.939+0000 7f4627c06700 1 -- 172.21.15.190:0/3112416610 >> 172.21.15.190:0/3112416610 conn(0x7f46200fa8a0 msgr2=0x7f46200fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:20.941 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.939+0000 7f4627c06700 1 -- 172.21.15.190:0/3112416610 shutdown_connections 2023-10-10T17:35:20.942 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.939+0000 7f4627c06700 1 -- 172.21.15.190:0/3112416610 wait complete. 2023-10-10T17:35:20.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.940+0000 7f4627c06700 1 Processor -- start 2023-10-10T17:35:20.944 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.940+0000 7f4627c06700 1 -- start start 2023-10-10T17:35:20.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.940+0000 7f4627c06700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 0x7f46201930f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:20.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.940+0000 7f4627c06700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 0x7f4620197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:20.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.940+0000 7f4627c06700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4620193bb0 con 0x7f460c0057d0 2023-10-10T17:35:20.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.940+0000 7f4627c06700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4620193d20 con 0x7f4620193630 2023-10-10T17:35:20.945 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46251a1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 0x7f4620197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:20.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46251a1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 0x7f4620197aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:51750/0 (socket says 172.21.15.190:51750) 2023-10-10T17:35:20.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46251a1700 1 -- 172.21.15.190:0/3419313770 learned_addr learned my addr 172.21.15.190:0/3419313770 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:35:20.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46259a2700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 0x7f46201930f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:20.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46251a1700 1 -- 172.21.15.190:0/3419313770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 msgr2=0x7f4620197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-10-10T17:35:20.946 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46251a1700 1 -- 172.21.15.190:0/3419313770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 msgr2=0x7f4620197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-10T17:35:20.947 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46251a1700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 0x7f4620197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-10T17:35:20.947 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46251a1700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 0x7f4620197aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:35:20.947 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46259a2700 1 -- 172.21.15.190:0/3419313770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 msgr2=0x7f4620197aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:35:20.947 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46259a2700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 0x7f4620197aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:20.947 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46259a2700 1 -- 172.21.15.190:0/3419313770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461000f900 con 0x7f460c0057d0 2023-10-10T17:35:20.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.941+0000 7f46259a2700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 0x7f46201930f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4610010010 tx=0x7f46100047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:20.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.942+0000 7f4616ffd700 1 -- 172.21.15.190:0/3419313770 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4610013960 con 0x7f460c0057d0 2023-10-10T17:35:20.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.942+0000 7f4616ffd700 1 -- 172.21.15.190:0/3419313770 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4610004030 con 0x7f460c0057d0 2023-10-10T17:35:20.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.942+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4620105bb0 con 0x7f460c0057d0 2023-10-10T17:35:20.948 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.942+0000 7f4616ffd700 1 -- 172.21.15.190:0/3419313770 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4610003a80 con 0x7f460c0057d0 2023-10-10T17:35:20.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.942+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4620069950 con 0x7f460c0057d0 2023-10-10T17:35:20.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.943+0000 7f4616ffd700 1 -- 172.21.15.190:0/3419313770 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f4610003d40 con 0x7f460c0057d0 2023-10-10T17:35:20.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.943+0000 7f4616ffd700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f46040386b0 0x7f460403ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:20.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.943+0000 7f4616ffd700 1 -- 172.21.15.190:0/3419313770 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4610000e40 con 0x7f460c0057d0 2023-10-10T17:35:20.949 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.943+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4620105d40 con 0x7f460c0057d0 2023-10-10T17:35:20.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.943+0000 7f46251a1700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f46040386b0 0x7f460403ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:20.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.944+0000 7f46251a1700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f46040386b0 0x7f460403ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f461c009180 tx=0x7f461c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:20.950 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:20.948+0000 7f4616ffd700 1 -- 172.21.15.190:0/3419313770 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f461001a980 con 0x7f460c0057d0 2023-10-10T17:35:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:20 smithi078 ceph-mon[99494]: Deploying daemon mon.b on smithi157 2023-10-10T17:35:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:20 smithi078 ceph-mon[99494]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:20 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:20 smithi190 ceph-mon[101220]: Deploying daemon mon.b on smithi157 2023-10-10T17:35:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:20 smithi190 ceph-mon[101220]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:21.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:20 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:21.215 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.215+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4620194550 con 0x7f460c0057d0 2023-10-10T17:35:21.216 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.215+0000 7f4616ffd700 1 -- 172.21.15.190:0/3419313770 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f461001a390 con 0x7f460c0057d0 2023-10-10T17:35:21.216 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:35:21.217 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":2,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","modified":"2023-10-10T17:35:14.593573Z","created":"2023-10-10T17:33:39.867990Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-10-10T17:35:21.219 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f46040386b0 msgr2=0x7f460403ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:21.219 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f46040386b0 0x7f460403ab60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f461c009180 tx=0x7f461c008040 comp rx=0 tx=0).stop 2023-10-10T17:35:21.219 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 msgr2=0x7f46201930f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:21.219 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 0x7f46201930f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4610010010 tx=0x7f46100047a0 comp rx=0 tx=0).stop 2023-10-10T17:35:21.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 shutdown_connections 2023-10-10T17:35:21.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f46040386b0 0x7f460403ab60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:21.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f460c0057d0 0x7f46201930f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:21.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 --2- 172.21.15.190:0/3419313770 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4620193630 0x7f4620197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:21.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 >> 172.21.15.190:0/3419313770 conn(0x7f46200fa8a0 msgr2=0x7f4620102910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:21.220 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 shutdown_connections 2023-10-10T17:35:21.221 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:21.218+0000 7f4627c06700 1 -- 172.21.15.190:0/3419313770 wait complete. 2023-10-10T17:35:21.221 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 2 2023-10-10T17:35:21.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:21 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/3419313770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:22.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:21 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/3419313770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:22.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 systemd[1]: Starting Ceph mon.b for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:35:22.698 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-10T17:35:22.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph mon dump -f json 2023-10-10T17:35:22.805 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 podman[100856]: 2023-10-10T17:35:22.805 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 bash[100856]: 7e7ac842ef1398d416ad81c41ef38398e284b5e81d7a4a8bf1ae6a075a8a9ee7 2023-10-10T17:35:22.805 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: set uid:gid to 167:167 (ceph:ceph) 2023-10-10T17:35:22.806 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev), process ceph-mon, pid 7 2023-10-10T17:35:22.806 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pidfile_write: ignore empty --pid-file 2023-10-10T17:35:22.806 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: load: jerasure load: lrc 2023-10-10T17:35:22.806 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: RocksDB version: 7.9.2 2023-10-10T17:35:22.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Git sha 0 2023-10-10T17:35:22.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Compile date 2023-09-28 19:19:12 2023-10-10T17:35:22.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: DB SUMMARY 2023-10-10T17:35:22.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: DB Session ID: YJWG9UPEAKKALA047292 2023-10-10T17:35:22.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: CURRENT file: CURRENT 2023-10-10T17:35:22.807 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: IDENTITY file: IDENTITY 2023-10-10T17:35:22.808 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-10-10T17:35:22.808 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-10-10T17:35:22.808 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-10-10T17:35:22.808 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.error_if_exists: 0 2023-10-10T17:35:22.808 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.create_if_missing: 0 2023-10-10T17:35:22.809 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.paranoid_checks: 1 2023-10-10T17:35:22.809 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-10T17:35:22.809 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-10T17:35:22.809 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-10T17:35:22.809 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.env: 0x555bf7c91720 2023-10-10T17:35:22.810 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.fs: PosixFileSystem 2023-10-10T17:35:22.810 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.info_log: 0x555bf985f3c0 2023-10-10T17:35:22.810 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_file_opening_threads: 16 2023-10-10T17:35:22.810 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.statistics: (nil) 2023-10-10T17:35:22.810 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.use_fsync: 0 2023-10-10T17:35:22.811 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_log_file_size: 0 2023-10-10T17:35:22.811 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-10T17:35:22.811 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-10T17:35:22.811 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.keep_log_file_num: 1000 2023-10-10T17:35:22.811 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.recycle_log_file_num: 0 2023-10-10T17:35:22.812 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.allow_fallocate: 1 2023-10-10T17:35:22.812 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.allow_mmap_reads: 0 2023-10-10T17:35:22.812 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.allow_mmap_writes: 0 2023-10-10T17:35:22.812 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.use_direct_reads: 0 2023-10-10T17:35:22.812 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-10T17:35:22.813 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.create_missing_column_families: 0 2023-10-10T17:35:22.813 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.db_log_dir: 2023-10-10T17:35:22.813 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.wal_dir: 2023-10-10T17:35:22.813 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-10T17:35:22.813 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-10T17:35:22.814 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-10T17:35:22.814 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-10T17:35:22.814 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-10T17:35:22.814 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-10T17:35:22.814 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.advise_random_on_open: 1 2023-10-10T17:35:22.815 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.db_write_buffer_size: 0 2023-10-10T17:35:22.815 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.write_buffer_manager: 0x555bf8ad03c0 2023-10-10T17:35:22.815 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-10T17:35:22.815 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-10T17:35:22.815 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-10T17:35:22.816 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.rate_limiter: (nil) 2023-10-10T17:35:22.816 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-10T17:35:22.816 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.wal_recovery_mode: 2 2023-10-10T17:35:22.816 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.enable_thread_tracking: 0 2023-10-10T17:35:22.816 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.enable_pipelined_write: 0 2023-10-10T17:35:22.817 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.unordered_write: 0 2023-10-10T17:35:22.817 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-10T17:35:22.817 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-10T17:35:22.817 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-10T17:35:22.817 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-10T17:35:22.818 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.row_cache: None 2023-10-10T17:35:22.818 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.wal_filter: None 2023-10-10T17:35:22.818 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-10T17:35:22.818 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.allow_ingest_behind: 0 2023-10-10T17:35:22.818 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.two_write_queues: 0 2023-10-10T17:35:22.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.manual_wal_flush: 0 2023-10-10T17:35:22.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.wal_compression: 0 2023-10-10T17:35:22.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.atomic_flush: 0 2023-10-10T17:35:22.819 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-10T17:35:22.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-10T17:35:22.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-10T17:35:22.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.log_readahead_size: 0 2023-10-10T17:35:22.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-10T17:35:22.820 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.best_efforts_recovery: 0 2023-10-10T17:35:22.821 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-10T17:35:22.821 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-10T17:35:22.822 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.allow_data_in_errors: 0 2023-10-10T17:35:22.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.db_host_id: __hostname__ 2023-10-10T17:35:22.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-10T17:35:22.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_background_jobs: 2 2023-10-10T17:35:22.823 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_background_compactions: -1 2023-10-10T17:35:22.824 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_subcompactions: 1 2023-10-10T17:35:22.824 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-10T17:35:22.824 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-10T17:35:22.824 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-10T17:35:22.824 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_total_wal_size: 0 2023-10-10T17:35:22.825 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-10T17:35:22.825 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-10T17:35:22.825 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-10T17:35:22.825 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-10T17:35:22.825 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_open_files: -1 2023-10-10T17:35:22.826 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bytes_per_sync: 0 2023-10-10T17:35:22.826 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-10T17:35:22.826 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-10T17:35:22.826 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_readahead_size: 0 2023-10-10T17:35:22.826 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_background_flushes: -1 2023-10-10T17:35:22.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Compression algorithms supported: 2023-10-10T17:35:22.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-10T17:35:22.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kZSTD supported: 0 2023-10-10T17:35:22.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kXpressCompression supported: 0 2023-10-10T17:35:22.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kLZ4HCCompression supported: 1 2023-10-10T17:35:22.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kZlibCompression supported: 1 2023-10-10T17:35:22.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kSnappyCompression supported: 1 2023-10-10T17:35:22.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kLZ4Compression supported: 1 2023-10-10T17:35:22.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: kBZip2Compression supported: 0 2023-10-10T17:35:22.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-10T17:35:22.829 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-10T17:35:22.829 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-10-10T17:35:22.829 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-10T17:35:22.829 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-10T17:35:22.829 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.merge_operator: 2023-10-10T17:35:22.830 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_filter: None 2023-10-10T17:35:22.830 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_filter_factory: None 2023-10-10T17:35:22.830 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.sst_partitioner_factory: None 2023-10-10T17:35:22.830 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-10T17:35:22.830 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-10T17:35:22.831 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555bf985f4e0) 2023-10-10T17:35:22.831 INFO:journalctl@ceph.mon.b.smithi157.stdout: cache_index_and_filter_blocks: 1 2023-10-10T17:35:22.831 INFO:journalctl@ceph.mon.b.smithi157.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-10T17:35:22.831 INFO:journalctl@ceph.mon.b.smithi157.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-10T17:35:22.831 INFO:journalctl@ceph.mon.b.smithi157.stdout: pin_top_level_index_and_filter: 1 2023-10-10T17:35:22.832 INFO:journalctl@ceph.mon.b.smithi157.stdout: index_type: 0 2023-10-10T17:35:22.832 INFO:journalctl@ceph.mon.b.smithi157.stdout: data_block_index_type: 0 2023-10-10T17:35:22.832 INFO:journalctl@ceph.mon.b.smithi157.stdout: index_shortening: 1 2023-10-10T17:35:22.832 INFO:journalctl@ceph.mon.b.smithi157.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-10T17:35:22.832 INFO:journalctl@ceph.mon.b.smithi157.stdout: checksum: 4 2023-10-10T17:35:22.833 INFO:journalctl@ceph.mon.b.smithi157.stdout: no_block_cache: 0 2023-10-10T17:35:22.833 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_cache: 0x555bf8b711f0 2023-10-10T17:35:22.833 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_cache_name: BinnedLRUCache 2023-10-10T17:35:22.833 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_cache_options: 2023-10-10T17:35:22.833 INFO:journalctl@ceph.mon.b.smithi157.stdout: capacity : 536870912 2023-10-10T17:35:22.834 INFO:journalctl@ceph.mon.b.smithi157.stdout: num_shard_bits : 4 2023-10-10T17:35:22.834 INFO:journalctl@ceph.mon.b.smithi157.stdout: strict_capacity_limit : 0 2023-10-10T17:35:22.834 INFO:journalctl@ceph.mon.b.smithi157.stdout: high_pri_pool_ratio: 0.000 2023-10-10T17:35:22.834 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_cache_compressed: (nil) 2023-10-10T17:35:22.835 INFO:journalctl@ceph.mon.b.smithi157.stdout: persistent_cache: (nil) 2023-10-10T17:35:22.835 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_size: 4096 2023-10-10T17:35:22.835 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_size_deviation: 10 2023-10-10T17:35:22.835 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_restart_interval: 16 2023-10-10T17:35:22.835 INFO:journalctl@ceph.mon.b.smithi157.stdout: index_block_restart_interval: 1 2023-10-10T17:35:22.835 INFO:journalctl@ceph.mon.b.smithi157.stdout: metadata_block_size: 4096 2023-10-10T17:35:22.836 INFO:journalctl@ceph.mon.b.smithi157.stdout: partition_filters: 0 2023-10-10T17:35:22.836 INFO:journalctl@ceph.mon.b.smithi157.stdout: use_delta_encoding: 1 2023-10-10T17:35:22.836 INFO:journalctl@ceph.mon.b.smithi157.stdout: filter_policy: bloomfilter 2023-10-10T17:35:22.836 INFO:journalctl@ceph.mon.b.smithi157.stdout: whole_key_filtering: 1 2023-10-10T17:35:22.836 INFO:journalctl@ceph.mon.b.smithi157.stdout: verify_compression: 0 2023-10-10T17:35:22.837 INFO:journalctl@ceph.mon.b.smithi157.stdout: read_amp_bytes_per_bit: 0 2023-10-10T17:35:22.837 INFO:journalctl@ceph.mon.b.smithi157.stdout: format_version: 5 2023-10-10T17:35:22.837 INFO:journalctl@ceph.mon.b.smithi157.stdout: enable_index_compression: 1 2023-10-10T17:35:22.837 INFO:journalctl@ceph.mon.b.smithi157.stdout: block_align: 0 2023-10-10T17:35:22.838 INFO:journalctl@ceph.mon.b.smithi157.stdout: max_auto_readahead_size: 262144 2023-10-10T17:35:22.838 INFO:journalctl@ceph.mon.b.smithi157.stdout: prepopulate_block_cache: 0 2023-10-10T17:35:22.838 INFO:journalctl@ceph.mon.b.smithi157.stdout: initial_auto_readahead_size: 8192 2023-10-10T17:35:22.838 INFO:journalctl@ceph.mon.b.smithi157.stdout: num_file_reads_for_auto_readahead: 2 2023-10-10T17:35:22.838 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.write_buffer_size: 33554432 2023-10-10T17:35:22.839 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_write_buffer_number: 2 2023-10-10T17:35:22.839 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression: NoCompression 2023-10-10T17:35:22.839 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression: Disabled 2023-10-10T17:35:22.839 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.prefix_extractor: nullptr 2023-10-10T17:35:22.839 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-10T17:35:22.840 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.num_levels: 7 2023-10-10T17:35:22.840 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-10T17:35:22.840 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-10T17:35:22.840 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-10T17:35:22.840 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-10T17:35:22.841 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-10T17:35:22.841 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-10T17:35:22.841 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-10T17:35:22.841 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-10T17:35:22.841 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-10T17:35:22.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-10T17:35:22.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-10T17:35:22.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-10T17:35:22.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-10T17:35:22.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.level: 32767 2023-10-10T17:35:22.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.strategy: 0 2023-10-10T17:35:22.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-10T17:35:22.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-10T17:35:22.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-10T17:35:22.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-10T17:35:22.844 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.enabled: false 2023-10-10T17:35:22.844 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-10T17:35:22.844 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-10T17:35:22.844 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-10T17:35:22.844 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-10T17:35:22.845 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.target_file_size_base: 67108864 2023-10-10T17:35:22.845 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-10T17:35:22.845 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-10T17:35:22.845 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-10T17:35:22.845 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-10T17:35:22.846 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-10T17:35:22.846 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-10T17:35:22.846 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-10T17:35:22.846 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-10T17:35:22.846 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-10T17:35:22.847 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-10T17:35:22.847 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-10T17:35:22.847 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-10T17:35:22.847 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-10T17:35:22.847 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-10T17:35:22.847 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.arena_block_size: 1048576 2023-10-10T17:35:22.848 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-10T17:35:22.848 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-10T17:35:22.848 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.disable_auto_compactions: 0 2023-10-10T17:35:22.848 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-10T17:35:22.848 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-10T17:35:22.849 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-10T17:35:22.849 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-10T17:35:22.849 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-10T17:35:22.849 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-10T17:35:22.849 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-10T17:35:22.849 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-10T17:35:22.850 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-10T17:35:22.850 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-10T17:35:22.850 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.table_properties_collectors: 2023-10-10T17:35:22.850 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.inplace_update_support: 0 2023-10-10T17:35:22.851 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-10T17:35:22.851 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-10T17:35:22.851 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-10T17:35:22.851 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-10T17:35:22.851 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.bloom_locality: 0 2023-10-10T17:35:22.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.max_successive_merges: 0 2023-10-10T17:35:22.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-10T17:35:22.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.paranoid_file_checks: 0 2023-10-10T17:35:22.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.force_consistency_checks: 1 2023-10-10T17:35:22.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.report_bg_io_stats: 0 2023-10-10T17:35:22.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.ttl: 2592000 2023-10-10T17:35:22.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-10T17:35:22.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-10T17:35:22.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-10T17:35:22.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.enable_blob_files: false 2023-10-10T17:35:22.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.min_blob_size: 0 2023-10-10T17:35:22.854 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.blob_file_size: 268435456 2023-10-10T17:35:22.854 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-10T17:35:22.854 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-10T17:35:22.854 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-10T17:35:22.854 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-10T17:35:22.855 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-10T17:35:22.855 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.blob_file_starting_level: 0 2023-10-10T17:35:22.855 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-10T17:35:22.855 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-10-10T17:35:22.855 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-10-10T17:35:22.856 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2c3d70fd-0b01-4355-9648-93cbff29e39a 2023-10-10T17:35:22.856 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959322735329, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-10-10T17:35:22.856 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-10-10T17:35:22.856 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959322735871, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1696959322, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2c3d70fd-0b01-4355-9648-93cbff29e39a", "db_session_id": "YJWG9UPEAKKALA047292", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-10-10T17:35:22.856 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696959322735958, "job": 1, "event": "recovery_finished"} 2023-10-10T17:35:22.857 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-10-10T17:35:22.857 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-10T17:35:22.857 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555bf8c0e000 2023-10-10T17:35:22.857 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: DB pointer 0x555bf8bfa000 2023-10-10T17:35:22.857 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-10T17:35:22.858 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-10T17:35:22.858 INFO:journalctl@ceph.mon.b.smithi157.stdout: ** DB Stats ** 2023-10-10T17:35:22.858 INFO:journalctl@ceph.mon.b.smithi157.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-10T17:35:22.858 INFO:journalctl@ceph.mon.b.smithi157.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-10T17:35:22.858 INFO:journalctl@ceph.mon.b.smithi157.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-10T17:35:22.859 INFO:journalctl@ceph.mon.b.smithi157.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-10T17:35:22.859 INFO:journalctl@ceph.mon.b.smithi157.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-10T17:35:22.859 INFO:journalctl@ceph.mon.b.smithi157.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-10T17:35:22.859 INFO:journalctl@ceph.mon.b.smithi157.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-10T17:35:22.859 INFO:journalctl@ceph.mon.b.smithi157.stdout: 2023-10-10T17:35:22.860 INFO:journalctl@ceph.mon.b.smithi157.stdout: ** Compaction Stats [default] ** 2023-10-10T17:35:22.860 INFO:journalctl@ceph.mon.b.smithi157.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-10T17:35:22.860 INFO:journalctl@ceph.mon.b.smithi157.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-10T17:35:22.860 INFO:journalctl@ceph.mon.b.smithi157.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:35:22.860 INFO:journalctl@ceph.mon.b.smithi157.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:35:22.861 INFO:journalctl@ceph.mon.b.smithi157.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:35:22.861 INFO:journalctl@ceph.mon.b.smithi157.stdout: 2023-10-10T17:35:22.861 INFO:journalctl@ceph.mon.b.smithi157.stdout: ** Compaction Stats [default] ** 2023-10-10T17:35:22.861 INFO:journalctl@ceph.mon.b.smithi157.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-10T17:35:22.861 INFO:journalctl@ceph.mon.b.smithi157.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-10T17:35:22.862 INFO:journalctl@ceph.mon.b.smithi157.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-10T17:35:22.862 INFO:journalctl@ceph.mon.b.smithi157.stdout: 2023-10-10T17:35:22.862 INFO:journalctl@ceph.mon.b.smithi157.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-10T17:35:22.862 INFO:journalctl@ceph.mon.b.smithi157.stdout: 2023-10-10T17:35:22.862 INFO:journalctl@ceph.mon.b.smithi157.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-10T17:35:22.863 INFO:journalctl@ceph.mon.b.smithi157.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-10T17:35:22.863 INFO:journalctl@ceph.mon.b.smithi157.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-10T17:35:22.863 INFO:journalctl@ceph.mon.b.smithi157.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-10T17:35:22.863 INFO:journalctl@ceph.mon.b.smithi157.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-10T17:35:22.863 INFO:journalctl@ceph.mon.b.smithi157.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-10T17:35:22.864 INFO:journalctl@ceph.mon.b.smithi157.stdout: Cumulative compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-10T17:35:22.864 INFO:journalctl@ceph.mon.b.smithi157.stdout: Interval compaction: 0.00 GB write, 0.50 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-10T17:35:22.864 INFO:journalctl@ceph.mon.b.smithi157.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-10T17:35:22.864 INFO:journalctl@ceph.mon.b.smithi157.stdout: Block cache BinnedLRUCache@0x555bf8b711f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-10-10T17:35:22.864 INFO:journalctl@ceph.mon.b.smithi157.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-10-10T17:35:22.864 INFO:journalctl@ceph.mon.b.smithi157.stdout: 2023-10-10T17:35:22.865 INFO:journalctl@ceph.mon.b.smithi157.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-10T17:35:22.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-10-10T17:35:22.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: using public_addr v2:172.21.15.157:0/0 -> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] 2023-10-10T17:35:22.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: starting mon.b rank -1 at public addrs [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] at bind addrs [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:35:22.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(???) e0 preinit fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:35:22.866 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).mds e1 new map 2023-10-10T17:35:22.866 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).mds e1 print_map 2023-10-10T17:35:22.866 INFO:journalctl@ceph.mon.b.smithi157.stdout: e1 2023-10-10T17:35:22.866 INFO:journalctl@ceph.mon.b.smithi157.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-10T17:35:22.866 INFO:journalctl@ceph.mon.b.smithi157.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-10T17:35:22.867 INFO:journalctl@ceph.mon.b.smithi157.stdout: legacy client fscid: -1 2023-10-10T17:35:22.867 INFO:journalctl@ceph.mon.b.smithi157.stdout: 2023-10-10T17:35:22.867 INFO:journalctl@ceph.mon.b.smithi157.stdout: No filesystems configured 2023-10-10T17:35:22.867 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-10-10T17:35:22.867 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-10-10T17:35:22.867 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-10-10T17:35:22.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-10-10T17:35:22.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-10-10T17:35:22.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-10-10T17:35:22.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-10T17:35:22.868 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:35:22.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:35:22.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-10T17:35:22.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:35:22.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.870 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.870 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.870 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.870 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:22.870 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Added host smithi190 2023-10-10T17:35:22.871 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.871 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.871 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.871 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.871 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:35:22.872 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.872 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1160213321' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-10T17:35:22.872 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.872 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.872 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.873 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.873 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:35:22.873 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:22.873 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:22.873 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:22.874 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:35:22.874 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.874 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:22.874 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1160213321' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-10T17:35:22.874 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:22.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:35:22.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.875 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:35:22.876 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.876 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.876 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:22.876 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:22.876 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:22.877 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.877 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:22.877 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:22.877 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:22.877 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c;count:3 2023-10-10T17:35:22.878 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:22.878 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:22.878 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Deploying daemon mon.c on smithi190 2023-10-10T17:35:22.878 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.878 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/3250204155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:22.879 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:35:22.879 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.a calling monitor election 2023-10-10T17:35:22.879 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:22.879 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:22.879 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.880 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:22.880 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.c calling monitor election 2023-10-10T17:35:22.880 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:22.880 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.880 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:22.881 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:22.881 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-10T17:35:22.881 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2023-10-10T17:35:22.881 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: fsmap 2023-10-10T17:35:22.881 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:22.882 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mgrmap e13: x(active, since 43s) 2023-10-10T17:35:22.882 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: overall HEALTH_OK 2023-10-10T17:35:22.882 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.882 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.882 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:22.883 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:22.883 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:22.883 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: Deploying daemon mon.b on smithi157 2023-10-10T17:35:22.883 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.883 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:22.884 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/3419313770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:22.884 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:22.884 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-10-10T17:35:22.884 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-10-10T17:35:22.884 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expand map: {default=false} 2023-10-10T17:35:22.885 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta from 'false' to 'false' 2023-10-10T17:35:22.885 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expanded map: {default=false} 2023-10-10T17:35:22.885 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expand map: {default=info} 2023-10-10T17:35:22.885 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta from 'info' to 'info' 2023-10-10T17:35:22.885 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expanded map: {default=info} 2023-10-10T17:35:22.886 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expand map: {default=daemon} 2023-10-10T17:35:22.886 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-10T17:35:22.886 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expanded map: {default=daemon} 2023-10-10T17:35:22.886 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expand map: {default=debug} 2023-10-10T17:35:22.886 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta from 'debug' to 'debug' 2023-10-10T17:35:22.887 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 ceph-mon[100896]: expand_channel_meta expanded map: {default=debug} 2023-10-10T17:35:22.887 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:22 smithi157 systemd[1]: Started Ceph mon.b for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:35:23.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:22 smithi078 ceph-mon[99494]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:22 smithi190 ceph-mon[101220]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:23.351 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:35:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:35:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: mon.a calling monitor election 2023-10-10T17:35:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: mon.c calling monitor election 2023-10-10T17:35:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: mon.b calling monitor election 2023-10-10T17:35:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-10T17:35:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2023-10-10T17:35:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: fsmap 2023-10-10T17:35:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:28.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: mgrmap e13: x(active, since 51s) 2023-10-10T17:35:28.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: overall HEALTH_OK 2023-10-10T17:35:28.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:35:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: mon.a calling monitor election 2023-10-10T17:35:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: mon.c calling monitor election 2023-10-10T17:35:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: mon.b calling monitor election 2023-10-10T17:35:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-10T17:35:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2023-10-10T17:35:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: fsmap 2023-10-10T17:35:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:28.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: mgrmap e13: x(active, since 51s) 2023-10-10T17:35:28.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: overall HEALTH_OK 2023-10-10T17:35:28.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.170 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.172 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 -- 172.21.15.190:0/163626191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e01045d0 msgr2=0x7fa4c8005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:28.172 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 --2- 172.21.15.190:0/163626191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e01045d0 0x7fa4c8005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4005fa0 tx=0x7fa4d400df90 comp rx=0 tx=0).stop 2023-10-10T17:35:28.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 -- 172.21.15.190:0/163626191 shutdown_connections 2023-10-10T17:35:28.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 --2- 172.21.15.190:0/163626191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e01045d0 0x7fa4c8005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 -- 172.21.15.190:0/163626191 >> 172.21.15.190:0/163626191 conn(0x7fa4e00742a0 msgr2=0x7fa4e00746a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:28.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 -- 172.21.15.190:0/163626191 shutdown_connections 2023-10-10T17:35:28.173 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 -- 172.21.15.190:0/163626191 wait complete. 2023-10-10T17:35:28.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.168+0000 7fa4e5267700 1 Processor -- start 2023-10-10T17:35:28.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4e5267700 1 -- start start 2023-10-10T17:35:28.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4e5267700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 0x7fa4e0195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:28.174 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4e5267700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e0195860 0x7fa4e0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4e5267700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 0x7fa4e019a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4e5267700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e0195f30 con 0x7fa4e0195860 2023-10-10T17:35:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4e5267700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01a1f10 con 0x7fa4e01045d0 2023-10-10T17:35:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4e5267700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01a2050 con 0x7fa4e019a1a0 2023-10-10T17:35:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4df7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 0x7fa4e019a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:28.175 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4df7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 0x7fa4e019a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:51774/0 (socket says 172.21.15.190:51774) 2023-10-10T17:35:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4df7fe700 1 -- 172.21.15.190:0/4181648577 learned_addr learned my addr 172.21.15.190:0/4181648577 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:35:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4de7fc700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e0195860 0x7fa4e0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4deffd700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 0x7fa4e0195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4df7fe700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 msgr2=0x7fa4e019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-10-10T17:35:28.176 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4df7fe700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 msgr2=0x7fa4e019a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-10T17:35:28.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4df7fe700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 0x7fa4e019a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-10T17:35:28.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4df7fe700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 0x7fa4e019a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:35:28.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4deffd700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 msgr2=0x7fa4e0195320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-10-10T17:35:28.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4deffd700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 msgr2=0x7fa4e0195320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-10T17:35:28.177 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4deffd700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 0x7fa4e0195320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-10T17:35:28.178 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.169+0000 7fa4deffd700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 0x7fa4e0195320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:35:28.178 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4de7fc700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 msgr2=0x7fa4e019a640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:35:28.178 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4de7fc700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 0x7fa4e019a640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.178 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4de7fc700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 msgr2=0x7fa4e0195320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:35:28.178 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4de7fc700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 0x7fa4e0195320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4de7fc700 1 -- 172.21.15.190:0/4181648577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d400d990 con 0x7fa4e0195860 2023-10-10T17:35:28.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4de7fc700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e0195860 0x7fa4e0199c60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000b580 tx=0x7fa4d000b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:28.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4c7fff700 1 -- 172.21.15.190:0/4181648577 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0004230 con 0x7fa4e0195860 2023-10-10T17:35:28.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4c7fff700 1 -- 172.21.15.190:0/4181648577 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4d0004390 con 0x7fa4e0195860 2023-10-10T17:35:28.179 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e0105b70 con 0x7fa4e0195860 2023-10-10T17:35:28.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4c7fff700 1 -- 172.21.15.190:0/4181648577 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0011a70 con 0x7fa4e0195860 2023-10-10T17:35:28.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.170+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e01059e0 con 0x7fa4e0195860 2023-10-10T17:35:28.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.171+0000 7fa4c7fff700 1 -- 172.21.15.190:0/4181648577 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa4d0011c10 con 0x7fa4e0195860 2023-10-10T17:35:28.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.171+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0105d00 con 0x7fa4e0195860 2023-10-10T17:35:28.180 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.171+0000 7fa4c7fff700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4cc038d00 0x7fa4cc03b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:28.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.171+0000 7fa4c7fff700 1 -- 172.21.15.190:0/4181648577 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa4d004bd60 con 0x7fa4e0195860 2023-10-10T17:35:28.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.171+0000 7fa4deffd700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4cc038d00 0x7fa4cc03b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:28.181 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.172+0000 7fa4deffd700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4cc038d00 0x7fa4cc03b1b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400d490 tx=0x7fa4d400d3e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:28.182 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.176+0000 7fa4c7fff700 1 -- 172.21.15.190:0/4181648577 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa4d00193a0 con 0x7fa4e0195860 2023-10-10T17:35:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:35:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: mon.a calling monitor election 2023-10-10T17:35:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: mon.c calling monitor election 2023-10-10T17:35:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:35:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: mon.b calling monitor election 2023-10-10T17:35:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-10T17:35:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0],c=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} 2023-10-10T17:35:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: fsmap 2023-10-10T17:35:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: osdmap e4: 0 total, 0 up, 0 in 2023-10-10T17:35:28.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: mgrmap e13: x(active, since 51s) 2023-10-10T17:35:28.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: overall HEALTH_OK 2023-10-10T17:35:28.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.233 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:28.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.447+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4e0062120 con 0x7fa4e0195860 2023-10-10T17:35:28.450 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.448+0000 7fa4c7fff700 1 -- 172.21.15.190:0/4181648577 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1276 (secure 0 0 0) 0x7fa4d000f2e0 con 0x7fa4e0195860 2023-10-10T17:35:28.450 INFO:teuthology.orchestra.run.smithi190.stdout: 2023-10-10T17:35:28.451 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":3,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","modified":"2023-10-10T17:35:22.760026Z","created":"2023-10-10T17:33:39.867990Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:3300","nonce":0},{"type":"v1","addr":"172.21.15.157:6789","nonce":0}]},"addr":"172.21.15.157:6789/0","public_addr":"172.21.15.157:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-10-10T17:35:28.451 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4cc038d00 msgr2=0x7fa4cc03b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:28.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4cc038d00 0x7fa4cc03b1b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa4d400d490 tx=0x7fa4d400d3e0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e0195860 msgr2=0x7fa4e0199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:28.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e0195860 0x7fa4e0199c60 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000b580 tx=0x7fa4d000b940 comp rx=0 tx=0).stop 2023-10-10T17:35:28.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 shutdown_connections 2023-10-10T17:35:28.452 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4cc038d00 0x7fa4cc03b1b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4e019a1a0 0x7fa4e019a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4e01045d0 0x7fa4e0195320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 --2- 172.21.15.190:0/4181648577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4e0195860 0x7fa4e0199c60 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:28.453 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 >> 172.21.15.190:0/4181648577 conn(0x7fa4e00742a0 msgr2=0x7fa4e0102960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:28.454 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 shutdown_connections 2023-10-10T17:35:28.454 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:28.450+0000 7fa4e5267700 1 -- 172.21.15.190:0/4181648577 wait complete. 2023-10-10T17:35:28.454 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 3 2023-10-10T17:35:28.817 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-10-10T17:35:28.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph config generate-minimal-conf 2023-10-10T17:35:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:28 smithi078 ceph-mon[99494]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/4181648577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:28 smithi190 ceph-mon[101220]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/4181648577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:28 smithi157 ceph-mon[100896]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/4181648577' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-10T17:35:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:35:29.474 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.737+0000 7f57fe283700 1 -- 172.21.15.78:0/463878036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80fee80 msgr2=0x7f57f80ff250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.737+0000 7f57fe283700 1 --2- 172.21.15.78:0/463878036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80fee80 0x7f57f80ff250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f57e8009910 tx=0x7f57e8009c20 comp rx=0 tx=0).stop 2023-10-10T17:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.737+0000 7f57fe283700 1 -- 172.21.15.78:0/463878036 shutdown_connections 2023-10-10T17:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.737+0000 7f57fe283700 1 --2- 172.21.15.78:0/463878036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f80fee80 0x7f57f80ff250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.737+0000 7f57fe283700 1 -- 172.21.15.78:0/463878036 >> 172.21.15.78:0/463878036 conn(0x7f57f80fa760 msgr2=0x7f57f80fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.737+0000 7f57fe283700 1 -- 172.21.15.78:0/463878036 shutdown_connections 2023-10-10T17:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.737+0000 7f57fe283700 1 -- 172.21.15.78:0/463878036 wait complete. 2023-10-10T17:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 Processor -- start 2023-10-10T17:35:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 -- start start 2023-10-10T17:35:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57f80fee80 0x7f57f8194260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57f81947a0 0x7f57f8198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:30.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f8199150 0x7f57f81995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f8194e70 con 0x7f57f8199150 2023-10-10T17:35:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f81a0ec0 con 0x7f57f81947a0 2023-10-10T17:35:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fe283700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57f81a1000 con 0x7f57f80fee80 2023-10-10T17:35:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fc820700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f8199150 0x7f57f81995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:30.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fc820700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f8199150 0x7f57f81995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41244/0 (socket says 172.21.15.78:41244) 2023-10-10T17:35:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.738+0000 7f57fc820700 1 -- 172.21.15.78:0/633348777 learned_addr learned my addr 172.21.15.78:0/633348777 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:35:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57f7fff700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57f80fee80 0x7f57f8194260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:30.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57f77fe700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57f81947a0 0x7f57f8198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:30.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fc820700 1 -- 172.21.15.78:0/633348777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57f80fee80 msgr2=0x7f57f8194260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:30.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fc820700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57f80fee80 0x7f57f8194260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fc820700 1 -- 172.21.15.78:0/633348777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57f81947a0 msgr2=0x7f57f8198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fc820700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57f81947a0 0x7f57f8198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fc820700 1 -- 172.21.15.78:0/633348777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e80095f0 con 0x7f57f8199150 2023-10-10T17:35:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57f77fe700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57f81947a0 0x7f57f8198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:35:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fc820700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f8199150 0x7f57f81995f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f57f000df70 tx=0x7f57f000e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57f57fa700 1 -- 172.21.15.78:0/633348777 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f000e7c0 con 0x7f57f8199150 2023-10-10T17:35:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f8109010 con 0x7f57f8199150 2023-10-10T17:35:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57f57fa700 1 -- 172.21.15.78:0/633348777 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57f000bb40 con 0x7f57f8199150 2023-10-10T17:35:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57f57fa700 1 -- 172.21.15.78:0/633348777 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f00163f0 con 0x7f57f8199150 2023-10-10T17:35:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.739+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57f8108e80 con 0x7f57f8199150 2023-10-10T17:35:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.740+0000 7f57f57fa700 1 -- 172.21.15.78:0/633348777 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f57f0016550 con 0x7f57f8199150 2023-10-10T17:35:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.740+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57f81091a0 con 0x7f57f8199150 2023-10-10T17:35:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.740+0000 7f57f57fa700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f57e0038cf0 0x7f57e003b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.740+0000 7f57f57fa700 1 -- 172.21.15.78:0/633348777 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f57f004c950 con 0x7f57f8199150 2023-10-10T17:35:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.740+0000 7f57f7fff700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f57e0038cf0 0x7f57e003b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:30.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.741+0000 7f57f7fff700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f57e0038cf0 0x7f57e003b1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006010 tx=0x7f57e800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:30.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.745+0000 7f57f57fa700 1 -- 172.21.15.78:0/633348777 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f57f0014710 con 0x7f57f8199150 2023-10-10T17:35:30.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:30.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:30.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:30.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:30.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:30.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:30.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:30.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:30.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:30.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:30.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.946+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f57f8062120 con 0x7f57f8199150 2023-10-10T17:35:30.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.947+0000 7f57f57fa700 1 -- 172.21.15.78:0/633348777 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f57f0015e80 con 0x7f57f8199150 2023-10-10T17:35:30.948 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:35:30.948 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2023-10-10T17:35:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:35:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2023-10-10T17:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f57e0038cf0 msgr2=0x7f57e003b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f57e0038cf0 0x7f57e003b1a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006010 tx=0x7f57e800b540 comp rx=0 tx=0).stop 2023-10-10T17:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f8199150 msgr2=0x7f57f81995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f8199150 0x7f57f81995f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f57f000df70 tx=0x7f57f000e0c0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 shutdown_connections 2023-10-10T17:35:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f57e0038cf0 0x7f57e003b1a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f57f80fee80 0x7f57f8194260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57f8199150 0x7f57f81995f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 --2- 172.21.15.78:0/633348777 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f57f81947a0 0x7f57f8198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 >> 172.21.15.78:0/633348777 conn(0x7f57f80fa760 msgr2=0x7f57f8104950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 shutdown_connections 2023-10-10T17:35:30.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:30.949+0000 7f57fe283700 1 -- 172.21.15.78:0/633348777 wait complete. 2023-10-10T17:35:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:30.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:30.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:30.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:30.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:30.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:31.419 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-10-10T17:35:31.419 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:35:31.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2023-10-10T17:35:31.450 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:35:31.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:35:31.517 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:35:31.518 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.conf 2023-10-10T17:35:31.554 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:35:31.554 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:35:31.625 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:35:31.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2023-10-10T17:35:31.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Updating smithi078:/etc/ceph/ceph.conf 2023-10-10T17:35:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Updating smithi157:/etc/ceph/ceph.conf 2023-10-10T17:35:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:35:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Reconfiguring mon.a (unknown last config time)... 2023-10-10T17:35:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: Reconfiguring daemon mon.a on smithi078 2023-10-10T17:35:31.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/633348777' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:31.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:31.662 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:35:31.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:35:31.735 INFO:tasks.cephadm:Adding mgr.x on smithi078 2023-10-10T17:35:31.736 INFO:tasks.cephadm:Adding mgr.y on smithi157 2023-10-10T17:35:31.736 INFO:tasks.cephadm:Adding mgr.z on smithi190 2023-10-10T17:35:31.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch apply mgr '3;smithi078=x;smithi157=y;smithi190=z' 2023-10-10T17:35:31.801 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:31.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Updating smithi078:/etc/ceph/ceph.conf 2023-10-10T17:35:31.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Updating smithi157:/etc/ceph/ceph.conf 2023-10-10T17:35:31.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:35:31.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:31.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:31.804 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Reconfiguring mon.a (unknown last config time)... 2023-10-10T17:35:31.804 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: Reconfiguring daemon mon.a on smithi078 2023-10-10T17:35:31.804 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/633348777' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:31.804 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Updating smithi078:/etc/ceph/ceph.conf 2023-10-10T17:35:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Updating smithi157:/etc/ceph/ceph.conf 2023-10-10T17:35:31.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:35:31.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:35:31.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:31.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:31.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Reconfiguring mon.a (unknown last config time)... 2023-10-10T17:35:31.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: Reconfiguring daemon mon.a on smithi078 2023-10-10T17:35:31.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/633348777' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:31.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:32.413 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:35:32.892 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:32.893 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:32.893 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: Reconfiguring mon.b (monmap changed)... 2023-10-10T17:35:32.893 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:32.893 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:32.893 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:32.894 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: Reconfiguring daemon mon.b on smithi157 2023-10-10T17:35:32.894 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:32 smithi190 ceph-mon[101220]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:33.006 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:33.007 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:33.007 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: Reconfiguring mon.b (monmap changed)... 2023-10-10T17:35:33.007 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:33.007 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:33.007 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:33.008 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: Reconfiguring daemon mon.b on smithi157 2023-10-10T17:35:33.008 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:32 smithi157 ceph-mon[100896]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: Reconfiguring mon.b (monmap changed)... 2023-10-10T17:35:33.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:33.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:33.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:33.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: Reconfiguring daemon mon.b on smithi157 2023-10-10T17:35:33.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:32 smithi078 ceph-mon[99494]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:33.661 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 -- 172.21.15.190:0/3695692129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc0ffe40 msgr2=0x7f9fcc100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:33.661 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/3695692129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc0ffe40 0x7f9fcc100250 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8009910 tx=0x7f9fc8009c20 comp rx=0 tx=0).stop 2023-10-10T17:35:33.661 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 -- 172.21.15.190:0/3695692129 shutdown_connections 2023-10-10T17:35:33.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/3695692129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc0ffe40 0x7f9fcc100250 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:33.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 -- 172.21.15.190:0/3695692129 >> 172.21.15.190:0/3695692129 conn(0x7f9fcc0fb3f0 msgr2=0x7f9fcc0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:33.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 -- 172.21.15.190:0/3695692129 shutdown_connections 2023-10-10T17:35:33.662 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 -- 172.21.15.190:0/3695692129 wait complete. 2023-10-10T17:35:33.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.658+0000 7f9fd3ced700 1 Processor -- start 2023-10-10T17:35:33.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd3ced700 1 -- start start 2023-10-10T17:35:33.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd3ced700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fcc0ffe40 0x7f9fcc195050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:33.663 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd3ced700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fcc195590 0x7f9fcc19a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:33.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd3ced700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc195a00 0x7f9fcc19cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:33.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd3ced700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc19d310 con 0x7f9fcc195a00 2023-10-10T17:35:33.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd3ced700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc19d480 con 0x7f9fcc0ffe40 2023-10-10T17:35:33.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd3ced700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc19d640 con 0x7f9fcc195590 2023-10-10T17:35:33.664 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fcc195590 0x7f9fcc19a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:33.665 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fcc195590 0x7f9fcc19a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:51782/0 (socket says 172.21.15.190:51782) 2023-10-10T17:35:33.665 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 -- 172.21.15.190:0/1813060350 learned_addr learned my addr 172.21.15.190:0/1813060350 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:35:33.665 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1a89700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fcc0ffe40 0x7f9fcc195050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:33.665 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd228a700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc195a00 0x7f9fcc19cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:33.665 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 -- 172.21.15.190:0/1813060350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fcc0ffe40 msgr2=0x7f9fcc195050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:33.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fcc0ffe40 0x7f9fcc195050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:33.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 -- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc195a00 msgr2=0x7f9fcc19cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:33.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc195a00 0x7f9fcc19cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:33.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 -- 172.21.15.190:0/1813060350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc80095f0 con 0x7f9fcc195590 2023-10-10T17:35:33.666 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1a89700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fcc0ffe40 0x7f9fcc195050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:35:33.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd228a700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc195a00 0x7f9fcc19cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:35:33.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fd1288700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fcc195590 0x7f9fcc19a600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00b780 tx=0x7f9fbc00bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:33.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.659+0000 7f9fc2ffd700 1 -- 172.21.15.190:0/1813060350 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc004230 con 0x7f9fcc195590 2023-10-10T17:35:33.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.660+0000 7f9fc2ffd700 1 -- 172.21.15.190:0/1813060350 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9fbc004390 con 0x7f9fcc195590 2023-10-10T17:35:33.667 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.660+0000 7f9fc2ffd700 1 -- 172.21.15.190:0/1813060350 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc011b30 con 0x7f9fcc195590 2023-10-10T17:35:33.668 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.660+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc105c40 con 0x7f9fcc195590 2023-10-10T17:35:33.668 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.660+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc105ab0 con 0x7f9fcc195590 2023-10-10T17:35:33.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.660+0000 7f9fc2ffd700 1 -- 172.21.15.190:0/1813060350 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9fbc011cd0 con 0x7f9fcc195590 2023-10-10T17:35:33.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.660+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fcc18e970 con 0x7f9fcc195590 2023-10-10T17:35:33.669 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.661+0000 7f9fc2ffd700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fb8038cf0 0x7f9fb803b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:33.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.661+0000 7f9fc2ffd700 1 -- 172.21.15.190:0/1813060350 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9fbc04acd0 con 0x7f9fcc195590 2023-10-10T17:35:33.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.661+0000 7f9fd1a89700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fb8038cf0 0x7f9fb803b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:33.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.661+0000 7f9fd1a89700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fb8038cf0 0x7f9fb803b1a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8000c00 tx=0x7f9fc800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:33.670 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.667+0000 7f9fc2ffd700 1 -- 172.21.15.190:0/1813060350 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9fbc050020 con 0x7f9fcc195590 2023-10-10T17:35:33.876 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.876+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi157=y;smithi190=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f9fcc061b20 con 0x7f9fb8038cf0 2023-10-10T17:35:33.883 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.882+0000 7f9fc2ffd700 1 -- 172.21.15.190:0/1813060350 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9fcc061b20 con 0x7f9fb8038cf0 2023-10-10T17:35:33.883 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mgr update... 2023-10-10T17:35:33.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fb8038cf0 msgr2=0x7f9fb803b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:33.885 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fb8038cf0 0x7f9fb803b1a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9fc8000c00 tx=0x7f9fc800b540 comp rx=0 tx=0).stop 2023-10-10T17:35:33.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fcc195590 msgr2=0x7f9fcc19a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:33.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fcc195590 0x7f9fcc19a600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00b780 tx=0x7f9fbc00bb40 comp rx=0 tx=0).stop 2023-10-10T17:35:33.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 shutdown_connections 2023-10-10T17:35:33.886 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9fb8038cf0 0x7f9fb803b1a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:33.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9fcc195a00 0x7f9fcc19cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:33.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9fcc0ffe40 0x7f9fcc195050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:33.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 --2- 172.21.15.190:0/1813060350 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9fcc195590 0x7f9fcc19a600 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:33.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 >> 172.21.15.190:0/1813060350 conn(0x7f9fcc0fb3f0 msgr2=0x7f9fcc0fc0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:33.887 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 shutdown_connections 2023-10-10T17:35:33.888 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:35:33.884+0000 7f9fd3ced700 1 -- 172.21.15.190:0/1813060350 wait complete. 2023-10-10T17:35:34.034 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.034 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: Reconfiguring mon.c (monmap changed)... 2023-10-10T17:35:34.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:34.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:34.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:34.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: Reconfiguring daemon mon.c on smithi190 2023-10-10T17:35:34.036 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: Reconfiguring mon.c (monmap changed)... 2023-10-10T17:35:34.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:34.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:34.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:34.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: Reconfiguring daemon mon.c on smithi190 2023-10-10T17:35:34.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: Reconfiguring mon.c (monmap changed)... 2023-10-10T17:35:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-10T17:35:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-10T17:35:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: Reconfiguring daemon mon.c on smithi190 2023-10-10T17:35:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:34.572 DEBUG:teuthology.orchestra.run.smithi157:mgr.y> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.y.service 2023-10-10T17:35:34.575 DEBUG:teuthology.orchestra.run.smithi190:mgr.z> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.z.service 2023-10-10T17:35:34.610 INFO:journalctl@ceph.mgr.y.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:35:34.613 INFO:tasks.cephadm:Deploying OSDs... 2023-10-10T17:35:34.614 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:35:34.614 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2023-10-10T17:35:34.636 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-10T17:35:34.637 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2023-10-10T17:35:34.642 INFO:journalctl@ceph.mgr.z.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:35:34.697 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-10T17:35:34.698 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:34.698 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 391994 Links: 1 2023-10-10T17:35:34.698 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:34.698 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:34.699 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-10 17:34:47.337128249 +0000 2023-10-10T17:35:34.699 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-10 17:29:27.918830045 +0000 2023-10-10T17:35:34.699 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-10 17:29:27.918830045 +0000 2023-10-10T17:35:34.699 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-10T17:35:34.699 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-10T17:35:34.770 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-10T17:35:34.771 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-10T17:35:34.771 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000263702 s, 1.9 MB/s 2023-10-10T17:35:34.772 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-10T17:35:34.835 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2023-10-10T17:35:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-10T17:35:34.896 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:34.896 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 390641 Links: 1 2023-10-10T17:35:34.897 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:34.897 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:34.897 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-10 17:34:47.337128249 +0000 2023-10-10T17:35:34.897 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-10 17:29:28.439821757 +0000 2023-10-10T17:35:34.897 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-10 17:29:28.439821757 +0000 2023-10-10T17:35:34.897 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-10T17:35:34.898 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-10T17:35:34.967 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-10T17:35:34.968 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-10T17:35:34.968 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000249268 s, 2.1 MB/s 2023-10-10T17:35:34.970 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-10T17:35:35.032 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2023-10-10T17:35:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-10T17:35:35.093 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:35.094 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 396451 Links: 1 2023-10-10T17:35:35.094 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:35.094 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:35.094 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-10 17:34:47.337128249 +0000 2023-10-10T17:35:35.095 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-10 17:29:28.944813724 +0000 2023-10-10T17:35:35.095 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-10 17:29:28.944813724 +0000 2023-10-10T17:35:35.095 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-10T17:35:35.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-10T17:35:35.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi157=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:35.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: Saving service mgr spec with placement smithi078=x;smithi157=y;smithi190=z;count:3 2023-10-10T17:35:35.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:35.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:35.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:35.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:35.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:35.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-10T17:35:35.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:35.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-10T17:35:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-10T17:35:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000224604 s, 2.3 MB/s 2023-10-10T17:35:35.162 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-10T17:35:35.224 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2023-10-10T17:35:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-10T17:35:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:35.285 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 394890 Links: 1 2023-10-10T17:35:35.286 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:35.286 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:35.286 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-10-10 17:34:47.337128249 +0000 2023-10-10T17:35:35.286 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-10 17:29:29.453805627 +0000 2023-10-10T17:35:35.287 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-10 17:29:29.453805627 +0000 2023-10-10T17:35:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-10-10T17:35:35.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-10T17:35:35.297 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi157=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:35.298 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: Saving service mgr spec with placement smithi078=x;smithi157=y;smithi190=z;count:3 2023-10-10T17:35:35.298 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:35.298 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.298 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.299 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:35.299 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:35.299 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:35.299 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.299 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:35.299 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-10T17:35:35.300 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:35.300 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:35.355 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-10-10T17:35:35.355 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-10-10T17:35:35.355 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000247188 s, 2.1 MB/s 2023-10-10T17:35:35.356 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-10T17:35:35.416 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2023-10-10T17:35:35.416 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/scratch_devs of=/dev/stdout 2023-10-10T17:35:35.438 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-10T17:35:35.438 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_1 2023-10-10T17:35:35.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi157=y;smithi190=z", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:35.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: Saving service mgr spec with placement smithi078=x;smithi157=y;smithi190=z;count:3 2023-10-10T17:35:35.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:35.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:35.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:35.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:35.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:35.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:35.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-10T17:35:35.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:35.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:35.500 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-10T17:35:35.500 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:35.500 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 394400 Links: 1 2023-10-10T17:35:35.501 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:35.501 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:35.501 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-10-10 17:35:22.171432344 +0000 2023-10-10T17:35:35.501 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-10-10 17:29:35.161672417 +0000 2023-10-10T17:35:35.501 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-10-10 17:29:35.161672417 +0000 2023-10-10T17:35:35.502 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-10-10T17:35:35.502 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-10T17:35:35.569 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-10-10T17:35:35.570 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-10-10T17:35:35.570 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000148669 s, 3.4 MB/s 2023-10-10T17:35:35.571 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-10T17:35:35.634 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_2 2023-10-10T17:35:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-10T17:35:35.693 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:35.693 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 387656 Links: 1 2023-10-10T17:35:35.693 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:35.694 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:35.694 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-10-10 17:35:22.171432344 +0000 2023-10-10T17:35:35.694 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-10-10 17:29:35.672663283 +0000 2023-10-10T17:35:35.694 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-10-10 17:29:35.672663283 +0000 2023-10-10T17:35:35.694 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-10-10T17:35:35.695 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-10T17:35:35.761 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-10-10T17:35:35.761 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-10-10T17:35:35.761 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000126267 s, 4.1 MB/s 2023-10-10T17:35:35.762 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-10T17:35:35.821 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_3 2023-10-10T17:35:35.881 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-10T17:35:35.881 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:35.881 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 387764 Links: 1 2023-10-10T17:35:35.882 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:35.882 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:35.882 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-10-10 17:35:22.171432344 +0000 2023-10-10T17:35:35.882 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-10-10 17:29:36.164654488 +0000 2023-10-10T17:35:35.882 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-10-10 17:29:36.164654488 +0000 2023-10-10T17:35:35.883 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-10-10T17:35:35.883 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-10T17:35:35.948 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-10-10T17:35:35.949 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-10-10T17:35:35.949 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 0.000134806 s, 3.8 MB/s 2023-10-10T17:35:35.950 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-10T17:35:36.009 DEBUG:teuthology.orchestra.run.smithi157:> stat /dev/vg_nvme/lv_4 2023-10-10T17:35:36.067 INFO:teuthology.orchestra.run.smithi157.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-10T17:35:36.067 INFO:teuthology.orchestra.run.smithi157.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:36.067 INFO:teuthology.orchestra.run.smithi157.stdout:Device: 6h/6d Inode: 395315 Links: 1 2023-10-10T17:35:36.068 INFO:teuthology.orchestra.run.smithi157.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:36.068 INFO:teuthology.orchestra.run.smithi157.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:36.068 INFO:teuthology.orchestra.run.smithi157.stdout:Access: 2023-10-10 17:35:22.171432344 +0000 2023-10-10T17:35:36.068 INFO:teuthology.orchestra.run.smithi157.stdout:Modify: 2023-10-10 17:29:36.680645264 +0000 2023-10-10T17:35:36.068 INFO:teuthology.orchestra.run.smithi157.stdout:Change: 2023-10-10 17:29:36.680645264 +0000 2023-10-10T17:35:36.068 INFO:teuthology.orchestra.run.smithi157.stdout: Birth: - 2023-10-10T17:35:36.069 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-10T17:35:36.133 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records in 2023-10-10T17:35:36.133 INFO:teuthology.orchestra.run.smithi157.stderr:1+0 records out 2023-10-10T17:35:36.134 INFO:teuthology.orchestra.run.smithi157.stderr:512 bytes copied, 9.5867e-05 s, 5.3 MB/s 2023-10-10T17:35:36.135 DEBUG:teuthology.orchestra.run.smithi157:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-10T17:35:36.193 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2023-10-10T17:35:36.193 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2023-10-10T17:35:36.212 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-10T17:35:36.212 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2023-10-10T17:35:36.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:35 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:36.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:35 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:35:36.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:35 smithi157 ceph-mon[100896]: Deploying daemon mgr.z on smithi190 2023-10-10T17:35:36.270 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-10T17:35:36.270 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:36.271 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 391212 Links: 1 2023-10-10T17:35:36.271 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:36.271 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:36.271 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-10-10 17:35:36.100287232 +0000 2023-10-10T17:35:36.271 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-10-10 17:29:20.008499155 +0000 2023-10-10T17:35:36.271 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-10-10 17:29:20.008499155 +0000 2023-10-10T17:35:36.272 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-10-10T17:35:36.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-10T17:35:36.338 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:35 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:36.339 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:35 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:35:36.339 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:35 smithi190 ceph-mon[101220]: Deploying daemon mgr.z on smithi190 2023-10-10T17:35:36.339 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:36 smithi190 systemd[1]: Starting Ceph mgr.z for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:35:36.342 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-10-10T17:35:36.342 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-10-10T17:35:36.342 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000241185 s, 2.1 MB/s 2023-10-10T17:35:36.343 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-10T17:35:36.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:35 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:36.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:35 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:35:36.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:35 smithi078 ceph-mon[99494]: Deploying daemon mgr.z on smithi190 2023-10-10T17:35:36.402 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2023-10-10T17:35:36.458 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-10T17:35:36.459 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:36.459 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 387926 Links: 1 2023-10-10T17:35:36.459 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:36.459 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:36.459 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-10-10 17:35:36.100287232 +0000 2023-10-10T17:35:36.460 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-10-10 17:29:20.522484869 +0000 2023-10-10T17:35:36.460 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-10-10 17:29:20.522484869 +0000 2023-10-10T17:35:36.460 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-10-10T17:35:36.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-10T17:35:36.526 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-10-10T17:35:36.526 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-10-10T17:35:36.527 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000299948 s, 1.7 MB/s 2023-10-10T17:35:36.527 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-10T17:35:36.585 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2023-10-10T17:35:36.600 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:36 smithi190 podman[103101]: 2023-10-10T17:35:36.617 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-10T17:35:36.617 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:36.617 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 393472 Links: 1 2023-10-10T17:35:36.617 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:36.617 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:36.618 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-10-10 17:35:36.100287232 +0000 2023-10-10T17:35:36.618 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-10-10 17:29:21.035470610 +0000 2023-10-10T17:35:36.618 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-10-10 17:29:21.035470610 +0000 2023-10-10T17:35:36.618 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-10-10T17:35:36.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-10T17:35:36.681 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-10-10T17:35:36.681 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-10-10T17:35:36.682 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00022804 s, 2.2 MB/s 2023-10-10T17:35:36.683 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-10T17:35:36.741 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2023-10-10T17:35:36.798 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-10T17:35:36.799 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-10T17:35:36.799 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 6h/6d Inode: 396555 Links: 1 2023-10-10T17:35:36.799 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:35:36.799 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2023-10-10T17:35:36.800 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2023-10-10 17:35:36.100287232 +0000 2023-10-10T17:35:36.800 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2023-10-10 17:29:21.526456963 +0000 2023-10-10T17:35:36.800 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2023-10-10 17:29:21.526456963 +0000 2023-10-10T17:35:36.800 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: - 2023-10-10T17:35:36.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-10T17:35:36.865 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2023-10-10T17:35:36.865 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2023-10-10T17:35:36.865 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000180905 s, 2.8 MB/s 2023-10-10T17:35:36.867 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-10T17:35:36.924 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2023-10-10T17:35:36.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-10T17:35:36.951 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:36 smithi190 bash[103101]: 26af3b7845378804b4cde21f65dd6e84ab363b991bd286615bceb66a0b3a701a 2023-10-10T17:35:36.951 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mgr[103286]: -- 172.21.15.190:0/3022693473 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x565541d81520 con 0x565541d57000 2023-10-10T17:35:36.952 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:36 smithi190 systemd[1]: Started Ceph mgr.z for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:35:36.952 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:36.864+0000 7fa62b1fc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:35:36.952 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:36.950+0000 7fa62b1fc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:35:37.300 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:37.039+0000 7fa62b1fc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:35:37.301 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:37.131+0000 7fa62b1fc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:35:37.301 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:37.299+0000 7fa62b1fc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:35:37.301 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:37.302 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.302 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.302 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.302 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:37.303 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-10T17:35:37.303 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:37.303 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:37.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:37.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-10T17:35:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:37.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:37.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:37.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:37.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-10-10T17:35:37.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:37.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:37.588 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:35:37.911 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:37.602+0000 7fa62b1fc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:35:37.911 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:37.690+0000 7fa62b1fc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:35:37.912 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:37.817+0000 7fa62b1fc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:35:38.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:37 smithi078 ceph-mon[99494]: Deploying daemon mgr.y on smithi157 2023-10-10T17:35:38.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-mon[101220]: Deploying daemon mgr.y on smithi157 2023-10-10T17:35:38.165 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:37.910+0000 7fa62b1fc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:35:38.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:37 smithi157 ceph-mon[100896]: Deploying daemon mgr.y on smithi157 2023-10-10T17:35:38.664 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:38 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:38.328+0000 7fa62b1fc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:35:38.722 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:38 smithi157 systemd[1]: Starting Ceph mgr.y for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:35:38.977 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:38 smithi157 podman[102331]: 2023-10-10T17:35:38.978 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:38 smithi157 ceph-mgr[102372]: -- 172.21.15.157:0/1318659176 <== mon.2 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5572364db520 con 0x5572364b0800 2023-10-10T17:35:39.164 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:38 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:38.894+0000 7fa62b1fc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:35:39.165 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:39.065+0000 7fa62b1fc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:35:39.256 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:38 smithi157 bash[102331]: 71f05455bfb5c50dc15323a4d4024df815c990e660f6bbeb3c2ee74368fb70fd 2023-10-10T17:35:39.256 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 systemd[1]: Started Ceph mgr.y for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:35:39.256 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:39.081+0000 7f1970af2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:35:39.257 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:39.163+0000 7f1970af2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:35:39.257 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:39.255+0000 7f1970af2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:35:39.264 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-10T17:35:39.264 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-10T17:35:39.264 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-10-10T17:35:39.264 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-10-10T17:35:39.265 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113959 s, 920 MB/s 2023-10-10T17:35:39.265 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-10-10T17:35:39.535 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:39.354+0000 7f1970af2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:35:39.535 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-mon[100896]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:39.535 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.536 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.536 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.536 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.536 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:39.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:39 smithi078 ceph-mon[99494]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:39.613 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.614 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:39.631 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2023-10-10T17:35:39.863 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:39.534+0000 7f1970af2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:35:39.878 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-mon[101220]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:39.878 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.878 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.879 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.879 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:39.879 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:40.153 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:39.876+0000 7fa62b1fc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:35:40.153 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:39.967+0000 7fa62b1fc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:35:40.190 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:39.862+0000 7f1970af2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:35:40.190 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:39.950+0000 7f1970af2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:35:40.190 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:40.087+0000 7f1970af2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:35:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:35:40.414 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:40.152+0000 7fa62b1fc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:35:40.476 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:40.189+0000 7f1970af2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:35:40.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:40 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:40.728 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:40.647+0000 7f1970af2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:35:40.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:40 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:40.836 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:40 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:40.836 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:40.441+0000 7fa62b1fc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:35:40.836 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:40.681+0000 7fa62b1fc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:35:40.836 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:40.834+0000 7fa62b1fc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:35:41.414 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:41.102+0000 7fa62b1fc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:35:41.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 -- 172.21.15.78:0/3038708336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 msgr2=0x7f7a50100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:41.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 --2- 172.21.15.78:0/3038708336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50100830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7a40009990 tx=0x7f7a40009ca0 comp rx=0 tx=0).stop 2023-10-10T17:35:41.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 -- 172.21.15.78:0/3038708336 shutdown_connections 2023-10-10T17:35:41.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 --2- 172.21.15.78:0/3038708336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a50100d70 0x7f7a50107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:41.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 --2- 172.21.15.78:0/3038708336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a500ff1e0 0x7f7a500ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:41.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 --2- 172.21.15.78:0/3038708336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50100830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:41.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 -- 172.21.15.78:0/3038708336 >> 172.21.15.78:0/3038708336 conn(0x7f7a500fa7f0 msgr2=0x7f7a500fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:41.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 -- 172.21.15.78:0/3038708336 shutdown_connections 2023-10-10T17:35:41.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 -- 172.21.15.78:0/3038708336 wait complete. 2023-10-10T17:35:41.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.456+0000 7f7a555d3700 1 Processor -- start 2023-10-10T17:35:41.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a555d3700 1 -- start start 2023-10-10T17:35:41.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a555d3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a500ff1e0 0x7f7a50194f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:41.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a555d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50195490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:41.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a555d3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a50100d70 0x7f7a5019c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:41.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a555d3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5019cae0 con 0x7f7a501003e0 2023-10-10T17:35:41.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a555d3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a5019cc20 con 0x7f7a500ff1e0 2023-10-10T17:35:41.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a555d3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50068630 con 0x7f7a50100d70 2023-10-10T17:35:41.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50195490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:41.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50195490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41254/0 (socket says 172.21.15.78:41254) 2023-10-10T17:35:41.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 -- 172.21.15.78:0/4275158985 learned_addr learned my addr 172.21.15.78:0/4275158985 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:35:41.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4effd700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a500ff1e0 0x7f7a50194f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:41.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4f7fe700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a50100d70 0x7f7a5019c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:41.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 -- 172.21.15.78:0/4275158985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a50100d70 msgr2=0x7f7a5019c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:41.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a50100d70 0x7f7a5019c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:41.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 -- 172.21.15.78:0/4275158985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a500ff1e0 msgr2=0x7f7a50194f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:41.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a500ff1e0 0x7f7a50194f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:41.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 -- 172.21.15.78:0/4275158985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a400095f0 con 0x7f7a501003e0 2023-10-10T17:35:41.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4f7fe700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a50100d70 0x7f7a5019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:35:41.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a4e7fc700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50195490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a400036c0 tx=0x7f7a4000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:41.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.457+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4001d070 con 0x7f7a501003e0 2023-10-10T17:35:41.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.458+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a5010ab80 con 0x7f7a501003e0 2023-10-10T17:35:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.458+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a4000fb80 con 0x7f7a501003e0 2023-10-10T17:35:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.458+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a40017650 con 0x7f7a501003e0 2023-10-10T17:35:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.458+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5010a9f0 con 0x7f7a501003e0 2023-10-10T17:35:41.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.458+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f7a400177b0 con 0x7f7a501003e0 2023-10-10T17:35:41.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.458+0000 7f7a37fff700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7a380389e0 0x7f7a3803ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:35:41.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.459+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7a4004d640 con 0x7f7a501003e0 2023-10-10T17:35:41.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.459+0000 7f7a4effd700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7a380389e0 0x7f7a3803ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:35:41.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.459+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a50066ea0 con 0x7f7a501003e0 2023-10-10T17:35:41.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.459+0000 7f7a4effd700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7a380389e0 0x7f7a3803ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a50100240 tx=0x7f7a4400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:35:41.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.463+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7a40027020 con 0x7f7a501003e0 2023-10-10T17:35:41.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:41.673+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a50068c20 con 0x7f7a380389e0 2023-10-10T17:35:41.727 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:41.247+0000 7f1970af2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:35:41.728 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:41.417+0000 7f1970af2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:35:41.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:41.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:41.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:41.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:41.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:41.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:41.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:41.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.766 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:41.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:41.767 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:41.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:41.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:41.768 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.801 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:41.801 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.801 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:41.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:41.802 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:41.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-10T17:35:41.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-10T17:35:41.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:41.803 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:42.119 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:41.800+0000 7fa62b1fc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:35:42.119 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:41.929+0000 7fa62b1fc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:35:42.414 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:42.118+0000 7fa62b1fc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:35:42.414 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:42.230+0000 7fa62b1fc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:35:42.444 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:42.183+0000 7f1970af2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:35:42.444 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:42.274+0000 7f1970af2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:35:42.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:42.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:35:42.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:42.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: Reconfiguring mgr.x (unknown last config time)... 2023-10-10T17:35:42.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: Reconfiguring daemon mgr.x on smithi078 2023-10-10T17:35:42.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:35:42.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:35:42.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:42.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:42.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:42.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:42 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:42.721 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:42.444+0000 7f1970af2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:35:42.721 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:42.722 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:35:42.722 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:42.722 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: Reconfiguring mgr.x (unknown last config time)... 2023-10-10T17:35:42.722 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: Reconfiguring daemon mgr.x on smithi078 2023-10-10T17:35:42.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:35:42.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:35:42.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:42.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:42.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:42.723 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:42.914 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:42.447+0000 7fa62b1fc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:35:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:35:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: Reconfiguring mgr.x (unknown last config time)... 2023-10-10T17:35:42.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: Reconfiguring daemon mgr.x on smithi078 2023-10-10T17:35:42.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:35:42.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:35:42.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:42.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:42.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:42.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:42 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:42.977 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:42.721+0000 7f1970af2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:35:42.978 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:42.948+0000 7f1970af2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:35:43.359 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:43.098+0000 7f1970af2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:35:43.600 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:43.600 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:43.601 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:43.601 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:43.601 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:43.601 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: Standby manager daemon z started 2023-10-10T17:35:43.601 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-10T17:35:43.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:35:43.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-10T17:35:43.602 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:43 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:35:43.664 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:35:43.308+0000 7fa62b1fc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:35:43.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:43.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:43.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:43.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:43.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:43.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: Standby manager daemon z started 2023-10-10T17:35:43.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-10T17:35:43.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:35:43.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-10T17:35:43.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:43 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:35:43.728 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:43.359+0000 7f1970af2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:35:43.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:35:43.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:43.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:43.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:43.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:43.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: Standby manager daemon z started 2023-10-10T17:35:43.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-10T17:35:43.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:35:43.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-10T17:35:43.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:43 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/2639062276' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:35:44.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:44.030+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f7a4002c7c0 con 0x7f7a501003e0 2023-10-10T17:35:44.385 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:44.063+0000 7f1970af2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:35:44.385 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:44.195+0000 7f1970af2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:35:44.713 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:44.384+0000 7f1970af2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:35:44.713 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:44.498+0000 7f1970af2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:35:44.713 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:44.712+0000 7f1970af2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:35:44.714 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:44.714 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:35:44.714 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:44.714 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: mgrmap e14: x(active, since 67s), standbys: z 2023-10-10T17:35:44.714 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-10T17:35:44.715 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2796933117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d74c4b96-ee1f-4f21-9424-d693d38f0335"}]: dispatch 2023-10-10T17:35:44.715 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2796933117' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d74c4b96-ee1f-4f21-9424-d693d38f0335"}]': finished 2023-10-10T17:35:44.715 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: osdmap e5: 1 total, 0 up, 1 in 2023-10-10T17:35:44.715 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:35:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:35:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: mgrmap e14: x(active, since 67s), standbys: z 2023-10-10T17:35:44.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-10T17:35:44.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2796933117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d74c4b96-ee1f-4f21-9424-d693d38f0335"}]: dispatch 2023-10-10T17:35:44.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2796933117' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d74c4b96-ee1f-4f21-9424-d693d38f0335"}]': finished 2023-10-10T17:35:44.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: osdmap e5: 1 total, 0 up, 1 in 2023-10-10T17:35:44.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:35:44.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:35:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: mgrmap e14: x(active, since 67s), standbys: z 2023-10-10T17:35:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-10T17:35:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2796933117' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d74c4b96-ee1f-4f21-9424-d693d38f0335"}]: dispatch 2023-10-10T17:35:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2796933117' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d74c4b96-ee1f-4f21-9424-d693d38f0335"}]': finished 2023-10-10T17:35:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: osdmap e5: 1 total, 0 up, 1 in 2023-10-10T17:35:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:35:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:45 smithi078 ceph-mon[99494]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:45 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/714968584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:35:45.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:45 smithi190 ceph-mon[101220]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:45 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/714968584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:35:45.978 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:35:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:35:45.595+0000 7f1970af2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:35:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:45 smithi157 ceph-mon[100896]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:45 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/714968584' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:35:46.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:46.498+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f7a40020370 con 0x7f7a501003e0 2023-10-10T17:35:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:46 smithi078 ceph-mon[99494]: Standby manager daemon y started 2023-10-10T17:35:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:46 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-10T17:35:46.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:46 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:35:46.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:46 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-10T17:35:46.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:46 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:35:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:46 smithi190 ceph-mon[101220]: Standby manager daemon y started 2023-10-10T17:35:46.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:46 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-10T17:35:46.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:46 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:35:46.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:46 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-10T17:35:46.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:46 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:35:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:46 smithi157 ceph-mon[100896]: Standby manager daemon y started 2023-10-10T17:35:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:46 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-10T17:35:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:46 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:35:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:46 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-10T17:35:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:46 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3702364543' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:35:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:47 smithi078 ceph-mon[99494]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:47 smithi078 ceph-mon[99494]: mgrmap e15: x(active, since 70s), standbys: z, y 2023-10-10T17:35:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-10T17:35:47.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:47 smithi190 ceph-mon[101220]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:47.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:47 smithi190 ceph-mon[101220]: mgrmap e15: x(active, since 70s), standbys: z, y 2023-10-10T17:35:47.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-10T17:35:47.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:47 smithi157 ceph-mon[100896]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:47 smithi157 ceph-mon[100896]: mgrmap e15: x(active, since 70s), standbys: z, y 2023-10-10T17:35:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-10T17:35:49.785 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:49 smithi078 ceph-mon[99494]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:49.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:49 smithi190 ceph-mon[101220]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:49.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:49 smithi157 ceph-mon[100896]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:51.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:51 smithi078 ceph-mon[99494]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:51.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-10T17:35:51.784 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:51.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:51 smithi190 ceph-mon[101220]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:51.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:51 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-10T17:35:51.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:51 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:51 smithi157 ceph-mon[100896]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:51 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-10T17:35:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:51 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:52.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:52 smithi078 ceph-mon[99494]: Deploying daemon osd.0 on smithi078 2023-10-10T17:35:52.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:52 smithi190 ceph-mon[101220]: Deploying daemon osd.0 on smithi078 2023-10-10T17:35:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:52 smithi157 ceph-mon[100896]: Deploying daemon osd.0 on smithi078 2023-10-10T17:35:53.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:53 smithi078 ceph-mon[99494]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:53.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:53 smithi190 ceph-mon[101220]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:53.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:53 smithi157 ceph-mon[100896]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:55.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:55 smithi078 ceph-mon[99494]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:55.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:55 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:55.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:55 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:55.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:55 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:55.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:55 smithi190 ceph-mon[101220]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:55.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:55 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:55.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:55 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:55.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:55 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:55.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:55 smithi157 ceph-mon[100896]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:55.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:55 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:55.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:55 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:55.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:55 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:57.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:57 smithi078 ceph-mon[99494]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:57.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:57 smithi190 ceph-mon[101220]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:57 smithi157 ceph-mon[100896]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:58.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.377+0000 7f7a37fff700 1 -- 172.21.15.78:0/4275158985 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7a50068c20 con 0x7f7a380389e0 2023-10-10T17:35:58.379 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2023-10-10T17:35:58.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.379+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7a380389e0 msgr2=0x7f7a3803ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:58.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.379+0000 7f7a555d3700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7a380389e0 0x7f7a3803ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a50100240 tx=0x7f7a4400b410 comp rx=0 tx=0).stop 2023-10-10T17:35:58.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 msgr2=0x7f7a50195490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:35:58.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50195490 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7a400036c0 tx=0x7f7a4000f690 comp rx=0 tx=0).stop 2023-10-10T17:35:58.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 shutdown_connections 2023-10-10T17:35:58.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7a380389e0 0x7f7a3803ae90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:58.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7a50100d70 0x7f7a5019c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:58.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7a500ff1e0 0x7f7a50194f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:58.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 --2- 172.21.15.78:0/4275158985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a501003e0 0x7f7a50195490 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:35:58.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 >> 172.21.15.78:0/4275158985 conn(0x7f7a500fa7f0 msgr2=0x7f7a50103930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:35:58.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 shutdown_connections 2023-10-10T17:35:58.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:35:58.380+0000 7f7a555d3700 1 -- 172.21.15.78:0/4275158985 wait complete. 2023-10-10T17:35:58.863 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:58.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:58.864 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.865 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:58.866 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.0.service 2023-10-10T17:35:58.868 INFO:tasks.cephadm:Deploying osd.1 on smithi157 with /dev/vg_nvme/lv_4... 2023-10-10T17:35:58.869 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-10T17:35:58.895 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.896 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.896 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:58.896 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:58.896 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.896 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.897 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.897 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:58.897 INFO:journalctl@ceph.osd.0.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:35:58.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:35:58.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:35:58.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:35:58.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:35:59.508 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:35:59.539 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:35:59 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-0[112519]: 2023-10-10T17:35:59.234+0000 7f60abe39640 -1 osd.0 0 log_to_monitors true 2023-10-10T17:35:59.793 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:59 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:59.794 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:59 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:35:59.794 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:59 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:35:59.794 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:59 smithi157 ceph-mon[100896]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:59.794 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:35:59 smithi157 ceph-mon[100896]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-10T17:35:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:59 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:59 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:35:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:59 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:35:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:59 smithi078 ceph-mon[99494]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:59.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:35:59 smithi078 ceph-mon[99494]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-10T17:35:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:59 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:59.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:59 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:35:59.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:59 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:35:59.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:59 smithi190 ceph-mon[101220]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:35:59.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:35:59 smithi190 ceph-mon[101220]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-10T17:36:00.577 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:00 smithi157 ceph-mon[100896]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-10T17:36:00.578 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:00 smithi157 ceph-mon[100896]: osdmap e6: 1 total, 0 up, 1 in 2023-10-10T17:36:00.578 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:00 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:00.578 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:00 smithi157 ceph-mon[100896]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:36:00.578 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:00 smithi157 ceph-mon[100896]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:36:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:00 smithi078 ceph-mon[99494]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-10T17:36:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:00 smithi078 ceph-mon[99494]: osdmap e6: 1 total, 0 up, 1 in 2023-10-10T17:36:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:00 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:00 smithi078 ceph-mon[99494]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:36:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:00 smithi078 ceph-mon[99494]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:36:00.879 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-0[112519]: 2023-10-10T17:36:00.555+0000 7f60a1cb1700 -1 osd.0 0 waiting for initial osdmap 2023-10-10T17:36:00.880 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-0[112519]: 2023-10-10T17:36:00.562+0000 7f609929d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:36:00.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:00 smithi190 ceph-mon[101220]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-10T17:36:00.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:00 smithi190 ceph-mon[101220]: osdmap e6: 1 total, 0 up, 1 in 2023-10-10T17:36:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:00 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:00 smithi190 ceph-mon[101220]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:36:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:00 smithi190 ceph-mon[101220]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:36:01.010 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-10T17:36:01.010 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-10T17:36:01.011 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2023-10-10T17:36:01.011 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2023-10-10T17:36:01.011 INFO:teuthology.orchestra.run.smithi157.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00821058 s, 1.3 GB/s 2023-10-10T17:36:01.011 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2023-10-10T17:36:01.349 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi157:vg_nvme/lv_4 2023-10-10T17:36:01.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:36:01.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: osdmap e7: 1 total, 0 up, 1 in 2023-10-10T17:36:01.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:01.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:01.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi078 2023-10-10T17:36:01.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:01.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi078 to 17168M 2023-10-10T17:36:01.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:01.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:01.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:01.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:36:01.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:01 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:36:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: osdmap e7: 1 total, 0 up, 1 in 2023-10-10T17:36:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi078 2023-10-10T17:36:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:01.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi078 to 17168M 2023-10-10T17:36:01.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:01.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:01.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:01.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:36:01.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:01 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:01.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:36:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: osdmap e7: 1 total, 0 up, 1 in 2023-10-10T17:36:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi078 2023-10-10T17:36:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi078 to 17168M 2023-10-10T17:36:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:01.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:01.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:01.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:01.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:36:01.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:01 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:02.031 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:36:02.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:02 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:36:02.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:02 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:36:02.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:02 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:02.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:02 smithi078 ceph-mon[99494]: osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] boot 2023-10-10T17:36:02.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:02 smithi078 ceph-mon[99494]: osdmap e8: 1 total, 1 up, 1 in 2023-10-10T17:36:02.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:02 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:02.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:02 smithi078 ceph-mon[99494]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:36:02.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:02 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:36:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:02 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:36:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:02 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:02 smithi190 ceph-mon[101220]: osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] boot 2023-10-10T17:36:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:02 smithi190 ceph-mon[101220]: osdmap e8: 1 total, 1 up, 1 in 2023-10-10T17:36:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:02 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:02.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:02 smithi190 ceph-mon[101220]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:36:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:02 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:36:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:02 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:36:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:02 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:02 smithi157 ceph-mon[100896]: osd.0 [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] boot 2023-10-10T17:36:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:02 smithi157 ceph-mon[100896]: osdmap e8: 1 total, 1 up, 1 in 2023-10-10T17:36:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:02 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:36:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:02 smithi157 ceph-mon[100896]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-10T17:36:03.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 -- 172.21.15.157:0/938966873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e680ff440 msgr2=0x7f9e680ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:03.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/938966873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e680ff440 0x7f9e680ff8b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9e580099c0 tx=0x7f9e58009cd0 comp rx=0 tx=0).stop 2023-10-10T17:36:03.186 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 -- 172.21.15.157:0/938966873 shutdown_connections 2023-10-10T17:36:03.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/938966873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e680ffdf0 0x7f9e681076a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:03.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/938966873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e681007e0 0x7f9e68100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:03.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/938966873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e680ff440 0x7f9e680ff8b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:03.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 -- 172.21.15.157:0/938966873 >> 172.21.15.157:0/938966873 conn(0x7f9e680fa000 msgr2=0x7f9e680fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:36:03.187 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 -- 172.21.15.157:0/938966873 shutdown_connections 2023-10-10T17:36:03.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.184+0000 7f9e6d61e700 1 -- 172.21.15.157:0/938966873 wait complete. 2023-10-10T17:36:03.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 Processor -- start 2023-10-10T17:36:03.188 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 -- start start 2023-10-10T17:36:03.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e680ff440 0x7f9e680feeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:03.190 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e680ffdf0 0x7f9e6819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:03.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e681007e0 0x7f9e6819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:03.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6819dc00 con 0x7f9e680ff440 2023-10-10T17:36:03.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6819dd70 con 0x7f9e681007e0 2023-10-10T17:36:03.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e6d61e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6819df30 con 0x7f9e680ffdf0 2023-10-10T17:36:03.191 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e681007e0 0x7f9e6819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:03.192 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e681007e0 0x7f9e6819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:58342/0 (socket says 172.21.15.157:58342) 2023-10-10T17:36:03.192 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 -- 172.21.15.157:0/1772965164 learned_addr learned my addr 172.21.15.157:0/1772965164 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-10-10T17:36:03.192 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e66ffd700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e680ff440 0x7f9e680feeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:03.192 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e667fc700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e680ffdf0 0x7f9e6819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:03.192 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 -- 172.21.15.157:0/1772965164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e680ffdf0 msgr2=0x7f9e6819ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:03.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e680ffdf0 0x7f9e6819ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:03.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 -- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e680ff440 msgr2=0x7f9e680feeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:03.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e680ff440 0x7f9e680feeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:03.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e677fe700 1 -- 172.21.15.157:0/1772965164 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e580095f0 con 0x7f9e681007e0 2023-10-10T17:36:03.193 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.185+0000 7f9e66ffd700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e680ff440 0x7f9e680feeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:36:03.194 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.186+0000 7f9e677fe700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e681007e0 0x7f9e6819d460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000b970 tx=0x7f9e6000bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:36:03.194 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.186+0000 7f9e5ffff700 1 -- 172.21.15.157:0/1772965164 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6000c9b0 con 0x7f9e681007e0 2023-10-10T17:36:03.194 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.186+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6810ac80 con 0x7f9e681007e0 2023-10-10T17:36:03.194 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.186+0000 7f9e5ffff700 1 -- 172.21.15.157:0/1772965164 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9e60014440 con 0x7f9e681007e0 2023-10-10T17:36:03.194 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.186+0000 7f9e5ffff700 1 -- 172.21.15.157:0/1772965164 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600075d0 con 0x7f9e681007e0 2023-10-10T17:36:03.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.186+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6810aaf0 con 0x7f9e681007e0 2023-10-10T17:36:03.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.187+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e681067a0 con 0x7f9e681007e0 2023-10-10T17:36:03.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.188+0000 7f9e5ffff700 1 -- 172.21.15.157:0/1772965164 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f9e600145b0 con 0x7f9e681007e0 2023-10-10T17:36:03.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.189+0000 7f9e5ffff700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9e540a10c0 0x7f9e540a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:03.195 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.189+0000 7f9e5ffff700 1 -- 172.21.15.157:0/1772965164 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f9e60006e40 con 0x7f9e681007e0 2023-10-10T17:36:03.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.189+0000 7f9e66ffd700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9e540a10c0 0x7f9e540a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:03.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.189+0000 7f9e66ffd700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9e540a10c0 0x7f9e540a3570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9e500099d0 tx=0x7f9e50008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:36:03.196 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.192+0000 7f9e5ffff700 1 -- 172.21.15.157:0/1772965164 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9e60095c30 con 0x7f9e681007e0 2023-10-10T17:36:03.386 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:03.385+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e6802cfe0 con 0x7f9e540a10c0 2023-10-10T17:36:03.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:03 smithi078 ceph-mon[99494]: osdmap e9: 1 total, 1 up, 1 in 2023-10-10T17:36:03.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:03 smithi078 ceph-mon[99494]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:03.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:03.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:03.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:03.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:03 smithi190 ceph-mon[101220]: osdmap e9: 1 total, 1 up, 1 in 2023-10-10T17:36:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:03 smithi190 ceph-mon[101220]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:03 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:03 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:03 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:03 smithi157 ceph-mon[100896]: osdmap e9: 1 total, 1 up, 1 in 2023-10-10T17:36:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:03 smithi157 ceph-mon[100896]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:03.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:03 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:03.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:03 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:03.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:03 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:04.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:04 smithi078 ceph-mon[99494]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:04.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:04 smithi190 ceph-mon[101220]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:04.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:04 smithi157 ceph-mon[100896]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:06.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:05 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/2514203422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]: dispatch 2023-10-10T17:36:06.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:05 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]: dispatch 2023-10-10T17:36:06.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:05 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]': finished 2023-10-10T17:36:06.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:05 smithi078 ceph-mon[99494]: osdmap e10: 2 total, 1 up, 2 in 2023-10-10T17:36:06.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:05 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:06.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:05 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/2514203422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]: dispatch 2023-10-10T17:36:06.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:05 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]: dispatch 2023-10-10T17:36:06.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:05 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]': finished 2023-10-10T17:36:06.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:05 smithi190 ceph-mon[101220]: osdmap e10: 2 total, 1 up, 2 in 2023-10-10T17:36:06.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:05 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:06.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:05 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/2514203422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]: dispatch 2023-10-10T17:36:06.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:05 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]: dispatch 2023-10-10T17:36:06.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:05 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c01e2eb5-d98b-4e22-9339-06aaa290b7ae"}]': finished 2023-10-10T17:36:06.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:05 smithi157 ceph-mon[100896]: osdmap e10: 2 total, 1 up, 2 in 2023-10-10T17:36:06.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:05 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:06 smithi078 ceph-mon[99494]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:06 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/3626093581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:06 smithi190 ceph-mon[101220]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:06 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/3626093581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:06 smithi157 ceph-mon[100896]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:06 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/3626093581' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:09.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:09 smithi157 ceph-mon[100896]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:09.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:09 smithi078 ceph-mon[99494]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:09.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:09 smithi190 ceph-mon[101220]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:11 smithi078 ceph-mon[99494]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:11.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:11 smithi190 ceph-mon[101220]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:11 smithi157 ceph-mon[100896]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:12.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-10T17:36:12.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:12.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-10T17:36:12.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:12.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-10T17:36:12.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:13.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:13 smithi157 ceph-mon[100896]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:13.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:13 smithi157 ceph-mon[100896]: Deploying daemon osd.1 on smithi157 2023-10-10T17:36:13.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:13 smithi078 ceph-mon[99494]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:13.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:13 smithi078 ceph-mon[99494]: Deploying daemon osd.1 on smithi157 2023-10-10T17:36:13.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:13 smithi190 ceph-mon[101220]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:13.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:13 smithi190 ceph-mon[101220]: Deploying daemon osd.1 on smithi157 2023-10-10T17:36:15.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:15 smithi157 ceph-mon[100896]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:15.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:15 smithi078 ceph-mon[99494]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:15.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:15 smithi190 ceph-mon[101220]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:17.753 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:17 smithi157 ceph-mon[100896]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:17.753 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:17.754 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:17.754 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:17 smithi078 ceph-mon[99494]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:17.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:17 smithi190 ceph-mon[101220]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:17.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:17.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:17.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:19.673 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:19 smithi157 ceph-mon[100896]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:19.674 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.674 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.674 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:19.674 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:19.675 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.799 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.797+0000 7f9e5ffff700 1 -- 172.21.15.157:0/1772965164 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9e6802cfe0 con 0x7f9e540a10c0 2023-10-10T17:36:19.799 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 1 on host 'smithi157' 2023-10-10T17:36:19.801 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9e540a10c0 msgr2=0x7f9e540a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:19.801 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9e540a10c0 0x7f9e540a3570 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9e500099d0 tx=0x7f9e50008040 comp rx=0 tx=0).stop 2023-10-10T17:36:19.801 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e681007e0 msgr2=0x7f9e6819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:19.801 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e681007e0 0x7f9e6819d460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000b970 tx=0x7f9e6000bd30 comp rx=0 tx=0).stop 2023-10-10T17:36:19.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 shutdown_connections 2023-10-10T17:36:19.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9e540a10c0 0x7f9e540a3570 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:19.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9e681007e0 0x7f9e6819d460 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:19.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9e680ff440 0x7f9e680feeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:19.802 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 --2- 172.21.15.157:0/1772965164 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9e680ffdf0 0x7f9e6819ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:19.803 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 >> 172.21.15.157:0/1772965164 conn(0x7f9e680fa000 msgr2=0x7f9e681038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:36:19.803 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.799+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 shutdown_connections 2023-10-10T17:36:19.803 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:36:19.800+0000 7f9e6d61e700 1 -- 172.21.15.157:0/1772965164 wait complete. 2023-10-10T17:36:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:19 smithi078 ceph-mon[99494]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:19 smithi190 ceph-mon[101220]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:19.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:20.207 DEBUG:teuthology.orchestra.run.smithi157:osd.1> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.1.service 2023-10-10T17:36:20.210 INFO:tasks.cephadm:Deploying osd.2 on smithi190 with /dev/vg_nvme/lv_4... 2023-10-10T17:36:20.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-10T17:36:20.242 INFO:journalctl@ceph.osd.1.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:36:20.728 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-1[105957]: 2023-10-10T17:36:20.460+0000 7fbc3880a640 -1 osd.1 0 log_to_monitors true 2023-10-10T17:36:20.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:20.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:36:20.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:20.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:20.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:20.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:20.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: from='osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-10T17:36:20.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:20 smithi157 ceph-mon[100896]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-10T17:36:20.856 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:36:20.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:20.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:36:20.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:20.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:20.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:20.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:20.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: from='osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-10T17:36:20.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:20 smithi078 ceph-mon[99494]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-10T17:36:20.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:20.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:36:20.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:20.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:20.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:20.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:20.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: from='osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-10T17:36:20.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:20 smithi190 ceph-mon[101220]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-10T17:36:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-mon[100896]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-mon[100896]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-10T17:36:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-mon[100896]: osdmap e11: 2 total, 1 up, 2 in 2023-10-10T17:36:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:21.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-mon[100896]: from='osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:36:21.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-mon[100896]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:36:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:21 smithi078 ceph-mon[99494]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:21 smithi078 ceph-mon[99494]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-10T17:36:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:21 smithi078 ceph-mon[99494]: osdmap e11: 2 total, 1 up, 2 in 2023-10-10T17:36:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:21.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:21 smithi078 ceph-mon[99494]: from='osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:36:21.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:21 smithi078 ceph-mon[99494]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:36:21.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:21 smithi190 ceph-mon[101220]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:21 smithi190 ceph-mon[101220]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-10T17:36:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:21 smithi190 ceph-mon[101220]: osdmap e11: 2 total, 1 up, 2 in 2023-10-10T17:36:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:21 smithi190 ceph-mon[101220]: from='osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:36:21.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:21 smithi190 ceph-mon[101220]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:36:22.045 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-1[105957]: 2023-10-10T17:36:21.774+0000 7fbc2de81700 -1 osd.1 0 waiting for initial osdmap 2023-10-10T17:36:22.045 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:21 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-1[105957]: 2023-10-10T17:36:21.778+0000 7fbc28c74700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:36:22.426 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-10T17:36:22.427 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-10T17:36:22.427 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2023-10-10T17:36:22.427 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2023-10-10T17:36:22.427 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109451 s, 958 MB/s 2023-10-10T17:36:22.427 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2023-10-10T17:36:22.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2023-10-10T17:36:23.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:36:23.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: osdmap e12: 2 total, 1 up, 2 in 2023-10-10T17:36:23.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:23.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:23.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:23.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi157 2023-10-10T17:36:23.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:23.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi157 to 17168M 2023-10-10T17:36:23.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:23.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:23.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:23.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:36:23.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:22 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:36:23.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: osdmap e12: 2 total, 1 up, 2 in 2023-10-10T17:36:23.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:23.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:23.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:23.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi157 2023-10-10T17:36:23.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:23.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi157 to 17168M 2023-10-10T17:36:23.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:23.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:23.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:23.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:36:23.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:22 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:23.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:36:23.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: osdmap e12: 2 total, 1 up, 2 in 2023-10-10T17:36:23.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:23.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:23.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: pgmap v55: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-10-10T17:36:23.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi157 2023-10-10T17:36:23.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:23.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi157 to 17168M 2023-10-10T17:36:23.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:23.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:23.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:23.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:23.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:36:23.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:22 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:23.541 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:36:24.034 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:23 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:36:24.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:23 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:36:24.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:23 smithi190 ceph-mon[101220]: osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] boot 2023-10-10T17:36:24.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:23 smithi190 ceph-mon[101220]: osdmap e13: 2 total, 2 up, 2 in 2023-10-10T17:36:24.035 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:24.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:23 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:36:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:23 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:36:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:23 smithi078 ceph-mon[99494]: osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] boot 2023-10-10T17:36:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:23 smithi078 ceph-mon[99494]: osdmap e13: 2 total, 2 up, 2 in 2023-10-10T17:36:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:24.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:23 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:36:24.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:23 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:36:24.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:23 smithi157 ceph-mon[100896]: osd.1 [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] boot 2023-10-10T17:36:24.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:23 smithi157 ceph-mon[100896]: osdmap e13: 2 total, 2 up, 2 in 2023-10-10T17:36:24.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:36:24.715 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 -- 172.21.15.190:0/3953173887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc1003e0 msgr2=0x7f18cc100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:24.715 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3953173887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc1003e0 0x7f18cc100830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009990 tx=0x7f18c8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:36:24.716 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 -- 172.21.15.190:0/3953173887 shutdown_connections 2023-10-10T17:36:24.716 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3953173887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18cc100d70 0x7f18cc107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:24.716 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3953173887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18cc0ff1e0 0x7f18cc0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:24.716 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3953173887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc1003e0 0x7f18cc100830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:24.716 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 -- 172.21.15.190:0/3953173887 >> 172.21.15.190:0/3953173887 conn(0x7f18cc0fa7f0 msgr2=0x7f18cc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:36:24.717 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 -- 172.21.15.190:0/3953173887 shutdown_connections 2023-10-10T17:36:24.717 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.711+0000 7f18d3a77700 1 -- 172.21.15.190:0/3953173887 wait complete. 2023-10-10T17:36:24.717 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 Processor -- start 2023-10-10T17:36:24.717 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 -- start start 2023-10-10T17:36:24.717 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18cc0ff1e0 0x7f18cc194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:24.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18cc1003e0 0x7f18cc1954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:24.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc100d70 0x7f18cc19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:24.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc19cb40 con 0x7f18cc1003e0 2023-10-10T17:36:24.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc19cc80 con 0x7f18cc0ff1e0 2023-10-10T17:36:24.718 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d3a77700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc19ce10 con 0x7f18cc100d70 2023-10-10T17:36:24.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d2014700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc100d70 0x7f18cc19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:24.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.712+0000 7f18d2014700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc100d70 0x7f18cc19c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:51816/0 (socket says 172.21.15.190:51816) 2023-10-10T17:36:24.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d2014700 1 -- 172.21.15.190:0/3199625486 learned_addr learned my addr 172.21.15.190:0/3199625486 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:36:24.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d1012700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18cc1003e0 0x7f18cc1954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:24.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d1813700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18cc0ff1e0 0x7f18cc194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:24.719 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d2014700 1 -- 172.21.15.190:0/3199625486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18cc0ff1e0 msgr2=0x7f18cc194fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:24.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d2014700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18cc0ff1e0 0x7f18cc194fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:24.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d2014700 1 -- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18cc1003e0 msgr2=0x7f18cc1954f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:24.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d2014700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18cc1003e0 0x7f18cc1954f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:24.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d2014700 1 -- 172.21.15.190:0/3199625486 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c80095f0 con 0x7f18cc100d70 2023-10-10T17:36:24.720 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d2014700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc100d70 0x7f18cc19c570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18c400b9a0 tx=0x7f18c400bcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:36:24.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18c2ffd700 1 -- 172.21.15.190:0/3199625486 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c400d850 con 0x7f18cc100d70 2023-10-10T17:36:24.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18c2ffd700 1 -- 172.21.15.190:0/3199625486 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18c400de70 con 0x7f18cc100d70 2023-10-10T17:36:24.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc10abb0 con 0x7f18cc100d70 2023-10-10T17:36:24.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18c2ffd700 1 -- 172.21.15.190:0/3199625486 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c4017400 con 0x7f18cc100d70 2023-10-10T17:36:24.721 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.713+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc10aa20 con 0x7f18cc100d70 2023-10-10T17:36:24.722 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.714+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc106200 con 0x7f18cc100d70 2023-10-10T17:36:24.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.715+0000 7f18c2ffd700 1 -- 172.21.15.190:0/3199625486 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f18c400e600 con 0x7f18cc100d70 2023-10-10T17:36:24.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.716+0000 7f18c2ffd700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18b80a1140 0x7f18b80a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:24.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.716+0000 7f18c2ffd700 1 -- 172.21.15.190:0/3199625486 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f18c4006e40 con 0x7f18cc100d70 2023-10-10T17:36:24.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.716+0000 7f18d1813700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18b80a1140 0x7f18b80a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:24.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.716+0000 7f18d1813700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18b80a1140 0x7f18b80a35f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f18cc100240 tx=0x7f18bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:36:24.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.720+0000 7f18c2ffd700 1 -- 172.21.15.190:0/3199625486 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f18c4095b00 con 0x7f18cc100d70 2023-10-10T17:36:24.916 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:24.914+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f18cc0692c0 con 0x7f18b80a1140 2023-10-10T17:36:25.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:24 smithi078 ceph-mon[99494]: osdmap e14: 2 total, 2 up, 2 in 2023-10-10T17:36:25.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:24 smithi078 ceph-mon[99494]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:24 smithi190 ceph-mon[101220]: osdmap e14: 2 total, 2 up, 2 in 2023-10-10T17:36:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:24 smithi190 ceph-mon[101220]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:25.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:24 smithi157 ceph-mon[100896]: osdmap e14: 2 total, 2 up, 2 in 2023-10-10T17:36:25.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:24 smithi157 ceph-mon[100896]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:25 smithi078 ceph-mon[99494]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:26.151 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:25 smithi190 ceph-mon[101220]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:26.151 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:26.151 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:26.151 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:26.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:25 smithi157 ceph-mon[100896]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:26.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:26.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:26.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:26 smithi078 ceph-mon[99494]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:27.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:26 smithi190 ceph-mon[101220]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:27.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:26 smithi157 ceph-mon[100896]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:27 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/2701607730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]: dispatch 2023-10-10T17:36:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:27 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]: dispatch 2023-10-10T17:36:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:27 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]': finished 2023-10-10T17:36:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:27 smithi078 ceph-mon[99494]: osdmap e15: 3 total, 2 up, 3 in 2023-10-10T17:36:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:28.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:27 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/2701607730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]: dispatch 2023-10-10T17:36:28.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:27 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]: dispatch 2023-10-10T17:36:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:27 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]': finished 2023-10-10T17:36:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:27 smithi190 ceph-mon[101220]: osdmap e15: 3 total, 2 up, 3 in 2023-10-10T17:36:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:27 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/2701607730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]: dispatch 2023-10-10T17:36:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:27 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]: dispatch 2023-10-10T17:36:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:27 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4c48ed5d-d817-45b6-9c2c-1c371df35f95"}]': finished 2023-10-10T17:36:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:27 smithi157 ceph-mon[100896]: osdmap e15: 3 total, 2 up, 3 in 2023-10-10T17:36:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/2845844773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:28 smithi078 ceph-mon[99494]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/2845844773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:28 smithi190 ceph-mon[101220]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/2845844773' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:28 smithi157 ceph-mon[100896]: pgmap v61: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:31.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:31 smithi190 ceph-mon[101220]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:31.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:31 smithi157 ceph-mon[100896]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:31 smithi078 ceph-mon[99494]: pgmap v62: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:33.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:33 smithi157 ceph-mon[100896]: pgmap v63: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:33.627 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:33 smithi190 ceph-mon[101220]: pgmap v63: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:33 smithi078 ceph-mon[99494]: pgmap v63: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:34.758 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-10T17:36:34.759 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:34.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-10T17:36:34.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:34.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-10T17:36:34.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:35 smithi078 ceph-mon[99494]: Deploying daemon osd.2 on smithi190 2023-10-10T17:36:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:35 smithi078 ceph-mon[99494]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:35 smithi190 ceph-mon[101220]: Deploying daemon osd.2 on smithi190 2023-10-10T17:36:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:35 smithi190 ceph-mon[101220]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:35 smithi157 ceph-mon[100896]: Deploying daemon osd.2 on smithi190 2023-10-10T17:36:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:35 smithi157 ceph-mon[100896]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:37.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:37 smithi078 ceph-mon[99494]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:37.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:37 smithi190 ceph-mon[101220]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:37.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:37 smithi157 ceph-mon[100896]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:39.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:39 smithi190 ceph-mon[101220]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:39.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:39.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:39.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:39.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:39 smithi157 ceph-mon[100896]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:39.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:39.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:39.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:39 smithi078 ceph-mon[99494]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:41.429 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 2 on host 'smithi190' 2023-10-10T17:36:41.429 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.428+0000 7f18c2ffd700 1 -- 172.21.15.190:0/3199625486 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f18cc0692c0 con 0x7f18b80a1140 2023-10-10T17:36:41.431 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18b80a1140 msgr2=0x7f18b80a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:41.431 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18b80a1140 0x7f18b80a35f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f18cc100240 tx=0x7f18bc00b410 comp rx=0 tx=0).stop 2023-10-10T17:36:41.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc100d70 msgr2=0x7f18cc19c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:41.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc100d70 0x7f18cc19c570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18c400b9a0 tx=0x7f18c400bcb0 comp rx=0 tx=0).stop 2023-10-10T17:36:41.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 shutdown_connections 2023-10-10T17:36:41.432 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18b80a1140 0x7f18b80a35f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:41.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18cc100d70 0x7f18cc19c570 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:41.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18cc0ff1e0 0x7f18cc194fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:41.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 --2- 172.21.15.190:0/3199625486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18cc1003e0 0x7f18cc1954f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:41.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.430+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 >> 172.21.15.190:0/3199625486 conn(0x7f18cc0fa7f0 msgr2=0x7f18cc103960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:36:41.433 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.431+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 shutdown_connections 2023-10-10T17:36:41.434 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:36:41.431+0000 7f18d3a77700 1 -- 172.21.15.190:0/3199625486 wait complete. 2023-10-10T17:36:41.806 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:41.807 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.807 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.807 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:41.807 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:36:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.808 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.809 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:41 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:41.832 DEBUG:teuthology.orchestra.run.smithi190:osd.2> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.2.service 2023-10-10T17:36:41.834 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_3... 2023-10-10T17:36:41.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-10T17:36:41.863 INFO:journalctl@ceph.osd.2.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:36:41.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:41.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:41.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:41.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:41.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.868 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:36:41.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:41.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:41 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:36:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:41 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:36:42.486 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:36:42.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:42 smithi190 ceph-mon[101220]: from='osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-10T17:36:42.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:42 smithi190 ceph-mon[101220]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-10T17:36:42.666 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-2[107039]: 2023-10-10T17:36:42.470+0000 7f7f4155f640 -1 osd.2 0 log_to_monitors true 2023-10-10T17:36:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:42 smithi078 ceph-mon[99494]: from='osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-10T17:36:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:42 smithi078 ceph-mon[99494]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-10T17:36:42.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:42 smithi157 ceph-mon[100896]: from='osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-10T17:36:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:42 smithi157 ceph-mon[100896]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-10T17:36:43.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:43 smithi078 ceph-mon[99494]: pgmap v68: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:43.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:43 smithi078 ceph-mon[99494]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-10T17:36:43.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:43 smithi078 ceph-mon[99494]: osdmap e16: 3 total, 2 up, 3 in 2023-10-10T17:36:43.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:43.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:43 smithi078 ceph-mon[99494]: from='osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:36:43.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:43 smithi078 ceph-mon[99494]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:36:43.823 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-2[107039]: 2023-10-10T17:36:43.542+0000 7f7f36bd6700 -1 osd.2 0 waiting for initial osdmap 2023-10-10T17:36:43.823 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-2[107039]: 2023-10-10T17:36:43.547+0000 7f7f329cb700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:36:43.823 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-mon[101220]: pgmap v68: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:43.824 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-mon[101220]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-10T17:36:43.824 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-mon[101220]: osdmap e16: 3 total, 2 up, 3 in 2023-10-10T17:36:43.824 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:43.824 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-mon[101220]: from='osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:36:43.825 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:43 smithi190 ceph-mon[101220]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:36:43.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:43 smithi157 ceph-mon[100896]: pgmap v68: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:43 smithi157 ceph-mon[100896]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-10T17:36:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:43 smithi157 ceph-mon[100896]: osdmap e16: 3 total, 2 up, 3 in 2023-10-10T17:36:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:43.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:43 smithi157 ceph-mon[100896]: from='osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:36:43.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:43 smithi157 ceph-mon[100896]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:36:43.993 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-10T17:36:43.994 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-10T17:36:43.994 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-10-10T17:36:43.994 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-10-10T17:36:43.994 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111939 s, 937 MB/s 2023-10-10T17:36:43.995 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-10-10T17:36:44.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2023-10-10T17:36:44.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:36:44.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: osdmap e17: 3 total, 2 up, 3 in 2023-10-10T17:36:44.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:44.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:44.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:44.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:36:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: osdmap e17: 3 total, 2 up, 3 in 2023-10-10T17:36:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:44.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:44.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:36:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: osdmap e17: 3 total, 2 up, 3 in 2023-10-10T17:36:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:44.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:36:44.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:44.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:44.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:36:44.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:45.028 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:36:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:36:45.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:36:45.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:45.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi190 2023-10-10T17:36:45.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi190 to 17168M 2023-10-10T17:36:45.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:45.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:36:45.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:36:45.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] boot 2023-10-10T17:36:45.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: osdmap e18: 3 total, 3 up, 3 in 2023-10-10T17:36:45.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:45 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:45.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:36:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:36:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi190 2023-10-10T17:36:45.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi190 to 17168M 2023-10-10T17:36:45.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:45.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:36:45.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:36:45.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] boot 2023-10-10T17:36:45.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: osdmap e18: 3 total, 3 up, 3 in 2023-10-10T17:36:45.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:45 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:36:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:36:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: pgmap v71: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-10-10T17:36:45.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi190 2023-10-10T17:36:45.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi190 to 17168M 2023-10-10T17:36:45.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:45.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:36:45.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:36:45.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: osd.2 [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] boot 2023-10-10T17:36:45.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: osdmap e18: 3 total, 3 up, 3 in 2023-10-10T17:36:45.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:45 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:36:46.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2197780226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8105fb0 msgr2=0x7fd2f8108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:46.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2197780226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8105fb0 0x7fd2f8108410 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd2f000b0a0 tx=0x7fd2f000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:36:46.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2197780226 shutdown_connections 2023-10-10T17:36:46.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2197780226 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2f8100d70 0x7fd2f8103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:46.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2197780226 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8105fb0 0x7fd2f8108410 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:46.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2197780226 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2f8103690 0x7fd2f8105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:46.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2197780226 >> 172.21.15.78:0/2197780226 conn(0x7fd2f80fa7f0 msgr2=0x7fd2f80fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:36:46.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2197780226 shutdown_connections 2023-10-10T17:36:46.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.461+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2197780226 wait complete. 2023-10-10T17:36:46.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 Processor -- start 2023-10-10T17:36:46.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 -- start start 2023-10-10T17:36:46.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2f8100d70 0x7fd2f81961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8103690 0x7fd2f81966e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2f8105fb0 0x7fd2f819d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f819dd30 con 0x7fd2f8103690 2023-10-10T17:36:46.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f819de70 con 0x7fd2f8100d70 2023-10-10T17:36:46.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2ff1e3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f819e000 con 0x7fd2f8105fb0 2023-10-10T17:36:46.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8103690 0x7fd2f81966e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:46.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8103690 0x7fd2f81966e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41322/0 (socket says 172.21.15.78:41322) 2023-10-10T17:36:46.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 -- 172.21.15.78:0/2629700510 learned_addr learned my addr 172.21.15.78:0/2629700510 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:36:46.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2fd780700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2f8105fb0 0x7fd2f819d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:46.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 -- 172.21.15.78:0/2629700510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2f8105fb0 msgr2=0x7fd2f819d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:46.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2fcf7f700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2f8100d70 0x7fd2f81961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:46.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2f8105fb0 0x7fd2f819d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:46.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 -- 172.21.15.78:0/2629700510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2f8100d70 msgr2=0x7fd2f81961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:36:46.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2f8100d70 0x7fd2f81961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:36:46.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 -- 172.21.15.78:0/2629700510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e4009520 con 0x7fd2f8103690 2023-10-10T17:36:46.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2fd780700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2f8105fb0 0x7fd2f819d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:36:46.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2fcf7f700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2f8100d70 0x7fd2f81961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:36:46.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f7fff700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8103690 0x7fd2f81966e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400eb60 tx=0x7fd2e400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:36:46.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.462+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e400cd40 con 0x7fd2f8103690 2023-10-10T17:36:46.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.463+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f0009d00 con 0x7fd2f8103690 2023-10-10T17:36:46.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.463+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd2e400cea0 con 0x7fd2f8103690 2023-10-10T17:36:46.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.463+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4010430 con 0x7fd2f8103690 2023-10-10T17:36:46.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.463+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f810bd00 con 0x7fd2f8103690 2023-10-10T17:36:46.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.464+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f810be90 con 0x7fd2f8103690 2023-10-10T17:36:46.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.465+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fd2e4003710 con 0x7fd2f8103690 2023-10-10T17:36:46.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.465+0000 7fd2f5ffb700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2e80a1140 0x7fd2e80a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:36:46.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.466+0000 7fd2fcf7f700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2e80a1140 0x7fd2e80a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:36:46.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.466+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7fd2e4014070 con 0x7fd2f8103690 2023-10-10T17:36:46.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.466+0000 7fd2fcf7f700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2e80a1140 0x7fd2e80a35f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009be0 tx=0x7fd2ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:36:46.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.469+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd2e40965b0 con 0x7fd2f8103690 2023-10-10T17:36:46.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:46.667+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2f8060f30 con 0x7fd2e80a1140 2023-10-10T17:36:46.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:46 smithi078 ceph-mon[99494]: osdmap e19: 3 total, 3 up, 3 in 2023-10-10T17:36:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:46 smithi078 ceph-mon[99494]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:46 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:36:46.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:46 smithi190 ceph-mon[101220]: osdmap e19: 3 total, 3 up, 3 in 2023-10-10T17:36:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:46 smithi190 ceph-mon[101220]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:46 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:36:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:46 smithi157 ceph-mon[100896]: osdmap e19: 3 total, 3 up, 3 in 2023-10-10T17:36:47.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:46 smithi157 ceph-mon[100896]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:47.001 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:46 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:36:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-10T17:36:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:47 smithi078 ceph-mon[99494]: osdmap e20: 3 total, 3 up, 3 in 2023-10-10T17:36:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:36:47.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:47 smithi078 ceph-mon[99494]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:47.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:47.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:47.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:47.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-10T17:36:47.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:47 smithi190 ceph-mon[101220]: osdmap e20: 3 total, 3 up, 3 in 2023-10-10T17:36:47.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:36:47.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:47 smithi190 ceph-mon[101220]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:47.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:47.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:47.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-10T17:36:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:47 smithi157 ceph-mon[100896]: osdmap e20: 3 total, 3 up, 3 in 2023-10-10T17:36:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:36:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:47 smithi157 ceph-mon[100896]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:36:47.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:36:47.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:36:47.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:48.175 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:47 smithi078 sudo[115513]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-10T17:36:48.175 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:47 smithi078 sudo[115513]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:48.177 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:47 smithi078 sudo[115513]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:48.177 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:47 smithi078 sudo[115513]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:48.468 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:48 smithi078 sudo[115517]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-10T17:36:48.469 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:48 smithi078 sudo[115517]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:48.469 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:48 smithi078 sudo[115517]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:48.469 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:36:48 smithi078 sudo[115517]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:48.696 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:48 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-10T17:36:48.697 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:48 smithi157 ceph-mon[100896]: osdmap e21: 3 total, 3 up, 3 in 2023-10-10T17:36:48.697 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:48 smithi157 ceph-mon[100896]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:48.697 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:48 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:48.697 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108267]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-10T17:36:48.698 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108267]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:48.698 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108267]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:48.698 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108267]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:48.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:48 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-10T17:36:48.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:48 smithi078 ceph-mon[99494]: osdmap e21: 3 total, 3 up, 3 in 2023-10-10T17:36:48.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:48 smithi078 ceph-mon[99494]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:48.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:48 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:48.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:48 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-10T17:36:48.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:48 smithi190 ceph-mon[101220]: osdmap e21: 3 total, 3 up, 3 in 2023-10-10T17:36:48.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:48 smithi190 ceph-mon[101220]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 15 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:48.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:48 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:48.978 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108270]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-10T17:36:48.978 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108270]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:48.979 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108270]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:48.979 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:36:48 smithi157 sudo[108270]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:49.213 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:48 smithi190 sudo[109279]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-10T17:36:49.214 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:48 smithi190 sudo[109279]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:49.214 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:48 smithi190 sudo[109279]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:49.214 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:48 smithi190 sudo[109279]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:49.664 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:49 smithi190 sudo[109282]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-10T17:36:49.665 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:49 smithi190 sudo[109282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:49.665 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:49 smithi190 sudo[109282]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:49.665 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:36:49 smithi190 sudo[109282]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:49.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115614]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-10T17:36:49.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115614]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:49.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115614]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:49.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115614]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:49.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:36:49.770+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd2e4096980 con 0x7fd2f8103690 2023-10-10T17:36:50.008 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: osdmap e22: 3 total, 3 up, 3 in 2023-10-10T17:36:50.009 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:50.009 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/283853232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b2e185b-92ed-4d3e-8d87-e8f34fc640e2"}]: dispatch 2023-10-10T17:36:50.009 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/283853232' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b2e185b-92ed-4d3e-8d87-e8f34fc640e2"}]': finished 2023-10-10T17:36:50.009 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: osdmap e23: 4 total, 3 up, 4 in 2023-10-10T17:36:50.009 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:36:50.010 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:50.010 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 ceph-mon[101220]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:50.010 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 sudo[109285]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-10T17:36:50.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 sudo[109285]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:50.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:49 smithi190 sudo[109285]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:50.011 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 sudo[109285]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:50.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115642]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-10T17:36:50.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115642]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:50.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115642]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:50.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 sudo[115642]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:50.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: osdmap e22: 3 total, 3 up, 3 in 2023-10-10T17:36:50.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:50.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/283853232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b2e185b-92ed-4d3e-8d87-e8f34fc640e2"}]: dispatch 2023-10-10T17:36:50.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/283853232' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b2e185b-92ed-4d3e-8d87-e8f34fc640e2"}]': finished 2023-10-10T17:36:50.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: osdmap e23: 4 total, 3 up, 4 in 2023-10-10T17:36:50.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:36:50.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:50.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:49 smithi078 ceph-mon[99494]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:50.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: osdmap e22: 3 total, 3 up, 3 in 2023-10-10T17:36:50.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:50.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/283853232' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b2e185b-92ed-4d3e-8d87-e8f34fc640e2"}]: dispatch 2023-10-10T17:36:50.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/283853232' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b2e185b-92ed-4d3e-8d87-e8f34fc640e2"}]': finished 2023-10-10T17:36:50.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: osdmap e23: 4 total, 3 up, 4 in 2023-10-10T17:36:50.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:36:50.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:36:50.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:49 smithi157 ceph-mon[100896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:50.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 sudo[109288]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-10T17:36:50.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 sudo[109288]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:50.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 sudo[109288]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:50.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 sudo[109288]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:50.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108273]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-10T17:36:50.738 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108273]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:50.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108273]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:50.739 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108273]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-10-10T17:36:51.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/66739781' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:51.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:51.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:51.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:50 smithi078 ceph-mon[99494]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:51.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-10-10T17:36:51.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/66739781' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:51.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:51.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:51.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:50 smithi190 ceph-mon[101220]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108276]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-10T17:36:51.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108276]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-10T17:36:51.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108276]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-10T17:36:51.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 sudo[108276]: pam_unix(sudo:session): session closed for user root 2023-10-10T17:36:51.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:51.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-10-10T17:36:51.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/66739781' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:36:51.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:51.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:51.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-10T17:36:51.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:36:51.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:36:51.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:36:51.232 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:50 smithi157 ceph-mon[100896]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-10T17:36:53.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:53 smithi078 ceph-mon[99494]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:53.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:53 smithi157 ceph-mon[100896]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:53.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:53 smithi190 ceph-mon[101220]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:36:55.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:55 smithi078 ceph-mon[99494]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-10-10T17:36:55.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:55 smithi190 ceph-mon[101220]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-10-10T17:36:55.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:55 smithi157 ceph-mon[100896]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 305 KiB/s wr, 20 op/s 2023-10-10T17:36:56.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:56 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-10T17:36:56.755 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:56 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:56.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:56 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-10T17:36:56.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:56 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:56 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-10T17:36:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:56 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:36:57.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:57 smithi078 ceph-mon[99494]: Deploying daemon osd.3 on smithi078 2023-10-10T17:36:57.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:57 smithi078 ceph-mon[99494]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-10-10T17:36:57.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:57 smithi190 ceph-mon[101220]: Deploying daemon osd.3 on smithi078 2023-10-10T17:36:57.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:57 smithi190 ceph-mon[101220]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-10-10T17:36:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:57 smithi157 ceph-mon[100896]: Deploying daemon osd.3 on smithi078 2023-10-10T17:36:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:57 smithi157 ceph-mon[100896]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 228 KiB/s wr, 15 op/s 2023-10-10T17:36:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:36:59 smithi078 ceph-mon[99494]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 190 KiB/s wr, 13 op/s 2023-10-10T17:36:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:36:59 smithi190 ceph-mon[101220]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 190 KiB/s wr, 13 op/s 2023-10-10T17:36:59.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:36:59 smithi157 ceph-mon[100896]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 190 KiB/s wr, 13 op/s 2023-10-10T17:37:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:01 smithi078 ceph-mon[99494]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-10-10T17:37:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:01.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:01 smithi190 ceph-mon[101220]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-10-10T17:37:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:01 smithi157 ceph-mon[100896]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-10-10T17:37:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:03.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.522+0000 7fd2f5ffb700 1 -- 172.21.15.78:0/2629700510 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd2f8060f30 con 0x7fd2e80a1140 2023-10-10T17:37:03.524 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2023-10-10T17:37:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2e80a1140 msgr2=0x7fd2e80a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2e80a1140 0x7fd2e80a35f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009be0 tx=0x7fd2ec009260 comp rx=0 tx=0).stop 2023-10-10T17:37:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8103690 msgr2=0x7fd2f81966e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:03.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8103690 0x7fd2f81966e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400eb60 tx=0x7fd2e400ef20 comp rx=0 tx=0).stop 2023-10-10T17:37:03.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 shutdown_connections 2023-10-10T17:37:03.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2e80a1140 0x7fd2e80a35f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:03.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2f8105fb0 0x7fd2f819d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:03.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2f8100d70 0x7fd2f81961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:03.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 --2- 172.21.15.78:0/2629700510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2f8103690 0x7fd2f81966e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:03.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 >> 172.21.15.78:0/2629700510 conn(0x7fd2f80fa7f0 msgr2=0x7fd2f81019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:37:03.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.524+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 shutdown_connections 2023-10-10T17:37:03.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:03.525+0000 7fd2ff1e3700 1 -- 172.21.15.78:0/2629700510 wait complete. 2023-10-10T17:37:03.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:03 smithi078 ceph-mon[99494]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-10-10T17:37:03.708 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:03 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:03 smithi190 ceph-mon[101220]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-10-10T17:37:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:03 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:03 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:03 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:03 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:03 smithi157 ceph-mon[100896]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-10-10T17:37:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:03 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:03 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:03 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:03.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:03 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:04.013 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.3.service 2023-10-10T17:37:04.015 INFO:tasks.cephadm:Deploying osd.4 on smithi157 with /dev/vg_nvme/lv_3... 2023-10-10T17:37:04.016 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-10T17:37:04.045 INFO:journalctl@ceph.osd.3.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:37:04.669 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:37:04.709 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:04 smithi078 ceph-mon[99494]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-10T17:37:04.710 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:37:04 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:37:04.434+0000 7f41ede1e640 -1 osd.3 0 log_to_monitors true 2023-10-10T17:37:04.861 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:04 smithi157 ceph-mon[100896]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-10T17:37:04.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:04 smithi190 ceph-mon[101220]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-10T17:37:05.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:05 smithi078 ceph-mon[99494]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-10-10T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:05 smithi078 ceph-mon[99494]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-10T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:05 smithi078 ceph-mon[99494]: osdmap e24: 4 total, 3 up, 4 in 2023-10-10T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:05 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:05.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:05 smithi078 ceph-mon[99494]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:37:05.880 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:37:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:37:05.532+0000 7f41e3495700 -1 osd.3 0 waiting for initial osdmap 2023-10-10T17:37:05.880 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:37:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:37:05.537+0000 7f41dca85700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:37:05.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:05 smithi190 ceph-mon[101220]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-10-10T17:37:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:05 smithi190 ceph-mon[101220]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-10T17:37:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:05 smithi190 ceph-mon[101220]: osdmap e24: 4 total, 3 up, 4 in 2023-10-10T17:37:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:05 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:05 smithi190 ceph-mon[101220]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:37:05.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:05 smithi157 ceph-mon[100896]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-10-10T17:37:05.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:05 smithi157 ceph-mon[100896]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-10T17:37:05.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:05 smithi157 ceph-mon[100896]: osdmap e24: 4 total, 3 up, 4 in 2023-10-10T17:37:05.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:05 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:05.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:05 smithi157 ceph-mon[100896]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:37:06.054 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-10T17:37:06.054 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-10T17:37:06.054 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2023-10-10T17:37:06.055 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2023-10-10T17:37:06.055 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123713 s, 848 MB/s 2023-10-10T17:37:06.055 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2023-10-10T17:37:06.409 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi157:vg_nvme/lv_3 2023-10-10T17:37:06.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:37:06.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: osdmap e25: 4 total, 3 up, 4 in 2023-10-10T17:37:06.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:06.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:06.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:06.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:06.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:06.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:06.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] boot 2023-10-10T17:37:06.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:06 smithi157 ceph-mon[100896]: osdmap e26: 4 total, 4 up, 4 in 2023-10-10T17:37:06.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:37:06.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: osdmap e25: 4 total, 3 up, 4 in 2023-10-10T17:37:06.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:06.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:06.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:06.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:06.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:06.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:06.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] boot 2023-10-10T17:37:06.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:06 smithi078 ceph-mon[99494]: osdmap e26: 4 total, 4 up, 4 in 2023-10-10T17:37:06.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:37:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: osdmap e25: 4 total, 3 up, 4 in 2023-10-10T17:37:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:06.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:06.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: osd.3 [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] boot 2023-10-10T17:37:06.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:06 smithi190 ceph-mon[101220]: osdmap e26: 4 total, 4 up, 4 in 2023-10-10T17:37:07.058 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:37:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:37:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:37:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:37:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi078 2023-10-10T17:37:07.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi078 to 8584M 2023-10-10T17:37:07.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:07.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:37:07.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:07.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:07.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:07.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:07.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:07.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:07.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:07 smithi078 ceph-mon[99494]: osdmap e27: 4 total, 4 up, 4 in 2023-10-10T17:37:07.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:37:07.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:37:07.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:37:07.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi078 2023-10-10T17:37:07.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi078 to 8584M 2023-10-10T17:37:07.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:07.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:37:07.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:07.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:07.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:07.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:07.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:07.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:07.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:07 smithi190 ceph-mon[101220]: osdmap e27: 4 total, 4 up, 4 in 2023-10-10T17:37:07.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:37:07.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:37:07.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-10-10T17:37:07.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi078 2023-10-10T17:37:07.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi078 to 8584M 2023-10-10T17:37:07.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:07.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:37:07.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:07.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:37:07.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:07.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:07.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:07.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:07.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:07 smithi157 ceph-mon[100896]: osdmap e27: 4 total, 4 up, 4 in 2023-10-10T17:37:08.239 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.235+0000 7fac9782b700 1 -- 172.21.15.157:0/2156044874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac900ff1e0 msgr2=0x7fac900ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:08.239 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.235+0000 7fac9782b700 1 --2- 172.21.15.157:0/2156044874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac900ff1e0 0x7fac900ff5f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac80009990 tx=0x7fac80009ca0 comp rx=0 tx=0).stop 2023-10-10T17:37:08.239 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.235+0000 7fac9782b700 1 -- 172.21.15.157:0/2156044874 shutdown_connections 2023-10-10T17:37:08.240 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.235+0000 7fac9782b700 1 --2- 172.21.15.157:0/2156044874 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac90100d70 0x7fac90107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:08.240 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.235+0000 7fac9782b700 1 --2- 172.21.15.157:0/2156044874 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac900ff1e0 0x7fac900ff5f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:08.240 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.235+0000 7fac9782b700 1 --2- 172.21.15.157:0/2156044874 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac901003e0 0x7fac90100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:08.240 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.235+0000 7fac9782b700 1 -- 172.21.15.157:0/2156044874 >> 172.21.15.157:0/2156044874 conn(0x7fac900fa7f0 msgr2=0x7fac900fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:37:08.240 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 -- 172.21.15.157:0/2156044874 shutdown_connections 2023-10-10T17:37:08.241 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 -- 172.21.15.157:0/2156044874 wait complete. 2023-10-10T17:37:08.241 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 Processor -- start 2023-10-10T17:37:08.241 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 -- start start 2023-10-10T17:37:08.241 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac900ff1e0 0x7fac900699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:08.241 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac901003e0 0x7fac9019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:08.242 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac90100d70 0x7fac9019d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:08.242 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9019daf0 con 0x7fac901003e0 2023-10-10T17:37:08.242 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9019dc60 con 0x7fac90100d70 2023-10-10T17:37:08.242 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac9782b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9019de20 con 0x7fac900ff1e0 2023-10-10T17:37:08.243 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac95dc8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac90100d70 0x7fac9019d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:08.243 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac95dc8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac90100d70 0x7fac9019d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:58414/0 (socket says 172.21.15.157:58414) 2023-10-10T17:37:08.243 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.236+0000 7fac95dc8700 1 -- 172.21.15.157:0/3000483196 learned_addr learned my addr 172.21.15.157:0/3000483196 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-10-10T17:37:08.243 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac94dc6700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac901003e0 0x7fac9019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:08.243 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac955c7700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac900ff1e0 0x7fac900699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:08.244 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac95dc8700 1 -- 172.21.15.157:0/3000483196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac900ff1e0 msgr2=0x7fac900699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:08.244 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac95dc8700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac900ff1e0 0x7fac900699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:08.244 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac95dc8700 1 -- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac901003e0 msgr2=0x7fac9019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:08.244 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac95dc8700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac901003e0 0x7fac9019ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:08.244 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac95dc8700 1 -- 172.21.15.157:0/3000483196 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac800095f0 con 0x7fac90100d70 2023-10-10T17:37:08.245 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac95dc8700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac90100d70 0x7fac9019d350 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fac88007ee0 tx=0x7fac8800d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:37:08.245 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac867fc700 1 -- 172.21.15.157:0/3000483196 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88017070 con 0x7fac90100d70 2023-10-10T17:37:08.245 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac867fc700 1 -- 172.21.15.157:0/3000483196 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac8800f040 con 0x7fac90100d70 2023-10-10T17:37:08.245 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9010ab30 con 0x7fac90100d70 2023-10-10T17:37:08.246 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac867fc700 1 -- 172.21.15.157:0/3000483196 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88005580 con 0x7fac90100d70 2023-10-10T17:37:08.246 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.237+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9010a9a0 con 0x7fac90100d70 2023-10-10T17:37:08.247 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.238+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac90194ee0 con 0x7fac90100d70 2023-10-10T17:37:08.247 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.239+0000 7fac867fc700 1 -- 172.21.15.157:0/3000483196 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fac8800db20 con 0x7fac90100d70 2023-10-10T17:37:08.247 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.240+0000 7fac867fc700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac7c0a1150 0x7fac7c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:08.248 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.240+0000 7fac867fc700 1 -- 172.21.15.157:0/3000483196 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fac880cb590 con 0x7fac90100d70 2023-10-10T17:37:08.248 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.240+0000 7fac955c7700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac7c0a1150 0x7fac7c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:08.248 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.240+0000 7fac955c7700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac7c0a1150 0x7fac7c0a3600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fac8000b5c0 tx=0x7fac80020040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:37:08.248 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.243+0000 7fac867fc700 1 -- 172.21.15.157:0/3000483196 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fac88095e40 con 0x7fac90100d70 2023-10-10T17:37:08.439 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:08.438+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fac90195c00 con 0x7fac7c0a1150 2023-10-10T17:37:08.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:08 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:08.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:08 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:37:08.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:08 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:08.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:08.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:08.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:08 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:08.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:08 smithi078 ceph-mon[99494]: osdmap e28: 4 total, 4 up, 4 in 2023-10-10T17:37:08.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:08 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:08.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:08 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:37:08.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:08 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:08.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:08 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:08.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:08 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:08.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:08 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:08.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:08 smithi190 ceph-mon[101220]: osdmap e28: 4 total, 4 up, 4 in 2023-10-10T17:37:08.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:08 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:08.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:08 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:37:08.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:08 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:08.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:08 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:08.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:08 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:08.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:08 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:08.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:08 smithi157 ceph-mon[100896]: osdmap e28: 4 total, 4 up, 4 in 2023-10-10T17:37:09.661 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:09 smithi157 ceph-mon[100896]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:09.662 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:09 smithi157 ceph-mon[100896]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:09.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:09 smithi078 ceph-mon[99494]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:09.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:09 smithi078 ceph-mon[99494]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:09.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:09 smithi190 ceph-mon[101220]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:09.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:09 smithi190 ceph-mon[101220]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:10.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:10 smithi078 ceph-mon[99494]: osdmap e29: 4 total, 4 up, 4 in 2023-10-10T17:37:10.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:10 smithi190 ceph-mon[101220]: osdmap e29: 4 total, 4 up, 4 in 2023-10-10T17:37:10.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:10 smithi157 ceph-mon[100896]: osdmap e29: 4 total, 4 up, 4 in 2023-10-10T17:37:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: osdmap e30: 4 total, 4 up, 4 in 2023-10-10T17:37:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/1295912096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]: dispatch 2023-10-10T17:37:11.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]: dispatch 2023-10-10T17:37:11.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]': finished 2023-10-10T17:37:11.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: osdmap e31: 5 total, 4 up, 5 in 2023-10-10T17:37:11.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:11.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:11 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/2334600236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:11.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:11.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: osdmap e30: 4 total, 4 up, 4 in 2023-10-10T17:37:11.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/1295912096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]: dispatch 2023-10-10T17:37:11.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]: dispatch 2023-10-10T17:37:11.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]': finished 2023-10-10T17:37:11.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: osdmap e31: 5 total, 4 up, 5 in 2023-10-10T17:37:11.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:11.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:11 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/2334600236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: osdmap e30: 4 total, 4 up, 4 in 2023-10-10T17:37:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/1295912096' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]: dispatch 2023-10-10T17:37:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]: dispatch 2023-10-10T17:37:11.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11514abf-4ffd-437b-9b35-e92d90c3a963"}]': finished 2023-10-10T17:37:11.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: osdmap e31: 5 total, 4 up, 5 in 2023-10-10T17:37:11.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:11.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:11 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/2334600236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:12.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:12 smithi078 ceph-mon[99494]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:12.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:12 smithi190 ceph-mon[101220]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:12.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:12 smithi157 ceph-mon[100896]: pgmap v99: 1 pgs: 1 remapped+peering; 577 KiB data, 22 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:15.801 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:15 smithi157 ceph-mon[100896]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-10-10T17:37:15.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:15 smithi078 ceph-mon[99494]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-10-10T17:37:15.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:15 smithi190 ceph-mon[101220]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-10-10T17:37:17.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:17 smithi157 ceph-mon[100896]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-10-10T17:37:17.762 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-10T17:37:17.763 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:17 smithi078 ceph-mon[99494]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-10-10T17:37:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-10T17:37:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:17.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:17 smithi190 ceph-mon[101220]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-10-10T17:37:17.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-10T17:37:17.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:18.578 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:18 smithi157 ceph-mon[100896]: Deploying daemon osd.4 on smithi157 2023-10-10T17:37:18.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:18 smithi078 ceph-mon[99494]: Deploying daemon osd.4 on smithi157 2023-10-10T17:37:18.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:18 smithi190 ceph-mon[101220]: Deploying daemon osd.4 on smithi157 2023-10-10T17:37:19.700 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:19 smithi157 ceph-mon[100896]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-10T17:37:19.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:19 smithi078 ceph-mon[99494]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-10T17:37:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:19 smithi190 ceph-mon[101220]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-10T17:37:21.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:21 smithi157 ceph-mon[100896]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2023-10-10T17:37:21.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:21.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:21.865 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:21.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:21 smithi078 ceph-mon[99494]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2023-10-10T17:37:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:21.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:21 smithi190 ceph-mon[101220]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2023-10-10T17:37:21.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:23.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:23 smithi078 ceph-mon[99494]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-10-10T17:37:23.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:23 smithi190 ceph-mon[101220]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-10-10T17:37:23.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:23 smithi157 ceph-mon[100896]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2023-10-10T17:37:24.654 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.653+0000 7fac867fc700 1 -- 172.21.15.157:0/3000483196 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fac90195c00 con 0x7fac7c0a1150 2023-10-10T17:37:24.654 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 4 on host 'smithi157' 2023-10-10T17:37:24.656 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac7c0a1150 msgr2=0x7fac7c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:24.656 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac7c0a1150 0x7fac7c0a3600 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fac8000b5c0 tx=0x7fac80020040 comp rx=0 tx=0).stop 2023-10-10T17:37:24.656 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac90100d70 msgr2=0x7fac9019d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:24.656 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac90100d70 0x7fac9019d350 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fac88007ee0 tx=0x7fac8800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:37:24.657 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 shutdown_connections 2023-10-10T17:37:24.657 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac7c0a1150 0x7fac7c0a3600 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:24.657 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac900ff1e0 0x7fac900699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:24.657 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac90100d70 0x7fac9019d350 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:24.657 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 --2- 172.21.15.157:0/3000483196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac901003e0 0x7fac9019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:24.658 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 >> 172.21.15.157:0/3000483196 conn(0x7fac900fa7f0 msgr2=0x7fac900fca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:37:24.658 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 shutdown_connections 2023-10-10T17:37:24.658 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:37:24.655+0000 7fac9782b700 1 -- 172.21.15.157:0/3000483196 wait complete. 2023-10-10T17:37:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:24.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:24.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:24.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:24.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:37:24.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:24.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-10T17:37:24.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:24.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:24.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:25.036 DEBUG:teuthology.orchestra.run.smithi157:osd.4> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.4.service 2023-10-10T17:37:25.038 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_3... 2023-10-10T17:37:25.039 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-10T17:37:25.067 INFO:journalctl@ceph.osd.4.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:37:25.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:25.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:25.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:25.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:37:25.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:25.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-10T17:37:25.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:25.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:25.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:25.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:37:25.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:25.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-10T17:37:25.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:25.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:25.689 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:37:25.727 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:37:25 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:37:25.416+0000 7f1ec74e7640 -1 osd.4 0 log_to_monitors true 2023-10-10T17:37:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:25 smithi078 ceph-mon[99494]: from='osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-10T17:37:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:25 smithi078 ceph-mon[99494]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-10T17:37:26.133 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:25 smithi157 ceph-mon[100896]: from='osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-10T17:37:26.134 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:25 smithi157 ceph-mon[100896]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-10T17:37:26.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:25 smithi190 ceph-mon[101220]: from='osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-10T17:37:26.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:25 smithi190 ceph-mon[101220]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-10T17:37:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:26 smithi078 ceph-mon[99494]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-10T17:37:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:26 smithi078 ceph-mon[99494]: osdmap e32: 5 total, 4 up, 5 in 2023-10-10T17:37:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:26 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:27.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:26 smithi078 ceph-mon[99494]: from='osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:37:27.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:26 smithi078 ceph-mon[99494]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:37:27.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:26 smithi078 ceph-mon[99494]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:26 smithi190 ceph-mon[101220]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-10T17:37:27.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:26 smithi190 ceph-mon[101220]: osdmap e32: 5 total, 4 up, 5 in 2023-10-10T17:37:27.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:26 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:27.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:26 smithi190 ceph-mon[101220]: from='osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:37:27.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:26 smithi190 ceph-mon[101220]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:37:27.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:26 smithi190 ceph-mon[101220]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:27.228 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:37:26.861+0000 7f1ebcb5e700 -1 osd.4 0 waiting for initial osdmap 2023-10-10T17:37:27.228 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:37:26.868+0000 7f1eb8953700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:37:27.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-mon[100896]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-10T17:37:27.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-mon[100896]: osdmap e32: 5 total, 4 up, 5 in 2023-10-10T17:37:27.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:27.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-mon[100896]: from='osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:37:27.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-mon[100896]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:37:27.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:26 smithi157 ceph-mon[100896]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:27.269 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-10T17:37:27.269 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-10T17:37:27.269 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2023-10-10T17:37:27.270 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2023-10-10T17:37:27.270 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136045 s, 771 MB/s 2023-10-10T17:37:27.270 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2023-10-10T17:37:27.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2023-10-10T17:37:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:37:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: osdmap e33: 5 total, 4 up, 5 in 2023-10-10T17:37:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi157 2023-10-10T17:37:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi157 to 8584M 2023-10-10T17:37:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:37:28.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:27 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:37:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: osdmap e33: 5 total, 4 up, 5 in 2023-10-10T17:37:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi157 2023-10-10T17:37:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:28.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi157 to 8584M 2023-10-10T17:37:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:28.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:37:28.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:27 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:37:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: osdmap e33: 5 total, 4 up, 5 in 2023-10-10T17:37:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi157 2023-10-10T17:37:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi157 to 8584M 2023-10-10T17:37:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:37:28.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:27 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:28.378 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:37:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:28 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:37:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:28 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:37:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:28 smithi078 ceph-mon[99494]: osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] boot 2023-10-10T17:37:29.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:28 smithi078 ceph-mon[99494]: osdmap e34: 5 total, 5 up, 5 in 2023-10-10T17:37:29.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:29.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:28 smithi078 ceph-mon[99494]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:28 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:37:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:28 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:37:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:28 smithi190 ceph-mon[101220]: osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] boot 2023-10-10T17:37:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:28 smithi190 ceph-mon[101220]: osdmap e34: 5 total, 5 up, 5 in 2023-10-10T17:37:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:29.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:28 smithi190 ceph-mon[101220]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:28 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:37:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:28 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:37:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:28 smithi157 ceph-mon[100896]: osd.4 [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] boot 2023-10-10T17:37:29.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:28 smithi157 ceph-mon[100896]: osdmap e34: 5 total, 5 up, 5 in 2023-10-10T17:37:29.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:37:29.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:28 smithi157 ceph-mon[100896]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2023-10-10T17:37:29.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1652391315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc1003e0 msgr2=0x7fc9bc100830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:29.725 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1652391315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc1003e0 0x7fc9bc100830 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8009990 tx=0x7fc9a8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:37:29.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1652391315 shutdown_connections 2023-10-10T17:37:29.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1652391315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9bc100d70 0x7fc9bc107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:29.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1652391315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9bc0ff1e0 0x7fc9bc0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:29.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1652391315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc1003e0 0x7fc9bc100830 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:29.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1652391315 >> 172.21.15.190:0/1652391315 conn(0x7fc9bc0fa7f0 msgr2=0x7fc9bc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:37:29.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1652391315 shutdown_connections 2023-10-10T17:37:29.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1652391315 wait complete. 2023-10-10T17:37:29.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 Processor -- start 2023-10-10T17:37:29.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 -- start start 2023-10-10T17:37:29.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.722+0000 7fc9c2e26700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc0ff1e0 0x7fc9bc069a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:29.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c2e26700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9bc1003e0 0x7fc9bc19aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:29.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c2e26700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9bc100d70 0x7fc9bc19d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:29.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c2e26700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc19dad0 con 0x7fc9bc100d70 2023-10-10T17:37:29.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c2e26700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc19dc40 con 0x7fc9bc1003e0 2023-10-10T17:37:29.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c2e26700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc19de00 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc0ff1e0 0x7fc9bc069a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:29.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc0ff1e0 0x7fc9bc069a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:51888/0 (socket says 172.21.15.190:51888) 2023-10-10T17:37:29.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 -- 172.21.15.190:0/1494720948 learned_addr learned my addr 172.21.15.190:0/1494720948 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:37:29.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9bbfff700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9bc1003e0 0x7fc9bc19aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:29.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c13c3700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9bc100d70 0x7fc9bc19d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:29.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 -- 172.21.15.190:0/1494720948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9bc1003e0 msgr2=0x7fc9bc19aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:29.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9bc1003e0 0x7fc9bc19aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:29.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 -- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9bc100d70 msgr2=0x7fc9bc19d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:29.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9bc100d70 0x7fc9bc19d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:29.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 -- 172.21.15.190:0/1494720948 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80095f0 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c13c3700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9bc100d70 0x7fc9bc19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:37:29.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c0bc2700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc0ff1e0 0x7fc9bc069a00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000d980 tx=0x7fc9b000dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:37:29.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9b9ffb700 1 -- 172.21.15.190:0/1494720948 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b000de60 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9b9ffb700 1 -- 172.21.15.190:0/1494720948 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9b00044e0 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc10ab40 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.723+0000 7fc9b9ffb700 1 -- 172.21.15.190:0/1494720948 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0003be0 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.724+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc10a9b0 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.724+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc106790 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.725+0000 7fc9b9ffb700 1 -- 172.21.15.190:0/1494720948 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc9b0009bb0 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.726+0000 7fc9b9ffb700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9ac0a1150 0x7fc9ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:29.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.726+0000 7fc9b9ffb700 1 -- 172.21.15.190:0/1494720948 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fc9b00cb650 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.726+0000 7fc9bbfff700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9ac0a1150 0x7fc9ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:29.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.727+0000 7fc9bbfff700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9ac0a1150 0x7fc9ac0a3600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8006010 tx=0x7fc9a800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:37:29.734 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.730+0000 7fc9b9ffb700 1 -- 172.21.15.190:0/1494720948 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc9b0095e10 con 0x7fc9bc0ff1e0 2023-10-10T17:37:29.927 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:29.926+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9bc061cb0 con 0x7fc9ac0a1150 2023-10-10T17:37:30.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:29 smithi078 ceph-mon[99494]: osdmap e35: 5 total, 5 up, 5 in 2023-10-10T17:37:30.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:29 smithi190 ceph-mon[101220]: osdmap e35: 5 total, 5 up, 5 in 2023-10-10T17:37:30.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:29 smithi157 ceph-mon[100896]: osdmap e35: 5 total, 5 up, 5 in 2023-10-10T17:37:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:30 smithi078 ceph-mon[99494]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:31.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:31.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:30 smithi078 ceph-mon[99494]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:31.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:30 smithi190 ceph-mon[101220]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:31.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:31.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:31.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:31.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:30 smithi190 ceph-mon[101220]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:31.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:30 smithi157 ceph-mon[100896]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:31.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:31.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:31.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:31.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:30 smithi157 ceph-mon[100896]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:33 smithi078 ceph-mon[99494]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:33 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/3922329519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]: dispatch 2023-10-10T17:37:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:33 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]: dispatch 2023-10-10T17:37:33.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:33 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]': finished 2023-10-10T17:37:33.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:33 smithi078 ceph-mon[99494]: osdmap e36: 6 total, 5 up, 6 in 2023-10-10T17:37:33.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:33.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:33 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/2725152304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:33.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:33 smithi190 ceph-mon[101220]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:33.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:33 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/3922329519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]: dispatch 2023-10-10T17:37:33.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:33 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]: dispatch 2023-10-10T17:37:33.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:33 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]': finished 2023-10-10T17:37:33.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:33 smithi190 ceph-mon[101220]: osdmap e36: 6 total, 5 up, 6 in 2023-10-10T17:37:33.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:33.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:33 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/2725152304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:33.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:33 smithi157 ceph-mon[100896]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:33.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:33 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/3922329519' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]: dispatch 2023-10-10T17:37:33.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:33 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]: dispatch 2023-10-10T17:37:33.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:33 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1a3f83a-3fe5-470e-a55b-76a851412013"}]': finished 2023-10-10T17:37:33.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:33 smithi157 ceph-mon[100896]: osdmap e36: 6 total, 5 up, 6 in 2023-10-10T17:37:33.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:33.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:33 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/2725152304' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:35 smithi078 ceph-mon[99494]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:35 smithi190 ceph-mon[101220]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:35 smithi157 ceph-mon[100896]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:37.790 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:37 smithi190 ceph-mon[101220]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:37 smithi078 ceph-mon[99494]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:37.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:37 smithi157 ceph-mon[100896]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:39 smithi078 ceph-mon[99494]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-10T17:37:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:39.895 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:39 smithi190 ceph-mon[101220]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:39.896 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-10T17:37:39.896 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:39 smithi157 ceph-mon[100896]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:39.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-10T17:37:39.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:40.829 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:40 smithi190 ceph-mon[101220]: Deploying daemon osd.5 on smithi190 2023-10-10T17:37:40.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:40 smithi078 ceph-mon[99494]: Deploying daemon osd.5 on smithi190 2023-10-10T17:37:40.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:40 smithi157 ceph-mon[100896]: Deploying daemon osd.5 on smithi190 2023-10-10T17:37:41.867 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:41 smithi190 ceph-mon[101220]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:41.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:41 smithi078 ceph-mon[99494]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:41.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:41 smithi157 ceph-mon[100896]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:43.727 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:43 smithi190 ceph-mon[101220]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:43.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:43 smithi078 ceph-mon[99494]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:43 smithi157 ceph-mon[100896]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:45.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:45.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:45.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:44 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:45.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:44 smithi078 ceph-mon[99494]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:45.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:45.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:44 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:45.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:44 smithi190 ceph-mon[101220]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:45.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:45.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:45.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:44 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:45.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:44 smithi157 ceph-mon[100896]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:46.837 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2023-10-10T17:37:46.837 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.836+0000 7fc9b9ffb700 1 -- 172.21.15.190:0/1494720948 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc9bc061cb0 con 0x7fc9ac0a1150 2023-10-10T17:37:46.839 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.838+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9ac0a1150 msgr2=0x7fc9ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:46.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.838+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9ac0a1150 0x7fc9ac0a3600 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8006010 tx=0x7fc9a800b540 comp rx=0 tx=0).stop 2023-10-10T17:37:46.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc0ff1e0 msgr2=0x7fc9bc069a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:46.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc0ff1e0 0x7fc9bc069a00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000d980 tx=0x7fc9b000dd40 comp rx=0 tx=0).stop 2023-10-10T17:37:46.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 shutdown_connections 2023-10-10T17:37:46.840 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9ac0a1150 0x7fc9ac0a3600 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:46.841 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9bc0ff1e0 0x7fc9bc069a00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:46.841 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9bc100d70 0x7fc9bc19d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:46.841 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 --2- 172.21.15.190:0/1494720948 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9bc1003e0 0x7fc9bc19aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:46.841 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 >> 172.21.15.190:0/1494720948 conn(0x7fc9bc0fa7f0 msgr2=0x7fc9bc1038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:37:46.841 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 shutdown_connections 2023-10-10T17:37:46.842 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:37:46.839+0000 7fc9c2e26700 1 -- 172.21.15.190:0/1494720948 wait complete. 2023-10-10T17:37:47.267 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.5.service 2023-10-10T17:37:47.269 INFO:tasks.cephadm:Deploying osd.6 on smithi078 with /dev/vg_nvme/lv_2... 2023-10-10T17:37:47.270 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-10T17:37:47.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:37:47.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:47.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:47.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:47.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:37:47.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:47.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:47.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:47.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:47.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:47.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:47.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:37:47.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:47.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:47.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:47.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:47.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:47.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:47.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:37:47.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:47.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:47.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:47.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:37:47.927 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:37:48.072 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:37:47 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:37:47.673+0000 7ff6f4b29640 -1 osd.5 0 log_to_monitors true 2023-10-10T17:37:48.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:48 smithi078 ceph-mon[99494]: from='osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-10T17:37:48.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:48 smithi078 ceph-mon[99494]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-10T17:37:48.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:48 smithi190 ceph-mon[101220]: from='osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-10T17:37:48.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:48 smithi190 ceph-mon[101220]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-10T17:37:48.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:48 smithi157 ceph-mon[100896]: from='osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-10T17:37:48.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:48 smithi157 ceph-mon[100896]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-10T17:37:49.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:49 smithi078 ceph-mon[99494]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-10T17:37:49.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:49 smithi078 ceph-mon[99494]: osdmap e37: 6 total, 5 up, 6 in 2023-10-10T17:37:49.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:49.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:49 smithi078 ceph-mon[99494]: from='osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:37:49.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:49 smithi078 ceph-mon[99494]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:37:49.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:49 smithi078 ceph-mon[99494]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:49.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:37:49.097+0000 7ff6ea1a0700 -1 osd.5 0 waiting for initial osdmap 2023-10-10T17:37:49.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:37:49.106+0000 7ff6e4792700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:37:49.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-mon[101220]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-10T17:37:49.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-mon[101220]: osdmap e37: 6 total, 5 up, 6 in 2023-10-10T17:37:49.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:49.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-mon[101220]: from='osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:37:49.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-mon[101220]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:37:49.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:49 smithi190 ceph-mon[101220]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:49.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:49 smithi157 ceph-mon[100896]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-10T17:37:49.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:49 smithi157 ceph-mon[100896]: osdmap e37: 6 total, 5 up, 6 in 2023-10-10T17:37:49.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:49.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:49 smithi157 ceph-mon[100896]: from='osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:37:49.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:49 smithi157 ceph-mon[100896]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:37:49.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:49 smithi157 ceph-mon[100896]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:49.513 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-10T17:37:49.513 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-10T17:37:49.513 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-10-10T17:37:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-10-10T17:37:49.514 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113528 s, 924 MB/s 2023-10-10T17:37:49.514 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-10-10T17:37:49.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2023-10-10T17:37:50.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:37:50.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: osdmap e38: 6 total, 5 up, 6 in 2023-10-10T17:37:50.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:50.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:50.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi190 2023-10-10T17:37:50.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:50.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:50.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi190 to 8584M 2023-10-10T17:37:50.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:50.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:50.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:50.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:37:50.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:50 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:37:50.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:37:50.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: osdmap e38: 6 total, 5 up, 6 in 2023-10-10T17:37:50.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:50.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:50.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi190 2023-10-10T17:37:50.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:50.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:50.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi190 to 8584M 2023-10-10T17:37:50.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:50.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:50.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:50.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:37:50.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:50 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:37:50.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:37:50.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: osdmap e38: 6 total, 5 up, 6 in 2023-10-10T17:37:50.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:50.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:50.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi190 2023-10-10T17:37:50.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:50.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:37:50.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi190 to 8584M 2023-10-10T17:37:50.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:50.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:37:50.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:50.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:37:50.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:37:50.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:50 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:37:50.625 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:37:51.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:51 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:37:51.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:51 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:37:51.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:51 smithi078 ceph-mon[99494]: osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] boot 2023-10-10T17:37:51.370 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:51 smithi078 ceph-mon[99494]: osdmap e39: 6 total, 6 up, 6 in 2023-10-10T17:37:51.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:51 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:51.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:51 smithi078 ceph-mon[99494]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:51.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:51 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:37:51.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:51 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:37:51.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:51 smithi190 ceph-mon[101220]: osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] boot 2023-10-10T17:37:51.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:51 smithi190 ceph-mon[101220]: osdmap e39: 6 total, 6 up, 6 in 2023-10-10T17:37:51.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:51 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:51.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:51 smithi190 ceph-mon[101220]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:51.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:51 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:51 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:51 smithi157 ceph-mon[100896]: osd.5 [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] boot 2023-10-10T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:51 smithi157 ceph-mon[100896]: osdmap e39: 6 total, 6 up, 6 in 2023-10-10T17:37:51.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:51 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:37:51.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:51 smithi157 ceph-mon[100896]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2023-10-10T17:37:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 -- 172.21.15.78:0/4268733670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc800ff1e0 msgr2=0x7fdc800ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/4268733670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc800ff1e0 0x7fdc800ff5f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009990 tx=0x7fdc70009ca0 comp rx=0 tx=0).stop 2023-10-10T17:37:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 -- 172.21.15.78:0/4268733670 shutdown_connections 2023-10-10T17:37:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/4268733670 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc80100d70 0x7fdc80107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/4268733670 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc800ff1e0 0x7fdc800ff5f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/4268733670 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc801003e0 0x7fdc80100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 -- 172.21.15.78:0/4268733670 >> 172.21.15.78:0/4268733670 conn(0x7fdc800fa7f0 msgr2=0x7fdc800fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:37:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 -- 172.21.15.78:0/4268733670 shutdown_connections 2023-10-10T17:37:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.806+0000 7fdc87bc0700 1 -- 172.21.15.78:0/4268733670 wait complete. 2023-10-10T17:37:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 Processor -- start 2023-10-10T17:37:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 -- start start 2023-10-10T17:37:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc800ff1e0 0x7fdc80195030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc801003e0 0x7fdc80195570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:51.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc80100d70 0x7fdc8019c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:51.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8019cbc0 con 0x7fdc801003e0 2023-10-10T17:37:51.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8019cd00 con 0x7fdc80100d70 2023-10-10T17:37:51.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc87bc0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8019ce90 con 0x7fdc800ff1e0 2023-10-10T17:37:51.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc801003e0 0x7fdc80195570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:51.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc801003e0 0x7fdc80195570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41458/0 (socket says 172.21.15.78:41458) 2023-10-10T17:37:51.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 -- 172.21.15.78:0/2185865150 learned_addr learned my addr 172.21.15.78:0/2185865150 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:37:51.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8595c700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc800ff1e0 0x7fdc80195030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:51.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8615d700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc80100d70 0x7fdc8019c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:51.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 -- 172.21.15.78:0/2185865150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc800ff1e0 msgr2=0x7fdc80195030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:51.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc800ff1e0 0x7fdc80195030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:51.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 -- 172.21.15.78:0/2185865150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc80100d70 msgr2=0x7fdc8019c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:37:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc80100d70 0x7fdc8019c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:37:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 -- 172.21.15.78:0/2185865150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc700095f0 con 0x7fdc801003e0 2023-10-10T17:37:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8615d700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc80100d70 0x7fdc8019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:37:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8595c700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc800ff1e0 0x7fdc80195030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:37:51.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.807+0000 7fdc8515b700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc801003e0 0x7fdc80195570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00b560 tx=0x7fdc7c00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:37:51.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.808+0000 7fdc76ffd700 1 -- 172.21.15.78:0/2185865150 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c011830 con 0x7fdc801003e0 2023-10-10T17:37:51.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.808+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8010ac00 con 0x7fdc801003e0 2023-10-10T17:37:51.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.808+0000 7fdc76ffd700 1 -- 172.21.15.78:0/2185865150 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc7c011e50 con 0x7fdc801003e0 2023-10-10T17:37:51.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.808+0000 7fdc76ffd700 1 -- 172.21.15.78:0/2185865150 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7c00f2e0 con 0x7fdc801003e0 2023-10-10T17:37:51.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.808+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc8010aa70 con 0x7fdc801003e0 2023-10-10T17:37:51.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.809+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc801061a0 con 0x7fdc801003e0 2023-10-10T17:37:51.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.810+0000 7fdc76ffd700 1 -- 172.21.15.78:0/2185865150 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdc7c011990 con 0x7fdc801003e0 2023-10-10T17:37:51.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.810+0000 7fdc76ffd700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc6c0a1150 0x7fdc6c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:37:51.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.810+0000 7fdc76ffd700 1 -- 172.21.15.78:0/2185865150 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fdc7c0cb550 con 0x7fdc801003e0 2023-10-10T17:37:51.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.810+0000 7fdc8595c700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc6c0a1150 0x7fdc6c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:37:51.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.810+0000 7fdc8595c700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc6c0a1150 0x7fdc6c0a3600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000b5c0 tx=0x7fdc70005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:37:51.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:51.814+0000 7fdc76ffd700 1 -- 172.21.15.78:0/2185865150 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdc7c095b40 con 0x7fdc801003e0 2023-10-10T17:37:52.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:37:52.008+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc80061bc0 con 0x7fdc6c0a1150 2023-10-10T17:37:52.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:52 smithi078 ceph-mon[99494]: osdmap e40: 6 total, 6 up, 6 in 2023-10-10T17:37:52.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:52.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:52.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:52.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:52 smithi190 ceph-mon[101220]: osdmap e40: 6 total, 6 up, 6 in 2023-10-10T17:37:52.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:52.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:52.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:52.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:52 smithi157 ceph-mon[100896]: osdmap e40: 6 total, 6 up, 6 in 2023-10-10T17:37:52.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:37:52.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:37:52.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:37:53.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:53 smithi078 ceph-mon[99494]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:53.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:53 smithi078 ceph-mon[99494]: osdmap e41: 6 total, 6 up, 6 in 2023-10-10T17:37:53.250 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:53 smithi078 ceph-mon[99494]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:37:53.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:53 smithi190 ceph-mon[101220]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:53.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:53 smithi190 ceph-mon[101220]: osdmap e41: 6 total, 6 up, 6 in 2023-10-10T17:37:53.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:53 smithi190 ceph-mon[101220]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:37:53.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:53 smithi157 ceph-mon[100896]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:37:53.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:53 smithi157 ceph-mon[100896]: osdmap e41: 6 total, 6 up, 6 in 2023-10-10T17:37:53.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:53 smithi157 ceph-mon[100896]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 33 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:37:54.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:54 smithi190 ceph-mon[101220]: osdmap e42: 6 total, 6 up, 6 in 2023-10-10T17:37:54.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:54 smithi157 ceph-mon[100896]: osdmap e42: 6 total, 6 up, 6 in 2023-10-10T17:37:54.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:54 smithi078 ceph-mon[99494]: osdmap e42: 6 total, 6 up, 6 in 2023-10-10T17:37:55.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:55 smithi190 ceph-mon[101220]: osdmap e43: 6 total, 6 up, 6 in 2023-10-10T17:37:55.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:55 smithi190 ceph-mon[101220]: pgmap v132: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1/6 objects misplaced (16.667%); 64 KiB/s, 0 objects/s recovering 2023-10-10T17:37:55.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:55 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1615298329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149dd00c-dd70-4d62-91db-f9a6830d918f"}]: dispatch 2023-10-10T17:37:55.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:55 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1615298329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149dd00c-dd70-4d62-91db-f9a6830d918f"}]': finished 2023-10-10T17:37:55.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:55 smithi190 ceph-mon[101220]: osdmap e44: 7 total, 6 up, 7 in 2023-10-10T17:37:55.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:55 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:37:55.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:55 smithi157 ceph-mon[100896]: osdmap e43: 6 total, 6 up, 6 in 2023-10-10T17:37:55.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:55 smithi157 ceph-mon[100896]: pgmap v132: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1/6 objects misplaced (16.667%); 64 KiB/s, 0 objects/s recovering 2023-10-10T17:37:55.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:55 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1615298329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149dd00c-dd70-4d62-91db-f9a6830d918f"}]: dispatch 2023-10-10T17:37:55.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:55 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1615298329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149dd00c-dd70-4d62-91db-f9a6830d918f"}]': finished 2023-10-10T17:37:55.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:55 smithi157 ceph-mon[100896]: osdmap e44: 7 total, 6 up, 7 in 2023-10-10T17:37:55.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:55 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:37:55.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:55 smithi078 ceph-mon[99494]: osdmap e43: 6 total, 6 up, 6 in 2023-10-10T17:37:55.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:55 smithi078 ceph-mon[99494]: pgmap v132: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1/6 objects misplaced (16.667%); 64 KiB/s, 0 objects/s recovering 2023-10-10T17:37:55.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:55 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1615298329' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "149dd00c-dd70-4d62-91db-f9a6830d918f"}]: dispatch 2023-10-10T17:37:55.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:55 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1615298329' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "149dd00c-dd70-4d62-91db-f9a6830d918f"}]': finished 2023-10-10T17:37:55.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:55 smithi078 ceph-mon[99494]: osdmap e44: 7 total, 6 up, 7 in 2023-10-10T17:37:55.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:55 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:37:56.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:56 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1121789850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:56.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:56 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1121789850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:56.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:56 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1121789850' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:37:57.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:57 smithi190 ceph-mon[101220]: pgmap v134: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1/6 objects misplaced (16.667%); 63 KiB/s, 0 objects/s recovering 2023-10-10T17:37:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:57 smithi157 ceph-mon[100896]: pgmap v134: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1/6 objects misplaced (16.667%); 63 KiB/s, 0 objects/s recovering 2023-10-10T17:37:57.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:57 smithi078 ceph-mon[99494]: pgmap v134: 1 pgs: 1 active+recovering+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 1/6 objects misplaced (16.667%); 63 KiB/s, 0 objects/s recovering 2023-10-10T17:37:59.796 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:37:59 smithi078 ceph-mon[99494]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:37:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:37:59 smithi190 ceph-mon[101220]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:37:59.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:37:59 smithi157 ceph-mon[100896]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:38:01.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:01 smithi078 ceph-mon[99494]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 45 KiB/s, 0 objects/s recovering 2023-10-10T17:38:01.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-10T17:38:01.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:01.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:01 smithi190 ceph-mon[101220]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 45 KiB/s, 0 objects/s recovering 2023-10-10T17:38:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-10T17:38:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:01 smithi157 ceph-mon[100896]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 45 KiB/s, 0 objects/s recovering 2023-10-10T17:38:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-10T17:38:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:02.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:02 smithi078 ceph-mon[99494]: Deploying daemon osd.6 on smithi078 2023-10-10T17:38:02.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:02 smithi190 ceph-mon[101220]: Deploying daemon osd.6 on smithi078 2023-10-10T17:38:02.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:02 smithi157 ceph-mon[100896]: Deploying daemon osd.6 on smithi078 2023-10-10T17:38:03.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:03 smithi078 ceph-mon[99494]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 40 KiB/s, 0 objects/s recovering 2023-10-10T17:38:03.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:03 smithi190 ceph-mon[101220]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 40 KiB/s, 0 objects/s recovering 2023-10-10T17:38:03.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:03 smithi157 ceph-mon[100896]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 40 KiB/s, 0 objects/s recovering 2023-10-10T17:38:05.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:05 smithi078 ceph-mon[99494]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 32 KiB/s, 0 objects/s recovering 2023-10-10T17:38:05.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:05 smithi190 ceph-mon[101220]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 32 KiB/s, 0 objects/s recovering 2023-10-10T17:38:05.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:05 smithi157 ceph-mon[100896]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 32 KiB/s, 0 objects/s recovering 2023-10-10T17:38:07.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:06 smithi078 ceph-mon[99494]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 28 KiB/s, 0 objects/s recovering 2023-10-10T17:38:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:07.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:07.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:07.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:06 smithi190 ceph-mon[101220]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 28 KiB/s, 0 objects/s recovering 2023-10-10T17:38:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:06 smithi157 ceph-mon[100896]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 28 KiB/s, 0 objects/s recovering 2023-10-10T17:38:08.860 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 6 on host 'smithi078' 2023-10-10T17:38:08.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.858+0000 7fdc76ffd700 1 -- 172.21.15.78:0/2185865150 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdc80061bc0 con 0x7fdc6c0a1150 2023-10-10T17:38:08.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc6c0a1150 msgr2=0x7fdc6c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:08.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc6c0a1150 0x7fdc6c0a3600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fdc7000b5c0 tx=0x7fdc70005fb0 comp rx=0 tx=0).stop 2023-10-10T17:38:08.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc801003e0 msgr2=0x7fdc80195570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:08.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc801003e0 0x7fdc80195570 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c00b560 tx=0x7fdc7c00b920 comp rx=0 tx=0).stop 2023-10-10T17:38:08.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 shutdown_connections 2023-10-10T17:38:08.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc6c0a1150 0x7fdc6c0a3600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:08.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc800ff1e0 0x7fdc80195030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc80100d70 0x7fdc8019c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 --2- 172.21.15.78:0/2185865150 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc801003e0 0x7fdc80195570 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 >> 172.21.15.78:0/2185865150 conn(0x7fdc800fa7f0 msgr2=0x7fdc800fcb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:38:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 shutdown_connections 2023-10-10T17:38:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:08.860+0000 7fdc87bc0700 1 -- 172.21.15.78:0/2185865150 wait complete. 2023-10-10T17:38:09.308 DEBUG:teuthology.orchestra.run.smithi078:osd.6> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.6.service 2023-10-10T17:38:09.310 INFO:tasks.cephadm:Deploying osd.7 on smithi157 with /dev/vg_nvme/lv_2... 2023-10-10T17:38:09.310 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-10T17:38:09.351 INFO:journalctl@ceph.osd.6.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:38:09.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:09 smithi157 ceph-mon[100896]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 27 KiB/s, 0 objects/s recovering 2023-10-10T17:38:09.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:09 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:09 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:09 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:09 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:09 smithi078 ceph-mon[99494]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 27 KiB/s, 0 objects/s recovering 2023-10-10T17:38:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:09 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:09 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:09 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:09 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:09 smithi190 ceph-mon[101220]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 27 KiB/s, 0 objects/s recovering 2023-10-10T17:38:09.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:09 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:09 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:09 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:09 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:09.971 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:38:10.113 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:38:09 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:38:09.780+0000 7fd2ea9a6640 -1 osd.6 0 log_to_monitors true 2023-10-10T17:38:10.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:10 smithi078 ceph-mon[99494]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-10T17:38:10.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:10 smithi157 ceph-mon[100896]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-10T17:38:10.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:10 smithi190 ceph-mon[101220]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-10T17:38:11.413 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-10T17:38:11.413 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-10T17:38:11.413 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2023-10-10T17:38:11.413 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2023-10-10T17:38:11.414 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00815958 s, 1.3 GB/s 2023-10-10T17:38:11.414 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2023-10-10T17:38:11.576 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:11 smithi157 ceph-mon[100896]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:38:11.577 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:11 smithi157 ceph-mon[100896]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-10T17:38:11.577 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:11 smithi157 ceph-mon[100896]: osdmap e45: 7 total, 6 up, 7 in 2023-10-10T17:38:11.577 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:11 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:11.577 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:11 smithi157 ceph-mon[100896]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:38:11.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:38:11 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:38:11.238+0000 7fd2e001d700 -1 osd.6 0 waiting for initial osdmap 2023-10-10T17:38:11.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:38:11 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:38:11.250+0000 7fd2dbe12700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:38:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:11 smithi078 ceph-mon[99494]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:38:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:11 smithi078 ceph-mon[99494]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-10T17:38:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:11 smithi078 ceph-mon[99494]: osdmap e45: 7 total, 6 up, 7 in 2023-10-10T17:38:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:11.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:11 smithi078 ceph-mon[99494]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:38:11.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:11 smithi190 ceph-mon[101220]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:38:11.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:11 smithi190 ceph-mon[101220]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-10T17:38:11.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:11 smithi190 ceph-mon[101220]: osdmap e45: 7 total, 6 up, 7 in 2023-10-10T17:38:11.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:11 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:11.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:11 smithi190 ceph-mon[101220]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:38:11.814 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi157:vg_nvme/lv_2 2023-10-10T17:38:12.462 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:38:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:38:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: osdmap e46: 7 total, 6 up, 7 in 2023-10-10T17:38:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:12.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:12.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:12.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:12.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:12.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:12.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:12.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:38:12.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: osdmap e46: 7 total, 6 up, 7 in 2023-10-10T17:38:12.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:12.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:12.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:12.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:12.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:12.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:12.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:12.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.674 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:38:12.675 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: osdmap e46: 7 total, 6 up, 7 in 2023-10-10T17:38:12.675 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:12.675 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:12.675 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:12.676 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.677 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:12.677 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:12.677 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:12.677 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:12.677 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:12.678 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:12.678 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:13.457 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:38:13.458 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:38:13.458 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi078 2023-10-10T17:38:13.458 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi078 to 5722M 2023-10-10T17:38:13.458 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.459 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:38:13.459 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.459 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.459 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:38:13.459 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.460 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:38:13.460 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] boot 2023-10-10T17:38:13.460 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: osdmap e47: 7 total, 7 up, 7 in 2023-10-10T17:38:13.460 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:13 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:38:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:38:13.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi078 2023-10-10T17:38:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi078 to 5722M 2023-10-10T17:38:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:38:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:38:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:38:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] boot 2023-10-10T17:38:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: osdmap e47: 7 total, 7 up, 7 in 2023-10-10T17:38:13.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:13 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:13.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:38:13.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:38:13.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi078 2023-10-10T17:38:13.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi078 to 5722M 2023-10-10T17:38:13.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:38:13.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:38:13.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:13.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2023-10-10T17:38:13.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: osd.6 [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] boot 2023-10-10T17:38:13.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: osdmap e47: 7 total, 7 up, 7 in 2023-10-10T17:38:13.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:13 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:38:13.693 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.691+0000 7f21d2ef8700 1 -- 172.21.15.157:0/1790510371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc100d70 msgr2=0x7f21cc107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:13.694 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.691+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/1790510371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc100d70 0x7f21cc107630 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f21c400b0a0 tx=0x7f21c400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:38:13.694 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 -- 172.21.15.157:0/1790510371 shutdown_connections 2023-10-10T17:38:13.694 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/1790510371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21cc0ff1e0 0x7f21cc0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:13.694 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/1790510371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc100d70 0x7f21cc107630 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:13.695 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/1790510371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21cc1003e0 0x7f21cc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:13.695 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 -- 172.21.15.157:0/1790510371 >> 172.21.15.157:0/1790510371 conn(0x7f21cc0fa7f0 msgr2=0x7f21cc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:38:13.695 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 -- 172.21.15.157:0/1790510371 shutdown_connections 2023-10-10T17:38:13.695 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 -- 172.21.15.157:0/1790510371 wait complete. 2023-10-10T17:38:13.695 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 Processor -- start 2023-10-10T17:38:13.695 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.692+0000 7f21d2ef8700 1 -- start start 2023-10-10T17:38:13.697 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d2ef8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc0ff1e0 0x7f21cc069110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:13.697 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d2ef8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21cc1003e0 0x7f21cc19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:13.697 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d2ef8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21cc100d70 0x7f21cc19d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:13.698 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d2ef8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc19dbf0 con 0x7f21cc100d70 2023-10-10T17:38:13.698 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d2ef8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc19dd60 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.698 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d2ef8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc19df20 con 0x7f21cc1003e0 2023-10-10T17:38:13.698 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc0ff1e0 0x7f21cc069110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:13.699 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc0ff1e0 0x7f21cc069110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:58518/0 (socket says 172.21.15.157:58518) 2023-10-10T17:38:13.699 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 -- 172.21.15.157:0/2184845905 learned_addr learned my addr 172.21.15.157:0/2184845905 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-10-10T17:38:13.699 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d1495700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21cc100d70 0x7f21cc19d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:13.699 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21cbfff700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21cc1003e0 0x7f21cc19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:13.699 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 -- 172.21.15.157:0/2184845905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21cc1003e0 msgr2=0x7f21cc19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:13.700 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21cc1003e0 0x7f21cc19ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:13.700 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 -- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21cc100d70 msgr2=0x7f21cc19d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:13.700 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21cc100d70 0x7f21cc19d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:13.700 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 -- 172.21.15.157:0/2184845905 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4009d00 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.700 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d1495700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21cc100d70 0x7f21cc19d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:38:13.701 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d0c94700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc0ff1e0 0x7f21cc069110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f21c000d980 tx=0x7f21c000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:38:13.701 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21c9ffb700 1 -- 172.21.15.157:0/2184845905 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c000de60 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.701 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21c9ffb700 1 -- 172.21.15.157:0/2184845905 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f21c00044e0 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.701 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc10ab30 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.701 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.693+0000 7f21c9ffb700 1 -- 172.21.15.157:0/2184845905 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0003be0 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.701 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.694+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc10a9a0 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.702 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.695+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc19e660 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.702 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.696+0000 7f21c9ffb700 1 -- 172.21.15.157:0/2184845905 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f21c0009bb0 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.703 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.696+0000 7f21c9ffb700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f21bc0a1150 0x7f21bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:13.703 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.697+0000 7f21c9ffb700 1 -- 172.21.15.157:0/2184845905 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f21c00cbde0 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.703 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.697+0000 7f21cbfff700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f21bc0a1150 0x7f21bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:13.703 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.697+0000 7f21cbfff700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f21bc0a1150 0x7f21bc0a3600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005950 tx=0x7f21b80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:38:13.704 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.700+0000 7f21c9ffb700 1 -- 172.21.15.157:0/2184845905 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f21c00961a0 con 0x7f21cc0ff1e0 2023-10-10T17:38:13.894 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:13.893+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f21cc19ec10 con 0x7f21bc0a1150 2023-10-10T17:38:14.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:14 smithi078 ceph-mon[99494]: osdmap e48: 7 total, 7 up, 7 in 2023-10-10T17:38:14.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:14.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:14.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:14.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:14 smithi190 ceph-mon[101220]: osdmap e48: 7 total, 7 up, 7 in 2023-10-10T17:38:14.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:14.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:14.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:14.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:14 smithi157 ceph-mon[100896]: osdmap e48: 7 total, 7 up, 7 in 2023-10-10T17:38:14.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:14.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:14.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:15.716 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:15 smithi157 ceph-mon[100896]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:15.716 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:15 smithi157 ceph-mon[100896]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:15.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:15 smithi078 ceph-mon[99494]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:15.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:15 smithi078 ceph-mon[99494]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:15.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:15 smithi190 ceph-mon[101220]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:15.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:15 smithi190 ceph-mon[101220]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:16.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:16 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/2267290048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]: dispatch 2023-10-10T17:38:16.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:16 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]: dispatch 2023-10-10T17:38:16.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:16 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]': finished 2023-10-10T17:38:16.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:16 smithi078 ceph-mon[99494]: osdmap e49: 8 total, 7 up, 8 in 2023-10-10T17:38:16.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:16 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:16.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:16 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/2267290048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]: dispatch 2023-10-10T17:38:16.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:16 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]: dispatch 2023-10-10T17:38:16.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:16 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]': finished 2023-10-10T17:38:16.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:16 smithi190 ceph-mon[101220]: osdmap e49: 8 total, 7 up, 8 in 2023-10-10T17:38:16.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:16 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:16.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:16 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/2267290048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]: dispatch 2023-10-10T17:38:16.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:16 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]: dispatch 2023-10-10T17:38:16.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:16 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2c72dbb4-e18a-4627-9871-2730cc29fcd9"}]': finished 2023-10-10T17:38:16.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:16 smithi157 ceph-mon[100896]: osdmap e49: 8 total, 7 up, 8 in 2023-10-10T17:38:16.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:16 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:17 smithi078 ceph-mon[99494]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:17.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:17 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/4162774357' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:38:17.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:17 smithi190 ceph-mon[101220]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:17.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:17 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/4162774357' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:38:17.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:17 smithi157 ceph-mon[100896]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:17.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:17 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/4162774357' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:38:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:19 smithi078 ceph-mon[99494]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:19 smithi190 ceph-mon[101220]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:19.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:19 smithi157 ceph-mon[100896]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:21 smithi157 ceph-mon[100896]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:21 smithi078 ceph-mon[99494]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:21.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:21 smithi190 ceph-mon[101220]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:23 smithi078 ceph-mon[99494]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-10T17:38:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:23.879 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:23 smithi157 ceph-mon[100896]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:23.879 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-10T17:38:23.880 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:23.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:23 smithi190 ceph-mon[101220]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:23.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-10T17:38:23.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:24.822 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:24 smithi157 ceph-mon[100896]: Deploying daemon osd.7 on smithi157 2023-10-10T17:38:24.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:24 smithi078 ceph-mon[99494]: Deploying daemon osd.7 on smithi157 2023-10-10T17:38:24.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:24 smithi190 ceph-mon[101220]: Deploying daemon osd.7 on smithi157 2023-10-10T17:38:25.862 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:25 smithi157 ceph-mon[100896]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:25.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:25 smithi078 ceph-mon[99494]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:25.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:25 smithi190 ceph-mon[101220]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:27.724 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:27 smithi157 ceph-mon[100896]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:27.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:27 smithi078 ceph-mon[99494]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:27.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:27 smithi190 ceph-mon[101220]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:28.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:28.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:28.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:28.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:28 smithi157 ceph-mon[100896]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:28 smithi078 ceph-mon[99494]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:28 smithi190 ceph-mon[101220]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:30.956 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 7 on host 'smithi157' 2023-10-10T17:38:30.957 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.956+0000 7f21c9ffb700 1 -- 172.21.15.157:0/2184845905 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f21cc19ec10 con 0x7f21bc0a1150 2023-10-10T17:38:30.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f21bc0a1150 msgr2=0x7f21bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:30.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f21bc0a1150 0x7f21bc0a3600 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f21b8005950 tx=0x7f21b80058e0 comp rx=0 tx=0).stop 2023-10-10T17:38:30.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc0ff1e0 msgr2=0x7f21cc069110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:30.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc0ff1e0 0x7f21cc069110 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f21c000d980 tx=0x7f21c000dd40 comp rx=0 tx=0).stop 2023-10-10T17:38:30.959 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 shutdown_connections 2023-10-10T17:38:30.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f21bc0a1150 0x7f21bc0a3600 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:30.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f21cc100d70 0x7f21cc19d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:30.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f21cc0ff1e0 0x7f21cc069110 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:30.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 --2- 172.21.15.157:0/2184845905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f21cc1003e0 0x7f21cc19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:30.960 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.958+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 >> 172.21.15.157:0/2184845905 conn(0x7f21cc0fa7f0 msgr2=0x7f21cc0fcac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:38:30.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.959+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 shutdown_connections 2023-10-10T17:38:30.961 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:38:30.959+0000 7f21d2ef8700 1 -- 172.21.15.157:0/2184845905 wait complete. 2023-10-10T17:38:31.353 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.353 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:31.353 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.353 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:31.354 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:31.354 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:31.354 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.354 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:38:31.354 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:31.355 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.355 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.355 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:31.356 DEBUG:teuthology.orchestra.run.smithi157:osd.7> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.7.service 2023-10-10T17:38:31.359 INFO:tasks.cephadm:Deploying osd.8 on smithi190 with /dev/vg_nvme/lv_2... 2023-10-10T17:38:31.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-10T17:38:31.387 INFO:journalctl@ceph.osd.7.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:38:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:38:31.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:31.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:31.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:31.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:31.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:31.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:31.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:38:31.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:31.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:31.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:32.021 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:38:32.054 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:38:31 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:38:31.731+0000 7f4265d5c640 -1 osd.7 0 log_to_monitors true 2023-10-10T17:38:32.416 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:32 smithi157 ceph-mon[100896]: from='osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-10T17:38:32.416 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:32 smithi157 ceph-mon[100896]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-10T17:38:32.561 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:32 smithi190 ceph-mon[101220]: from='osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-10T17:38:32.561 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:32 smithi190 ceph-mon[101220]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-10T17:38:32.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:32 smithi078 ceph-mon[99494]: from='osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-10T17:38:32.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:32 smithi078 ceph-mon[99494]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-10T17:38:33.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:33 smithi190 ceph-mon[101220]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:33.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:33 smithi190 ceph-mon[101220]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-10T17:38:33.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:33 smithi190 ceph-mon[101220]: osdmap e50: 8 total, 7 up, 8 in 2023-10-10T17:38:33.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:33.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:33 smithi190 ceph-mon[101220]: from='osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:38:33.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:33 smithi190 ceph-mon[101220]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:38:33.478 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:38:33.209+0000 7f425b3d3700 -1 osd.7 0 waiting for initial osdmap 2023-10-10T17:38:33.478 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:38:33.219+0000 7f42539c1700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:38:33.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-mon[100896]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:33.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-mon[100896]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-10T17:38:33.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-mon[100896]: osdmap e50: 8 total, 7 up, 8 in 2023-10-10T17:38:33.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:33.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-mon[100896]: from='osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:38:33.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:33 smithi157 ceph-mon[100896]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:38:33.492 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-10T17:38:33.492 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-10T17:38:33.493 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2023-10-10T17:38:33.493 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2023-10-10T17:38:33.493 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110694 s, 947 MB/s 2023-10-10T17:38:33.493 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 2023-10-10T17:38:33.493 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2023-10-10T17:38:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:33 smithi078 ceph-mon[99494]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:33 smithi078 ceph-mon[99494]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-10T17:38:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:33 smithi078 ceph-mon[99494]: osdmap e50: 8 total, 7 up, 8 in 2023-10-10T17:38:33.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:33.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:33 smithi078 ceph-mon[99494]: from='osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:38:33.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:33 smithi078 ceph-mon[99494]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:38:33.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2023-10-10T17:38:34.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:38:34.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: osdmap e51: 8 total, 7 up, 8 in 2023-10-10T17:38:34.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] boot 2023-10-10T17:38:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: osdmap e52: 8 total, 8 up, 8 in 2023-10-10T17:38:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi157 2023-10-10T17:38:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi157 to 5722M 2023-10-10T17:38:34.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:34.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:34.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:34.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:38:34.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:34 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:34.514 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:38:34.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:38:34.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: osdmap e51: 8 total, 7 up, 8 in 2023-10-10T17:38:34.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] boot 2023-10-10T17:38:34.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: osdmap e52: 8 total, 8 up, 8 in 2023-10-10T17:38:34.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi157 2023-10-10T17:38:34.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi157 to 5722M 2023-10-10T17:38:34.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:34.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:34.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:34.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:38:34.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:34 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:34.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:38:34.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: osdmap e51: 8 total, 7 up, 8 in 2023-10-10T17:38:34.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: osd.7 [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] boot 2023-10-10T17:38:34.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: osdmap e52: 8 total, 8 up, 8 in 2023-10-10T17:38:34.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:38:34.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi157 2023-10-10T17:38:34.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:34.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi157 to 5722M 2023-10-10T17:38:34.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:34.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:34.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:34.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:34.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:38:34.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:34 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:35.722 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 -- 172.21.15.190:0/1089072093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9834100d70 msgr2=0x7f9834107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:35.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 --2- 172.21.15.190:0/1089072093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9834100d70 0x7f9834107630 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f982c00b0a0 tx=0x7f982c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:38:35.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 -- 172.21.15.190:0/1089072093 shutdown_connections 2023-10-10T17:38:35.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 --2- 172.21.15.190:0/1089072093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9834100d70 0x7f9834107630 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:35.723 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 --2- 172.21.15.190:0/1089072093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98340ff1e0 0x7f98340ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:35.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 --2- 172.21.15.190:0/1089072093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98341003e0 0x7f9834100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:35.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 -- 172.21.15.190:0/1089072093 >> 172.21.15.190:0/1089072093 conn(0x7f98340fa7f0 msgr2=0x7f98340fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:38:35.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.721+0000 7f9838bdd700 1 -- 172.21.15.190:0/1089072093 shutdown_connections 2023-10-10T17:38:35.724 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.722+0000 7f9838bdd700 1 -- 172.21.15.190:0/1089072093 wait complete. 2023-10-10T17:38:35.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 Processor -- start 2023-10-10T17:38:35.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 -- start start 2023-10-10T17:38:35.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98340ff1e0 0x7f98340699c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:35.726 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98341003e0 0x7f983419afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:35.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9834100d70 0x7f983419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:35.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983419dd90 con 0x7f9834100d70 2023-10-10T17:38:35.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983419df00 con 0x7f98340ff1e0 2023-10-10T17:38:35.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9838bdd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983419e0c0 con 0x7f98341003e0 2023-10-10T17:38:35.727 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98341003e0 0x7f983419afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:35.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98341003e0 0x7f983419afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:52008/0 (socket says 172.21.15.190:52008) 2023-10-10T17:38:35.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 -- 172.21.15.190:0/2461896048 learned_addr learned my addr 172.21.15.190:0/2461896048 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:38:35.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f983259c700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98340ff1e0 0x7f98340699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:35.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9832d9d700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9834100d70 0x7f983419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:35.728 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 -- 172.21.15.190:0/2461896048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98340ff1e0 msgr2=0x7f98340699c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:35.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98340ff1e0 0x7f98340699c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:35.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 -- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9834100d70 msgr2=0x7f983419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:35.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9834100d70 0x7f983419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:35.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 -- 172.21.15.190:0/2461896048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c009d00 con 0x7f98341003e0 2023-10-10T17:38:35.729 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f9831d9b700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98341003e0 0x7f983419afa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f982400b530 tx=0x7f982400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:38:35.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.723+0000 7f98237fe700 1 -- 172.21.15.190:0/2461896048 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824011830 con 0x7f98341003e0 2023-10-10T17:38:35.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.724+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983410ac50 con 0x7f98341003e0 2023-10-10T17:38:35.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.724+0000 7f98237fe700 1 -- 172.21.15.190:0/2461896048 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9824011e50 con 0x7f98341003e0 2023-10-10T17:38:35.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.724+0000 7f98237fe700 1 -- 172.21.15.190:0/2461896048 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982400f2e0 con 0x7f98341003e0 2023-10-10T17:38:35.730 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.724+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983410aac0 con 0x7f98341003e0 2023-10-10T17:38:35.731 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.724+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98341067d0 con 0x7f98341003e0 2023-10-10T17:38:35.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.726+0000 7f98237fe700 1 -- 172.21.15.190:0/2461896048 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f98240105a0 con 0x7f98341003e0 2023-10-10T17:38:35.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.727+0000 7f98237fe700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f981c0a11d0 0x7f981c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:35.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.727+0000 7f98237fe700 1 -- 172.21.15.190:0/2461896048 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f98240cbdd0 con 0x7f98341003e0 2023-10-10T17:38:35.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.727+0000 7f983259c700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f981c0a11d0 0x7f981c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:35.732 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.727+0000 7f983259c700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f981c0a11d0 0x7f981c0a3680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9834100240 tx=0x7f9828009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:38:35.733 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.730+0000 7f98237fe700 1 -- 172.21.15.190:0/2461896048 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9824095fc0 con 0x7f98341003e0 2023-10-10T17:38:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:35 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:38:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:35 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:38:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:35 smithi078 ceph-mon[99494]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:35.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:35 smithi078 ceph-mon[99494]: osdmap e53: 8 total, 8 up, 8 in 2023-10-10T17:38:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:35 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:38:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:35 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:38:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:35 smithi190 ceph-mon[101220]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:35 smithi190 ceph-mon[101220]: osdmap e53: 8 total, 8 up, 8 in 2023-10-10T17:38:35.928 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:35.927+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9834061b40 con 0x7f981c0a11d0 2023-10-10T17:38:35.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:35 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:38:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:35 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:38:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:35 smithi157 ceph-mon[100896]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2023-10-10T17:38:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:35 smithi157 ceph-mon[100896]: osdmap e53: 8 total, 8 up, 8 in 2023-10-10T17:38:36.811 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:36 smithi190 ceph-mon[101220]: osdmap e54: 8 total, 8 up, 8 in 2023-10-10T17:38:36.811 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:36.811 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:36.812 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:36 smithi078 ceph-mon[99494]: osdmap e54: 8 total, 8 up, 8 in 2023-10-10T17:38:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:36 smithi157 ceph-mon[100896]: osdmap e54: 8 total, 8 up, 8 in 2023-10-10T17:38:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:37 smithi078 ceph-mon[99494]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:37 smithi078 ceph-mon[99494]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:37 smithi078 ceph-mon[99494]: osdmap e55: 8 total, 8 up, 8 in 2023-10-10T17:38:37.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:37 smithi190 ceph-mon[101220]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:37.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:37 smithi190 ceph-mon[101220]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:37.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:37 smithi190 ceph-mon[101220]: osdmap e55: 8 total, 8 up, 8 in 2023-10-10T17:38:37.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:37 smithi157 ceph-mon[100896]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:37.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:37 smithi157 ceph-mon[100896]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:37.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:37 smithi157 ceph-mon[100896]: osdmap e55: 8 total, 8 up, 8 in 2023-10-10T17:38:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:38 smithi078 ceph-mon[99494]: osdmap e56: 8 total, 8 up, 8 in 2023-10-10T17:38:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:38 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/2065495815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]: dispatch 2023-10-10T17:38:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:38 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]: dispatch 2023-10-10T17:38:38.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:38 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]': finished 2023-10-10T17:38:38.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:38 smithi078 ceph-mon[99494]: osdmap e57: 9 total, 8 up, 9 in 2023-10-10T17:38:38.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:38 smithi190 ceph-mon[101220]: osdmap e56: 8 total, 8 up, 8 in 2023-10-10T17:38:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:38 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/2065495815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]: dispatch 2023-10-10T17:38:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:38 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]: dispatch 2023-10-10T17:38:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:38 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]': finished 2023-10-10T17:38:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:38 smithi190 ceph-mon[101220]: osdmap e57: 9 total, 8 up, 9 in 2023-10-10T17:38:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:38.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:38 smithi157 ceph-mon[100896]: osdmap e56: 8 total, 8 up, 8 in 2023-10-10T17:38:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:38 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/2065495815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]: dispatch 2023-10-10T17:38:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:38 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]: dispatch 2023-10-10T17:38:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:38 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "82521098-f51f-4c32-ad81-5d1c12c67a24"}]': finished 2023-10-10T17:38:38.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:38 smithi157 ceph-mon[100896]: osdmap e57: 9 total, 8 up, 9 in 2023-10-10T17:38:38.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:39 smithi078 ceph-mon[99494]: pgmap v167: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:39 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/77176008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:38:39.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:39 smithi190 ceph-mon[101220]: pgmap v167: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:39.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:39 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/77176008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:38:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:39 smithi157 ceph-mon[100896]: pgmap v167: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:39 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/77176008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:38:41.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:41 smithi078 ceph-mon[99494]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:41.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:41 smithi190 ceph-mon[101220]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:41.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:41 smithi157 ceph-mon[100896]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:43.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:43 smithi078 ceph-mon[99494]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:43.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:43 smithi190 ceph-mon[101220]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:43 smithi157 ceph-mon[100896]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:45.871 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:45 smithi190 ceph-mon[101220]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:45.871 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:45 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-10-10T17:38:45.871 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:45 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:45.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:45 smithi078 ceph-mon[99494]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:45 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-10-10T17:38:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:45 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:45 smithi157 ceph-mon[100896]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:45 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-10-10T17:38:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:45 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:46.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:46 smithi190 ceph-mon[101220]: Deploying daemon osd.8 on smithi190 2023-10-10T17:38:46.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:46 smithi078 ceph-mon[99494]: Deploying daemon osd.8 on smithi190 2023-10-10T17:38:46.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:46 smithi157 ceph-mon[100896]: Deploying daemon osd.8 on smithi190 2023-10-10T17:38:47.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:47 smithi190 ceph-mon[101220]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:47.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:47 smithi078 ceph-mon[99494]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:47.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:47 smithi157 ceph-mon[100896]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:49.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:49 smithi078 ceph-mon[99494]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:49.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:49.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:49.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:49.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:49 smithi190 ceph-mon[101220]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:49.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:49.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:49.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:49 smithi157 ceph-mon[100896]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:50.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:50 smithi190 ceph-mon[101220]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:50.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:50 smithi078 ceph-mon[99494]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:50 smithi157 ceph-mon[100896]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:51.986 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.984+0000 7f98237fe700 1 -- 172.21.15.190:0/2461896048 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9834061b40 con 0x7f981c0a11d0 2023-10-10T17:38:51.986 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 8 on host 'smithi190' 2023-10-10T17:38:51.988 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f981c0a11d0 msgr2=0x7f981c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:51.988 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f981c0a11d0 0x7f981c0a3680 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9834100240 tx=0x7f9828009190 comp rx=0 tx=0).stop 2023-10-10T17:38:51.988 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98341003e0 msgr2=0x7f983419afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98341003e0 0x7f983419afa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f982400b530 tx=0x7f982400b840 comp rx=0 tx=0).stop 2023-10-10T17:38:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 shutdown_connections 2023-10-10T17:38:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f981c0a11d0 0x7f981c0a3680 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9834100d70 0x7f983419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f98340ff1e0 0x7f98340699c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:51.990 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 --2- 172.21.15.190:0/2461896048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f98341003e0 0x7f983419afa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:51.990 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 >> 172.21.15.190:0/2461896048 conn(0x7f98340fa7f0 msgr2=0x7f9834103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:38:51.990 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 shutdown_connections 2023-10-10T17:38:51.990 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:38:51.986+0000 7f9838bdd700 1 -- 172.21.15.190:0/2461896048 wait complete. 2023-10-10T17:38:52.499 DEBUG:teuthology.orchestra.run.smithi190:osd.8> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.8.service 2023-10-10T17:38:52.501 INFO:tasks.cephadm:Deploying osd.9 on smithi078 with /dev/vg_nvme/lv_1... 2023-10-10T17:38:52.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-10-10T17:38:52.529 INFO:journalctl@ceph.osd.8.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:38:52.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:52.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:52.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:52.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:38:52.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:52.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:52.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:52 smithi078 ceph-mon[99494]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:52.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:52.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:52.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:52.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:38:52.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:52.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:52.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:52 smithi190 ceph-mon[101220]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:38:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:38:52.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:52.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:52 smithi157 ceph-mon[100896]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:38:53.186 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:38:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:38:53.116+0000 7feff373b640 -1 osd.8 0 log_to_monitors true 2023-10-10T17:38:54.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:53 smithi078 ceph-mon[99494]: from='osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-10T17:38:54.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:53 smithi078 ceph-mon[99494]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-10T17:38:54.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:53 smithi157 ceph-mon[100896]: from='osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-10T17:38:54.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:53 smithi157 ceph-mon[100896]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-10T17:38:54.253 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:53 smithi190 ceph-mon[101220]: from='osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-10T17:38:54.253 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:53 smithi190 ceph-mon[101220]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-10-10T17:38:54.812 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-10-10T17:38:54.812 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-10-10T17:38:54.813 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-10-10T17:38:54.813 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-10-10T17:38:54.813 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115547 s, 907 MB/s 2023-10-10T17:38:54.813 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-10-10T17:38:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-10-10T17:38:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: osdmap e58: 9 total, 8 up, 9 in 2023-10-10T17:38:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:55.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:38:55.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:38:55.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:55.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi190 2023-10-10T17:38:55.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi190 to 5722M 2023-10-10T17:38:55.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:55.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:55.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:55.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:38:55.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:54 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:38:55.314 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2023-10-10T17:38:55.414 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:38:54.985+0000 7fefe8db2700 -1 osd.8 0 waiting for initial osdmap 2023-10-10T17:38:55.415 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:38:54.998+0000 7fefe23a2700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:38:55.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-10-10T17:38:55.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: osdmap e58: 9 total, 8 up, 9 in 2023-10-10T17:38:55.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:55.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:38:55.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:38:55.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:55.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi190 2023-10-10T17:38:55.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi190 to 5722M 2023-10-10T17:38:55.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:55.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:55.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:55.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.419 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:38:55.419 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:54 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:38:55.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-10-10T17:38:55.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: osdmap e58: 9 total, 8 up, 9 in 2023-10-10T17:38:55.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:55.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:38:55.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:38:55.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:55.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi190 2023-10-10T17:38:55.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:38:55.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi190 to 5722M 2023-10-10T17:38:55.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:55.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:38:55.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:55.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:38:55.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:38:55.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:54 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:38:55.958 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:38:56.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:55 smithi078 ceph-mon[99494]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:38:56.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:55 smithi078 ceph-mon[99494]: osdmap e59: 9 total, 8 up, 9 in 2023-10-10T17:38:56.357 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:55 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:56.358 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:55 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:56.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:55 smithi190 ceph-mon[101220]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:38:56.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:55 smithi190 ceph-mon[101220]: osdmap e59: 9 total, 8 up, 9 in 2023-10-10T17:38:56.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:55 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:56.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:55 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:56.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:55 smithi157 ceph-mon[100896]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:38:56.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:55 smithi157 ceph-mon[100896]: osdmap e59: 9 total, 8 up, 9 in 2023-10-10T17:38:56.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:55 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:56.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:55 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1879530315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e41002f0 msgr2=0x7fd7e4100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1879530315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e41002f0 0x7fd7e4100700 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009990 tx=0x7fd7d4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:38:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1879530315 shutdown_connections 2023-10-10T17:38:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1879530315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7e4101e80 0x7fd7e4108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1879530315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e41002f0 0x7fd7e4100700 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1879530315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e41014f0 0x7fd7e4101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:57.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1879530315 >> 172.21.15.78:0/1879530315 conn(0x7fd7e40fb900 msgr2=0x7fd7e40fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:38:57.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.146+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1879530315 shutdown_connections 2023-10-10T17:38:57.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1879530315 wait complete. 2023-10-10T17:38:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 Processor -- start 2023-10-10T17:38:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 -- start start 2023-10-10T17:38:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7e41002f0 0x7fd7e4074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e41014f0 0x7fd7e4072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e4101e80 0x7fd7e4073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4073ba0 con 0x7fd7e4101e80 2023-10-10T17:38:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4073d10 con 0x7fd7e41002f0 2023-10-10T17:38:57.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ec1f5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e41a0030 con 0x7fd7e41014f0 2023-10-10T17:38:57.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ea792700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e4101e80 0x7fd7e4073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:57.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ea792700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e4101e80 0x7fd7e4073420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41600/0 (socket says 172.21.15.78:41600) 2023-10-10T17:38:57.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7ea792700 1 -- 172.21.15.78:0/1567011634 learned_addr learned my addr 172.21.15.78:0/1567011634 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:38:57.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7e9790700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e41014f0 0x7fd7e4072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:57.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.147+0000 7fd7e9f91700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7e41002f0 0x7fd7e4074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:57.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ea792700 1 -- 172.21.15.78:0/1567011634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e41014f0 msgr2=0x7fd7e4072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:57.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ea792700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e41014f0 0x7fd7e4072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:57.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ea792700 1 -- 172.21.15.78:0/1567011634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7e41002f0 msgr2=0x7fd7e4074890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:38:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ea792700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7e41002f0 0x7fd7e4074890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:38:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ea792700 1 -- 172.21.15.78:0/1567011634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d40095f0 con 0x7fd7e4101e80 2023-10-10T17:38:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ea792700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e4101e80 0x7fd7e4073420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc007d70 tx=0x7fd7dc00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:38:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7daffd700 1 -- 172.21.15.78:0/1567011634 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc00ad40 con 0x7fd7e4101e80 2023-10-10T17:38:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e410bc60 con 0x7fd7e4101e80 2023-10-10T17:38:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7daffd700 1 -- 172.21.15.78:0/1567011634 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7dc0128e0 con 0x7fd7e4101e80 2023-10-10T17:38:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7daffd700 1 -- 172.21.15.78:0/1567011634 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc0116b0 con 0x7fd7e4101e80 2023-10-10T17:38:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.148+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e410bad0 con 0x7fd7e4101e80 2023-10-10T17:38:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.149+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e4107840 con 0x7fd7e4101e80 2023-10-10T17:38:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.150+0000 7fd7daffd700 1 -- 172.21.15.78:0/1567011634 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd7dc012420 con 0x7fd7e4101e80 2023-10-10T17:38:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.151+0000 7fd7daffd700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7d00a11d0 0x7fd7d00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:38:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.151+0000 7fd7e9f91700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7d00a11d0 0x7fd7d00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:38:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.151+0000 7fd7daffd700 1 -- 172.21.15.78:0/1567011634 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fd7dc0cc1e0 con 0x7fd7e4101e80 2023-10-10T17:38:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.151+0000 7fd7e9f91700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7d00a11d0 0x7fd7d00a3680 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400b2e0 tx=0x7fd7d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:38:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.155+0000 7fd7daffd700 1 -- 172.21.15.78:0/1567011634 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd7dc096220 con 0x7fd7e4101e80 2023-10-10T17:38:57.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:38:57.353+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7e4061c70 con 0x7fd7d00a11d0 2023-10-10T17:38:57.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:56 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:38:57.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:56 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:38:57.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:56 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:57.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:56 smithi078 ceph-mon[99494]: osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] boot 2023-10-10T17:38:57.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:56 smithi078 ceph-mon[99494]: osdmap e60: 9 total, 9 up, 9 in 2023-10-10T17:38:57.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:56 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:57.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:56 smithi078 ceph-mon[99494]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:57.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:56 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:38:57.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:56 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:38:57.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:56 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:57.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:56 smithi190 ceph-mon[101220]: osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] boot 2023-10-10T17:38:57.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:56 smithi190 ceph-mon[101220]: osdmap e60: 9 total, 9 up, 9 in 2023-10-10T17:38:57.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:56 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:57.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:56 smithi190 ceph-mon[101220]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:56 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:38:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:56 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:38:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:56 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:56 smithi157 ceph-mon[100896]: osd.8 [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] boot 2023-10-10T17:38:57.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:56 smithi157 ceph-mon[100896]: osdmap e60: 9 total, 9 up, 9 in 2023-10-10T17:38:57.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:56 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:38:57.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:56 smithi157 ceph-mon[100896]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-10-10T17:38:58.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:58 smithi078 ceph-mon[99494]: osdmap e61: 9 total, 9 up, 9 in 2023-10-10T17:38:58.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:58 smithi078 ceph-mon[99494]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:58.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:58.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:58.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:58 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:58.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:58 smithi190 ceph-mon[101220]: osdmap e61: 9 total, 9 up, 9 in 2023-10-10T17:38:58.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:58 smithi190 ceph-mon[101220]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:58.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:58.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:58.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:58 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:58.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:58 smithi157 ceph-mon[100896]: osdmap e61: 9 total, 9 up, 9 in 2023-10-10T17:38:58.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:58 smithi157 ceph-mon[100896]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:38:58.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:38:58.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:38:58.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:58 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:38:59.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:59 smithi078 ceph-mon[99494]: osdmap e62: 9 total, 9 up, 9 in 2023-10-10T17:38:59.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:59 smithi078 ceph-mon[99494]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:38:59.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:38:59 smithi078 ceph-mon[99494]: osdmap e63: 9 total, 9 up, 9 in 2023-10-10T17:38:59.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:59 smithi190 ceph-mon[101220]: osdmap e62: 9 total, 9 up, 9 in 2023-10-10T17:38:59.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:59 smithi190 ceph-mon[101220]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:38:59.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:38:59 smithi190 ceph-mon[101220]: osdmap e63: 9 total, 9 up, 9 in 2023-10-10T17:38:59.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:59 smithi157 ceph-mon[100896]: osdmap e62: 9 total, 9 up, 9 in 2023-10-10T17:38:59.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:59 smithi157 ceph-mon[100896]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 52 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:38:59.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:38:59 smithi157 ceph-mon[100896]: osdmap e63: 9 total, 9 up, 9 in 2023-10-10T17:39:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:00 smithi078 ceph-mon[99494]: osdmap e64: 9 total, 9 up, 9 in 2023-10-10T17:39:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:00 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2181424587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7670b692-faa9-4a49-8d19-32418fa64d2a"}]: dispatch 2023-10-10T17:39:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:00 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2181424587' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7670b692-faa9-4a49-8d19-32418fa64d2a"}]': finished 2023-10-10T17:39:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:00 smithi078 ceph-mon[99494]: osdmap e65: 10 total, 9 up, 10 in 2023-10-10T17:39:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:00 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:00.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:00 smithi190 ceph-mon[101220]: osdmap e64: 9 total, 9 up, 9 in 2023-10-10T17:39:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:00 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2181424587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7670b692-faa9-4a49-8d19-32418fa64d2a"}]: dispatch 2023-10-10T17:39:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:00 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2181424587' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7670b692-faa9-4a49-8d19-32418fa64d2a"}]': finished 2023-10-10T17:39:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:00 smithi190 ceph-mon[101220]: osdmap e65: 10 total, 9 up, 10 in 2023-10-10T17:39:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:00 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:00.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:00 smithi157 ceph-mon[100896]: osdmap e64: 9 total, 9 up, 9 in 2023-10-10T17:39:00.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:00 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2181424587' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7670b692-faa9-4a49-8d19-32418fa64d2a"}]: dispatch 2023-10-10T17:39:00.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:00 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2181424587' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7670b692-faa9-4a49-8d19-32418fa64d2a"}]': finished 2023-10-10T17:39:00.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:00 smithi157 ceph-mon[100896]: osdmap e65: 10 total, 9 up, 10 in 2023-10-10T17:39:00.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:00 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:01 smithi078 ceph-mon[99494]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 180 KiB/s, 0 objects/s recovering 2023-10-10T17:39:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:01 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1487276705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:01.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:01 smithi190 ceph-mon[101220]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 180 KiB/s, 0 objects/s recovering 2023-10-10T17:39:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:01 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1487276705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:01.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:01 smithi157 ceph-mon[100896]: pgmap v187: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 180 KiB/s, 0 objects/s recovering 2023-10-10T17:39:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:01 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1487276705' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:03.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:03 smithi078 ceph-mon[99494]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-10-10T17:39:03.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:03 smithi190 ceph-mon[101220]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-10-10T17:39:03.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:03 smithi157 ceph-mon[100896]: pgmap v188: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 137 KiB/s, 0 objects/s recovering 2023-10-10T17:39:05.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:05 smithi078 ceph-mon[99494]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:39:05.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:05 smithi190 ceph-mon[101220]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:39:05.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:05 smithi157 ceph-mon[100896]: pgmap v189: 1 pgs: 1 peering; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:39:06.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-10-10T17:39:06.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:06 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-10-10T17:39:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:06 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:06.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-10-10T17:39:06.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:06 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:07.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:07 smithi078 ceph-mon[99494]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2023-10-10T17:39:07.839 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:07 smithi078 ceph-mon[99494]: Deploying daemon osd.9 on smithi078 2023-10-10T17:39:07.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:07 smithi190 ceph-mon[101220]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2023-10-10T17:39:07.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:07 smithi190 ceph-mon[101220]: Deploying daemon osd.9 on smithi078 2023-10-10T17:39:07.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:07 smithi157 ceph-mon[100896]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 53 MiB used, 805 GiB / 805 GiB avail; 75 KiB/s, 0 objects/s recovering 2023-10-10T17:39:07.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:07 smithi157 ceph-mon[100896]: Deploying daemon osd.9 on smithi078 2023-10-10T17:39:09.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:09 smithi078 ceph-mon[99494]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:09.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:09 smithi190 ceph-mon[101220]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:09.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:09 smithi157 ceph-mon[100896]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:11.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:11 smithi078 ceph-mon[99494]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:11.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:11 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:11.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:11 smithi190 ceph-mon[101220]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:11.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:11 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:11.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:11 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:11.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:11 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:11 smithi157 ceph-mon[100896]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:11 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:11 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:11 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:13.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:13 smithi078 ceph-mon[99494]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:13.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.900+0000 7fd7daffd700 1 -- 172.21.15.78:0/1567011634 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd7e4061c70 con 0x7fd7d00a11d0 2023-10-10T17:39:13.902 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 9 on host 'smithi078' 2023-10-10T17:39:13.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7d00a11d0 msgr2=0x7fd7d00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:13.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7d00a11d0 0x7fd7d00a3680 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400b2e0 tx=0x7fd7d4005fb0 comp rx=0 tx=0).stop 2023-10-10T17:39:13.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e4101e80 msgr2=0x7fd7e4073420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:13.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e4101e80 0x7fd7e4073420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc007d70 tx=0x7fd7dc00a340 comp rx=0 tx=0).stop 2023-10-10T17:39:13.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 shutdown_connections 2023-10-10T17:39:13.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7d00a11d0 0x7fd7d00a3680 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:13.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7e4101e80 0x7fd7e4073420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:13.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7e41002f0 0x7fd7e4074890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:13.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 --2- 172.21.15.78:0/1567011634 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7e41014f0 0x7fd7e4072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:13.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 >> 172.21.15.78:0/1567011634 conn(0x7fd7e40fb900 msgr2=0x7fd7e4104980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:39:13.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 shutdown_connections 2023-10-10T17:39:13.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:13.903+0000 7fd7ec1f5700 1 -- 172.21.15.78:0/1567011634 wait complete. 2023-10-10T17:39:13.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:13 smithi190 ceph-mon[101220]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:13.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:13 smithi157 ceph-mon[100896]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:14.343 DEBUG:teuthology.orchestra.run.smithi078:osd.9> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.9.service 2023-10-10T17:39:14.346 INFO:tasks.cephadm:Deploying osd.10 on smithi157 with /dev/vg_nvme/lv_1... 2023-10-10T17:39:14.346 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-10-10T17:39:14.375 INFO:journalctl@ceph.osd.9.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:39:14.995 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:39:15.127 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:39:14 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:39:14.863+0000 7f8d842df640 -1 osd.9 0 log_to_monitors true 2023-10-10T17:39:15.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:14 smithi078 ceph-mon[99494]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:14 smithi190 ceph-mon[101220]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:15.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:14 smithi157 ceph-mon[100896]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:16.109 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:15 smithi157 ceph-mon[100896]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-10-10T17:39:16.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:15 smithi078 ceph-mon[99494]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-10-10T17:39:16.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:15 smithi190 ceph-mon[101220]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-10-10T17:39:16.568 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-10-10T17:39:16.568 INFO:teuthology.orchestra.run.smithi157.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-10-10T17:39:16.568 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10+0 records in 2023-10-10T17:39:16.569 INFO:teuthology.orchestra.run.smithi157.stderr:10+0 records out 2023-10-10T17:39:16.569 INFO:teuthology.orchestra.run.smithi157.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00821527 s, 1.3 GB/s 2023-10-10T17:39:16.569 INFO:teuthology.orchestra.run.smithi157.stderr:--> Zapping successful for: 2023-10-10T17:39:17.042 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi157:vg_nvme/lv_1 2023-10-10T17:39:17.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:16 smithi078 ceph-mon[99494]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-10-10T17:39:17.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:16 smithi078 ceph-mon[99494]: osdmap e66: 10 total, 9 up, 10 in 2023-10-10T17:39:17.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:16 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:17.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:16 smithi078 ceph-mon[99494]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:39:17.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:16 smithi078 ceph-mon[99494]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:17.130 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:39:16 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:39:16.872+0000 7f8d79956700 -1 osd.9 0 waiting for initial osdmap 2023-10-10T17:39:17.130 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:39:16 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:39:16.890+0000 7f8d73747700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:39:17.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:16 smithi190 ceph-mon[101220]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-10-10T17:39:17.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:16 smithi190 ceph-mon[101220]: osdmap e66: 10 total, 9 up, 10 in 2023-10-10T17:39:17.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:16 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:17.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:16 smithi190 ceph-mon[101220]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:39:17.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:16 smithi190 ceph-mon[101220]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:16 smithi157 ceph-mon[100896]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-10-10T17:39:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:16 smithi157 ceph-mon[100896]: osdmap e66: 10 total, 9 up, 10 in 2023-10-10T17:39:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:16 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:16 smithi157 ceph-mon[100896]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-10-10T17:39:17.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:16 smithi157 ceph-mon[100896]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:17.684 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:39:18.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:39:18.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:39:18.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:39:18.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: osdmap e67: 10 total, 9 up, 10 in 2023-10-10T17:39:18.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi078 2023-10-10T17:39:18.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi078 to 4292M 2023-10-10T17:39:18.170 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.170 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:18.170 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:18.171 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.171 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.172 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:39:18.172 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.172 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:18.172 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:18.172 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:18.172 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.173 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.173 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:39:18.173 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:17 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.207 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:39:18.207 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:39:18.207 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:39:18.208 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: osdmap e67: 10 total, 9 up, 10 in 2023-10-10T17:39:18.208 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.208 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.208 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi078 2023-10-10T17:39:18.208 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.208 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.209 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.209 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.209 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.209 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.209 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi078 to 4292M 2023-10-10T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.210 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:18.211 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:39:18.212 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:17 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:39:18.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:39:18.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-10-10T17:39:18.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: osdmap e67: 10 total, 9 up, 10 in 2023-10-10T17:39:18.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi078 2023-10-10T17:39:18.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:18.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi078 to 4292M 2023-10-10T17:39:18.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:18.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:18.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:39:18.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:18.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:18.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:18.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:18.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:39:18.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:17 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:18.875 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.870+0000 7f0e42bec700 1 -- 172.21.15.157:0/3500474600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c101e80 msgr2=0x7f0e3c108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:18.876 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.870+0000 7f0e42bec700 1 --2- 172.21.15.157:0/3500474600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c101e80 0x7f0e3c108740 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0e3400b0a0 tx=0x7f0e3400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:39:18.876 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.870+0000 7f0e42bec700 1 -- 172.21.15.157:0/3500474600 shutdown_connections 2023-10-10T17:39:18.876 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.870+0000 7f0e42bec700 1 --2- 172.21.15.157:0/3500474600 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c101e80 0x7f0e3c108740 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:18.877 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.870+0000 7f0e42bec700 1 --2- 172.21.15.157:0/3500474600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1002f0 0x7f0e3c100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:18.877 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.870+0000 7f0e42bec700 1 --2- 172.21.15.157:0/3500474600 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e3c1014f0 0x7f0e3c101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:18.877 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.870+0000 7f0e42bec700 1 -- 172.21.15.157:0/3500474600 >> 172.21.15.157:0/3500474600 conn(0x7f0e3c0fb900 msgr2=0x7f0e3c0fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:39:18.878 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 -- 172.21.15.157:0/3500474600 shutdown_connections 2023-10-10T17:39:18.878 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 -- 172.21.15.157:0/3500474600 wait complete. 2023-10-10T17:39:18.879 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 Processor -- start 2023-10-10T17:39:18.879 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 -- start start 2023-10-10T17:39:18.880 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c1002f0 0x7f0e3c074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:18.880 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1014f0 0x7f0e3c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:18.882 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e3c101e80 0x7f0e3c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:18.882 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c073ba0 con 0x7f0e3c1014f0 2023-10-10T17:39:18.882 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c073d10 con 0x7f0e3c1002f0 2023-10-10T17:39:18.883 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e42bec700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3c1a0140 con 0x7f0e3c101e80 2023-10-10T17:39:18.883 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e40988700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c1002f0 0x7f0e3c074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:18.883 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e40988700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c1002f0 0x7f0e3c074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.157:58658/0 (socket says 172.21.15.157:58658) 2023-10-10T17:39:18.883 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.871+0000 7f0e40988700 1 -- 172.21.15.157:0/1168440330 learned_addr learned my addr 172.21.15.157:0/1168440330 (peer_addr_for_me v2:172.21.15.157:0/0) 2023-10-10T17:39:18.884 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e3bfff700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1014f0 0x7f0e3c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:18.884 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e41189700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e3c101e80 0x7f0e3c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:18.884 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e40988700 1 -- 172.21.15.157:0/1168440330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e3c101e80 msgr2=0x7f0e3c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:18.885 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e40988700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e3c101e80 0x7f0e3c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:18.885 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e40988700 1 -- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1014f0 msgr2=0x7f0e3c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:18.885 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e40988700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1014f0 0x7f0e3c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:18.885 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e40988700 1 -- 172.21.15.157:0/1168440330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e34009d00 con 0x7f0e3c1002f0 2023-10-10T17:39:18.886 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e41189700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e3c101e80 0x7f0e3c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:39:18.886 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e40988700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c1002f0 0x7f0e3c074890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000d980 tx=0x7f0e3000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:39:18.886 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e39ffb700 1 -- 172.21.15.157:0/1168440330 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3000b950 con 0x7f0e3c1002f0 2023-10-10T17:39:18.886 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e39ffb700 1 -- 172.21.15.157:0/1168440330 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e30007620 con 0x7f0e3c1002f0 2023-10-10T17:39:18.887 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3c10bc50 con 0x7f0e3c1002f0 2023-10-10T17:39:18.887 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e39ffb700 1 -- 172.21.15.157:0/1168440330 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e3000f3c0 con 0x7f0e3c1002f0 2023-10-10T17:39:18.887 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.872+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3c10bac0 con 0x7f0e3c1002f0 2023-10-10T17:39:18.887 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.873+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e3c107850 con 0x7f0e3c1002f0 2023-10-10T17:39:18.887 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.874+0000 7f0e39ffb700 1 -- 172.21.15.157:0/1168440330 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0e30010640 con 0x7f0e3c1002f0 2023-10-10T17:39:18.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.875+0000 7f0e39ffb700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e2c0a1150 0x7f0e2c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:18.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.875+0000 7f0e39ffb700 1 -- 172.21.15.157:0/1168440330 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f0e300cbff0 con 0x7f0e3c1002f0 2023-10-10T17:39:18.888 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.875+0000 7f0e3bfff700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e2c0a1150 0x7f0e2c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:18.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.875+0000 7f0e3bfff700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e2c0a1150 0x7f0e2c0a3600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005950 tx=0x7f0e2800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:39:18.889 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:18.879+0000 7f0e39ffb700 1 -- 172.21.15.157:0/1168440330 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0e30095f40 con 0x7f0e3c1002f0 2023-10-10T17:39:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:18 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:18 smithi157 ceph-mon[100896]: osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] boot 2023-10-10T17:39:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:18 smithi157 ceph-mon[100896]: osdmap e68: 10 total, 10 up, 10 in 2023-10-10T17:39:18.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:18 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:18.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:18 smithi157 ceph-mon[100896]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:19.077 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:19.077+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e3c061a20 con 0x7f0e2c0a1150 2023-10-10T17:39:19.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:18 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:18 smithi190 ceph-mon[101220]: osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] boot 2023-10-10T17:39:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:18 smithi190 ceph-mon[101220]: osdmap e68: 10 total, 10 up, 10 in 2023-10-10T17:39:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:18 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:18 smithi190 ceph-mon[101220]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:19.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:18 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:19.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:18 smithi078 ceph-mon[99494]: osd.9 [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] boot 2023-10-10T17:39:19.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:18 smithi078 ceph-mon[99494]: osdmap e68: 10 total, 10 up, 10 in 2023-10-10T17:39:19.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:18 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:39:19.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:18 smithi078 ceph-mon[99494]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-10-10T17:39:20.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:19 smithi190 ceph-mon[101220]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:39:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:39:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:39:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:20.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:19 smithi190 ceph-mon[101220]: osdmap e69: 10 total, 10 up, 10 in 2023-10-10T17:39:20.215 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:19 smithi157 ceph-mon[100896]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:39:20.215 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:39:20.216 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:39:20.216 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:20.216 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:19 smithi157 ceph-mon[100896]: osdmap e69: 10 total, 10 up, 10 in 2023-10-10T17:39:20.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:19 smithi078 ceph-mon[99494]: from='client.24332 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi157:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:39:20.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:39:20.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:39:20.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:20.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:19 smithi078 ceph-mon[99494]: osdmap e69: 10 total, 10 up, 10 in 2023-10-10T17:39:20.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:20 smithi157 ceph-mon[100896]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:20 smithi190 ceph-mon[101220]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:21.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:20 smithi078 ceph-mon[99494]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 59 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:22.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:21 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/2879878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]: dispatch 2023-10-10T17:39:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:21 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]: dispatch 2023-10-10T17:39:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:21 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]': finished 2023-10-10T17:39:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:21 smithi190 ceph-mon[101220]: osdmap e70: 11 total, 10 up, 11 in 2023-10-10T17:39:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:22.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:21 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/2879878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]: dispatch 2023-10-10T17:39:22.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:21 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]: dispatch 2023-10-10T17:39:22.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:21 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]': finished 2023-10-10T17:39:22.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:21 smithi157 ceph-mon[100896]: osdmap e70: 11 total, 10 up, 11 in 2023-10-10T17:39:22.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:22.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:21 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/2879878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]: dispatch 2023-10-10T17:39:22.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:21 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]: dispatch 2023-10-10T17:39:22.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:21 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "998b645c-fda2-4969-9917-da58dede0c03"}]': finished 2023-10-10T17:39:22.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:21 smithi078 ceph-mon[99494]: osdmap e70: 11 total, 10 up, 11 in 2023-10-10T17:39:22.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:22 smithi190 ceph-mon[101220]: from='client.? 172.21.15.157:0/3835341511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:22 smithi190 ceph-mon[101220]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:23.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.157:0/3835341511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:23.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:22 smithi157 ceph-mon[100896]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:23.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:22 smithi078 ceph-mon[99494]: from='client.? 172.21.15.157:0/3835341511' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:23.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:22 smithi078 ceph-mon[99494]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:25.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:25 smithi157 ceph-mon[100896]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:25.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:25 smithi078 ceph-mon[99494]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:25.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:25 smithi190 ceph-mon[101220]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:27.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:27 smithi078 ceph-mon[99494]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:27.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:27 smithi190 ceph-mon[101220]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:27.928 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:27 smithi157 ceph-mon[100896]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:28.827 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-10-10T17:39:28.828 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:28.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-10-10T17:39:28.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:28.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-10-10T17:39:28.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:29.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:29 smithi157 ceph-mon[100896]: Deploying daemon osd.10 on smithi157 2023-10-10T17:39:29.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:29 smithi157 ceph-mon[100896]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:29.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:29 smithi078 ceph-mon[99494]: Deploying daemon osd.10 on smithi157 2023-10-10T17:39:29.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:29 smithi078 ceph-mon[99494]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:29.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:29 smithi190 ceph-mon[101220]: Deploying daemon osd.10 on smithi157 2023-10-10T17:39:29.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:29 smithi190 ceph-mon[101220]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:31.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:31 smithi157 ceph-mon[100896]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:31.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:31 smithi078 ceph-mon[99494]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:31.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:31 smithi190 ceph-mon[101220]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:33.696 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:33 smithi157 ceph-mon[100896]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:33.697 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:33.697 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:33.697 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:33 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:33.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:33 smithi078 ceph-mon[99494]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:33.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:33.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:33.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:33 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:33.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:33 smithi190 ceph-mon[101220]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:33.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:33.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:33.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:33 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:35.335 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.334+0000 7f0e39ffb700 1 -- 172.21.15.157:0/1168440330 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f0e3c061a20 con 0x7f0e2c0a1150 2023-10-10T17:39:35.336 INFO:teuthology.orchestra.run.smithi157.stdout:Created osd(s) 10 on host 'smithi157' 2023-10-10T17:39:35.337 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e2c0a1150 msgr2=0x7f0e2c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:35.338 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e2c0a1150 0x7f0e2c0a3600 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0e28005950 tx=0x7f0e2800b410 comp rx=0 tx=0).stop 2023-10-10T17:39:35.338 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c1002f0 msgr2=0x7f0e3c074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:35.338 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c1002f0 0x7f0e3c074890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000d980 tx=0x7f0e3000dc90 comp rx=0 tx=0).stop 2023-10-10T17:39:35.338 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 shutdown_connections 2023-10-10T17:39:35.339 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e2c0a1150 0x7f0e2c0a3600 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:35.339 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e3c101e80 0x7f0e3c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:35.339 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e3c1002f0 0x7f0e3c074890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:35.339 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 --2- 172.21.15.157:0/1168440330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e3c1014f0 0x7f0e3c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:35.339 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 >> 172.21.15.157:0/1168440330 conn(0x7f0e3c0fb900 msgr2=0x7f0e3c104990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:39:35.340 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 shutdown_connections 2023-10-10T17:39:35.340 INFO:teuthology.orchestra.run.smithi157.stderr:2023-10-10T17:39:35.336+0000 7f0e42bec700 1 -- 172.21.15.157:0/1168440330 wait complete. 2023-10-10T17:39:35.638 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:35.639 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.639 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.639 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:35.639 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:35.640 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.640 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.640 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.640 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:35 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:35.677 DEBUG:teuthology.orchestra.run.smithi157:osd.10> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.10.service 2023-10-10T17:39:35.680 INFO:tasks.cephadm:Deploying osd.11 on smithi190 with /dev/vg_nvme/lv_1... 2023-10-10T17:39:35.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-10-10T17:39:35.710 INFO:journalctl@ceph.osd.10.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:39:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:35.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:35.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:35 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:35.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:35 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:36.333 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:39:36.539 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:39:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:39:36.359+0000 7fc82bf15640 -1 osd.10 0 log_to_monitors true 2023-10-10T17:39:36.863 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:36 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:36.864 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:36 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:39:36.864 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:36 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:36.864 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:36 smithi190 ceph-mon[101220]: from='osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-10T17:39:36.864 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:36 smithi190 ceph-mon[101220]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-10T17:39:36.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:36 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:36 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:39:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:36 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:36 smithi078 ceph-mon[99494]: from='osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-10T17:39:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:36 smithi078 ceph-mon[99494]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-10T17:39:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:36 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:36 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:39:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:36 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:36 smithi157 ceph-mon[100896]: from='osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-10T17:39:36.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:36 smithi157 ceph-mon[100896]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-10-10T17:39:37.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:37 smithi190 ceph-mon[101220]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:37.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:37 smithi190 ceph-mon[101220]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-10-10T17:39:37.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:37 smithi190 ceph-mon[101220]: osdmap e71: 11 total, 10 up, 11 in 2023-10-10T17:39:37.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:37.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:37 smithi190 ceph-mon[101220]: from='osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:39:37.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:37 smithi190 ceph-mon[101220]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:39:37.768 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-10-10T17:39:37.768 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-10-10T17:39:37.768 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2023-10-10T17:39:37.769 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2023-10-10T17:39:37.769 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111934 s, 937 MB/s 2023-10-10T17:39:37.769 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2023-10-10T17:39:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:37 smithi078 ceph-mon[99494]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:37 smithi078 ceph-mon[99494]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-10-10T17:39:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:37 smithi078 ceph-mon[99494]: osdmap e71: 11 total, 10 up, 11 in 2023-10-10T17:39:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:37.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:37 smithi078 ceph-mon[99494]: from='osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:39:37.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:37 smithi078 ceph-mon[99494]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:39:37.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:39:37.549+0000 7fc82158c700 -1 osd.10 0 waiting for initial osdmap 2023-10-10T17:39:37.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:39:37.566+0000 7fc81d381700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:39:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-mon[100896]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-mon[100896]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-10-10T17:39:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-mon[100896]: osdmap e71: 11 total, 10 up, 11 in 2023-10-10T17:39:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:37.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-mon[100896]: from='osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:39:37.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:37 smithi157 ceph-mon[100896]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]: dispatch 2023-10-10T17:39:38.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch daemon add osd smithi190:vg_nvme/lv_1 2023-10-10T17:39:38.824 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:39:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:39:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: osdmap e72: 11 total, 10 up, 11 in 2023-10-10T17:39:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:38.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:38.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] boot 2023-10-10T17:39:38.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: osdmap e73: 11 total, 11 up, 11 in 2023-10-10T17:39:38.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:38 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:39:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: osdmap e72: 11 total, 10 up, 11 in 2023-10-10T17:39:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:38.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:38.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] boot 2023-10-10T17:39:38.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: osdmap e73: 11 total, 11 up, 11 in 2023-10-10T17:39:38.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:38 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi157", "root=default"]}]': finished 2023-10-10T17:39:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: osdmap e72: 11 total, 10 up, 11 in 2023-10-10T17:39:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:38.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:38.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:38.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:38.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:38.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: osd.10 [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] boot 2023-10-10T17:39:38.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: osdmap e73: 11 total, 11 up, 11 in 2023-10-10T17:39:38.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:38 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:39:39.576 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:39:39.576 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:39:39.576 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi157 2023-10-10T17:39:39.576 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi157 to 4292M 2023-10-10T17:39:39.577 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:39.577 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:39.577 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:39:39.577 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:39:39.577 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:39 smithi190 ceph-mon[101220]: osdmap e74: 11 total, 11 up, 11 in 2023-10-10T17:39:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:39:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:39:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi157 2023-10-10T17:39:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi157 to 4292M 2023-10-10T17:39:39.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:39.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:39.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:39:39.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:39:39.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:39 smithi078 ceph-mon[99494]: osdmap e74: 11 total, 11 up, 11 in 2023-10-10T17:39:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:39:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:39:39.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi157 2023-10-10T17:39:39.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi157 to 4292M 2023-10-10T17:39:39.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:39.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-10-10T17:39:39.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:39:39.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:39:39.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:39 smithi157 ceph-mon[100896]: osdmap e74: 11 total, 11 up, 11 in 2023-10-10T17:39:40.002 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 -- 172.21.15.190:0/2222722800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77881008b0 msgr2=0x7f7788100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:40.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 --2- 172.21.15.190:0/2222722800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77881008b0 0x7f7788100cc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f777c009990 tx=0x7f777c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:39:40.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 -- 172.21.15.190:0/2222722800 shutdown_connections 2023-10-10T17:39:40.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 --2- 172.21.15.190:0/2222722800 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77881008b0 0x7f7788100cc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:40.003 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 --2- 172.21.15.190:0/2222722800 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77880ffe30 0x7f77881076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:40.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 --2- 172.21.15.190:0/2222722800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77880ff480 0x7f77880ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:40.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 -- 172.21.15.190:0/2222722800 >> 172.21.15.190:0/2222722800 conn(0x7f77880fa000 msgr2=0x7f77880fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:39:40.004 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 -- 172.21.15.190:0/2222722800 shutdown_connections 2023-10-10T17:39:40.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:39.999+0000 7f778d9da700 1 -- 172.21.15.190:0/2222722800 wait complete. 2023-10-10T17:39:40.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 Processor -- start 2023-10-10T17:39:40.005 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 -- start start 2023-10-10T17:39:40.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77880ff480 0x7f77881981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:40.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77880ffe30 0x7f7788198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:40.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77881008b0 0x7f778819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:40.006 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778819fd70 con 0x7f77880ffe30 2023-10-10T17:39:40.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778819feb0 con 0x7f77881008b0 2023-10-10T17:39:40.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f778d9da700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77881a0040 con 0x7f77880ff480 2023-10-10T17:39:40.007 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f7786ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77880ff480 0x7f77881981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:40.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f7786ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77880ff480 0x7f77881981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:52148/0 (socket says 172.21.15.190:52148) 2023-10-10T17:39:40.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f7786ffd700 1 -- 172.21.15.190:0/2432213916 learned_addr learned my addr 172.21.15.190:0/2432213916 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:39:40.008 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f77867fc700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77880ffe30 0x7f7788198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:40.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.000+0000 7f77877fe700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77881008b0 0x7f778819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:40.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f7786ffd700 1 -- 172.21.15.190:0/2432213916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77881008b0 msgr2=0x7f778819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:40.009 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f7786ffd700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77881008b0 0x7f778819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:40.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f7786ffd700 1 -- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77880ffe30 msgr2=0x7f7788198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:40.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f7786ffd700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77880ffe30 0x7f7788198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:40.010 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f7786ffd700 1 -- 172.21.15.190:0/2432213916 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c0095f0 con 0x7f77880ff480 2023-10-10T17:39:40.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f7786ffd700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77880ff480 0x7f77881981e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f777c005910 tx=0x7f777c00f710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:39:40.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f778c9d8700 1 -- 172.21.15.190:0/2432213916 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c01d070 con 0x7f77880ff480 2023-10-10T17:39:40.011 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f778c9d8700 1 -- 172.21.15.190:0/2432213916 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f777c022470 con 0x7f77880ff480 2023-10-10T17:39:40.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f778c9d8700 1 -- 172.21.15.190:0/2432213916 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c017650 con 0x7f77880ff480 2023-10-10T17:39:40.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778810abe0 con 0x7f77880ff480 2023-10-10T17:39:40.012 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.001+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778810aa50 con 0x7f77880ff480 2023-10-10T17:39:40.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.002+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7788105170 con 0x7f77880ff480 2023-10-10T17:39:40.014 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.003+0000 7f778c9d8700 1 -- 172.21.15.190:0/2432213916 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f777c0225e0 con 0x7f77880ff480 2023-10-10T17:39:40.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.004+0000 7f778c9d8700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77700a1150 0x7f77700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:40.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.004+0000 7f778c9d8700 1 -- 172.21.15.190:0/2432213916 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f777c0cdc80 con 0x7f77880ff480 2023-10-10T17:39:40.015 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.004+0000 7f77867fc700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77700a1150 0x7f77700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:40.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.004+0000 7f77867fc700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77700a1150 0x7f77700a3600 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7778005950 tx=0x7f777800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:39:40.016 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.007+0000 7f778c9d8700 1 -- 172.21.15.190:0/2432213916 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f777c097a90 con 0x7f77880ff480 2023-10-10T17:39:40.203 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:40.202+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7788061c90 con 0x7f77700a1150 2023-10-10T17:39:40.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:39:40.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:39:40.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:40.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:40 smithi078 ceph-mon[99494]: osdmap e75: 11 total, 11 up, 11 in 2023-10-10T17:39:40.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:40 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:39:40.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:40 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:39:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:40 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:40 smithi190 ceph-mon[101220]: osdmap e75: 11 total, 11 up, 11 in 2023-10-10T17:39:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:40 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-10T17:39:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:40 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-10T17:39:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:40 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:40 smithi157 ceph-mon[100896]: osdmap e75: 11 total, 11 up, 11 in 2023-10-10T17:39:41.777 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:41 smithi190 ceph-mon[101220]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:41.777 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:41 smithi190 ceph-mon[101220]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:39:41.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:41 smithi078 ceph-mon[99494]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:41.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:41 smithi078 ceph-mon[99494]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:39:41.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:41 smithi157 ceph-mon[100896]: pgmap v216: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:41 smithi157 ceph-mon[100896]: from='client.24355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:39:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:42 smithi078 ceph-mon[99494]: osdmap e76: 11 total, 11 up, 11 in 2023-10-10T17:39:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/385939912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]: dispatch 2023-10-10T17:39:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:42 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]: dispatch 2023-10-10T17:39:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:42 smithi078 ceph-mon[99494]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]': finished 2023-10-10T17:39:42.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:42 smithi078 ceph-mon[99494]: osdmap e77: 12 total, 11 up, 12 in 2023-10-10T17:39:42.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:42 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:39:42.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:42 smithi190 ceph-mon[101220]: osdmap e76: 11 total, 11 up, 11 in 2023-10-10T17:39:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:42 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/385939912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]: dispatch 2023-10-10T17:39:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:42 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]: dispatch 2023-10-10T17:39:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:42 smithi190 ceph-mon[101220]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]': finished 2023-10-10T17:39:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:42 smithi190 ceph-mon[101220]: osdmap e77: 12 total, 11 up, 12 in 2023-10-10T17:39:42.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:42 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:39:42.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:42 smithi157 ceph-mon[100896]: osdmap e76: 11 total, 11 up, 11 in 2023-10-10T17:39:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:42 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/385939912' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]: dispatch 2023-10-10T17:39:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:42 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]: dispatch 2023-10-10T17:39:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:42 smithi157 ceph-mon[100896]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d638b1e0-6b53-45f2-9ea0-719cdf4ff100"}]': finished 2023-10-10T17:39:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:42 smithi157 ceph-mon[100896]: osdmap e77: 12 total, 11 up, 12 in 2023-10-10T17:39:42.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:42 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:39:43.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:43 smithi078 ceph-mon[99494]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:43.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:43 smithi078 ceph-mon[99494]: from='client.? 172.21.15.190:0/1947361019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:43.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:43 smithi190 ceph-mon[101220]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:43.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:43 smithi190 ceph-mon[101220]: from='client.? 172.21.15.190:0/1947361019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:43 smithi157 ceph-mon[100896]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 66 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:43 smithi157 ceph-mon[100896]: from='client.? 172.21.15.190:0/1947361019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-10T17:39:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:45 smithi078 ceph-mon[99494]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:45.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:45 smithi190 ceph-mon[101220]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:45.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:45 smithi157 ceph-mon[100896]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:47.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:47 smithi190 ceph-mon[101220]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:39:47.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:47 smithi078 ceph-mon[99494]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:39:47.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:47 smithi157 ceph-mon[100896]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-10-10T17:39:49.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:49 smithi078 ceph-mon[99494]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2023-10-10T17:39:49.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-10-10T17:39:49.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:49 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:49.886 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:49 smithi190 ceph-mon[101220]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2023-10-10T17:39:49.886 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-10-10T17:39:49.887 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:49 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:49.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:49 smithi157 ceph-mon[100896]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2023-10-10T17:39:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-10-10T17:39:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:49 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:50.835 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:50 smithi190 ceph-mon[101220]: Deploying daemon osd.11 on smithi190 2023-10-10T17:39:50.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:50 smithi078 ceph-mon[99494]: Deploying daemon osd.11 on smithi190 2023-10-10T17:39:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:50 smithi157 ceph-mon[100896]: Deploying daemon osd.11 on smithi190 2023-10-10T17:39:51.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:51 smithi078 ceph-mon[99494]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-10-10T17:39:51.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:51 smithi190 ceph-mon[101220]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-10-10T17:39:51.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:51 smithi157 ceph-mon[100896]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 67 KiB/s, 0 objects/s recovering 2023-10-10T17:39:52.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:52 smithi190 ceph-mon[101220]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-10-10T17:39:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:52 smithi157 ceph-mon[100896]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-10-10T17:39:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:52 smithi078 ceph-mon[99494]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-10-10T17:39:54.593 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:54.594 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:54.594 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:54 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:54.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:54.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:54.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:54 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:54.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:54.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:54.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:54 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:55.835 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:55 smithi190 ceph-mon[101220]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-10-10T17:39:55.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:55 smithi078 ceph-mon[99494]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-10-10T17:39:55.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:55 smithi157 ceph-mon[100896]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-10-10T17:39:56.161 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.160+0000 7f778c9d8700 1 -- 172.21.15.190:0/2432213916 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f7788061c90 con 0x7f77700a1150 2023-10-10T17:39:56.161 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 11 on host 'smithi190' 2023-10-10T17:39:56.163 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77700a1150 msgr2=0x7f77700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:56.163 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77700a1150 0x7f77700a3600 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7778005950 tx=0x7f777800b410 comp rx=0 tx=0).stop 2023-10-10T17:39:56.163 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77880ff480 msgr2=0x7f77881981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:56.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77880ff480 0x7f77881981e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f777c005910 tx=0x7f777c00f710 comp rx=0 tx=0).stop 2023-10-10T17:39:56.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 shutdown_connections 2023-10-10T17:39:56.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77700a1150 0x7f77700a3600 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:56.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77880ff480 0x7f77881981e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:56.164 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77881008b0 0x7f778819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:56.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 --2- 172.21.15.190:0/2432213916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77880ffe30 0x7f7788198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:56.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 >> 172.21.15.190:0/2432213916 conn(0x7f77880fa000 msgr2=0x7f77880fc210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:39:56.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 shutdown_connections 2023-10-10T17:39:56.165 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:39:56.162+0000 7f778d9da700 1 -- 172.21.15.190:0/2432213916 wait complete. 2023-10-10T17:39:56.521 DEBUG:teuthology.orchestra.run.smithi190:osd.11> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.11.service 2023-10-10T17:39:56.524 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-10-10T17:39:56.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd stat -f json 2023-10-10T17:39:56.562 INFO:journalctl@ceph.osd.11.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:39:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:39:57.335 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:39:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:39:57.267+0000 7fe1c21af640 -1 osd.11 0 log_to_monitors true 2023-10-10T17:39:57.336 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:57 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.336 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:57 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.337 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:57 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.337 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:57 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.338 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:57 smithi190 ceph-mon[101220]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-10T17:39:57.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:57 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:57 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:57 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:57 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:57 smithi078 ceph-mon[99494]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-10T17:39:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:57 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:57 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:57 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:57 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:57.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:57 smithi157 ceph-mon[100896]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-10-10T17:39:58.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 -- 172.21.15.78:0/3247490187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45240ffb60 msgr2=0x7f45240fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 --2- 172.21.15.78:0/3247490187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45240ffb60 0x7f45240fffb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4514009990 tx=0x7f4514009ca0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 -- 172.21.15.78:0/3247490187 shutdown_connections 2023-10-10T17:39:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 --2- 172.21.15.78:0/3247490187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240fe960 0x7f45240fed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 --2- 172.21.15.78:0/3247490187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45241004f0 0x7f4524106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 --2- 172.21.15.78:0/3247490187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45240ffb60 0x7f45240fffb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 -- 172.21.15.78:0/3247490187 >> 172.21.15.78:0/3247490187 conn(0x7f45240f9f70 msgr2=0x7f45240fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:39:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 -- 172.21.15.78:0/3247490187 shutdown_connections 2023-10-10T17:39:58.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.386+0000 7f452c0ca700 1 -- 172.21.15.78:0/3247490187 wait complete. 2023-10-10T17:39:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 Processor -- start 2023-10-10T17:39:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 -- start start 2023-10-10T17:39:58.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45240fe960 0x7f45241028f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240ffb60 0x7f4524100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45241004f0 0x7f4524101480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524101c00 con 0x7f45241004f0 2023-10-10T17:39:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524101d70 con 0x7f45240fe960 2023-10-10T17:39:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452c0ca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45241a0260 con 0x7f45240ffb60 2023-10-10T17:39:58.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452a667700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45241004f0 0x7f4524101480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:58.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f4529665700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240ffb60 0x7f4524100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:58.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452a667700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45241004f0 0x7f4524101480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41760/0 (socket says 172.21.15.78:41760) 2023-10-10T17:39:58.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f4529665700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240ffb60 0x7f4524100f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:33288/0 (socket says 172.21.15.78:33288) 2023-10-10T17:39:58.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.387+0000 7f452a667700 1 -- 172.21.15.78:0/1549942366 learned_addr learned my addr 172.21.15.78:0/1549942366 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:39:58.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f4529e66700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45240fe960 0x7f45241028f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:58.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452a667700 1 -- 172.21.15.78:0/1549942366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240ffb60 msgr2=0x7f4524100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:58.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452a667700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240ffb60 0x7f4524100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452a667700 1 -- 172.21.15.78:0/1549942366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45240fe960 msgr2=0x7f45241028f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:58.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452a667700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45240fe960 0x7f45241028f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452a667700 1 -- 172.21.15.78:0/1549942366 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140095f0 con 0x7f45241004f0 2023-10-10T17:39:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f4529665700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240ffb60 0x7f4524100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:39:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452a667700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45241004f0 0x7f4524101480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f451c007ee0 tx=0x7f451c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:39:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f451affd700 1 -- 172.21.15.78:0/1549942366 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c017070 con 0x7f45241004f0 2023-10-10T17:39:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f451affd700 1 -- 172.21.15.78:0/1549942366 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f451c00f040 con 0x7f45241004f0 2023-10-10T17:39:58.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f451affd700 1 -- 172.21.15.78:0/1549942366 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451c005740 con 0x7f45241004f0 2023-10-10T17:39:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f452410a2b0 con 0x7f45241004f0 2023-10-10T17:39:58.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.388+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f452410a120 con 0x7f45241004f0 2023-10-10T17:39:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.389+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4524106740 con 0x7f45241004f0 2023-10-10T17:39:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.390+0000 7f451affd700 1 -- 172.21.15.78:0/1549942366 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f451c00dc60 con 0x7f45241004f0 2023-10-10T17:39:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.391+0000 7f451affd700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f45100a11d0 0x7f45100a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:39:58.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.391+0000 7f451affd700 1 -- 172.21.15.78:0/1549942366 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f451c0cc120 con 0x7f45241004f0 2023-10-10T17:39:58.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.391+0000 7f4529e66700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f45100a11d0 0x7f45100a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:39:58.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.391+0000 7f4529e66700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f45100a11d0 0x7f45100a3680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f45240ff9c0 tx=0x7f4520009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:39:58.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.394+0000 7f451affd700 1 -- 172.21.15.78:0/1549942366 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f451c095e40 con 0x7f45241004f0 2023-10-10T17:39:58.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:58 smithi190 ceph-mon[101220]: from='osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-10T17:39:58.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:58 smithi190 ceph-mon[101220]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-10T17:39:58.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:58 smithi157 ceph-mon[100896]: from='osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-10T17:39:58.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:58 smithi157 ceph-mon[100896]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-10T17:39:58.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.584+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f452402cfe0 con 0x7f45241004f0 2023-10-10T17:39:58.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.584+0000 7f451affd700 1 -- 172.21.15.78:0/1549942366 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v78) v1 ==== 74+0+133 (secure 0 0 0) 0x7f451c099460 con 0x7f45241004f0 2023-10-10T17:39:58.586 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:39:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f45100a11d0 msgr2=0x7f45100a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:58.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f45100a11d0 0x7f45100a3680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f45240ff9c0 tx=0x7f4520009260 comp rx=0 tx=0).stop 2023-10-10T17:39:58.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45241004f0 msgr2=0x7f4524101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:39:58.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45241004f0 0x7f4524101480 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f451c007ee0 tx=0x7f451c00d3b0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 shutdown_connections 2023-10-10T17:39:58.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f45100a11d0 0x7f45100a3680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f45241004f0 0x7f4524101480 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f45240fe960 0x7f45241028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 --2- 172.21.15.78:0/1549942366 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f45240ffb60 0x7f4524100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:39:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 >> 172.21.15.78:0/1549942366 conn(0x7f45240f9f70 msgr2=0x7f452410b3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:39:58.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 shutdown_connections 2023-10-10T17:39:58.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:39:58.586+0000 7f452c0ca700 1 -- 172.21.15.78:0/1549942366 wait complete. 2023-10-10T17:39:58.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:58 smithi078 ceph-mon[99494]: from='osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-10T17:39:58.608 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:58 smithi078 ceph-mon[99494]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-10-10T17:39:59.041 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":78,"num_osds":12,"num_up_osds":11,"osd_up_since":1696959578,"num_in_osds":12,"osd_in_since":1696959582,"num_remapped_pgs":0} 2023-10-10T17:39:59.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-10-10T17:39:59.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: osdmap e78: 12 total, 11 up, 12 in 2023-10-10T17:39:59.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:39:59.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:39:59.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:39:59.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:59.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1549942366' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-10T17:39:59.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:59.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:59.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:59.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:59.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:59.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:39:59 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-10-10T17:39:59.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: osdmap e78: 12 total, 11 up, 12 in 2023-10-10T17:39:59.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:39:59.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:39:59.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:39:59.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:59.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1549942366' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-10T17:39:59.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:59.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:59.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:59.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:59.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:59.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:39:59 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:39:59.170+0000 7fe1b7826700 -1 osd.11 0 waiting for initial osdmap 2023-10-10T17:39:59.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:39:59.185+0000 7fe1b2e1a700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-10T17:39:59.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-10-10T17:39:59.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: osdmap e78: 12 total, 11 up, 12 in 2023-10-10T17:39:59.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:39:59.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:39:59.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2023-10-10T17:39:59.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:39:59.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1549942366' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-10T17:39:59.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:39:59.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:59.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:59.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:39:59.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:39:59.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:39:59.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:39:59.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:39:59 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:00.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd stat -f json 2023-10-10T17:40:00.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Detected new or changed devices on smithi190 2023-10-10T17:40:00.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi190 to 4292M 2023-10-10T17:40:00.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:40:00.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:40:00.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:40:00.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: osdmap e79: 12 total, 11 up, 12 in 2023-10-10T17:40:00.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:00.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:00.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:00 smithi078 ceph-mon[99494]: overall HEALTH_OK 2023-10-10T17:40:00.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Detected new or changed devices on smithi190 2023-10-10T17:40:00.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi190 to 4292M 2023-10-10T17:40:00.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:40:00.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:40:00.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:40:00.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: osdmap e79: 12 total, 11 up, 12 in 2023-10-10T17:40:00.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:00.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:00.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:00 smithi157 ceph-mon[100896]: overall HEALTH_OK 2023-10-10T17:40:00.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Detected new or changed devices on smithi190 2023-10-10T17:40:00.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi190 to 4292M 2023-10-10T17:40:00.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:40:00.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:40:00.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:00.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2023-10-10T17:40:00.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: osdmap e79: 12 total, 11 up, 12 in 2023-10-10T17:40:00.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:00.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:00.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:00 smithi190 ceph-mon[101220]: overall HEALTH_OK 2023-10-10T17:40:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:40:01.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:01 smithi078 ceph-mon[99494]: purged_snaps scrub starts 2023-10-10T17:40:01.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:01 smithi078 ceph-mon[99494]: purged_snaps scrub ok 2023-10-10T17:40:01.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:01 smithi078 ceph-mon[99494]: osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] boot 2023-10-10T17:40:01.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:01 smithi078 ceph-mon[99494]: osdmap e80: 12 total, 12 up, 12 in 2023-10-10T17:40:01.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:01 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:01.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:01 smithi078 ceph-mon[99494]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:40:01.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:01 smithi157 ceph-mon[100896]: purged_snaps scrub starts 2023-10-10T17:40:01.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:01 smithi157 ceph-mon[100896]: purged_snaps scrub ok 2023-10-10T17:40:01.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:01 smithi157 ceph-mon[100896]: osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] boot 2023-10-10T17:40:01.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:01 smithi157 ceph-mon[100896]: osdmap e80: 12 total, 12 up, 12 in 2023-10-10T17:40:01.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:01 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:01.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:01 smithi157 ceph-mon[100896]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:40:01.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:01 smithi190 ceph-mon[101220]: purged_snaps scrub starts 2023-10-10T17:40:01.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:01 smithi190 ceph-mon[101220]: purged_snaps scrub ok 2023-10-10T17:40:01.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:01 smithi190 ceph-mon[101220]: osd.11 [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] boot 2023-10-10T17:40:01.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:01 smithi190 ceph-mon[101220]: osdmap e80: 12 total, 12 up, 12 in 2023-10-10T17:40:01.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:01 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:40:01.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:01 smithi190 ceph-mon[101220]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-10-10T17:40:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 -- 172.21.15.78:0/3406162601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c1081a0 msgr2=0x7fd55c10a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:01.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3406162601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c1081a0 0x7fd55c10a620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd55400b0a0 tx=0x7fd55400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 -- 172.21.15.78:0/3406162601 shutdown_connections 2023-10-10T17:40:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3406162601 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c1081a0 0x7fd55c10a620 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3406162601 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd55c074ac0 0x7fd55c072f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3406162601 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c0734f0 0x7fd55c073960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:01.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 -- 172.21.15.78:0/3406162601 >> 172.21.15.78:0/3406162601 conn(0x7fd55c0fb910 msgr2=0x7fd55c0fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 -- 172.21.15.78:0/3406162601 shutdown_connections 2023-10-10T17:40:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.911+0000 7fd561bbb700 1 -- 172.21.15.78:0/3406162601 wait complete. 2023-10-10T17:40:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 Processor -- start 2023-10-10T17:40:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 -- start start 2023-10-10T17:40:01.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0734f0 0x7fd55c198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c074ac0 0x7fd55c198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd55c1081a0 0x7fd55c19f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c19fde0 con 0x7fd55c0734f0 2023-10-10T17:40:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c19ff20 con 0x7fd55c1081a0 2023-10-10T17:40:01.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd561bbb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55c1a00b0 con 0x7fd55c074ac0 2023-10-10T17:40:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd55b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0734f0 0x7fd55c198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd55b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0734f0 0x7fd55c198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41774/0 (socket says 172.21.15.78:41774) 2023-10-10T17:40:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd55b7fe700 1 -- 172.21.15.78:0/3290878739 learned_addr learned my addr 172.21.15.78:0/3290878739 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.912+0000 7fd55bfff700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd55c1081a0 0x7fd55c19f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:01.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55affd700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c074ac0 0x7fd55c198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:01.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55b7fe700 1 -- 172.21.15.78:0/3290878739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c074ac0 msgr2=0x7fd55c198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:01.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55b7fe700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c074ac0 0x7fd55c198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:01.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55b7fe700 1 -- 172.21.15.78:0/3290878739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd55c1081a0 msgr2=0x7fd55c19f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:01.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55b7fe700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd55c1081a0 0x7fd55c19f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:01.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55b7fe700 1 -- 172.21.15.78:0/3290878739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554009d00 con 0x7fd55c0734f0 2023-10-10T17:40:01.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55affd700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c074ac0 0x7fd55c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd55b7fe700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0734f0 0x7fd55c198250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd55000d980 tx=0x7fd55000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd558ff9700 1 -- 172.21.15.78:0/3290878739 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd550009b40 con 0x7fd55c0734f0 2023-10-10T17:40:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd558ff9700 1 -- 172.21.15.78:0/3290878739 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd55000b6e0 con 0x7fd55c0734f0 2023-10-10T17:40:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd55c10de50 con 0x7fd55c0734f0 2023-10-10T17:40:01.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd558ff9700 1 -- 172.21.15.78:0/3290878739 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55000f620 con 0x7fd55c0734f0 2023-10-10T17:40:01.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.913+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd55c10dcc0 con 0x7fd55c0734f0 2023-10-10T17:40:01.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.914+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd55c106ff0 con 0x7fd55c0734f0 2023-10-10T17:40:01.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.915+0000 7fd558ff9700 1 -- 172.21.15.78:0/3290878739 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd550010460 con 0x7fd55c0734f0 2023-10-10T17:40:01.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.916+0000 7fd558ff9700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd5440a11d0 0x7fd5440a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:01.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.916+0000 7fd55affd700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd5440a11d0 0x7fd5440a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:01.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.916+0000 7fd558ff9700 1 -- 172.21.15.78:0/3290878739 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd5500cc730 con 0x7fd55c0734f0 2023-10-10T17:40:01.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.916+0000 7fd55affd700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd5440a11d0 0x7fd5440a3680 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd54c005fd0 tx=0x7fd54c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:01.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:01.919+0000 7fd558ff9700 1 -- 172.21.15.78:0/3290878739 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd550096280 con 0x7fd55c0734f0 2023-10-10T17:40:02.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.110+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd55c02cfe0 con 0x7fd55c0734f0 2023-10-10T17:40:02.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.111+0000 7fd558ff9700 1 -- 172.21.15.78:0/3290878739 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7fd5500998a0 con 0x7fd55c0734f0 2023-10-10T17:40:02.114 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:02.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd5440a11d0 msgr2=0x7fd5440a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd5440a11d0 0x7fd5440a3680 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd54c005fd0 tx=0x7fd54c005e20 comp rx=0 tx=0).stop 2023-10-10T17:40:02.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0734f0 msgr2=0x7fd55c198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:02.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0734f0 0x7fd55c198250 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd55000d980 tx=0x7fd55000dc90 comp rx=0 tx=0).stop 2023-10-10T17:40:02.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 shutdown_connections 2023-10-10T17:40:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd5440a11d0 0x7fd5440a3680 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd55c1081a0 0x7fd55c19f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd55c0734f0 0x7fd55c198250 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 --2- 172.21.15.78:0/3290878739 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd55c074ac0 0x7fd55c198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 >> 172.21.15.78:0/3290878739 conn(0x7fd55c0fb910 msgr2=0x7fd55c104130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:02.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 shutdown_connections 2023-10-10T17:40:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:02.113+0000 7fd561bbb700 1 -- 172.21.15.78:0/3290878739 wait complete. 2023-10-10T17:40:02.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:02 smithi078 ceph-mon[99494]: osdmap e81: 12 total, 12 up, 12 in 2023-10-10T17:40:02.465 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:02 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3290878739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-10T17:40:02.467 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1696959600,"num_in_osds":12,"osd_in_since":1696959582,"num_remapped_pgs":0} 2023-10-10T17:40:02.467 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd dump --format=json 2023-10-10T17:40:02.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:02 smithi157 ceph-mon[100896]: osdmap e81: 12 total, 12 up, 12 in 2023-10-10T17:40:02.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:02 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3290878739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-10T17:40:02.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:02 smithi190 ceph-mon[101220]: osdmap e81: 12 total, 12 up, 12 in 2023-10-10T17:40:02.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:02 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3290878739' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-10T17:40:03.109 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:40:03.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:03 smithi157 ceph-mon[100896]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:03.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:03 smithi078 ceph-mon[99494]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:03.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:03 smithi190 ceph-mon[101220]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:04.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 -- 172.21.15.78:0/4053230981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ffe00 msgr2=0x7f25c8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 --2- 172.21.15.78:0/4053230981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ffe00 0x7f25c8103dd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f25b80099c0 tx=0x7f25b8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 -- 172.21.15.78:0/4053230981 shutdown_connections 2023-10-10T17:40:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 --2- 172.21.15.78:0/4053230981 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25c80ff460 0x7f25c80ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 --2- 172.21.15.78:0/4053230981 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25c8104500 0x7f25c810c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 --2- 172.21.15.78:0/4053230981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ffe00 0x7f25c8103dd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 -- 172.21.15.78:0/4053230981 >> 172.21.15.78:0/4053230981 conn(0x7f25c80770a0 msgr2=0x7f25c80774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:04.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 -- 172.21.15.78:0/4053230981 shutdown_connections 2023-10-10T17:40:04.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.410+0000 7f25ccc23700 1 -- 172.21.15.78:0/4053230981 wait complete. 2023-10-10T17:40:04.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 Processor -- start 2023-10-10T17:40:04.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 -- start start 2023-10-10T17:40:04.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ff460 0x7f25c8100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:04.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25c80ffe00 0x7f25c81013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:04.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25c8104500 0x7f25c8101930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:04.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c81a4870 con 0x7f25c80ff460 2023-10-10T17:40:04.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c81a49e0 con 0x7f25c8104500 2023-10-10T17:40:04.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25ccc23700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c81a4ba0 con 0x7f25c80ffe00 2023-10-10T17:40:04.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ff460 0x7f25c8100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:04.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ff460 0x7f25c8100e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41794/0 (socket says 172.21.15.78:41794) 2023-10-10T17:40:04.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 -- 172.21.15.78:0/3460489274 learned_addr learned my addr 172.21.15.78:0/3460489274 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:04.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c5d9b700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25c80ffe00 0x7f25c81013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:04.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c6d9d700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25c8104500 0x7f25c8101930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:04.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 -- 172.21.15.78:0/3460489274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25c80ffe00 msgr2=0x7f25c81013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:04.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25c80ffe00 0x7f25c81013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 -- 172.21.15.78:0/3460489274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25c8104500 msgr2=0x7f25c8101930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:04.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25c8104500 0x7f25c8101930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 -- 172.21.15.78:0/3460489274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b80095f0 con 0x7f25c80ff460 2023-10-10T17:40:04.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c6d9d700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25c8104500 0x7f25c8101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:04.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c5d9b700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25c80ffe00 0x7f25c81013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:04.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.411+0000 7f25c659c700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ff460 0x7f25c8100e70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00c920 tx=0x7f25bc00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:04.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.412+0000 7f25b77fe700 1 -- 172.21.15.78:0/3460489274 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc007b00 con 0x7f25c80ff460 2023-10-10T17:40:04.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.412+0000 7f25b77fe700 1 -- 172.21.15.78:0/3460489274 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f25bc004cf0 con 0x7f25c80ff460 2023-10-10T17:40:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.412+0000 7f25b77fe700 1 -- 172.21.15.78:0/3460489274 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc005610 con 0x7f25c80ff460 2023-10-10T17:40:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.412+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c810f230 con 0x7f25c80ff460 2023-10-10T17:40:04.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.412+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c810f0a0 con 0x7f25c80ff460 2023-10-10T17:40:04.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.413+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c810f3c0 con 0x7f25c80ff460 2023-10-10T17:40:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.414+0000 7f25b77fe700 1 -- 172.21.15.78:0/3460489274 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f25bc004750 con 0x7f25c80ff460 2023-10-10T17:40:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.414+0000 7f25b77fe700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f25b00a1150 0x7f25b00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.415+0000 7f25c5d9b700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f25b00a1150 0x7f25b00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:04.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.415+0000 7f25b77fe700 1 -- 172.21.15.78:0/3460489274 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f25bc0cc410 con 0x7f25c80ff460 2023-10-10T17:40:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.415+0000 7f25c5d9b700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f25b00a1150 0x7f25b00a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f25b8000c00 tx=0x7f25b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:04.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.418+0000 7f25b77fe700 1 -- 172.21.15.78:0/3460489274 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f25bc096000 con 0x7f25c80ff460 2023-10-10T17:40:04.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.606+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25c81a55f0 con 0x7f25c80ff460 2023-10-10T17:40:04.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.607+0000 7f25b77fe700 1 -- 172.21.15.78:0/3460489274 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f25bc099620 con 0x7f25c80ff460 2023-10-10T17:40:04.608 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:04.608 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:01.178954+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:04.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f25b00a1150 msgr2=0x7f25b00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f25b00a1150 0x7f25b00a3600 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f25b8000c00 tx=0x7f25b8005fb0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ff460 msgr2=0x7f25c8100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ff460 0x7f25c8100e70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00c920 tx=0x7f25bc00cce0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 shutdown_connections 2023-10-10T17:40:04.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f25b00a1150 0x7f25b00a3600 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f25c8104500 0x7f25c8101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f25c80ff460 0x7f25c8100e70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 --2- 172.21.15.78:0/3460489274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f25c80ffe00 0x7f25c81013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:04.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 >> 172.21.15.78:0/3460489274 conn(0x7f25c80770a0 msgr2=0x7f25c80691c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:04.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 shutdown_connections 2023-10-10T17:40:04.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:04.609+0000 7f25ccc23700 1 -- 172.21.15.78:0/3460489274 wait complete. 2023-10-10T17:40:05.051 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-10T17:40:05.052 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd pool get .mgr pg_num 2023-10-10T17:40:05.695 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:40:05.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:05 smithi078 ceph-mon[99494]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:05.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:05 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3460489274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:05.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:05 smithi190 ceph-mon[101220]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:05 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3460489274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:05.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:05 smithi157 ceph-mon[100896]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 73 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:05.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:05 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3460489274' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:06.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 -- 172.21.15.78:0/3675532758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 msgr2=0x7f0338104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3675532758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338104f50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0334009990 tx=0x7f0334009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 -- 172.21.15.78:0/3675532758 shutdown_connections 2023-10-10T17:40:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3675532758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0338105630 0x7f033810d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3675532758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f03380fead0 0x7f03380feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:06.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3675532758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338104f50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:06.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 -- 172.21.15.78:0/3675532758 >> 172.21.15.78:0/3675532758 conn(0x7f03380758a0 msgr2=0x7f0338075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:06.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 -- 172.21.15.78:0/3675532758 shutdown_connections 2023-10-10T17:40:06.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.933+0000 7f033f2a3700 1 -- 172.21.15.78:0/3675532758 wait complete. 2023-10-10T17:40:06.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 Processor -- start 2023-10-10T17:40:06.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 -- start start 2023-10-10T17:40:06.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03380fead0 0x7f03381962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:06.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338196820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:06.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0338105630 0x7f033819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:06.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033819b2a0 con 0x7f03380ff470 2023-10-10T17:40:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033819b410 con 0x7f0338105630 2023-10-10T17:40:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033f2a3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033819b5d0 con 0x7f03380fead0 2023-10-10T17:40:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338196820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338196820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41810/0 (socket says 172.21.15.78:41810) 2023-10-10T17:40:06.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 -- 172.21.15.78:0/3248947903 learned_addr learned my addr 172.21.15.78:0/3248947903 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033d840700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0338105630 0x7f033819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033d03f700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03380fead0 0x7f03381962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 -- 172.21.15.78:0/3248947903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03380fead0 msgr2=0x7f03381962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03380fead0 0x7f03381962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:06.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 -- 172.21.15.78:0/3248947903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0338105630 msgr2=0x7f033819ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:06.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0338105630 0x7f033819ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:06.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 -- 172.21.15.78:0/3248947903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03340095f0 con 0x7f03380ff470 2023-10-10T17:40:06.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.934+0000 7f033c83e700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338196820 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0334004760 tx=0x7f0334009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:06.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.935+0000 7f032e7fc700 1 -- 172.21.15.78:0/3248947903 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033401d070 con 0x7f03380ff470 2023-10-10T17:40:06.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.935+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0338110340 con 0x7f03380ff470 2023-10-10T17:40:06.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.935+0000 7f032e7fc700 1 -- 172.21.15.78:0/3248947903 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0334004cf0 con 0x7f03380ff470 2023-10-10T17:40:06.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.935+0000 7f032e7fc700 1 -- 172.21.15.78:0/3248947903 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0334003d00 con 0x7f03380ff470 2023-10-10T17:40:06.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.935+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03381101b0 con 0x7f03380ff470 2023-10-10T17:40:06.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.935+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03381104d0 con 0x7f03380ff470 2023-10-10T17:40:06.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.937+0000 7f032e7fc700 1 -- 172.21.15.78:0/3248947903 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0334005000 con 0x7f03380ff470 2023-10-10T17:40:06.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.937+0000 7f032e7fc700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03240a1150 0x7f03240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:06.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.937+0000 7f033d03f700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03240a1150 0x7f03240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:06.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.938+0000 7f032e7fc700 1 -- 172.21.15.78:0/3248947903 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f03340cd8f0 con 0x7f03380ff470 2023-10-10T17:40:06.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.938+0000 7f033d03f700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03240a1150 0x7f03240a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0328005950 tx=0x7f03280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:06.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:06.941+0000 7f032e7fc700 1 -- 172.21.15.78:0/3248947903 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0334097570 con 0x7f03380ff470 2023-10-10T17:40:07.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.127+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f03381a3360 con 0x7f03380ff470 2023-10-10T17:40:07.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.127+0000 7f032e7fc700 1 -- 172.21.15.78:0/3248947903 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0334004e60 con 0x7f03380ff470 2023-10-10T17:40:07.128 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:40:07.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03240a1150 msgr2=0x7f03240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:07.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03240a1150 0x7f03240a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0328005950 tx=0x7f03280058e0 comp rx=0 tx=0).stop 2023-10-10T17:40:07.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 msgr2=0x7f0338196820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:07.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338196820 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0334004760 tx=0x7f0334009580 comp rx=0 tx=0).stop 2023-10-10T17:40:07.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 shutdown_connections 2023-10-10T17:40:07.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03240a1150 0x7f03240a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:07.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f03380fead0 0x7f03381962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:07.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0338105630 0x7f033819ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:07.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 --2- 172.21.15.78:0/3248947903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f03380ff470 0x7f0338196820 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:07.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 >> 172.21.15.78:0/3248947903 conn(0x7f03380758a0 msgr2=0x7f03380fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:07.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 shutdown_connections 2023-10-10T17:40:07.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:07.129+0000 7f033f2a3700 1 -- 172.21.15.78:0/3248947903 wait complete. 2023-10-10T17:40:07.548 INFO:tasks.cephadm:Adding mds.a on smithi078 2023-10-10T17:40:07.549 INFO:tasks.cephadm:Adding mds.d on smithi078 2023-10-10T17:40:07.549 INFO:tasks.cephadm:Adding mds.g on smithi078 2023-10-10T17:40:07.549 INFO:tasks.cephadm:Adding mds.j on smithi078 2023-10-10T17:40:07.549 INFO:tasks.cephadm:Adding mds.b on smithi157 2023-10-10T17:40:07.550 INFO:tasks.cephadm:Adding mds.e on smithi157 2023-10-10T17:40:07.550 INFO:tasks.cephadm:Adding mds.h on smithi157 2023-10-10T17:40:07.550 INFO:tasks.cephadm:Adding mds.k on smithi157 2023-10-10T17:40:07.550 INFO:tasks.cephadm:Adding mds.c on smithi190 2023-10-10T17:40:07.551 INFO:tasks.cephadm:Adding mds.f on smithi190 2023-10-10T17:40:07.551 INFO:tasks.cephadm:Adding mds.i on smithi190 2023-10-10T17:40:07.551 INFO:tasks.cephadm:Adding mds.l on smithi190 2023-10-10T17:40:07.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph orch apply mds all '12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l' 2023-10-10T17:40:07.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:07 smithi078 ceph-mon[99494]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:07 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3248947903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:07.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:07 smithi190 ceph-mon[101220]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:07.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:07 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3248947903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:07.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:07 smithi157 ceph-mon[100896]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:07.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:07 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3248947903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:08.206 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:40:09.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 -- 172.21.15.190:0/4184773566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 msgr2=0x7f13cc103150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:09.385 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/4184773566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc103150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009990 tx=0x7f13c0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 -- 172.21.15.190:0/4184773566 shutdown_connections 2023-10-10T17:40:09.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/4184773566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc103150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/4184773566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13cc105fb0 0x7f13cc108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/4184773566 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13cc103690 0x7f13cc105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.386 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 -- 172.21.15.190:0/4184773566 >> 172.21.15.190:0/4184773566 conn(0x7f13cc0fa7f0 msgr2=0x7f13cc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:09.387 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 -- 172.21.15.190:0/4184773566 shutdown_connections 2023-10-10T17:40:09.387 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.384+0000 7f13d0c9b700 1 -- 172.21.15.190:0/4184773566 wait complete. 2023-10-10T17:40:09.389 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 Processor -- start 2023-10-10T17:40:09.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 -- start start 2023-10-10T17:40:09.390 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc1006d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:09.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13cc103690 0x7f13cc0fed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:09.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13cc105fb0 0x7f13cc0ff260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:09.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc0ff9e0 con 0x7f13cc105fb0 2023-10-10T17:40:09.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc0ffb50 con 0x7f13cc103690 2023-10-10T17:40:09.391 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13d0c9b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc1a00a0 con 0x7f13cc100d70 2023-10-10T17:40:09.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc1006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:09.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13ca59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc1006d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.190:52280/0 (socket says 172.21.15.190:52280) 2023-10-10T17:40:09.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13ca59c700 1 -- 172.21.15.190:0/679911187 learned_addr learned my addr 172.21.15.190:0/679911187 (peer_addr_for_me v2:172.21.15.190:0/0) 2023-10-10T17:40:09.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13c9d9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13cc103690 0x7f13cc0fed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:09.392 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13cad9d700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13cc105fb0 0x7f13cc0ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:09.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.385+0000 7f13ca59c700 1 -- 172.21.15.190:0/679911187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13cc103690 msgr2=0x7f13cc0fed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:09.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13ca59c700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13cc103690 0x7f13cc0fed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13ca59c700 1 -- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13cc105fb0 msgr2=0x7f13cc0ff260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:09.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13ca59c700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13cc105fb0 0x7f13cc0ff260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.393 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13ca59c700 1 -- 172.21.15.190:0/679911187 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c00095f0 con 0x7f13cc100d70 2023-10-10T17:40:09.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13cad9d700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13cc105fb0 0x7f13cc0ff260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:09.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13ca59c700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc1006d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13c0006010 tx=0x7f13c000f6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:09.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13bb7fe700 1 -- 172.21.15.190:0/679911187 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c001d070 con 0x7f13cc100d70 2023-10-10T17:40:09.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13cc10bcd0 con 0x7f13cc100d70 2023-10-10T17:40:09.394 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13bb7fe700 1 -- 172.21.15.190:0/679911187 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f13c000fbd0 con 0x7f13cc100d70 2023-10-10T17:40:09.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13bb7fe700 1 -- 172.21.15.190:0/679911187 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0017650 con 0x7f13cc100d70 2023-10-10T17:40:09.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.386+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13cc10bb40 con 0x7f13cc100d70 2023-10-10T17:40:09.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.387+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13cc0fc360 con 0x7f13cc100d70 2023-10-10T17:40:09.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.388+0000 7f13bb7fe700 1 -- 172.21.15.190:0/679911187 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f13c000fd40 con 0x7f13cc100d70 2023-10-10T17:40:09.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.389+0000 7f13bb7fe700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f13b40a1150 0x7f13b40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:09.395 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.389+0000 7f13bb7fe700 1 -- 172.21.15.190:0/679911187 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f13c00ce120 con 0x7f13cc100d70 2023-10-10T17:40:09.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.389+0000 7f13c9d9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f13b40a1150 0x7f13b40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:09.396 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.389+0000 7f13c9d9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f13b40a1150 0x7f13b40a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00ac50 tx=0x7f13bc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:09.397 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.392+0000 7f13bb7fe700 1 -- 172.21.15.190:0/679911187 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f13c0097da0 con 0x7f13cc100d70 2023-10-10T17:40:09.589 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.588+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f13cc061970 con 0x7f13b40a1150 2023-10-10T17:40:09.597 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.596+0000 7f13bb7fe700 1 -- 172.21.15.190:0/679911187 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f13cc061970 con 0x7f13b40a1150 2023-10-10T17:40:09.597 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mds.all update... 2023-10-10T17:40:09.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f13b40a1150 msgr2=0x7f13b40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:09.599 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f13b40a1150 0x7f13b40a3600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f13bc00ac50 tx=0x7f13bc00a380 comp rx=0 tx=0).stop 2023-10-10T17:40:09.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 msgr2=0x7f13cc1006d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:09.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc1006d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f13c0006010 tx=0x7f13c000f6e0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 shutdown_connections 2023-10-10T17:40:09.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f13b40a1150 0x7f13b40a3600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.600 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f13cc100d70 0x7f13cc1006d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f13cc105fb0 0x7f13cc0ff260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 --2- 172.21.15.190:0/679911187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f13cc103690 0x7f13cc0fed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:09.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 >> 172.21.15.190:0/679911187 conn(0x7f13cc0fa7f0 msgr2=0x7f13cc1019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:09.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 shutdown_connections 2023-10-10T17:40:09.601 INFO:teuthology.orchestra.run.smithi190.stderr:2023-10-10T17:40:09.598+0000 7f13d0c9b700 1 -- 172.21.15.190:0/679911187 wait complete. 2023-10-10T17:40:09.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:09 smithi078 ceph-mon[99494]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:09.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:09 smithi190 ceph-mon[101220]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:09.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:09 smithi157 ceph-mon[100896]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:10.061 DEBUG:teuthology.orchestra.run.smithi078:mds.a> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.a.service 2023-10-10T17:40:10.064 DEBUG:teuthology.orchestra.run.smithi078:mds.d> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.d.service 2023-10-10T17:40:10.102 INFO:journalctl@ceph.mds.a.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:40:10.103 DEBUG:teuthology.orchestra.run.smithi078:mds.g> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.g.service 2023-10-10T17:40:10.136 INFO:journalctl@ceph.mds.d.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:40:10.137 DEBUG:teuthology.orchestra.run.smithi078:mds.j> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.j.service 2023-10-10T17:40:10.165 INFO:journalctl@ceph.mds.g.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:40:10.166 DEBUG:teuthology.orchestra.run.smithi157:mds.b> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.b.service 2023-10-10T17:40:10.169 DEBUG:teuthology.orchestra.run.smithi157:mds.e> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.e.service 2023-10-10T17:40:10.194 INFO:journalctl@ceph.mds.j.smithi078.stdout:-- Logs begin at Tue 2023-10-10 17:18:46 UTC. -- 2023-10-10T17:40:10.207 INFO:journalctl@ceph.mds.b.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:40:10.208 DEBUG:teuthology.orchestra.run.smithi157:mds.h> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.h.service 2023-10-10T17:40:10.237 INFO:journalctl@ceph.mds.e.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:40:10.238 DEBUG:teuthology.orchestra.run.smithi157:mds.k> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.k.service 2023-10-10T17:40:10.265 INFO:journalctl@ceph.mds.h.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:40:10.266 DEBUG:teuthology.orchestra.run.smithi190:mds.c> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.c.service 2023-10-10T17:40:10.269 DEBUG:teuthology.orchestra.run.smithi190:mds.f> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.f.service 2023-10-10T17:40:10.293 INFO:journalctl@ceph.mds.k.smithi157.stdout:-- Logs begin at Tue 2023-10-10 17:19:31 UTC. -- 2023-10-10T17:40:10.297 INFO:journalctl@ceph.mds.c.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:40:10.298 DEBUG:teuthology.orchestra.run.smithi190:mds.i> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.i.service 2023-10-10T17:40:10.325 INFO:journalctl@ceph.mds.f.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:40:10.326 DEBUG:teuthology.orchestra.run.smithi190:mds.l> sudo journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.l.service 2023-10-10T17:40:10.351 INFO:journalctl@ceph.mds.i.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:40:10.352 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-10-10T17:40:10.353 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:10.378 INFO:journalctl@ceph.mds.l.smithi190.stdout:-- Logs begin at Tue 2023-10-10 17:18:51 UTC. -- 2023-10-10T17:40:10.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 -- 172.21.15.78:0/1179450617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9650100d10 msgr2=0x7f96501075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 --2- 172.21.15.78:0/1179450617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9650100d10 0x7f96501075d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f964800b0a0 tx=0x7f964800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 -- 172.21.15.78:0/1179450617 shutdown_connections 2023-10-10T17:40:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 --2- 172.21.15.78:0/1179450617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96500ff190 0x7f96500ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 --2- 172.21.15.78:0/1179450617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9650100d10 0x7f96501075d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 --2- 172.21.15.78:0/1179450617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9650100380 0x7f96501007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 -- 172.21.15.78:0/1179450617 >> 172.21.15.78:0/1179450617 conn(0x7f96500fa770 msgr2=0x7f96500fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 -- 172.21.15.78:0/1179450617 shutdown_connections 2023-10-10T17:40:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.512+0000 7f9657355700 1 -- 172.21.15.78:0/1179450617 wait complete. 2023-10-10T17:40:10.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 Processor -- start 2023-10-10T17:40:10.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 -- start start 2023-10-10T17:40:10.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96500ff190 0x7f9650194f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9650100380 0x7f9650195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9650100d10 0x7f965019c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965019cab0 con 0x7f96500ff190 2023-10-10T17:40:10.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965019cbf0 con 0x7f9650100d10 2023-10-10T17:40:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f9657355700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965019cd80 con 0x7f9650100380 2023-10-10T17:40:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f96550f1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96500ff190 0x7f9650194f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f96550f1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96500ff190 0x7f9650194f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41822/0 (socket says 172.21.15.78:41822) 2023-10-10T17:40:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.513+0000 7f96550f1700 1 -- 172.21.15.78:0/1917576686 learned_addr learned my addr 172.21.15.78:0/1917576686 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:10.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96548f0700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9650100380 0x7f9650195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96558f2700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9650100d10 0x7f965019c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96550f1700 1 -- 172.21.15.78:0/1917576686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9650100380 msgr2=0x7f9650195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96550f1700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9650100380 0x7f9650195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96550f1700 1 -- 172.21.15.78:0/1917576686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9650100d10 msgr2=0x7f965019c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96550f1700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9650100d10 0x7f965019c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96550f1700 1 -- 172.21.15.78:0/1917576686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648009d00 con 0x7f96500ff190 2023-10-10T17:40:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96550f1700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96500ff190 0x7f9650194f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f964000e970 tx=0x7f964000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96467fc700 1 -- 172.21.15.78:0/1917576686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640009790 con 0x7f96500ff190 2023-10-10T17:40:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f965010ab10 con 0x7f96500ff190 2023-10-10T17:40:10.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96467fc700 1 -- 172.21.15.78:0/1917576686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f964000c7e0 con 0x7f96500ff190 2023-10-10T17:40:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f96467fc700 1 -- 172.21.15.78:0/1917576686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640003c70 con 0x7f96500ff190 2023-10-10T17:40:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.514+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965010a980 con 0x7f96500ff190 2023-10-10T17:40:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.515+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96501066f0 con 0x7f96500ff190 2023-10-10T17:40:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.517+0000 7f96467fc700 1 -- 172.21.15.78:0/1917576686 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f964000cc50 con 0x7f96500ff190 2023-10-10T17:40:10.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.517+0000 7f96467fc700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f963c0a1150 0x7f963c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.517+0000 7f96548f0700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f963c0a1150 0x7f963c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.517+0000 7f96467fc700 1 -- 172.21.15.78:0/1917576686 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9640014070 con 0x7f96500ff190 2023-10-10T17:40:10.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.518+0000 7f96548f0700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f963c0a1150 0x7f963c0a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f964c009c60 tx=0x7f964c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:10.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.521+0000 7f96467fc700 1 -- 172.21.15.78:0/1917576686 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9640096350 con 0x7f96500ff190 2023-10-10T17:40:10.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.715+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f965002cfe0 con 0x7f96500ff190 2023-10-10T17:40:10.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.716+0000 7f96467fc700 1 -- 172.21.15.78:0/1917576686 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f9640003dd0 con 0x7f96500ff190 2023-10-10T17:40:10.718 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:10.718 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:01.178954+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:10.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f963c0a1150 msgr2=0x7f963c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f963c0a1150 0x7f963c0a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f964c009c60 tx=0x7f964c009260 comp rx=0 tx=0).stop 2023-10-10T17:40:10.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96500ff190 msgr2=0x7f9650194f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96500ff190 0x7f9650194f30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f964000e970 tx=0x7f964000ec80 comp rx=0 tx=0).stop 2023-10-10T17:40:10.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 shutdown_connections 2023-10-10T17:40:10.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f963c0a1150 0x7f963c0a3600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9650100d10 0x7f965019c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96500ff190 0x7f9650194f30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 --2- 172.21.15.78:0/1917576686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9650100380 0x7f9650195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 >> 172.21.15.78:0/1917576686 conn(0x7f96500fa770 msgr2=0x7f9650103830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:10.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 shutdown_connections 2023-10-10T17:40:10.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.718+0000 7f9657355700 1 -- 172.21.15.78:0/1917576686 wait complete. 2023-10-10T17:40:10.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:40:10.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2023-10-10T17:40:10.731 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:10.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:10.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:10.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:10.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:10.732 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:10.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:40:10.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:10.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:10.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:10.733 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:10.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:10.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: Deploying daemon mds.l on smithi190 2023-10-10T17:40:10.734 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:10 smithi078 ceph-mon[99494]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:10.735 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-10T17:40:10.736 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:40:10.846 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:40:10.847 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2023-10-10T17:40:10.847 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:10.847 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:10.847 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:10.848 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:10.848 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:10.848 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:10.848 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:40:10.848 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:10.849 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:10.849 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:10.849 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:10.849 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:10.849 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: Deploying daemon mds.l on smithi190 2023-10-10T17:40:10.850 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:10 smithi190 ceph-mon[101220]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:10.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/316801863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 msgr2=0x7ff59c107710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/316801863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c107710 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff59400b0a0 tx=0x7ff59400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/316801863 shutdown_connections 2023-10-10T17:40:10.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/316801863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c107710 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/316801863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff59c100810 0x7ff59c100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/316801863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff59c0ff4b0 0x7ff59c0ff920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/316801863 >> 172.21.15.78:0/316801863 conn(0x7ff59c0f9fb0 msgr2=0x7ff59c0fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/316801863 shutdown_connections 2023-10-10T17:40:10.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.860+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/316801863 wait complete. 2023-10-10T17:40:10.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 Processor -- start 2023-10-10T17:40:10.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 -- start start 2023-10-10T17:40:10.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff59c0ff4b0 0x7ff59c195020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff59c100810 0x7ff59c19c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c19cba0 con 0x7ff59c0ffe60 2023-10-10T17:40:10.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c19cce0 con 0x7ff59c0ff4b0 2023-10-10T17:40:10.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a3f2f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff59c19ce70 con 0x7ff59c100810 2023-10-10T17:40:10.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c195560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41838/0 (socket says 172.21.15.78:41838) 2023-10-10T17:40:10.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 -- 172.21.15.78:0/2165729644 learned_addr learned my addr 172.21.15.78:0/2165729644 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:10.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a24cc700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff59c100810 0x7ff59c19c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a1ccb700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff59c0ff4b0 0x7ff59c195020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 -- 172.21.15.78:0/2165729644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff59c100810 msgr2=0x7ff59c19c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff59c100810 0x7ff59c19c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 -- 172.21.15.78:0/2165729644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff59c0ff4b0 msgr2=0x7ff59c195020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:10.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff59c0ff4b0 0x7ff59c195020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:10.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 -- 172.21.15.78:0/2165729644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594009d00 con 0x7ff59c0ffe60 2023-10-10T17:40:10.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a1ccb700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff59c0ff4b0 0x7ff59c195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:10.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff5a14ca700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c195560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff59800d980 tx=0x7ff59800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:10.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff592ffd700 1 -- 172.21.15.78:0/2165729644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598004020 con 0x7ff59c0ffe60 2023-10-10T17:40:10.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.861+0000 7ff592ffd700 1 -- 172.21.15.78:0/2165729644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff598009730 con 0x7ff59c0ffe60 2023-10-10T17:40:10.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.862+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c10ab90 con 0x7ff59c0ffe60 2023-10-10T17:40:10.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.862+0000 7ff592ffd700 1 -- 172.21.15.78:0/2165729644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598003a80 con 0x7ff59c0ffe60 2023-10-10T17:40:10.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.862+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c10aa00 con 0x7ff59c0ffe60 2023-10-10T17:40:10.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.863+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c106160 con 0x7ff59c0ffe60 2023-10-10T17:40:10.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.864+0000 7ff592ffd700 1 -- 172.21.15.78:0/2165729644 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff5980098a0 con 0x7ff59c0ffe60 2023-10-10T17:40:10.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.864+0000 7ff592ffd700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5880a1150 0x7ff5880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:10.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.864+0000 7ff5a1ccb700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5880a1150 0x7ff5880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:10.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.864+0000 7ff592ffd700 1 -- 172.21.15.78:0/2165729644 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff5980cc4b0 con 0x7ff59c0ffe60 2023-10-10T17:40:10.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.865+0000 7ff5a1ccb700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5880a1150 0x7ff5880a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff58c005fd0 tx=0x7ff58c005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:10.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:10.868+0000 7ff592ffd700 1 -- 172.21.15.78:0/2165729644 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff5980960a0 con 0x7ff59c0ffe60 2023-10-10T17:40:10.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:40:10.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2023-10-10T17:40:10.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:10.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:10.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:10.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:10.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:10.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:10.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:40:10.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:10.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:10.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:10.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:10.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:10.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: Deploying daemon mds.l on smithi190 2023-10-10T17:40:10.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:10 smithi157 ceph-mon[100896]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:11.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.054+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff59c061700 con 0x7ff59c0ffe60 2023-10-10T17:40:11.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.054+0000 7ff592ffd700 1 -- 172.21.15.78:0/2165729644 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff5980996c0 con 0x7ff59c0ffe60 2023-10-10T17:40:11.056 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:40:11.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5880a1150 msgr2=0x7ff5880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:11.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5880a1150 0x7ff5880a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff58c005fd0 tx=0x7ff58c005dc0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 msgr2=0x7ff59c195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:11.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c195560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff59800d980 tx=0x7ff59800dc90 comp rx=0 tx=0).stop 2023-10-10T17:40:11.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 shutdown_connections 2023-10-10T17:40:11.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5880a1150 0x7ff5880a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff59c100810 0x7ff59c19c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff59c0ff4b0 0x7ff59c195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 --2- 172.21.15.78:0/2165729644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff59c0ffe60 0x7ff59c195560 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 >> 172.21.15.78:0/2165729644 conn(0x7ff59c0f9fb0 msgr2=0x7ff59c0fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:11.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 shutdown_connections 2023-10-10T17:40:11.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.056+0000 7ff5a3f2f700 1 -- 172.21.15.78:0/2165729644 wait complete. 2023-10-10T17:40:11.070 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 -- 172.21.15.78:0/2793664540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb01015e0 msgr2=0x7f3bb0101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2793664540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb01015e0 0x7f3bb0101a30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009990 tx=0x7f3ba4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 -- 172.21.15.78:0/2793664540 shutdown_connections 2023-10-10T17:40:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2793664540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb00fe610 0x7f3bb00fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2793664540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb0101f70 0x7f3bb0108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2793664540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb01015e0 0x7f3bb0101a30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 -- 172.21.15.78:0/2793664540 >> 172.21.15.78:0/2793664540 conn(0x7f3bb00f9fb0 msgr2=0x7f3bb00fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 -- 172.21.15.78:0/2793664540 shutdown_connections 2023-10-10T17:40:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.190+0000 7f3bb6449700 1 -- 172.21.15.78:0/2793664540 wait complete. 2023-10-10T17:40:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.191+0000 7f3bb6449700 1 Processor -- start 2023-10-10T17:40:11.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.191+0000 7f3bb6449700 1 -- start start 2023-10-10T17:40:11.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.191+0000 7f3bb6449700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb00fe610 0x7f3bb0195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:11.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.191+0000 7f3bb6449700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb01015e0 0x7f3bb0196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:11.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb6449700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb0101f70 0x7f3bb019d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:11.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb6449700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb019dad0 con 0x7f3bb0101f70 2023-10-10T17:40:11.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb6449700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb019dc10 con 0x7f3bb01015e0 2023-10-10T17:40:11.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb6449700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb019dda0 con 0x7f3bb00fe610 2023-10-10T17:40:11.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb0101f70 0x7f3bb019d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb0101f70 0x7f3bb019d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41854/0 (socket says 172.21.15.78:41854) 2023-10-10T17:40:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 -- 172.21.15.78:0/2782035151 learned_addr learned my addr 172.21.15.78:0/2782035151 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3baffff700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb00fe610 0x7f3bb0195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3baf7fe700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb01015e0 0x7f3bb0196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 -- 172.21.15.78:0/2782035151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb00fe610 msgr2=0x7f3bb0195f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:11.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb00fe610 0x7f3bb0195f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 -- 172.21.15.78:0/2782035151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb01015e0 msgr2=0x7f3bb0196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb01015e0 0x7f3bb0196490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 -- 172.21.15.78:0/2782035151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40095f0 con 0x7f3bb0101f70 2023-10-10T17:40:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3baf7fe700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb01015e0 0x7f3bb0196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:11.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3baffff700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb00fe610 0x7f3bb0195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:40:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb49e6700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb0101f70 0x7f3bb019d500 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800ada0 tx=0x7f3ba800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bad7fa700 1 -- 172.21.15.78:0/2782035151 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba800f070 con 0x7f3bb0101f70 2023-10-10T17:40:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb010bc50 con 0x7f3bb0101f70 2023-10-10T17:40:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bad7fa700 1 -- 172.21.15.78:0/2782035151 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ba8007b30 con 0x7f3bb0101f70 2023-10-10T17:40:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bad7fa700 1 -- 172.21.15.78:0/2782035151 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8013ba0 con 0x7f3bb0101f70 2023-10-10T17:40:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.192+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb010bac0 con 0x7f3bb0101f70 2023-10-10T17:40:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.193+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb0107340 con 0x7f3bb0101f70 2023-10-10T17:40:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.194+0000 7f3bad7fa700 1 -- 172.21.15.78:0/2782035151 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3ba800c980 con 0x7f3bb0101f70 2023-10-10T17:40:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.195+0000 7f3bad7fa700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3b980a1150 0x7f3b980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.195+0000 7f3bad7fa700 1 -- 172.21.15.78:0/2782035151 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3ba80cc580 con 0x7f3bb0101f70 2023-10-10T17:40:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.195+0000 7f3baffff700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3b980a1150 0x7f3b980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.195+0000 7f3baffff700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3b980a1150 0x7f3b980a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0068e90 tx=0x7f3ba0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.198+0000 7f3bad7fa700 1 -- 172.21.15.78:0/2782035151 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3ba8096200 con 0x7f3bb0101f70 2023-10-10T17:40:11.415 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:40:11 smithi190 systemd[1]: Starting Ceph mds.l for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:11.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.429+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb002cfe0 con 0x7f3bb0101f70 2023-10-10T17:40:11.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.429+0000 7f3bad7fa700 1 -- 172.21.15.78:0/2782035151 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f3ba8099820 con 0x7f3bb0101f70 2023-10-10T17:40:11.431 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:11.432 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-10-10T17:40:11.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.431+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3b980a1150 msgr2=0x7f3b980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:11.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.431+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3b980a1150 0x7f3b980a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0068e90 tx=0x7f3ba0009260 comp rx=0 tx=0).stop 2023-10-10T17:40:11.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.431+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb0101f70 msgr2=0x7f3bb019d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:11.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.431+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb0101f70 0x7f3bb019d500 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3ba800ada0 tx=0x7f3ba800c330 comp rx=0 tx=0).stop 2023-10-10T17:40:11.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 shutdown_connections 2023-10-10T17:40:11.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3b980a1150 0x7f3b980a3600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3bb00fe610 0x7f3bb0195f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bb0101f70 0x7f3bb019d500 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 --2- 172.21.15.78:0/2782035151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3bb01015e0 0x7f3bb0196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:11.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 >> 172.21.15.78:0/2782035151 conn(0x7f3bb00f9fb0 msgr2=0x7f3bb00fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:11.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 shutdown_connections 2023-10-10T17:40:11.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:11.432+0000 7f3bb6449700 1 -- 172.21.15.78:0/2782035151 wait complete. 2023-10-10T17:40:11.436 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 1 2023-10-10T17:40:11.706 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:40:11 smithi190 podman[126420]: 2023-10-10T17:40:11.706 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:40:11 smithi190 ceph-mds[126460]: -- 172.21.15.190:0/969920446 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56461e977520 con 0x56461e94d000 2023-10-10T17:40:11.707 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:40:11 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-l[126436]: starting mds.l at 2023-10-10T17:40:11.707 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:11 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1917576686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:11.707 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:11 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2165729644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:11.707 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:11 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2782035151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:11 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1917576686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:11 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2165729644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:11 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2782035151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:11.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:11 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1917576686' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:11 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2165729644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:11 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2782035151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:12.164 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:40:11 smithi190 bash[126420]: 65090a020cc495b5375645f0514280bda521407e1d689f88c82e1eca60c04d87 2023-10-10T17:40:12.165 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:40:11 smithi190 systemd[1]: Started Ceph mds.l for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:12.781 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6832/1708598976,v1:172.21.15.190:6833/1708598976] up:boot 2023-10-10T17:40:12.781 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: fsmap 1 up:standby 2023-10-10T17:40:12.782 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-10T17:40:12.782 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:12.782 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:12.782 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:12.782 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:12.783 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:12.783 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:12.783 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: Deploying daemon mds.k on smithi157 2023-10-10T17:40:12.783 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:12 smithi157 ceph-mon[100896]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6832/1708598976,v1:172.21.15.190:6833/1708598976] up:boot 2023-10-10T17:40:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: fsmap 1 up:standby 2023-10-10T17:40:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-10T17:40:13.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:13.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:13.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:13.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:13.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:13.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:13.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: Deploying daemon mds.k on smithi157 2023-10-10T17:40:13.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:12 smithi078 ceph-mon[99494]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:13.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6832/1708598976,v1:172.21.15.190:6833/1708598976] up:boot 2023-10-10T17:40:13.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: fsmap 1 up:standby 2023-10-10T17:40:13.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-10T17:40:13.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:13.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:13.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:13.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:13.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:13.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:13.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: Deploying daemon mds.k on smithi157 2023-10-10T17:40:13.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:12 smithi190 ceph-mon[101220]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:13.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:13.478 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:40:13 smithi157 systemd[1]: Starting Ceph mds.k for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:13.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 -- 172.21.15.78:0/2397964630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84100290 msgr2=0x7f0f841006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:13.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/2397964630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84100290 0x7f0f841006a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009990 tx=0x7f0f78009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 -- 172.21.15.78:0/2397964630 shutdown_connections 2023-10-10T17:40:13.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/2397964630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f84101e10 0x7f0f841086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/2397964630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84100290 0x7f0f841006a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/2397964630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f84101480 0x7f0f841018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 -- 172.21.15.78:0/2397964630 >> 172.21.15.78:0/2397964630 conn(0x7f0f840fb870 msgr2=0x7f0f840fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:13.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 -- 172.21.15.78:0/2397964630 shutdown_connections 2023-10-10T17:40:13.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 -- 172.21.15.78:0/2397964630 wait complete. 2023-10-10T17:40:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 Processor -- start 2023-10-10T17:40:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.586+0000 7f0f8aab1700 1 -- start start 2023-10-10T17:40:13.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8aab1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f84100290 0x7f0f84195f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8aab1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84101480 0x7f0f84196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8aab1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f84101e10 0x7f0f8419d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8aab1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8419da90 con 0x7f0f84101480 2023-10-10T17:40:13.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8aab1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8419dbd0 con 0x7f0f84100290 2023-10-10T17:40:13.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8aab1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8419dd60 con 0x7f0f84101e10 2023-10-10T17:40:13.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84101480 0x7f0f84196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:13.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84101480 0x7f0f84196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41866/0 (socket says 172.21.15.78:41866) 2023-10-10T17:40:13.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 -- 172.21.15.78:0/3380582475 learned_addr learned my addr 172.21.15.78:0/3380582475 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:13.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8904e700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f84101e10 0x7f0f8419d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:13.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8884d700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f84100290 0x7f0f84195f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:13.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 -- 172.21.15.78:0/3380582475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f84101e10 msgr2=0x7f0f8419d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:13.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f84101e10 0x7f0f8419d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 -- 172.21.15.78:0/3380582475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f84100290 msgr2=0x7f0f84195f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:13.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f84100290 0x7f0f84195f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 -- 172.21.15.78:0/3380582475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780095f0 con 0x7f0f84101480 2023-10-10T17:40:13.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f83fff700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84101480 0x7f0f84196450 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c9b0 tx=0x7f0f7000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:13.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f81ffb700 1 -- 172.21.15.78:0/3380582475 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70007b00 con 0x7f0f84101480 2023-10-10T17:40:13.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f81ffb700 1 -- 172.21.15.78:0/3380582475 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f7000ce50 con 0x7f0f84101480 2023-10-10T17:40:13.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8410bc40 con 0x7f0f84101480 2023-10-10T17:40:13.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.587+0000 7f0f81ffb700 1 -- 172.21.15.78:0/3380582475 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70018660 con 0x7f0f84101480 2023-10-10T17:40:13.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.588+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f8410bab0 con 0x7f0f84101480 2023-10-10T17:40:13.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.589+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8419e3f0 con 0x7f0f84101480 2023-10-10T17:40:13.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.590+0000 7f0f81ffb700 1 -- 172.21.15.78:0/3380582475 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f7000f450 con 0x7f0f84101480 2023-10-10T17:40:13.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.590+0000 7f0f81ffb700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0f740a1150 0x7f0f740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:13.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.590+0000 7f0f8884d700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0f740a1150 0x7f0f740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:13.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.590+0000 7f0f81ffb700 1 -- 172.21.15.78:0/3380582475 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0f700cc370 con 0x7f0f84101480 2023-10-10T17:40:13.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.591+0000 7f0f8884d700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0f740a1150 0x7f0f740a3600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800b2e0 tx=0x7f0f78005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:13.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.594+0000 7f0f81ffb700 1 -- 172.21.15.78:0/3380582475 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0f70095f60 con 0x7f0f84101480 2023-10-10T17:40:13.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.826+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f84107160 con 0x7f0f84101480 2023-10-10T17:40:13.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.826+0000 7f0f81ffb700 1 -- 172.21.15.78:0/3380582475 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f0f70007e60 con 0x7f0f84101480 2023-10-10T17:40:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:13.829 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-10-10T17:40:13.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0f740a1150 msgr2=0x7f0f740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:13.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0f740a1150 0x7f0f740a3600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800b2e0 tx=0x7f0f78005fb0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84101480 msgr2=0x7f0f84196450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:13.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84101480 0x7f0f84196450 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0f7000c9b0 tx=0x7f0f7000ccc0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 shutdown_connections 2023-10-10T17:40:13.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0f740a1150 0x7f0f740a3600 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f84101e10 0x7f0f8419d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f84100290 0x7f0f84195f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 --2- 172.21.15.78:0/3380582475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f84101480 0x7f0f84196450 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:13.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 >> 172.21.15.78:0/3380582475 conn(0x7f0f840fb870 msgr2=0x7f0f840fdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:13.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 shutdown_connections 2023-10-10T17:40:13.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:13.828+0000 7f0f8aab1700 1 -- 172.21.15.78:0/3380582475 wait complete. 2023-10-10T17:40:13.833 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2023-10-10T17:40:13.852 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:40:13 smithi157 podman[125171]: 2023-10-10T17:40:14.228 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:40:13 smithi157 ceph-mds[125213]: -- 172.21.15.157:0/726144585 <== mon.2 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cd12f63520 con 0x55cd12f38c00 2023-10-10T17:40:14.228 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:40:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-k[125187]: starting mds.k at 2023-10-10T17:40:14.228 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:40:13 smithi157 bash[125171]: 644474bea76460b74bf3e416f4b5111aa3ad4da16e32e3f5182637e900943139 2023-10-10T17:40:14.228 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:40:13 smithi157 systemd[1]: Started Ceph mds.k for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:14.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:14 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3380582475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:14.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.874 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:14.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:14.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:14 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:14.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:14 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3380582475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:14.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:14.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:14 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:14 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3380582475' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:14.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:14.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:14.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:14 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:15.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:15.866 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:15 smithi078 ceph-mon[99494]: Deploying daemon mds.j on smithi078 2023-10-10T17:40:15.866 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:15 smithi078 ceph-mon[99494]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:15.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:15 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6832/1712500424,v1:172.21.15.157:6833/1712500424] up:boot 2023-10-10T17:40:15.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:15 smithi078 ceph-mon[99494]: fsmap 2 up:standby 2023-10-10T17:40:15.867 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:15 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:40:15.867 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:40:15 smithi078 systemd[1]: Starting Ceph mds.j for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:15.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:15 smithi190 ceph-mon[101220]: Deploying daemon mds.j on smithi078 2023-10-10T17:40:15.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:15 smithi190 ceph-mon[101220]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:15.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:15 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6832/1712500424,v1:172.21.15.157:6833/1712500424] up:boot 2023-10-10T17:40:15.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:15 smithi190 ceph-mon[101220]: fsmap 2 up:standby 2023-10-10T17:40:15.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:15 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:40:15.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:15 smithi157 ceph-mon[100896]: Deploying daemon mds.j on smithi078 2023-10-10T17:40:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:15 smithi157 ceph-mon[100896]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:15 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6832/1712500424,v1:172.21.15.157:6833/1712500424] up:boot 2023-10-10T17:40:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:15 smithi157 ceph-mon[100896]: fsmap 2 up:standby 2023-10-10T17:40:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:15 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:40:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.978+0000 7fd7e24a0700 1 -- 172.21.15.78:0/3160012143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 msgr2=0x7fd7dc100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:15.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.978+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/3160012143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc100870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0009990 tx=0x7fd7d0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 -- 172.21.15.78:0/3160012143 shutdown_connections 2023-10-10T17:40:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/3160012143 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc0ff170 0x7fd7dc0ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/3160012143 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc100db0 0x7fd7dc107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/3160012143 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc100870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:15.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 -- 172.21.15.78:0/3160012143 >> 172.21.15.78:0/3160012143 conn(0x7fd7dc0fa750 msgr2=0x7fd7dc0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:15.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 -- 172.21.15.78:0/3160012143 shutdown_connections 2023-10-10T17:40:15.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 -- 172.21.15.78:0/3160012143 wait complete. 2023-10-10T17:40:15.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 Processor -- start 2023-10-10T17:40:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 -- start start 2023-10-10T17:40:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc0ff170 0x7fd7dc068940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.979+0000 7fd7e24a0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc19d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:15.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7e24a0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc100db0 0x7fd7dc19f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7e24a0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc19fcb0 con 0x7fd7dc100400 2023-10-10T17:40:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7e24a0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc19fe20 con 0x7fd7dc100db0 2023-10-10T17:40:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7e24a0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc19ffe0 con 0x7fd7dc0ff170 2023-10-10T17:40:15.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc19d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc19d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41880/0 (socket says 172.21.15.78:41880) 2023-10-10T17:40:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 -- 172.21.15.78:0/1706995258 learned_addr learned my addr 172.21.15.78:0/1706995258 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7dbfff700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc0ff170 0x7fd7dc068940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 -- 172.21.15.78:0/1706995258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc0ff170 msgr2=0x7fd7dc068940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:15.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc0ff170 0x7fd7dc068940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 -- 172.21.15.78:0/1706995258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc100db0 msgr2=0x7fd7dc19f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc100db0 0x7fd7dc19f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 -- 172.21.15.78:0/1706995258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d00095f0 con 0x7fd7dc100400 2023-10-10T17:40:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7dbfff700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc0ff170 0x7fd7dc068940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:15.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7db7fe700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc19d0b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0000c00 tx=0x7fd7d000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7d97fa700 1 -- 172.21.15.78:0/1706995258 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d001d070 con 0x7fd7dc100400 2023-10-10T17:40:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7d97fa700 1 -- 172.21.15.78:0/1706995258 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7d000fc00 con 0x7fd7dc100400 2023-10-10T17:40:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc10ab70 con 0x7fd7dc100400 2023-10-10T17:40:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7d97fa700 1 -- 172.21.15.78:0/1706995258 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0017650 con 0x7fd7dc100400 2023-10-10T17:40:15.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.980+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc10a9e0 con 0x7fd7dc100400 2023-10-10T17:40:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.981+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7dc106160 con 0x7fd7dc100400 2023-10-10T17:40:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.982+0000 7fd7d97fa700 1 -- 172.21.15.78:0/1706995258 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd7d0022470 con 0x7fd7dc100400 2023-10-10T17:40:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.983+0000 7fd7d97fa700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7c40a1150 0x7fd7c40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.983+0000 7fd7dbfff700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7c40a1150 0x7fd7c40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:15.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.983+0000 7fd7d97fa700 1 -- 172.21.15.78:0/1706995258 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd7d00cd9e0 con 0x7fd7dc100400 2023-10-10T17:40:15.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.983+0000 7fd7dbfff700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7c40a1150 0x7fd7c40a3600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc005fd0 tx=0x7fd7cc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:15.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:15.986+0000 7fd7d97fa700 1 -- 172.21.15.78:0/1706995258 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd7d0097660 con 0x7fd7dc100400 2023-10-10T17:40:16.127 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:40:15 smithi078 podman[132704]: 2023-10-10T17:40:16.127 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:40:16 smithi078 ceph-mds[132787]: -- 172.21.15.78:0/2430416562 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a787543520 con 0x55a787519000 2023-10-10T17:40:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.246+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7dc061700 con 0x7fd7dc100400 2023-10-10T17:40:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.246+0000 7fd7d97fa700 1 -- 172.21.15.78:0/1706995258 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7fd7d009ac80 con 0x7fd7dc100400 2023-10-10T17:40:16.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:16.249 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-10T17:40:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7c40a1150 msgr2=0x7fd7c40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7c40a1150 0x7fd7c40a3600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc005fd0 tx=0x7fd7cc005ee0 comp rx=0 tx=0).stop 2023-10-10T17:40:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 msgr2=0x7fd7dc19d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc19d0b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0000c00 tx=0x7fd7d000f710 comp rx=0 tx=0).stop 2023-10-10T17:40:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 shutdown_connections 2023-10-10T17:40:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd7c40a1150 0x7fd7c40a3600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc0ff170 0x7fd7dc068940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:16.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc100db0 0x7fd7dc19f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 --2- 172.21.15.78:0/1706995258 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc100400 0x7fd7dc19d0b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 >> 172.21.15.78:0/1706995258 conn(0x7fd7dc0fa750 msgr2=0x7fd7dc0fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:16.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 shutdown_connections 2023-10-10T17:40:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:16.249+0000 7fd7e24a0700 1 -- 172.21.15.78:0/1706995258 wait complete. 2023-10-10T17:40:16.256 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2023-10-10T17:40:16.378 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:40:16 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-j[132743]: starting mds.j at 2023-10-10T17:40:16.378 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:40:16 smithi078 bash[132704]: d26cd2cd8f32e1a6af921f3e0c2a1fb40456a3fa9bae493edda18e84bf745ab8 2023-10-10T17:40:16.378 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:40:16 smithi078 systemd[1]: Started Ceph mds.j for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:17.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.274 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.275 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:17.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:17.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:17.276 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: Deploying daemon mds.d on smithi078 2023-10-10T17:40:17.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:17.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1706995258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:17.277 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.78:6834/1904698406,v1:172.21.15.78:6835/1904698406] up:boot 2023-10-10T17:40:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: fsmap 3 up:standby 2023-10-10T17:40:17.278 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:17 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-10T17:40:17.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:17.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:17.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:17.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: Deploying daemon mds.d on smithi078 2023-10-10T17:40:17.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:17.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1706995258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:17.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.78:6834/1904698406,v1:172.21.15.78:6835/1904698406] up:boot 2023-10-10T17:40:17.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: fsmap 3 up:standby 2023-10-10T17:40:17.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:17 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-10T17:40:17.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:17.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:17.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:17.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:17.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: Deploying daemon mds.d on smithi078 2023-10-10T17:40:17.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:17.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1706995258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:17.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.78:6834/1904698406,v1:172.21.15.78:6835/1904698406] up:boot 2023-10-10T17:40:17.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: fsmap 3 up:standby 2023-10-10T17:40:17.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:17 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-10T17:40:18.098 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:40:17 smithi078 systemd[1]: Starting Ceph mds.d for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:18.265 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:18.349 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:40:18 smithi078 podman[133086]: 2023-10-10T17:40:18.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:40:18 smithi078 bash[133086]: 9e0dda29a3b67c48300f5d76755be6791eb46f2bd055e4fc158b0d73e602b747 2023-10-10T17:40:18.350 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:40:18 smithi078 ceph-mds[133149]: -- 172.21.15.78:0/261522772 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562fa7b83520 con 0x562fa7b59000 2023-10-10T17:40:18.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3984577036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 msgr2=0x7fc0580ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:18.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3984577036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580ff850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc044009990 tx=0x7fc044009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3984577036 shutdown_connections 2023-10-10T17:40:18.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3984577036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ffd90 0x7fc058107640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3984577036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc058100790 0x7fc058100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3984577036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580ff850 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3984577036 >> 172.21.15.78:0/3984577036 conn(0x7fc0580f9f20 msgr2=0x7fc0580fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:18.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3984577036 shutdown_connections 2023-10-10T17:40:18.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3984577036 wait complete. 2023-10-10T17:40:18.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 Processor -- start 2023-10-10T17:40:18.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- start start 2023-10-10T17:40:18.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580fee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:18.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ffd90 0x7fc05819ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:18.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc058100790 0x7fc05819d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:18.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05819dc60 con 0x7fc0580ff3e0 2023-10-10T17:40:18.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05819ddd0 con 0x7fc058100790 2023-10-10T17:40:18.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.398+0000 7fc05f0fe700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc05819df90 con 0x7fc0580ffd90 2023-10-10T17:40:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580fee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41914/0 (socket says 172.21.15.78:41914) 2023-10-10T17:40:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 -- 172.21.15.78:0/3310683830 learned_addr learned my addr 172.21.15.78:0/3310683830 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05d69b700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc058100790 0x7fc05819d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:18.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc057fff700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ffd90 0x7fc05819ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 -- 172.21.15.78:0/3310683830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ffd90 msgr2=0x7fc05819ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ffd90 0x7fc05819ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 -- 172.21.15.78:0/3310683830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc058100790 msgr2=0x7fc05819d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:18.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc058100790 0x7fc05819d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 -- 172.21.15.78:0/3310683830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0440095f0 con 0x7fc0580ff3e0 2023-10-10T17:40:18.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05ce9a700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580fee60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00b500 tx=0x7fc04c00b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:18.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc055ffb700 1 -- 172.21.15.78:0/3310683830 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c011830 con 0x7fc0580ff3e0 2023-10-10T17:40:18.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc05810ac20 con 0x7fc0580ff3e0 2023-10-10T17:40:18.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc055ffb700 1 -- 172.21.15.78:0/3310683830 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc04c011e50 con 0x7fc0580ff3e0 2023-10-10T17:40:18.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc055ffb700 1 -- 172.21.15.78:0/3310683830 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c00f2e0 con 0x7fc0580ff3e0 2023-10-10T17:40:18.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.399+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc05810aa90 con 0x7fc0580ff3e0 2023-10-10T17:40:18.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.400+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0581067e0 con 0x7fc0580ff3e0 2023-10-10T17:40:18.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.402+0000 7fc055ffb700 1 -- 172.21.15.78:0/3310683830 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc04c010580 con 0x7fc0580ff3e0 2023-10-10T17:40:18.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.402+0000 7fc055ffb700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc0480a1150 0x7fc0480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:18.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.402+0000 7fc055ffb700 1 -- 172.21.15.78:0/3310683830 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc04c0cbf70 con 0x7fc0580ff3e0 2023-10-10T17:40:18.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.403+0000 7fc057fff700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc0480a1150 0x7fc0480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:18.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.403+0000 7fc057fff700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc0480a1150 0x7fc0480a3600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc04400b5c0 tx=0x7fc044005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:18.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.407+0000 7fc055ffb700 1 -- 172.21.15.78:0/3310683830 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc04c095ac0 con 0x7fc0580ff3e0 2023-10-10T17:40:18.627 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:40:18 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-d[133102]: starting mds.d at 2023-10-10T17:40:18.628 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:40:18 smithi078 systemd[1]: Started Ceph mds.d for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:18.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.646+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc05802cfe0 con 0x7fc0580ff3e0 2023-10-10T17:40:18.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.646+0000 7fc055ffb700 1 -- 172.21.15.78:0/3310683830 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7fc04c0990e0 con 0x7fc0580ff3e0 2023-10-10T17:40:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:18.649 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-10T17:40:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc0480a1150 msgr2=0x7fc0480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc0480a1150 0x7fc0480a3600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc04400b5c0 tx=0x7fc044005fb0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 msgr2=0x7fc0580fee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580fee60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc04c00b500 tx=0x7fc04c00b810 comp rx=0 tx=0).stop 2023-10-10T17:40:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 shutdown_connections 2023-10-10T17:40:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc0480a1150 0x7fc0480a3600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc058100790 0x7fc05819d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0580ff3e0 0x7fc0580fee60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 --2- 172.21.15.78:0/3310683830 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc0580ffd90 0x7fc05819ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:18.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 >> 172.21.15.78:0/3310683830 conn(0x7fc0580f9f20 msgr2=0x7fc058103920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:18.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 shutdown_connections 2023-10-10T17:40:18.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:18.648+0000 7fc05f0fe700 1 -- 172.21.15.78:0/3310683830 wait complete. 2023-10-10T17:40:18.653 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 4 2023-10-10T17:40:19.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:19.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:19.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:19.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:19.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: Deploying daemon mds.e on smithi157 2023-10-10T17:40:19.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3310683830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:19.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.78:6836/4004555102,v1:172.21.15.78:6837/4004555102] up:boot 2023-10-10T17:40:19.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: fsmap 4 up:standby 2023-10-10T17:40:19.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:19 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-10T17:40:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: Deploying daemon mds.e on smithi157 2023-10-10T17:40:19.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3310683830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:19.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.78:6836/4004555102,v1:172.21.15.78:6837/4004555102] up:boot 2023-10-10T17:40:19.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: fsmap 4 up:standby 2023-10-10T17:40:19.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:19 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-10T17:40:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:19.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: Deploying daemon mds.e on smithi157 2023-10-10T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3310683830' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:19.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.78:6836/4004555102,v1:172.21.15.78:6837/4004555102] up:boot 2023-10-10T17:40:19.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: fsmap 4 up:standby 2023-10-10T17:40:19.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:19 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-10T17:40:20.303 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:40:20 smithi157 systemd[1]: Starting Ceph mds.e for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:20.578 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:40:20 smithi157 podman[125511]: 2023-10-10T17:40:20.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:20.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 -- 172.21.15.78:0/2086659725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec0fe5a0 msgr2=0x7fc3ec0fe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:20.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 --2- 172.21.15.78:0/2086659725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec0fe5a0 0x7fc3ec0fe9b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc3e0009990 tx=0x7fc3e0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:20.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 -- 172.21.15.78:0/2086659725 shutdown_connections 2023-10-10T17:40:20.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 --2- 172.21.15.78:0/2086659725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3ec101fa0 0x7fc3ec108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:20.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 --2- 172.21.15.78:0/2086659725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec0fe5a0 0x7fc3ec0fe9b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:20.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 --2- 172.21.15.78:0/2086659725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3ec101610 0x7fc3ec101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 -- 172.21.15.78:0/2086659725 >> 172.21.15.78:0/2086659725 conn(0x7fc3ec0f9f20 msgr2=0x7fc3ec0fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 -- 172.21.15.78:0/2086659725 shutdown_connections 2023-10-10T17:40:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.816+0000 7fc3f098f700 1 -- 172.21.15.78:0/2086659725 wait complete. 2023-10-10T17:40:20.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.817+0000 7fc3f098f700 1 Processor -- start 2023-10-10T17:40:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.817+0000 7fc3f098f700 1 -- start start 2023-10-10T17:40:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.817+0000 7fc3f098f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3ec0fe5a0 0x7fc3ec195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.817+0000 7fc3f098f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3ec101610 0x7fc3ec1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.817+0000 7fc3f098f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec101fa0 0x7fc3ec19d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3f098f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec19daf0 con 0x7fc3ec101fa0 2023-10-10T17:40:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3f098f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec19dc30 con 0x7fc3ec101610 2023-10-10T17:40:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3f098f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec19ddc0 con 0x7fc3ec0fe5a0 2023-10-10T17:40:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec101fa0 0x7fc3ec19d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec101fa0 0x7fc3ec19d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41934/0 (socket says 172.21.15.78:41934) 2023-10-10T17:40:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 -- 172.21.15.78:0/1139856780 learned_addr learned my addr 172.21.15.78:0/1139856780 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3e9d9b700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3ec101610 0x7fc3ec1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ea59c700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3ec0fe5a0 0x7fc3ec195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 -- 172.21.15.78:0/1139856780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3ec0fe5a0 msgr2=0x7fc3ec195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3ec0fe5a0 0x7fc3ec195f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 -- 172.21.15.78:0/1139856780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3ec101610 msgr2=0x7fc3ec1964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3ec101610 0x7fc3ec1964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 -- 172.21.15.78:0/1139856780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e00095f0 con 0x7fc3ec101fa0 2023-10-10T17:40:20.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3ead9d700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec101fa0 0x7fc3ec19d520 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400ef30 tx=0x7fc3e400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:20.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3db7fe700 1 -- 172.21.15.78:0/1139856780 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4018070 con 0x7fc3ec101fa0 2023-10-10T17:40:20.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3db7fe700 1 -- 172.21.15.78:0/1139856780 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc3e400f040 con 0x7fc3ec101fa0 2023-10-10T17:40:20.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec10bc10 con 0x7fc3ec101fa0 2023-10-10T17:40:20.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3db7fe700 1 -- 172.21.15.78:0/1139856780 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e400cd00 con 0x7fc3ec101fa0 2023-10-10T17:40:20.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.818+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec10ba80 con 0x7fc3ec101fa0 2023-10-10T17:40:20.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.819+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec19e450 con 0x7fc3ec101fa0 2023-10-10T17:40:20.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.820+0000 7fc3db7fe700 1 -- 172.21.15.78:0/1139856780 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc3e4003a00 con 0x7fc3ec101fa0 2023-10-10T17:40:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.821+0000 7fc3db7fe700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc3d40a1150 0x7fc3d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.821+0000 7fc3db7fe700 1 -- 172.21.15.78:0/1139856780 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc3e40cbee0 con 0x7fc3ec101fa0 2023-10-10T17:40:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.821+0000 7fc3ea59c700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc3d40a1150 0x7fc3d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:20.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.821+0000 7fc3ea59c700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc3d40a1150 0x7fc3d40a3600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000b2e0 tx=0x7fc3e0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:20.825+0000 7fc3db7fe700 1 -- 172.21.15.78:0/1139856780 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc3e4095b60 con 0x7fc3ec101fa0 2023-10-10T17:40:20.978 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:40:20 smithi157 ceph-mds[125551]: -- 172.21.15.157:0/3613349891 <== mon.2 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5589fb97f520 con 0x5589fb955000 2023-10-10T17:40:20.978 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:40:20 smithi157 bash[125511]: f3cf017e3dd8e3882ff64cb6301b78abcd6ed714906127c4f0d30b595d850391 2023-10-10T17:40:20.979 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:40:20 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-e[125527]: starting mds.e at 2023-10-10T17:40:20.979 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:40:20 smithi157 systemd[1]: Started Ceph mds.e for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.057+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3ec107370 con 0x7fc3ec101fa0 2023-10-10T17:40:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.058+0000 7fc3db7fe700 1 -- 172.21.15.78:0/1139856780 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3534 (secure 0 0 0) 0x7fc3e4099180 con 0x7fc3ec101fa0 2023-10-10T17:40:21.060 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:21.060 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-10T17:40:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.059+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc3d40a1150 msgr2=0x7fc3d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.059+0000 7fc3f098f700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc3d40a1150 0x7fc3d40a3600 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000b2e0 tx=0x7fc3e0005fb0 comp rx=0 tx=0).stop 2023-10-10T17:40:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.059+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec101fa0 msgr2=0x7fc3ec19d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.059+0000 7fc3f098f700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec101fa0 0x7fc3ec19d520 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400ef30 tx=0x7fc3e400c3a0 comp rx=0 tx=0).stop 2023-10-10T17:40:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 shutdown_connections 2023-10-10T17:40:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc3d40a1150 0x7fc3d40a3600 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc3ec0fe5a0 0x7fc3ec195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3ec101fa0 0x7fc3ec19d520 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 --2- 172.21.15.78:0/1139856780 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc3ec101610 0x7fc3ec1964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 >> 172.21.15.78:0/1139856780 conn(0x7fc3ec0f9f20 msgr2=0x7fc3ec0fc200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 shutdown_connections 2023-10-10T17:40:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:21.060+0000 7fc3f098f700 1 -- 172.21.15.78:0/1139856780 wait complete. 2023-10-10T17:40:21.065 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 5 2023-10-10T17:40:21.682 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:21.686 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.686 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.687 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.687 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:21.688 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:21.688 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:21.688 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:21 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1139856780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:21.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:21 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1139856780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:21.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:21.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:21.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:21.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:21 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1139856780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:22.407 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:40:22 smithi190 systemd[1]: Starting Ceph mds.f for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:22.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:22 smithi190 ceph-mon[101220]: Deploying daemon mds.f on smithi190 2023-10-10T17:40:22.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:22 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6834/2501992479,v1:172.21.15.157:6835/2501992479] up:boot 2023-10-10T17:40:22.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:22 smithi190 ceph-mon[101220]: fsmap 5 up:standby 2023-10-10T17:40:22.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:22 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:40:22.666 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:40:22 smithi190 podman[126758]: 2023-10-10T17:40:22.666 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:40:22 smithi190 ceph-mds[126798]: -- 172.21.15.190:0/3656376912 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5560438e3520 con 0x5560438b8c00 2023-10-10T17:40:22.666 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:40:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-f[126774]: starting mds.f at 2023-10-10T17:40:22.666 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:40:22 smithi190 bash[126758]: 94978f176263275b39c69e2e70c4771172b524aa30d2cf6d9e853f2eea7eea41 2023-10-10T17:40:22.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:22 smithi157 ceph-mon[100896]: Deploying daemon mds.f on smithi190 2023-10-10T17:40:22.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:22 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6834/2501992479,v1:172.21.15.157:6835/2501992479] up:boot 2023-10-10T17:40:22.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:22 smithi157 ceph-mon[100896]: fsmap 5 up:standby 2023-10-10T17:40:22.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:22 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:40:22.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:22 smithi078 ceph-mon[99494]: Deploying daemon mds.f on smithi190 2023-10-10T17:40:22.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:22 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6834/2501992479,v1:172.21.15.157:6835/2501992479] up:boot 2023-10-10T17:40:22.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:22 smithi078 ceph-mon[99494]: fsmap 5 up:standby 2023-10-10T17:40:22.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:22 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:40:23.075 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:23.164 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:40:22 smithi190 systemd[1]: Started Ceph mds.f for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:23.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 -- 172.21.15.78:0/3470864168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 msgr2=0x7fcc300ff5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:23.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 --2- 172.21.15.78:0/3470864168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc300ff5a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcc20009990 tx=0x7fcc20009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 -- 172.21.15.78:0/3470864168 shutdown_connections 2023-10-10T17:40:23.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 --2- 172.21.15.78:0/3470864168 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc30100d10 0x7fcc301075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 --2- 172.21.15.78:0/3470864168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc300ff5a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 --2- 172.21.15.78:0/3470864168 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc30100380 0x7fcc301007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 -- 172.21.15.78:0/3470864168 >> 172.21.15.78:0/3470864168 conn(0x7fcc300fa770 msgr2=0x7fcc300fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:23.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 -- 172.21.15.78:0/3470864168 shutdown_connections 2023-10-10T17:40:23.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 -- 172.21.15.78:0/3470864168 wait complete. 2023-10-10T17:40:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.211+0000 7fcc368db700 1 Processor -- start 2023-10-10T17:40:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc368db700 1 -- start start 2023-10-10T17:40:23.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc368db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc30109a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:23.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc368db700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc30100380 0x7fcc301080c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:23.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc368db700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc30100d10 0x7fcc30108600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:23.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc368db700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30108da0 con 0x7fcc300ff190 2023-10-10T17:40:23.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc368db700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30108f10 con 0x7fcc30100d10 2023-10-10T17:40:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc368db700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc301a0000 con 0x7fcc30100380 2023-10-10T17:40:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc30109a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:23.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc30109a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41944/0 (socket says 172.21.15.78:41944) 2023-10-10T17:40:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 -- 172.21.15.78:0/1481806615 learned_addr learned my addr 172.21.15.78:0/1481806615 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:23.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2f7fe700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc30100380 0x7fcc301080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 -- 172.21.15.78:0/1481806615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc30100380 msgr2=0x7fcc301080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc30100380 0x7fcc301080c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 -- 172.21.15.78:0/1481806615 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc30100d10 msgr2=0x7fcc30108600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:23.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc30100d10 0x7fcc30108600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 -- 172.21.15.78:0/1481806615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc200095f0 con 0x7fcc300ff190 2023-10-10T17:40:23.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.212+0000 7fcc2ffff700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc30109a80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcc20004760 tx=0x7fcc20009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:23.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.213+0000 7fcc2d7fa700 1 -- 172.21.15.78:0/1481806615 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2001d070 con 0x7fcc300ff190 2023-10-10T17:40:23.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.213+0000 7fcc2d7fa700 1 -- 172.21.15.78:0/1481806615 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc20004120 con 0x7fcc300ff190 2023-10-10T17:40:23.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.213+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3010bcc0 con 0x7fcc300ff190 2023-10-10T17:40:23.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.213+0000 7fcc2d7fa700 1 -- 172.21.15.78:0/1481806615 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc20005290 con 0x7fcc300ff190 2023-10-10T17:40:23.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.213+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3010bb30 con 0x7fcc300ff190 2023-10-10T17:40:23.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.214+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc301a05a0 con 0x7fcc300ff190 2023-10-10T17:40:23.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.215+0000 7fcc2d7fa700 1 -- 172.21.15.78:0/1481806615 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcc20004810 con 0x7fcc300ff190 2023-10-10T17:40:23.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.215+0000 7fcc2d7fa700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcc180a1150 0x7fcc180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.215+0000 7fcc2f7fe700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcc180a1150 0x7fcc180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:23.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.215+0000 7fcc2d7fa700 1 -- 172.21.15.78:0/1481806615 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcc200cd4b0 con 0x7fcc300ff190 2023-10-10T17:40:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.216+0000 7fcc2f7fe700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcc180a1150 0x7fcc180a3600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400bbd0 tx=0x7fcc2400b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:23.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.219+0000 7fcc2d7fa700 1 -- 172.21.15.78:0/1481806615 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcc20097130 con 0x7fcc300ff190 2023-10-10T17:40:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.454+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc301060d0 con 0x7fcc300ff190 2023-10-10T17:40:23.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.455+0000 7fcc2d7fa700 1 -- 172.21.15.78:0/1481806615 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5054 (secure 0 0 0) 0x7fcc2009a750 con 0x7fcc300ff190 2023-10-10T17:40:23.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:23.457 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-10-10T17:40:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.456+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcc180a1150 msgr2=0x7fcc180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcc180a1150 0x7fcc180a3600 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400bbd0 tx=0x7fcc2400b5b0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 msgr2=0x7fcc30109a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc30109a80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcc20004760 tx=0x7fcc20009580 comp rx=0 tx=0).stop 2023-10-10T17:40:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 shutdown_connections 2023-10-10T17:40:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcc180a1150 0x7fcc180a3600 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcc30100d10 0x7fcc30108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcc300ff190 0x7fcc30109a80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 --2- 172.21.15.78:0/1481806615 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcc30100380 0x7fcc301080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:23.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 >> 172.21.15.78:0/1481806615 conn(0x7fcc300fa770 msgr2=0x7fcc300fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 shutdown_connections 2023-10-10T17:40:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:23.457+0000 7fcc368db700 1 -- 172.21.15.78:0/1481806615 wait complete. 2023-10-10T17:40:23.461 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 7 2023-10-10T17:40:23.687 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:23 smithi190 ceph-mon[101220]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:23.687 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.687 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.688 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.688 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:23.688 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:23.688 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:23 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:23.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:23 smithi157 ceph-mon[100896]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:23.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:23.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:23 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:23.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:23 smithi078 ceph-mon[99494]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:23.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:23.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:23 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:24.414 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:40:24 smithi190 systemd[1]: Starting Ceph mds.c for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:24.698 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:24 smithi190 ceph-mon[101220]: Deploying daemon mds.c on smithi190 2023-10-10T17:40:24.698 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:24 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6834/1269112994,v1:172.21.15.190:6835/1269112994] up:boot 2023-10-10T17:40:24.698 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:24 smithi190 ceph-mon[101220]: fsmap 6 up:standby 2023-10-10T17:40:24.699 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:24 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:40:24.699 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:24 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1481806615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:24.699 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:40:24 smithi190 podman[127099]: 2023-10-10T17:40:24.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:24 smithi157 ceph-mon[100896]: Deploying daemon mds.c on smithi190 2023-10-10T17:40:24.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:24 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6834/1269112994,v1:172.21.15.190:6835/1269112994] up:boot 2023-10-10T17:40:24.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:24 smithi157 ceph-mon[100896]: fsmap 6 up:standby 2023-10-10T17:40:24.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:24 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:40:24.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:24 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1481806615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:24.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:24 smithi078 ceph-mon[99494]: Deploying daemon mds.c on smithi190 2023-10-10T17:40:24.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:24 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6834/1269112994,v1:172.21.15.190:6835/1269112994] up:boot 2023-10-10T17:40:24.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:24 smithi078 ceph-mon[99494]: fsmap 6 up:standby 2023-10-10T17:40:24.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:24 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:40:24.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:24 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1481806615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:25.164 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:40:24 smithi190 ceph-mds[127139]: -- 172.21.15.190:0/3256021369 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559d31b91520 con 0x559d31b66800 2023-10-10T17:40:25.164 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:40:24 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-c[127115]: starting mds.c at 2023-10-10T17:40:25.165 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:40:24 smithi190 bash[127099]: 760d849e7fb8f10eb3c7ef7bf0492fee4472a45f592e6779cd9dfe37c5f704fa 2023-10-10T17:40:25.165 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:40:24 smithi190 systemd[1]: Started Ceph mds.c for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:25.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:25.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 -- 172.21.15.78:0/4086774061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ff430 msgr2=0x7f19f40ff8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:25.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 --2- 172.21.15.78:0/4086774061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ff430 0x7f19f40ff8a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009990 tx=0x7f19e4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 -- 172.21.15.78:0/4086774061 shutdown_connections 2023-10-10T17:40:25.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 --2- 172.21.15.78:0/4086774061 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19f40ffde0 0x7f19f4107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 --2- 172.21.15.78:0/4086774061 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19f4100870 0x7f19f4100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 --2- 172.21.15.78:0/4086774061 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ff430 0x7f19f40ff8a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 -- 172.21.15.78:0/4086774061 >> 172.21.15.78:0/4086774061 conn(0x7f19f40f9fb0 msgr2=0x7f19f40fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:25.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 -- 172.21.15.78:0/4086774061 shutdown_connections 2023-10-10T17:40:25.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.613+0000 7f19f9303700 1 -- 172.21.15.78:0/4086774061 wait complete. 2023-10-10T17:40:25.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 Processor -- start 2023-10-10T17:40:25.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 -- start start 2023-10-10T17:40:25.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19f40ff430 0x7f19f41034f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:25.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ffde0 0x7f19f4101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:25.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19f4100870 0x7f19f4102070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:25.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4102810 con 0x7f19f40ffde0 2023-10-10T17:40:25.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4102980 con 0x7f19f40ff430 2023-10-10T17:40:25.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f9303700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f419ffa0 con 0x7f19f4100870 2023-10-10T17:40:25.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ffde0 0x7f19f4101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:25.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ffde0 0x7f19f4101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41960/0 (socket says 172.21.15.78:41960) 2023-10-10T17:40:25.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f27fc700 1 -- 172.21.15.78:0/3792734326 learned_addr learned my addr 172.21.15.78:0/3792734326 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:25.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.614+0000 7f19f2ffd700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19f40ff430 0x7f19f41034f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:25.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f37fe700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19f4100870 0x7f19f4102070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:25.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f27fc700 1 -- 172.21.15.78:0/3792734326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19f4100870 msgr2=0x7f19f4102070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:25.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f27fc700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19f4100870 0x7f19f4102070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f27fc700 1 -- 172.21.15.78:0/3792734326 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19f40ff430 msgr2=0x7f19f41034f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:25.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f27fc700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19f40ff430 0x7f19f41034f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f27fc700 1 -- 172.21.15.78:0/3792734326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e40095f0 con 0x7f19f40ffde0 2023-10-10T17:40:25.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f2ffd700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19f40ff430 0x7f19f41034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:40:25.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f27fc700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ffde0 0x7f19f4101b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f19e40059d0 tx=0x7f19e40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:25.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19dbfff700 1 -- 172.21.15.78:0/3792734326 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e401d070 con 0x7f19f40ffde0 2023-10-10T17:40:25.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19dbfff700 1 -- 172.21.15.78:0/3792734326 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f19e400fb90 con 0x7f19f40ffde0 2023-10-10T17:40:25.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f410ab90 con 0x7f19f40ffde0 2023-10-10T17:40:25.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19dbfff700 1 -- 172.21.15.78:0/3792734326 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e4017650 con 0x7f19f40ffde0 2023-10-10T17:40:25.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.615+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f410aa00 con 0x7f19f40ffde0 2023-10-10T17:40:25.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.616+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f4107330 con 0x7f19f40ffde0 2023-10-10T17:40:25.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.617+0000 7f19dbfff700 1 -- 172.21.15.78:0/3792734326 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f19e400fd00 con 0x7f19f40ffde0 2023-10-10T17:40:25.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.618+0000 7f19dbfff700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19dc0a1150 0x7f19dc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:25.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.618+0000 7f19f2ffd700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19dc0a1150 0x7f19dc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:25.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.618+0000 7f19dbfff700 1 -- 172.21.15.78:0/3792734326 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f19e40cdc20 con 0x7f19f40ffde0 2023-10-10T17:40:25.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.618+0000 7f19f2ffd700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19dc0a1150 0x7f19dc0a3600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f19e8005950 tx=0x7f19e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:25.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.622+0000 7f19dbfff700 1 -- 172.21.15.78:0/3792734326 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f19e4097800 con 0x7f19f40ffde0 2023-10-10T17:40:25.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:25 smithi157 ceph-mon[100896]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:25.842 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:25.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:25.843 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:25 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:25.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.857+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19f402cfe0 con 0x7f19f40ffde0 2023-10-10T17:40:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.858+0000 7f19dbfff700 1 -- 172.21.15.78:0/3792734326 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5811 (secure 0 0 0) 0x7f19e409ae20 con 0x7f19f40ffde0 2023-10-10T17:40:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:25.860 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-10-10T17:40:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19dc0a1150 msgr2=0x7f19dc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19dc0a1150 0x7f19dc0a3600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f19e8005950 tx=0x7f19e800b410 comp rx=0 tx=0).stop 2023-10-10T17:40:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ffde0 msgr2=0x7f19f4101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ffde0 0x7f19f4101b30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f19e40059d0 tx=0x7f19e40047a0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 shutdown_connections 2023-10-10T17:40:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19dc0a1150 0x7f19dc0a3600 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19f4100870 0x7f19f4102070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19f40ff430 0x7f19f41034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 --2- 172.21.15.78:0/3792734326 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19f40ffde0 0x7f19f4101b30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 >> 172.21.15.78:0/3792734326 conn(0x7f19f40f9fb0 msgr2=0x7f19f40fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 shutdown_connections 2023-10-10T17:40:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:25.860+0000 7f19f9303700 1 -- 172.21.15.78:0/3792734326 wait complete. 2023-10-10T17:40:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-10-10T17:40:25.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:25 smithi078 ceph-mon[99494]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:25.872 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:25.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:25.873 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:25 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:25.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:25 smithi190 ceph-mon[101220]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:25.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:25.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:25.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:25.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:25 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:26.479 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:40:26 smithi157 systemd[1]: Starting Ceph mds.h for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:26.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:26 smithi078 ceph-mon[99494]: Deploying daemon mds.h on smithi157 2023-10-10T17:40:26.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:26 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] up:boot 2023-10-10T17:40:26.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:26 smithi078 ceph-mon[99494]: fsmap 7 up:standby 2023-10-10T17:40:26.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:26 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:40:26.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:26 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3792734326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:26.910 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:40:26 smithi157 podman[125850]: 2023-10-10T17:40:26.910 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:26 smithi157 ceph-mon[100896]: Deploying daemon mds.h on smithi157 2023-10-10T17:40:26.910 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:26 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] up:boot 2023-10-10T17:40:26.911 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:26 smithi157 ceph-mon[100896]: fsmap 7 up:standby 2023-10-10T17:40:26.911 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:26 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:40:26.911 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:26 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3792734326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:26.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:26 smithi190 ceph-mon[101220]: Deploying daemon mds.h on smithi157 2023-10-10T17:40:26.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:26 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] up:boot 2023-10-10T17:40:26.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:26 smithi190 ceph-mon[101220]: fsmap 7 up:standby 2023-10-10T17:40:26.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:26 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:40:26.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:26 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3792734326' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:27.227 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:40:26 smithi157 ceph-mds[125890]: -- 172.21.15.157:0/2615865650 <== mon.2 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556739b91520 con 0x556739b66800 2023-10-10T17:40:27.228 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:40:26 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-h[125866]: starting mds.h at 2023-10-10T17:40:27.228 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:40:26 smithi157 bash[125850]: e1aa27419eb0a028a4e530a10c3968511781c1f47d42d8b332225082466dc0a2 2023-10-10T17:40:27.228 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:40:26 smithi157 systemd[1]: Started Ceph mds.h for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:27.838 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:27 smithi190 ceph-mon[101220]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:27.838 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.839 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.839 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.839 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:27.839 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:27.840 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:27 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:27.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:27.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:27 smithi078 ceph-mon[99494]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:27.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:27.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:27.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:27 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:27.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:27 smithi157 ceph-mon[100896]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:27.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:27.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:27.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:27.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:27 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:28.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.015+0000 7f106b5b9700 1 -- 172.21.15.78:0/2598104861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ff370 msgr2=0x7f10640ff7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:28.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.015+0000 7f106b5b9700 1 --2- 172.21.15.78:0/2598104861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ff370 0x7f10640ff7e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1060009990 tx=0x7f1060009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.015+0000 7f106b5b9700 1 -- 172.21.15.78:0/2598104861 shutdown_connections 2023-10-10T17:40:28.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.015+0000 7f106b5b9700 1 --2- 172.21.15.78:0/2598104861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10640ffd20 0x7f10641045d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.015+0000 7f106b5b9700 1 --2- 172.21.15.78:0/2598104861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10640fe610 0x7f10640fea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.015+0000 7f106b5b9700 1 --2- 172.21.15.78:0/2598104861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ff370 0x7f10640ff7e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.015+0000 7f106b5b9700 1 -- 172.21.15.78:0/2598104861 >> 172.21.15.78:0/2598104861 conn(0x7f10640f9fb0 msgr2=0x7f10640fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:28.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 -- 172.21.15.78:0/2598104861 shutdown_connections 2023-10-10T17:40:28.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 -- 172.21.15.78:0/2598104861 wait complete. 2023-10-10T17:40:28.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 Processor -- start 2023-10-10T17:40:28.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 -- start start 2023-10-10T17:40:28.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10640fe610 0x7f106419c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:28.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10640ff370 0x7f106419caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:28.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ffd20 0x7f10641a3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:28.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a4130 con 0x7f10640ffd20 2023-10-10T17:40:28.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a4270 con 0x7f10640ff370 2023-10-10T17:40:28.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f106b5b9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10641a4400 con 0x7f10640fe610 2023-10-10T17:40:28.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f1069355700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10640fe610 0x7f106419c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:28.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f1069b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ffd20 0x7f10641a3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:28.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.016+0000 7f1069355700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10640fe610 0x7f106419c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:33500/0 (socket says 172.21.15.78:33500) 2023-10-10T17:40:28.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ffd20 0x7f10641a3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41974/0 (socket says 172.21.15.78:41974) 2023-10-10T17:40:28.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069355700 1 -- 172.21.15.78:0/29876887 learned_addr learned my addr 172.21.15.78:0/29876887 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:28.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1068b54700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10640ff370 0x7f106419caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:28.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069b56700 1 -- 172.21.15.78:0/29876887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10640fe610 msgr2=0x7f106419c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:28.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069b56700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10640fe610 0x7f106419c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069b56700 1 -- 172.21.15.78:0/29876887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10640ff370 msgr2=0x7f106419caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:28.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069b56700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10640ff370 0x7f106419caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069b56700 1 -- 172.21.15.78:0/29876887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10600095f0 con 0x7f10640ffd20 2023-10-10T17:40:28.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f1069b56700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ffd20 0x7f10641a3b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f105c00cb70 tx=0x7f105c00ce80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:28.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f105a7fc700 1 -- 172.21.15.78:0/29876887 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00a860 con 0x7f10640ffd20 2023-10-10T17:40:28.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f105a7fc700 1 -- 172.21.15.78:0/29876887 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f105c00ae80 con 0x7f10640ffd20 2023-10-10T17:40:28.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1064112340 con 0x7f10640ffd20 2023-10-10T17:40:28.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f105a7fc700 1 -- 172.21.15.78:0/29876887 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c011dd0 con 0x7f10640ffd20 2023-10-10T17:40:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.017+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10641121b0 con 0x7f10640ffd20 2023-10-10T17:40:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.018+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10641a4b00 con 0x7f10640ffd20 2023-10-10T17:40:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.019+0000 7f105a7fc700 1 -- 172.21.15.78:0/29876887 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f105c011400 con 0x7f10640ffd20 2023-10-10T17:40:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.020+0000 7f105a7fc700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10500a11d0 0x7f10500a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.020+0000 7f1069355700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10500a11d0 0x7f10500a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:28.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.020+0000 7f105a7fc700 1 -- 172.21.15.78:0/29876887 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f105c0cc450 con 0x7f10640ffd20 2023-10-10T17:40:28.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.020+0000 7f1069355700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10500a11d0 0x7f10500a3680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1054005950 tx=0x7f105400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:28.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.023+0000 7f105a7fc700 1 -- 172.21.15.78:0/29876887 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f105c096030 con 0x7f10640ffd20 2023-10-10T17:40:28.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.254+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10641a50b0 con 0x7f10640ffd20 2023-10-10T17:40:28.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.255+0000 7f105a7fc700 1 -- 172.21.15.78:0/29876887 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6571 (secure 0 0 0) 0x7f105c099650 con 0x7f10640ffd20 2023-10-10T17:40:28.257 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:28.257 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-10-10T17:40:28.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10500a11d0 msgr2=0x7f10500a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10500a11d0 0x7f10500a3680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1054005950 tx=0x7f105400b500 comp rx=0 tx=0).stop 2023-10-10T17:40:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ffd20 msgr2=0x7f10641a3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ffd20 0x7f10641a3b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f105c00cb70 tx=0x7f105c00ce80 comp rx=0 tx=0).stop 2023-10-10T17:40:28.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 shutdown_connections 2023-10-10T17:40:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f10500a11d0 0x7f10500a3680 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f10640fe610 0x7f106419c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10640ffd20 0x7f10641a3b60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 --2- 172.21.15.78:0/29876887 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f10640ff370 0x7f106419caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:28.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 >> 172.21.15.78:0/29876887 conn(0x7f10640f9fb0 msgr2=0x7f10640fac10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 shutdown_connections 2023-10-10T17:40:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:28.257+0000 7f106b5b9700 1 -- 172.21.15.78:0/29876887 wait complete. 2023-10-10T17:40:28.261 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 9 2023-10-10T17:40:28.844 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:40:28 smithi190 systemd[1]: Starting Ceph mds.i for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:28.845 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:28 smithi190 ceph-mon[101220]: Deploying daemon mds.i on smithi190 2023-10-10T17:40:28.845 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:28 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6836/1666360072,v1:172.21.15.157:6837/1666360072] up:boot 2023-10-10T17:40:28.845 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:28 smithi190 ceph-mon[101220]: fsmap 8 up:standby 2023-10-10T17:40:28.845 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:28 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:40:28.846 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/29876887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:28.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:28 smithi078 ceph-mon[99494]: Deploying daemon mds.i on smithi190 2023-10-10T17:40:28.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:28 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6836/1666360072,v1:172.21.15.157:6837/1666360072] up:boot 2023-10-10T17:40:28.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:28 smithi078 ceph-mon[99494]: fsmap 8 up:standby 2023-10-10T17:40:28.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:28 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:40:28.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/29876887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:28.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:28 smithi157 ceph-mon[100896]: Deploying daemon mds.i on smithi190 2023-10-10T17:40:28.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:28 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6836/1666360072,v1:172.21.15.157:6837/1666360072] up:boot 2023-10-10T17:40:28.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:28 smithi157 ceph-mon[100896]: fsmap 8 up:standby 2023-10-10T17:40:28.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:28 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:40:28.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/29876887' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:29.148 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:40:28 smithi190 podman[127441]: 2023-10-10T17:40:29.149 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mds[127482]: -- 172.21.15.190:0/2301545002 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56333189d520 con 0x563331872800 2023-10-10T17:40:29.149 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-i[127457]: starting mds.i at 2023-10-10T17:40:29.149 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:40:29 smithi190 bash[127441]: c770cd9e493e742a4364f536c07ce23ce907180eacaf2631b1fca8702c448ae3 2023-10-10T17:40:29.414 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:40:29 smithi190 systemd[1]: Started Ceph mds.i for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:29.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:29 smithi078 ceph-mon[99494]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:29.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:29 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:29 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:29 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:29 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:29 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:29 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:29.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mon[101220]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:29.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:29.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:29.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:29 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:29.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:29 smithi157 ceph-mon[100896]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:29.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:29 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:29 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:29 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:29.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:29 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:29.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:29 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:29.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:29 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:30.271 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:30.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 -- 172.21.15.78:0/1827280713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0100290 msgr2=0x7f7ed01006a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:30.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 --2- 172.21.15.78:0/1827280713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0100290 0x7f7ed01006a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4009990 tx=0x7f7ec4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 -- 172.21.15.78:0/1827280713 shutdown_connections 2023-10-10T17:40:30.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 --2- 172.21.15.78:0/1827280713 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed0101e10 0x7f7ed01086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 --2- 172.21.15.78:0/1827280713 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0100290 0x7f7ed01006a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 --2- 172.21.15.78:0/1827280713 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed0101480 0x7f7ed01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 -- 172.21.15.78:0/1827280713 >> 172.21.15.78:0/1827280713 conn(0x7f7ed00fb870 msgr2=0x7f7ed00fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:30.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 -- 172.21.15.78:0/1827280713 shutdown_connections 2023-10-10T17:40:30.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.395+0000 7f7ed7043700 1 -- 172.21.15.78:0/1827280713 wait complete. 2023-10-10T17:40:30.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 Processor -- start 2023-10-10T17:40:30.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 -- start start 2023-10-10T17:40:30.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed0100290 0x7f7ed0198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:30.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0101480 0x7f7ed0198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:30.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed0101e10 0x7f7ed019f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:30.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed019fcd0 con 0x7f7ed0101480 2023-10-10T17:40:30.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed019fe10 con 0x7f7ed0100290 2023-10-10T17:40:30.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed7043700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed019ffa0 con 0x7f7ed0101e10 2023-10-10T17:40:30.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0101480 0x7f7ed0198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:30.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0101480 0x7f7ed0198690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:41988/0 (socket says 172.21.15.78:41988) 2023-10-10T17:40:30.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 -- 172.21.15.78:0/2906625792 learned_addr learned my addr 172.21.15.78:0/2906625792 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:30.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed55e0700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed0101e10 0x7f7ed019f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:30.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ed4ddf700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed0100290 0x7f7ed0198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:30.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 -- 172.21.15.78:0/2906625792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed0101e10 msgr2=0x7f7ed019f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:30.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed0101e10 0x7f7ed019f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 -- 172.21.15.78:0/2906625792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed0100290 msgr2=0x7f7ed0198150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:30.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed0100290 0x7f7ed0198150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.396+0000 7f7ecffff700 1 -- 172.21.15.78:0/2906625792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec40095f0 con 0x7f7ed0101480 2023-10-10T17:40:30.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.397+0000 7f7ecffff700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0101480 0x7f7ed0198690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00d980 tx=0x7f7ebc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:30.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.397+0000 7f7ecdffb700 1 -- 172.21.15.78:0/2906625792 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc004220 con 0x7f7ed0101480 2023-10-10T17:40:30.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.397+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed0073dc0 con 0x7f7ed0101480 2023-10-10T17:40:30.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.397+0000 7f7ecdffb700 1 -- 172.21.15.78:0/2906625792 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ebc007640 con 0x7f7ed0101480 2023-10-10T17:40:30.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.397+0000 7f7ecdffb700 1 -- 172.21.15.78:0/2906625792 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc0055c0 con 0x7f7ed0101480 2023-10-10T17:40:30.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.397+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed0073c30 con 0x7f7ed0101480 2023-10-10T17:40:30.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.398+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed0106e60 con 0x7f7ed0101480 2023-10-10T17:40:30.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.399+0000 7f7ecdffb700 1 -- 172.21.15.78:0/2906625792 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7ebc004380 con 0x7f7ed0101480 2023-10-10T17:40:30.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.399+0000 7f7ecdffb700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ec00a1150 0x7f7ec00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:30.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.399+0000 7f7ed4ddf700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ec00a1150 0x7f7ec00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:30.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.399+0000 7f7ecdffb700 1 -- 172.21.15.78:0/2906625792 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7ebc0cc2f0 con 0x7f7ed0101480 2023-10-10T17:40:30.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.400+0000 7f7ed4ddf700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ec00a1150 0x7f7ec00a3600 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400b5c0 tx=0x7f7ec4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:30.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.404+0000 7f7ecdffb700 1 -- 172.21.15.78:0/2906625792 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7ebc095ee0 con 0x7f7ed0101480 2023-10-10T17:40:30.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.657+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ed0061700 con 0x7f7ed0101480 2023-10-10T17:40:30.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.658+0000 7f7ecdffb700 1 -- 172.21.15.78:0/2906625792 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7330 (secure 0 0 0) 0x7f7ebc099500 con 0x7f7ed0101480 2023-10-10T17:40:30.661 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:30.661 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-10-10T17:40:30.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ec00a1150 msgr2=0x7f7ec00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:30.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ec00a1150 0x7f7ec00a3600 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400b5c0 tx=0x7f7ec4005960 comp rx=0 tx=0).stop 2023-10-10T17:40:30.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0101480 msgr2=0x7f7ed0198690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:30.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0101480 0x7f7ed0198690 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc00d980 tx=0x7f7ebc00dc90 comp rx=0 tx=0).stop 2023-10-10T17:40:30.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 shutdown_connections 2023-10-10T17:40:30.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ec00a1150 0x7f7ec00a3600 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7ed0101e10 0x7f7ed019f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7ed0100290 0x7f7ed0198150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 --2- 172.21.15.78:0/2906625792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7ed0101480 0x7f7ed0198690 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:30.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.660+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 >> 172.21.15.78:0/2906625792 conn(0x7f7ed00fb870 msgr2=0x7f7ed00fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:30.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.661+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 shutdown_connections 2023-10-10T17:40:30.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:30.661+0000 7f7ed7043700 1 -- 172.21.15.78:0/2906625792 wait complete. 2023-10-10T17:40:30.665 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 10 2023-10-10T17:40:30.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:30 smithi078 ceph-mon[99494]: Deploying daemon mds.g on smithi078 2023-10-10T17:40:30.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:30 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] up:boot 2023-10-10T17:40:30.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:30 smithi078 ceph-mon[99494]: fsmap 9 up:standby 2023-10-10T17:40:30.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:30 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:40:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:30 smithi190 ceph-mon[101220]: Deploying daemon mds.g on smithi078 2023-10-10T17:40:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:30 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] up:boot 2023-10-10T17:40:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:30 smithi190 ceph-mon[101220]: fsmap 9 up:standby 2023-10-10T17:40:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:30 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:40:30.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:30 smithi157 ceph-mon[100896]: Deploying daemon mds.g on smithi078 2023-10-10T17:40:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:30 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] up:boot 2023-10-10T17:40:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:30 smithi157 ceph-mon[100896]: fsmap 9 up:standby 2023-10-10T17:40:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:30 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:40:31.127 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:40:30 smithi078 systemd[1]: Starting Ceph mds.g for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:31.420 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:40:31 smithi078 podman[133696]: 2023-10-10T17:40:31.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:31 smithi078 ceph-mon[99494]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:31.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:31 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2906625792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:31.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:31.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:31 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:31.879 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:40:31 smithi078 ceph-mds[133736]: -- 172.21.15.78:0/338732397 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55709f217520 con 0x55709f1ed000 2023-10-10T17:40:31.879 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:40:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-g[133712]: starting mds.g at 2023-10-10T17:40:31.879 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:40:31 smithi078 bash[133696]: 26867ac92b4a84d18c9dd636d1a10c568d638cdc39e3acdaea0699e6b30b2b8d 2023-10-10T17:40:31.879 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:40:31 smithi078 systemd[1]: Started Ceph mds.g for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:31.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:31 smithi190 ceph-mon[101220]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:31.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:31 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2906625792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:31.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:31.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:31 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:31 smithi157 ceph-mon[100896]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:31 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2906625792' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:31.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:31 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:32.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:32.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 -- 172.21.15.78:0/2265333898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72680ff170 msgr2=0x7f72680ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:32.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 --2- 172.21.15.78:0/2265333898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72680ff170 0x7f72680ff580 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f7258009990 tx=0x7f7258009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:32.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 -- 172.21.15.78:0/2265333898 shutdown_connections 2023-10-10T17:40:32.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 --2- 172.21.15.78:0/2265333898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7268100db0 0x7f7268107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:32.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 --2- 172.21.15.78:0/2265333898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72680ff170 0x7f72680ff580 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:32.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 --2- 172.21.15.78:0/2265333898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7268100400 0x7f7268100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:32.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 -- 172.21.15.78:0/2265333898 >> 172.21.15.78:0/2265333898 conn(0x7f72680fa750 msgr2=0x7f72680fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:32.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 -- 172.21.15.78:0/2265333898 shutdown_connections 2023-10-10T17:40:32.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.807+0000 7f726c887700 1 -- 172.21.15.78:0/2265333898 wait complete. 2023-10-10T17:40:32.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 Processor -- start 2023-10-10T17:40:32.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 -- start start 2023-10-10T17:40:32.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72680ff170 0x7f7268194ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:32.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7268100400 0x7f7268195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:32.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7268100db0 0x7f726819c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:32.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726819ca60 con 0x7f7268100400 2023-10-10T17:40:32.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726819cba0 con 0x7f7268100db0 2023-10-10T17:40:32.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726c887700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f726819cd30 con 0x7f72680ff170 2023-10-10T17:40:32.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7268100400 0x7f7268195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:32.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7268100400 0x7f7268195420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42012/0 (socket says 172.21.15.78:42012) 2023-10-10T17:40:32.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f726659c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72680ff170 0x7f7268194ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:32.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 -- 172.21.15.78:0/642411204 learned_addr learned my addr 172.21.15.78:0/642411204 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:32.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7266d9d700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7268100db0 0x7f726819c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:32.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 -- 172.21.15.78:0/642411204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72680ff170 msgr2=0x7f7268194ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:32.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72680ff170 0x7f7268194ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:32.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 -- 172.21.15.78:0/642411204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7268100db0 msgr2=0x7f726819c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:32.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7268100db0 0x7f726819c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:32.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 -- 172.21.15.78:0/642411204 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580095f0 con 0x7f7268100400 2023-10-10T17:40:32.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7266d9d700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7268100db0 0x7f726819c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:32.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.808+0000 7f7265d9b700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7268100400 0x7f7268195420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f725c00c920 tx=0x7f725c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:32.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.809+0000 7f72577fe700 1 -- 172.21.15.78:0/642411204 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c007b00 con 0x7f7268100400 2023-10-10T17:40:32.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.809+0000 7f72577fe700 1 -- 172.21.15.78:0/642411204 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f725c00ce70 con 0x7f7268100400 2023-10-10T17:40:32.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.809+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726810ab70 con 0x7f7268100400 2023-10-10T17:40:32.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.809+0000 7f72577fe700 1 -- 172.21.15.78:0/642411204 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c0054e0 con 0x7f7268100400 2023-10-10T17:40:32.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.809+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f726810a9e0 con 0x7f7268100400 2023-10-10T17:40:32.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.809+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7268106190 con 0x7f7268100400 2023-10-10T17:40:32.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.810+0000 7f72577fe700 1 -- 172.21.15.78:0/642411204 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f725c004750 con 0x7f7268100400 2023-10-10T17:40:32.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.811+0000 7f72577fe700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72500a1150 0x7f72500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:32.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.811+0000 7f72577fe700 1 -- 172.21.15.78:0/642411204 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f725c0cc3c0 con 0x7f7268100400 2023-10-10T17:40:32.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.811+0000 7f726659c700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72500a1150 0x7f72500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:32.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.811+0000 7f726659c700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72500a1150 0x7f72500a3600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f725800b5c0 tx=0x7f7258005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:32.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:32.814+0000 7f72577fe700 1 -- 172.21.15.78:0/642411204 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f725c095fb0 con 0x7f7268100400 2023-10-10T17:40:32.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.78:6838/943709499,v1:172.21.15.78:6839/943709499] up:boot 2023-10-10T17:40:32.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: fsmap 10 up:standby 2023-10-10T17:40:32.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-10T17:40:32.822 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:32.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:32.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:32.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:32.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: Deploying daemon mds.a on smithi078 2023-10-10T17:40:32.823 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:32 smithi078 ceph-mon[99494]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:32.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.78:6838/943709499,v1:172.21.15.78:6839/943709499] up:boot 2023-10-10T17:40:32.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: fsmap 10 up:standby 2023-10-10T17:40:32.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-10T17:40:32.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:32.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:32.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:32.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:32.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: Deploying daemon mds.a on smithi078 2023-10-10T17:40:32.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:32 smithi190 ceph-mon[101220]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:32.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.78:6838/943709499,v1:172.21.15.78:6839/943709499] up:boot 2023-10-10T17:40:32.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: fsmap 10 up:standby 2023-10-10T17:40:32.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-10T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:32.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: Deploying daemon mds.a on smithi078 2023-10-10T17:40:32.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:32 smithi157 ceph-mon[100896]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:33.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.052+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7268061700 con 0x7f7268100400 2023-10-10T17:40:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.052+0000 7f72577fe700 1 -- 172.21.15.78:0/642411204 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8085 (secure 0 0 0) 0x7f725c0995d0 con 0x7f7268100400 2023-10-10T17:40:33.054 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:33.055 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-10-10T17:40:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.054+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72500a1150 msgr2=0x7f72500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.054+0000 7f726c887700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72500a1150 0x7f72500a3600 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f725800b5c0 tx=0x7f7258005fd0 comp rx=0 tx=0).stop 2023-10-10T17:40:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.054+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7268100400 msgr2=0x7f7268195420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:33.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.054+0000 7f726c887700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7268100400 0x7f7268195420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f725c00c920 tx=0x7f725c00cc30 comp rx=0 tx=0).stop 2023-10-10T17:40:33.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 shutdown_connections 2023-10-10T17:40:33.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72500a1150 0x7f72500a3600 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:33.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72680ff170 0x7f7268194ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:33.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7268100db0 0x7f726819c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:33.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 --2- 172.21.15.78:0/642411204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7268100400 0x7f7268195420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:33.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 >> 172.21.15.78:0/642411204 conn(0x7f72680fa750 msgr2=0x7f72680fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:33.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 shutdown_connections 2023-10-10T17:40:33.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:33.055+0000 7f726c887700 1 -- 172.21.15.78:0/642411204 wait complete. 2023-10-10T17:40:33.061 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 11 2023-10-10T17:40:33.586 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:40:33 smithi078 systemd[1]: Starting Ceph mds.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:33.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:33 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/642411204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:33.837 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:40:33 smithi078 podman[134079]: 2023-10-10T17:40:33.837 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:40:33 smithi078 ceph-mds[134120]: -- 172.21.15.78:0/1958830089 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55986dc49520 con 0x55986dc1e800 2023-10-10T17:40:33.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:33 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/642411204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:33.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:33 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/642411204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:34.127 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:40:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-a[134095]: starting mds.a at 2023-10-10T17:40:34.128 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:40:33 smithi078 bash[134079]: 1e06a33ed0cd8d09834952b5b3083194e7874357f9abd58531b1a6d0bb52208f 2023-10-10T17:40:34.128 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:40:33 smithi078 systemd[1]: Started Ceph mds.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:35.070 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:35.195 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.195 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.195 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.196 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:35.196 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:35.196 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:35.196 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: Deploying daemon mds.b on smithi157 2023-10-10T17:40:35.197 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:35.197 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.78:6840/504515550,v1:172.21.15.78:6841/504515550] up:boot 2023-10-10T17:40:35.197 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: fsmap 11 up:standby 2023-10-10T17:40:35.197 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:34 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-10T17:40:35.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 -- 172.21.15.78:0/3921155244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fee90 msgr2=0x7f0d341015b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:35.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 --2- 172.21.15.78:0/3921155244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fee90 0x7f0d341015b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0d28009990 tx=0x7f0d28009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 -- 172.21.15.78:0/3921155244 shutdown_connections 2023-10-10T17:40:35.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 --2- 172.21.15.78:0/3921155244 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d340fe540 0x7f0d340fe950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 --2- 172.21.15.78:0/3921155244 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d34101c40 0x7f0d34108800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 --2- 172.21.15.78:0/3921155244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fee90 0x7f0d341015b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 -- 172.21.15.78:0/3921155244 >> 172.21.15.78:0/3921155244 conn(0x7f0d340f9ee0 msgr2=0x7f0d340fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 -- 172.21.15.78:0/3921155244 shutdown_connections 2023-10-10T17:40:35.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 -- 172.21.15.78:0/3921155244 wait complete. 2023-10-10T17:40:35.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.207+0000 7f0d39efb700 1 Processor -- start 2023-10-10T17:40:35.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 -- start start 2023-10-10T17:40:35.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fe540 0x7f0d34195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:35.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d340fee90 0x7f0d34196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:35.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d34101c40 0x7f0d3419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:35.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3419db50 con 0x7f0d340fe540 2023-10-10T17:40:35.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3419dc90 con 0x7f0d340fee90 2023-10-10T17:40:35.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3419de20 con 0x7f0d34101c40 2023-10-10T17:40:35.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fe540 0x7f0d34195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:35.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fe540 0x7f0d34195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42036/0 (socket says 172.21.15.78:42036) 2023-10-10T17:40:35.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 -- 172.21.15.78:0/4216178072 learned_addr learned my addr 172.21.15.78:0/4216178072 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:35.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d33fff700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d34101c40 0x7f0d3419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:35.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d32ffd700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d340fee90 0x7f0d34196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:35.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 -- 172.21.15.78:0/4216178072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d34101c40 msgr2=0x7f0d3419d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:35.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d34101c40 0x7f0d3419d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 -- 172.21.15.78:0/4216178072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d340fee90 msgr2=0x7f0d34196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:35.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d340fee90 0x7f0d34196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 -- 172.21.15.78:0/4216178072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d280095f0 con 0x7f0d340fe540 2023-10-10T17:40:35.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d337fe700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fe540 0x7f0d34195fd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400d950 tx=0x7f0d2400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:35.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d30ff9700 1 -- 172.21.15.78:0/4216178072 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2400f890 con 0x7f0d340fe540 2023-10-10T17:40:35.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d30ff9700 1 -- 172.21.15.78:0/4216178072 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d24010460 con 0x7f0d340fe540 2023-10-10T17:40:35.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3410bc10 con 0x7f0d340fe540 2023-10-10T17:40:35.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d30ff9700 1 -- 172.21.15.78:0/4216178072 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2400e610 con 0x7f0d340fe540 2023-10-10T17:40:35.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.208+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3410ba80 con 0x7f0d340fe540 2023-10-10T17:40:35.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.209+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d341079a0 con 0x7f0d340fe540 2023-10-10T17:40:35.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.210+0000 7f0d30ff9700 1 -- 172.21.15.78:0/4216178072 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0d240108d0 con 0x7f0d340fe540 2023-10-10T17:40:35.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.211+0000 7f0d30ff9700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d1c0a1150 0x7f0d1c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:35.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.211+0000 7f0d32ffd700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d1c0a1150 0x7f0d1c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:35.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.211+0000 7f0d30ff9700 1 -- 172.21.15.78:0/4216178072 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0d240cc230 con 0x7f0d340fe540 2023-10-10T17:40:35.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.211+0000 7f0d32ffd700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d1c0a1150 0x7f0d1c0a3600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0d28006010 tx=0x7f0d28005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:35.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.215+0000 7f0d30ff9700 1 -- 172.21.15.78:0/4216178072 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0d24095e20 con 0x7f0d340fe540 2023-10-10T17:40:35.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:35.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:35.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:35.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: Deploying daemon mds.b on smithi157 2023-10-10T17:40:35.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:35.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.78:6840/504515550,v1:172.21.15.78:6841/504515550] up:boot 2023-10-10T17:40:35.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: fsmap 11 up:standby 2023-10-10T17:40:35.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:34 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-10T17:40:35.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-10-10T17:40:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-10-10T17:40:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: Deploying daemon mds.b on smithi157 2023-10-10T17:40:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.78:6840/504515550,v1:172.21.15.78:6841/504515550] up:boot 2023-10-10T17:40:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: fsmap 11 up:standby 2023-10-10T17:40:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:34 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-10T17:40:35.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.446+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d3402cfe0 con 0x7f0d340fe540 2023-10-10T17:40:35.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.447+0000 7f0d30ff9700 1 -- 172.21.15.78:0/4216178072 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8840 (secure 0 0 0) 0x7f0d24099440 con 0x7f0d340fe540 2023-10-10T17:40:35.449 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:35.449 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-10-10T17:40:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d1c0a1150 msgr2=0x7f0d1c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d1c0a1150 0x7f0d1c0a3600 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0d28006010 tx=0x7f0d28005960 comp rx=0 tx=0).stop 2023-10-10T17:40:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fe540 msgr2=0x7f0d34195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:35.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fe540 0x7f0d34195fd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400d950 tx=0x7f0d2400dc60 comp rx=0 tx=0).stop 2023-10-10T17:40:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 shutdown_connections 2023-10-10T17:40:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d1c0a1150 0x7f0d1c0a3600 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0d34101c40 0x7f0d3419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0d340fe540 0x7f0d34195fd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 --2- 172.21.15.78:0/4216178072 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0d340fee90 0x7f0d34196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 >> 172.21.15.78:0/4216178072 conn(0x7f0d340f9ee0 msgr2=0x7f0d34104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 shutdown_connections 2023-10-10T17:40:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:35.449+0000 7f0d39efb700 1 -- 172.21.15.78:0/4216178072 wait complete. 2023-10-10T17:40:35.453 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 12 2023-10-10T17:40:35.714 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:40:35 smithi157 systemd[1]: Starting Ceph mds.b for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:40:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:35 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4216178072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:35.978 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:40:35 smithi157 podman[126190]: 2023-10-10T17:40:35.979 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:40:35 smithi157 ceph-mds[126230]: -- 172.21.15.157:0/1206246497 <== mon.2 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563297317520 con 0x5632972ed000 2023-10-10T17:40:35.979 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:40:35 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: starting mds.b at 2023-10-10T17:40:35.979 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:40:35 smithi157 bash[126190]: 8b1e2f0706d175750976d029c649ea2d5b91faf7448006986e3c43a4016e420b 2023-10-10T17:40:36.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:35 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4216178072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:36.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:35 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4216178072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:36.477 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:40:36 smithi157 systemd[1]: Started Ceph mds.b for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:40:37.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] up:boot 2023-10-10T17:40:37.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: fsmap 12 up:standby 2023-10-10T17:40:37.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:40:37.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:37.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:37.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] up:boot 2023-10-10T17:40:37.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: fsmap 12 up:standby 2023-10-10T17:40:37.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:40:37.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:37.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:37.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] up:boot 2023-10-10T17:40:37.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: fsmap 12 up:standby 2023-10-10T17:40:37.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:40:37.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:37.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:37.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:37.464 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:37.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 -- 172.21.15.78:0/2240784674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 msgr2=0x7ff9100ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:37.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 --2- 172.21.15.78:0/2240784674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9100ff580 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff904009990 tx=0x7ff904009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 -- 172.21.15.78:0/2240784674 shutdown_connections 2023-10-10T17:40:37.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 --2- 172.21.15.78:0/2240784674 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100db0 0x7ff910107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 --2- 172.21.15.78:0/2240784674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9100ff580 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 --2- 172.21.15.78:0/2240784674 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff910100400 0x7ff910100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 -- 172.21.15.78:0/2240784674 >> 172.21.15.78:0/2240784674 conn(0x7ff9100fa750 msgr2=0x7ff9100fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:37.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 -- 172.21.15.78:0/2240784674 shutdown_connections 2023-10-10T17:40:37.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.594+0000 7ff916b7c700 1 -- 172.21.15.78:0/2240784674 wait complete. 2023-10-10T17:40:37.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 Processor -- start 2023-10-10T17:40:37.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 -- start start 2023-10-10T17:40:37.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9101031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:37.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100400 0x7ff9101017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:37.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff910100db0 0x7ff910101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:37.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9101024c0 con 0x7ff9100ff170 2023-10-10T17:40:37.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff910102630 con 0x7ff910100400 2023-10-10T17:40:37.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff916b7c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91019ffd0 con 0x7ff910100db0 2023-10-10T17:40:37.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9101031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:37.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9101031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42048/0 (socket says 172.21.15.78:42048) 2023-10-10T17:40:37.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 -- 172.21.15.78:0/120799942 learned_addr learned my addr 172.21.15.78:0/120799942 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:37.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 -- 172.21.15.78:0/120799942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff910100db0 msgr2=0x7ff910101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:37.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff915119700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff910100db0 0x7ff910101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:37.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff910100db0 0x7ff910101d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 -- 172.21.15.78:0/120799942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100400 msgr2=0x7ff9101017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:37.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff90ffff700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100400 0x7ff9101017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:37.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100400 0x7ff9101017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 -- 172.21.15.78:0/120799942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9040095f0 con 0x7ff9100ff170 2023-10-10T17:40:37.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff90ffff700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100400 0x7ff9101017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:37.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.595+0000 7ff914918700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9101031a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff904000c00 tx=0x7ff90400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:37.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.596+0000 7ff90dffb700 1 -- 172.21.15.78:0/120799942 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90401d070 con 0x7ff9100ff170 2023-10-10T17:40:37.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.596+0000 7ff90dffb700 1 -- 172.21.15.78:0/120799942 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff90400fc00 con 0x7ff9100ff170 2023-10-10T17:40:37.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.596+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91010ab10 con 0x7ff9100ff170 2023-10-10T17:40:37.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.596+0000 7ff90dffb700 1 -- 172.21.15.78:0/120799942 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904017650 con 0x7ff9100ff170 2023-10-10T17:40:37.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.596+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91010a980 con 0x7ff9100ff170 2023-10-10T17:40:37.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.597+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9101a06b0 con 0x7ff9100ff170 2023-10-10T17:40:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.598+0000 7ff90dffb700 1 -- 172.21.15.78:0/120799942 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff904022470 con 0x7ff9100ff170 2023-10-10T17:40:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.598+0000 7ff90dffb700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9000a11d0 0x7ff9000a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.598+0000 7ff90dffb700 1 -- 172.21.15.78:0/120799942 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9040cd9a0 con 0x7ff9100ff170 2023-10-10T17:40:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.598+0000 7ff90ffff700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9000a11d0 0x7ff9000a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.599+0000 7ff90ffff700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9000a11d0 0x7ff9000a3680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc005fd0 tx=0x7ff8fc005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.602+0000 7ff90dffb700 1 -- 172.21.15.78:0/120799942 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff904097620 con 0x7ff9100ff170 2023-10-10T17:40:37.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.854+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9101a0c60 con 0x7ff9100ff170 2023-10-10T17:40:37.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.855+0000 7ff90dffb700 1 -- 172.21.15.78:0/120799942 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9601 (secure 0 0 0) 0x7ff90409ac40 con 0x7ff9100ff170 2023-10-10T17:40:37.856 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:37.857 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24500,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-10-10T17:40:37.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9000a11d0 msgr2=0x7ff9000a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:37.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9000a11d0 0x7ff9000a3680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc005fd0 tx=0x7ff8fc005ee0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 msgr2=0x7ff9101031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:37.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9101031a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff904000c00 tx=0x7ff90400f710 comp rx=0 tx=0).stop 2023-10-10T17:40:37.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 shutdown_connections 2023-10-10T17:40:37.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9000a11d0 0x7ff9000a3680 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff910100db0 0x7ff910101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9100ff170 0x7ff9101031a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 --2- 172.21.15.78:0/120799942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff910100400 0x7ff9101017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 >> 172.21.15.78:0/120799942 conn(0x7ff9100fa750 msgr2=0x7ff910067e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:37.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 shutdown_connections 2023-10-10T17:40:37.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.857+0000 7ff916b7c700 1 -- 172.21.15.78:0/120799942 wait complete. 2023-10-10T17:40:37.862 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 13 2023-10-10T17:40:37.872 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:37.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 -- 172.21.15.78:0/2983967090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2948101f10 msgr2=0x7f29481087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 --2- 172.21.15.78:0/2983967090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2948101f10 0x7f29481087d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f294000b0a0 tx=0x7f294000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 -- 172.21.15.78:0/2983967090 shutdown_connections 2023-10-10T17:40:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 --2- 172.21.15.78:0/2983967090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2948101f10 0x7f29481087d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 --2- 172.21.15.78:0/2983967090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29480fe5b0 0x7f29480fe9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 --2- 172.21.15.78:0/2983967090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2948101580 0x7f29481019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 -- 172.21.15.78:0/2983967090 >> 172.21.15.78:0/2983967090 conn(0x7f29480f9fb0 msgr2=0x7f29480fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 -- 172.21.15.78:0/2983967090 shutdown_connections 2023-10-10T17:40:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 -- 172.21.15.78:0/2983967090 wait complete. 2023-10-10T17:40:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 Processor -- start 2023-10-10T17:40:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.996+0000 7f295040c700 1 -- start start 2023-10-10T17:40:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f295040c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29480fe5b0 0x7f2948100ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f295040c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2948101580 0x7f29480ff530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f295040c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2948101f10 0x7f29480ffa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f295040c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948100210 con 0x7f29480fe5b0 2023-10-10T17:40:38.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f295040c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2948100380 con 0x7f2948101f10 2023-10-10T17:40:38.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f295040c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29481a0110 con 0x7f2948101580 2023-10-10T17:40:38.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29480fe5b0 0x7f2948100ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29480fe5b0 0x7f2948100ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42062/0 (socket says 172.21.15.78:42062) 2023-10-10T17:40:38.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 -- 172.21.15.78:0/2559821437 learned_addr learned my addr 172.21.15.78:0/2559821437 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:38.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294d9a7700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2948101580 0x7f29480ff530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e9a9700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2948101f10 0x7f29480ffa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 -- 172.21.15.78:0/2559821437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2948101580 msgr2=0x7f29480ff530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2948101580 0x7f29480ff530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 -- 172.21.15.78:0/2559821437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2948101f10 msgr2=0x7f29480ffa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2948101f10 0x7f29480ffa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 -- 172.21.15.78:0/2559821437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2940009d00 con 0x7f29480fe5b0 2023-10-10T17:40:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f294e1a8700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29480fe5b0 0x7f2948100ef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f294400e970 tx=0x7f294400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f293f7fe700 1 -- 172.21.15.78:0/2559821437 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944009790 con 0x7f29480fe5b0 2023-10-10T17:40:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294810bcc0 con 0x7f29480fe5b0 2023-10-10T17:40:38.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f293f7fe700 1 -- 172.21.15.78:0/2559821437 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f294400c7e0 con 0x7f29480fe5b0 2023-10-10T17:40:38.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.997+0000 7f293f7fe700 1 -- 172.21.15.78:0/2559821437 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944003c70 con 0x7f29480fe5b0 2023-10-10T17:40:38.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.998+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294810bb30 con 0x7f29480fe5b0 2023-10-10T17:40:38.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:37.998+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29481a0830 con 0x7f29480fe5b0 2023-10-10T17:40:38.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.000+0000 7f293f7fe700 1 -- 172.21.15.78:0/2559821437 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f294400cc50 con 0x7f29480fe5b0 2023-10-10T17:40:38.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.000+0000 7f293f7fe700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f29340a1150 0x7f29340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.001+0000 7f294d9a7700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f29340a1150 0x7f29340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.001+0000 7f293f7fe700 1 -- 172.21.15.78:0/2559821437 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2944014070 con 0x7f29480fe5b0 2023-10-10T17:40:38.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.001+0000 7f294d9a7700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f29340a1150 0x7f29340a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2938009c60 tx=0x7f2938009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:38.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.005+0000 7f293f7fe700 1 -- 172.21.15.78:0/2559821437 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2944096370 con 0x7f29480fe5b0 2023-10-10T17:40:38.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.204+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2948107210 con 0x7f29480fe5b0 2023-10-10T17:40:38.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.205+0000 7f293f7fe700 1 -- 172.21.15.78:0/2559821437 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f2944099990 con 0x7f29480fe5b0 2023-10-10T17:40:38.206 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:38.206 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":81,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:01.178954+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:38.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f29340a1150 msgr2=0x7f29340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f29340a1150 0x7f29340a3600 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f2938009c60 tx=0x7f2938009260 comp rx=0 tx=0).stop 2023-10-10T17:40:38.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29480fe5b0 msgr2=0x7f2948100ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29480fe5b0 0x7f2948100ef0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f294400e970 tx=0x7f294400ec80 comp rx=0 tx=0).stop 2023-10-10T17:40:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 shutdown_connections 2023-10-10T17:40:38.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f29340a1150 0x7f29340a3600 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2948101f10 0x7f29480ffa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29480fe5b0 0x7f2948100ef0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 --2- 172.21.15.78:0/2559821437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2948101580 0x7f29480ff530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 >> 172.21.15.78:0/2559821437 conn(0x7f29480f9fb0 msgr2=0x7f29480fc180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:38.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 shutdown_connections 2023-10-10T17:40:38.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.207+0000 7f295040c700 1 -- 172.21.15.78:0/2559821437 wait complete. 2023-10-10T17:40:38.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-10T17:40:38.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:40:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 -- 172.21.15.78:0/739611395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 msgr2=0x7f7acc100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 --2- 172.21.15.78:0/739611395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc100800 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7abc009990 tx=0x7f7abc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 -- 172.21.15.78:0/739611395 shutdown_connections 2023-10-10T17:40:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 --2- 172.21.15.78:0/739611395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7acc100d40 0x7f7acc107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 --2- 172.21.15.78:0/739611395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7acc0ff1c0 0x7f7acc0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 --2- 172.21.15.78:0/739611395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc100800 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 -- 172.21.15.78:0/739611395 >> 172.21.15.78:0/739611395 conn(0x7f7acc0fa7c0 msgr2=0x7f7acc0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.357+0000 7f7ad3398700 1 -- 172.21.15.78:0/739611395 shutdown_connections 2023-10-10T17:40:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.358+0000 7f7ad3398700 1 -- 172.21.15.78:0/739611395 wait complete. 2023-10-10T17:40:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.358+0000 7f7ad3398700 1 Processor -- start 2023-10-10T17:40:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.358+0000 7f7ad3398700 1 -- start start 2023-10-10T17:40:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.358+0000 7f7ad3398700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7acc0ff1c0 0x7f7acc194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.358+0000 7f7ad3398700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.358+0000 7f7ad3398700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7acc100d40 0x7f7acc19c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.358+0000 7f7ad3398700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc19cb40 con 0x7f7acc1003b0 2023-10-10T17:40:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad3398700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc19cc80 con 0x7f7acc100d40 2023-10-10T17:40:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad3398700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc19ce10 con 0x7f7acc0ff1c0 2023-10-10T17:40:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc195500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42078/0 (socket says 172.21.15.78:42078) 2023-10-10T17:40:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 -- 172.21.15.78:0/1899365660 learned_addr learned my addr 172.21.15.78:0/1899365660 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad1935700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7acc100d40 0x7f7acc19c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad1134700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7acc0ff1c0 0x7f7acc194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 -- 172.21.15.78:0/1899365660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7acc0ff1c0 msgr2=0x7f7acc194fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7acc0ff1c0 0x7f7acc194fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 -- 172.21.15.78:0/1899365660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7acc100d40 msgr2=0x7f7acc19c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7acc100d40 0x7f7acc19c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 -- 172.21.15.78:0/1899365660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc0095f0 con 0x7f7acc1003b0 2023-10-10T17:40:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad1134700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7acc0ff1c0 0x7f7acc194fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad1935700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7acc100d40 0x7f7acc19c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad0933700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc195500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7abc006010 tx=0x7f7abc003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ac27fc700 1 -- 172.21.15.78:0/1899365660 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc01d070 con 0x7f7acc1003b0 2023-10-10T17:40:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ac27fc700 1 -- 172.21.15.78:0/1899365660 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7abc022470 con 0x7f7acc1003b0 2023-10-10T17:40:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc10ab30 con 0x7f7acc1003b0 2023-10-10T17:40:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ac27fc700 1 -- 172.21.15.78:0/1899365660 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc00f6c0 con 0x7f7acc1003b0 2023-10-10T17:40:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.359+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc10a9a0 con 0x7f7acc1003b0 2023-10-10T17:40:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.360+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc1067d0 con 0x7f7acc1003b0 2023-10-10T17:40:38.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.361+0000 7f7ac27fc700 1 -- 172.21.15.78:0/1899365660 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7abc0225e0 con 0x7f7acc1003b0 2023-10-10T17:40:38.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.362+0000 7f7ac27fc700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ab80a1150 0x7f7ab80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.362+0000 7f7ad1134700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ab80a1150 0x7f7ab80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.362+0000 7f7ac27fc700 1 -- 172.21.15.78:0/1899365660 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7abc0cdaa0 con 0x7f7acc1003b0 2023-10-10T17:40:38.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.362+0000 7f7ad1134700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ab80a1150 0x7f7ab80a3600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8005fd0 tx=0x7f7ac8005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:38.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.365+0000 7f7ac27fc700 1 -- 172.21.15.78:0/1899365660 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7abc097720 con 0x7f7acc1003b0 2023-10-10T17:40:38.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:38 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/120799942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:38.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:38 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/120799942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:38.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.561+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7acc061700 con 0x7f7acc1003b0 2023-10-10T17:40:38.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.561+0000 7f7ac27fc700 1 -- 172.21.15.78:0/1899365660 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7abc09ad40 con 0x7f7acc1003b0 2023-10-10T17:40:38.562 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:40:38.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ab80a1150 msgr2=0x7f7ab80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ab80a1150 0x7f7ab80a3600 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8005fd0 tx=0x7f7ac8005e40 comp rx=0 tx=0).stop 2023-10-10T17:40:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 msgr2=0x7f7acc195500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc195500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7abc006010 tx=0x7f7abc003890 comp rx=0 tx=0).stop 2023-10-10T17:40:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 shutdown_connections 2023-10-10T17:40:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7ab80a1150 0x7f7ab80a3600 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7acc0ff1c0 0x7f7acc194fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7acc100d40 0x7f7acc19c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 --2- 172.21.15.78:0/1899365660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7acc1003b0 0x7f7acc195500 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.563+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 >> 172.21.15.78:0/1899365660 conn(0x7f7acc0fa7c0 msgr2=0x7f7acc103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.564+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 shutdown_connections 2023-10-10T17:40:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.564+0000 7f7ad3398700 1 -- 172.21.15.78:0/1899365660 wait complete. 2023-10-10T17:40:38.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:38 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/120799942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:38.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-10T17:40:38.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 -- 172.21.15.78:0/1600450866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff370100540 msgr2=0x7ff370106de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 --2- 172.21.15.78:0/1600450866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff370100540 0x7ff370106de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff36800b0a0 tx=0x7ff36800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 -- 172.21.15.78:0/1600450866 shutdown_connections 2023-10-10T17:40:38.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 --2- 172.21.15.78:0/1600450866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3700fe900 0x7ff3700fed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 --2- 172.21.15.78:0/1600450866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff370100540 0x7ff370106de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 --2- 172.21.15.78:0/1600450866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff3700ffb90 0x7ff370100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 -- 172.21.15.78:0/1600450866 >> 172.21.15.78:0/1600450866 conn(0x7ff3700f9ee0 msgr2=0x7ff3700fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:38.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 -- 172.21.15.78:0/1600450866 shutdown_connections 2023-10-10T17:40:38.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.700+0000 7ff377d4b700 1 -- 172.21.15.78:0/1600450866 wait complete. 2023-10-10T17:40:38.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 Processor -- start 2023-10-10T17:40:38.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 -- start start 2023-10-10T17:40:38.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3700fe900 0x7ff370196090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3700ffb90 0x7ff3701965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff370100540 0x7ff37019d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37019dc10 con 0x7ff3700fe900 2023-10-10T17:40:38.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37019dd50 con 0x7ff370100540 2023-10-10T17:40:38.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff377d4b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37019dee0 con 0x7ff3700ffb90 2023-10-10T17:40:38.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff375ae7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3700fe900 0x7ff370196090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff375ae7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3700fe900 0x7ff370196090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42090/0 (socket says 172.21.15.78:42090) 2023-10-10T17:40:38.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff375ae7700 1 -- 172.21.15.78:0/3710031449 learned_addr learned my addr 172.21.15.78:0/3710031449 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:38.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff3762e8700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff370100540 0x7ff37019d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff3752e6700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3700ffb90 0x7ff3701965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff375ae7700 1 -- 172.21.15.78:0/3710031449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3700ffb90 msgr2=0x7ff3701965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff375ae7700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3700ffb90 0x7ff3701965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff375ae7700 1 -- 172.21.15.78:0/3710031449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff370100540 msgr2=0x7ff37019d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.701+0000 7ff375ae7700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff370100540 0x7ff37019d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.702+0000 7ff375ae7700 1 -- 172.21.15.78:0/3710031449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368009d00 con 0x7ff3700fe900 2023-10-10T17:40:38.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.702+0000 7ff375ae7700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3700fe900 0x7ff370196090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff36000d950 tx=0x7ff36000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:38.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.702+0000 7ff366ffd700 1 -- 172.21.15.78:0/3710031449 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360009940 con 0x7ff3700fe900 2023-10-10T17:40:38.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.702+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37010bcd0 con 0x7ff3700fe900 2023-10-10T17:40:38.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.702+0000 7ff366ffd700 1 -- 172.21.15.78:0/3710031449 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff360010460 con 0x7ff3700fe900 2023-10-10T17:40:38.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.702+0000 7ff366ffd700 1 -- 172.21.15.78:0/3710031449 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36000f620 con 0x7ff3700fe900 2023-10-10T17:40:38.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.702+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37010bb40 con 0x7ff3700fe900 2023-10-10T17:40:38.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.703+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37019e620 con 0x7ff3700fe900 2023-10-10T17:40:38.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.704+0000 7ff366ffd700 1 -- 172.21.15.78:0/3710031449 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff3600108d0 con 0x7ff3700fe900 2023-10-10T17:40:38.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.704+0000 7ff366ffd700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff35c0a1150 0x7ff35c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:38.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.704+0000 7ff3752e6700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff35c0a1150 0x7ff35c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:38.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.704+0000 7ff366ffd700 1 -- 172.21.15.78:0/3710031449 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff3600cc300 con 0x7ff3700fe900 2023-10-10T17:40:38.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.705+0000 7ff3752e6700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff35c0a1150 0x7ff35c0a3600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff36c005950 tx=0x7ff36c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:38.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.708+0000 7ff366ffd700 1 -- 172.21.15.78:0/3710031449 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff360095ef0 con 0x7ff3700fe900 2023-10-10T17:40:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.916+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff370105800 con 0x7ff3700fe900 2023-10-10T17:40:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.916+0000 7ff366ffd700 1 -- 172.21.15.78:0/3710031449 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff360099510 con 0x7ff3700fe900 2023-10-10T17:40:38.917 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:38.918 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2023-10-10T17:40:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:40:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2023-10-10T17:40:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2023-10-10T17:40:38.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:40:38.919 INFO:teuthology.orchestra.run.smithi078.stdout:] 2023-10-10T17:40:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff35c0a1150 msgr2=0x7ff35c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff35c0a1150 0x7ff35c0a3600 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff36c005950 tx=0x7ff36c0058e0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3700fe900 msgr2=0x7ff370196090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3700fe900 0x7ff370196090 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff36000d950 tx=0x7ff36000dc60 comp rx=0 tx=0).stop 2023-10-10T17:40:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 shutdown_connections 2023-10-10T17:40:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff35c0a1150 0x7ff35c0a3600 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff370100540 0x7ff37019d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff3700fe900 0x7ff370196090 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 --2- 172.21.15.78:0/3710031449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff3700ffb90 0x7ff3701965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 >> 172.21.15.78:0/3710031449 conn(0x7ff3700f9ee0 msgr2=0x7ff3700fc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 shutdown_connections 2023-10-10T17:40:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:38.919+0000 7ff377d4b700 1 -- 172.21.15.78:0/3710031449 wait complete. 2023-10-10T17:40:38.932 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-10T17:40:38.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-10T17:40:39.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 -- 172.21.15.78:0/4227955934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac105fe0 msgr2=0x7f0dac1083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:39.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 --2- 172.21.15.78:0/4227955934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac105fe0 0x7f0dac1083e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0da400b0a0 tx=0x7f0da400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:39.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 -- 172.21.15.78:0/4227955934 shutdown_connections 2023-10-10T17:40:39.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 --2- 172.21.15.78:0/4227955934 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac105fe0 0x7f0dac1083e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:39.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 --2- 172.21.15.78:0/4227955934 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dac100da0 0x7f0dac103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:39.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 --2- 172.21.15.78:0/4227955934 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dac1036c0 0x7f0dac105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:39.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 -- 172.21.15.78:0/4227955934 >> 172.21.15.78:0/4227955934 conn(0x7f0dac0fa7c0 msgr2=0x7f0dac0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:39.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 -- 172.21.15.78:0/4227955934 shutdown_connections 2023-10-10T17:40:39.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.060+0000 7f0db09d8700 1 -- 172.21.15.78:0/4227955934 wait complete. 2023-10-10T17:40:39.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 Processor -- start 2023-10-10T17:40:39.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 -- start start 2023-10-10T17:40:39.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac100da0 0x7f0dac196230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:39.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dac1036c0 0x7f0dac196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:39.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dac105fe0 0x7f0dac19d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac19ddb0 con 0x7f0dac100da0 2023-10-10T17:40:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac19def0 con 0x7f0dac1036c0 2023-10-10T17:40:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0db09d8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac19e080 con 0x7f0dac105fe0 2023-10-10T17:40:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac100da0 0x7f0dac196230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac100da0 0x7f0dac196230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42102/0 (socket says 172.21.15.78:42102) 2023-10-10T17:40:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 -- 172.21.15.78:0/1859847754 learned_addr learned my addr 172.21.15.78:0/1859847754 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0da9d9b700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dac1036c0 0x7f0dac196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 -- 172.21.15.78:0/1859847754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dac105fe0 msgr2=0x7f0dac19d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dac105fe0 0x7f0dac19d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 -- 172.21.15.78:0/1859847754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dac1036c0 msgr2=0x7f0dac196770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dac1036c0 0x7f0dac196770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0daa59c700 1 -- 172.21.15.78:0/1859847754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da4009d00 con 0x7f0dac100da0 2023-10-10T17:40:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.061+0000 7f0da9d9b700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dac1036c0 0x7f0dac196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.062+0000 7f0daa59c700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac100da0 0x7f0dac196230 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0da000d950 tx=0x7f0da000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.062+0000 7f0d9b7fe700 1 -- 172.21.15.78:0/1859847754 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0009b40 con 0x7f0dac100da0 2023-10-10T17:40:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.062+0000 7f0d9b7fe700 1 -- 172.21.15.78:0/1859847754 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0da0009ca0 con 0x7f0dac100da0 2023-10-10T17:40:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.062+0000 7f0d9b7fe700 1 -- 172.21.15.78:0/1859847754 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da000f620 con 0x7f0dac100da0 2023-10-10T17:40:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.062+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac10bcd0 con 0x7f0dac100da0 2023-10-10T17:40:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.062+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dac10bb40 con 0x7f0dac100da0 2023-10-10T17:40:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.063+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dac0fc330 con 0x7f0dac100da0 2023-10-10T17:40:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.064+0000 7f0d9b7fe700 1 -- 172.21.15.78:0/1859847754 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0da0010460 con 0x7f0dac100da0 2023-10-10T17:40:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.064+0000 7f0d9b7fe700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d940a11d0 0x7f0d940a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.064+0000 7f0d9b7fe700 1 -- 172.21.15.78:0/1859847754 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0da00cc890 con 0x7f0dac100da0 2023-10-10T17:40:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.064+0000 7f0da9d9b700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d940a11d0 0x7f0d940a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.065+0000 7f0da9d9b700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d940a11d0 0x7f0d940a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0099d0 tx=0x7f0d9c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.068+0000 7f0d9b7fe700 1 -- 172.21.15.78:0/1859847754 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0da00963e0 con 0x7f0dac100da0 2023-10-10T17:40:39.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:39.272+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0dac02cfe0 con 0x7f0dac100da0 2023-10-10T17:40:39.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2559821437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:39.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:39.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1899365660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:39.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3710031449' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-10-10T17:40:39.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:39 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2559821437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:39.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:39.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1899365660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:39.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3710031449' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-10-10T17:40:39.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:39 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2559821437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:39.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:39.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1899365660' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:39.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3710031449' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-10-10T17:40:39.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:39.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:39 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.146+0000 7f0d9b7fe700 1 -- 172.21.15.78:0/1859847754 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0da0099a00 con 0x7f0dac100da0 2023-10-10T17:40:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d940a11d0 msgr2=0x7f0d940a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:40.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d940a11d0 0x7f0d940a3680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c0099d0 tx=0x7f0d9c008040 comp rx=0 tx=0).stop 2023-10-10T17:40:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac100da0 msgr2=0x7f0dac196230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:40.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac100da0 0x7f0dac196230 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0da000d950 tx=0x7f0da000dd10 comp rx=0 tx=0).stop 2023-10-10T17:40:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 shutdown_connections 2023-10-10T17:40:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0d940a11d0 0x7f0d940a3680 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0dac105fe0 0x7f0dac19d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0dac100da0 0x7f0dac196230 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0da000d950 tx=0x7f0da000dd10 comp rx=0 tx=0).stop 2023-10-10T17:40:40.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 --2- 172.21.15.78:0/1859847754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0dac1036c0 0x7f0dac196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 >> 172.21.15.78:0/1859847754 conn(0x7f0dac0fa7c0 msgr2=0x7f0dac101aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:40.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.150+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 shutdown_connections 2023-10-10T17:40:40.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.151+0000 7f0db09d8700 1 -- 172.21.15.78:0/1859847754 wait complete. 2023-10-10T17:40:40.154 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2023-10-10T17:40:40.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-10-10T17:40:40.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 -- 172.21.15.78:0/2050902115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 msgr2=0x7f9bac105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/2050902115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 0x7f9bac105a40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c009990 tx=0x7f9b9c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 -- 172.21.15.78:0/2050902115 shutdown_connections 2023-10-10T17:40:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/2050902115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 0x7f9bac1083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/2050902115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bac100d40 0x7f9bac103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/2050902115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 0x7f9bac105a40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 -- 172.21.15.78:0/2050902115 >> 172.21.15.78:0/2050902115 conn(0x7f9bac0fa7a0 msgr2=0x7f9bac0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:40.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 -- 172.21.15.78:0/2050902115 shutdown_connections 2023-10-10T17:40:40.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.303+0000 7f9bb15e8700 1 -- 172.21.15.78:0/2050902115 wait complete. 2023-10-10T17:40:40.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 Processor -- start 2023-10-10T17:40:40.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 -- start start 2023-10-10T17:40:40.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bac100d40 0x7f9bac195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:40.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 0x7f9bac196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:40.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 0x7f9bac19d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:40.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac19db70 con 0x7f9bac105f80 2023-10-10T17:40:40.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac19dcb0 con 0x7f9bac100d40 2023-10-10T17:40:40.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bb15e8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac19de40 con 0x7f9bac103660 2023-10-10T17:40:40.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 0x7f9bac19d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:40.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 0x7f9bac19d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42120/0 (socket says 172.21.15.78:42120) 2023-10-10T17:40:40.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bab7fe700 1 -- 172.21.15.78:0/275380693 learned_addr learned my addr 172.21.15.78:0/275380693 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:40.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9baa7fc700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 0x7f9bac196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:40.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9baaffd700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bac100d40 0x7f9bac195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:40.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.304+0000 7f9bab7fe700 1 -- 172.21.15.78:0/275380693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 msgr2=0x7f9bac196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:40.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9bab7fe700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 0x7f9bac196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9bab7fe700 1 -- 172.21.15.78:0/275380693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bac100d40 msgr2=0x7f9bac195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:40.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9bab7fe700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bac100d40 0x7f9bac195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:40.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9bab7fe700 1 -- 172.21.15.78:0/275380693 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c0095f0 con 0x7f9bac105f80 2023-10-10T17:40:40.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9baa7fc700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 0x7f9bac196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:40:40.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9bab7fe700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 0x7f9bac19d5a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400ef30 tx=0x7f9ba400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:40.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9b93fff700 1 -- 172.21.15.78:0/275380693 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba4018070 con 0x7f9bac105f80 2023-10-10T17:40:40.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac10bc80 con 0x7f9bac105f80 2023-10-10T17:40:40.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9b93fff700 1 -- 172.21.15.78:0/275380693 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ba400f040 con 0x7f9bac105f80 2023-10-10T17:40:40.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9b93fff700 1 -- 172.21.15.78:0/275380693 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba400cd00 con 0x7f9bac105f80 2023-10-10T17:40:40.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.305+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac10baf0 con 0x7f9bac105f80 2023-10-10T17:40:40.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.306+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac19e4d0 con 0x7f9bac105f80 2023-10-10T17:40:40.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.307+0000 7f9b93fff700 1 -- 172.21.15.78:0/275380693 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9ba4003a00 con 0x7f9bac105f80 2023-10-10T17:40:40.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.307+0000 7f9b93fff700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9b940a11d0 0x7f9b940a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:40.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.308+0000 7f9baaffd700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9b940a11d0 0x7f9b940a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:40.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.308+0000 7f9b93fff700 1 -- 172.21.15.78:0/275380693 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f9ba40ccb80 con 0x7f9bac105f80 2023-10-10T17:40:40.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.308+0000 7f9baaffd700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9b940a11d0 0x7f9b940a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009c60 tx=0x7f9ba0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:40.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.311+0000 7f9b93fff700 1 -- 172.21.15.78:0/275380693 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9ba40965f0 con 0x7f9bac105f80 2023-10-10T17:40:40.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:40 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1859847754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-10-10T17:40:40.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:40 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:40.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:40 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:40.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:40 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:40.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:40 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:40.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:40 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1859847754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-10-10T17:40:40.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:40 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:40.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:40 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:40.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:40 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:40.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:40 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:40.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:40.506+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f9bac19eb70 con 0x7f9bac105f80 2023-10-10T17:40:40.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:40 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1859847754' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-10-10T17:40:40.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:40.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:40.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:40.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:40 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:40:41.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.156+0000 7f9b93fff700 1 -- 172.21.15.78:0/275380693 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f9ba4099c10 con 0x7f9bac105f80 2023-10-10T17:40:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9b940a11d0 msgr2=0x7f9b940a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9b940a11d0 0x7f9b940a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009c60 tx=0x7f9ba0009260 comp rx=0 tx=0).stop 2023-10-10T17:40:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 msgr2=0x7f9bac19d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 0x7f9bac19d5a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9ba400ef30 tx=0x7f9ba400c3a0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 shutdown_connections 2023-10-10T17:40:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9b940a11d0 0x7f9b940a3680 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9bac105f80 0x7f9bac19d5a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9bac100d40 0x7f9bac195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 --2- 172.21.15.78:0/275380693 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9bac103660 0x7f9bac196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 >> 172.21.15.78:0/275380693 conn(0x7f9bac0fa7a0 msgr2=0x7f9bac102180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:41.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 shutdown_connections 2023-10-10T17:40:41.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.171+0000 7f9bb15e8700 1 -- 172.21.15.78:0/275380693 wait complete. 2023-10-10T17:40:41.174 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2023-10-10T17:40:41.184 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-10T17:40:41.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 -- 172.21.15.78:0/2142315261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8105fe0 msgr2=0x7f62b81083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 --2- 172.21.15.78:0/2142315261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8105fe0 0x7f62b81083e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f62b000b0a0 tx=0x7f62b000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 -- 172.21.15.78:0/2142315261 shutdown_connections 2023-10-10T17:40:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 --2- 172.21.15.78:0/2142315261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62b8100da0 0x7f62b8103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 --2- 172.21.15.78:0/2142315261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8105fe0 0x7f62b81083e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 --2- 172.21.15.78:0/2142315261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62b81036c0 0x7f62b8105aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 -- 172.21.15.78:0/2142315261 >> 172.21.15.78:0/2142315261 conn(0x7f62b80fa7c0 msgr2=0x7f62b80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 -- 172.21.15.78:0/2142315261 shutdown_connections 2023-10-10T17:40:41.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.319+0000 7f62bcc99700 1 -- 172.21.15.78:0/2142315261 wait complete. 2023-10-10T17:40:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 Processor -- start 2023-10-10T17:40:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 -- start start 2023-10-10T17:40:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8100da0 0x7f62b8196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62b81036c0 0x7f62b8196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62b8105fe0 0x7f62b819d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819dcc0 con 0x7f62b8100da0 2023-10-10T17:40:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819de00 con 0x7f62b8105fe0 2023-10-10T17:40:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62bcc99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b819df90 con 0x7f62b81036c0 2023-10-10T17:40:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8100da0 0x7f62b8196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8100da0 0x7f62b8196140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42128/0 (socket says 172.21.15.78:42128) 2023-10-10T17:40:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 -- 172.21.15.78:0/3306114666 learned_addr learned my addr 172.21.15.78:0/3306114666 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b6d9d700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62b8105fe0 0x7f62b819d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b5d9b700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62b81036c0 0x7f62b8196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 -- 172.21.15.78:0/3306114666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62b81036c0 msgr2=0x7f62b8196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62b81036c0 0x7f62b8196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 -- 172.21.15.78:0/3306114666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62b8105fe0 msgr2=0x7f62b819d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62b8105fe0 0x7f62b819d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 -- 172.21.15.78:0/3306114666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0009d00 con 0x7f62b8100da0 2023-10-10T17:40:41.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.320+0000 7f62b659c700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8100da0 0x7f62b8196140 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00d980 tx=0x7f62ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:41.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.321+0000 7f62a77fe700 1 -- 172.21.15.78:0/3306114666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac004220 con 0x7f62b8100da0 2023-10-10T17:40:41.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.321+0000 7f62a77fe700 1 -- 172.21.15.78:0/3306114666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62ac007640 con 0x7f62b8100da0 2023-10-10T17:40:41.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.321+0000 7f62a77fe700 1 -- 172.21.15.78:0/3306114666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac005610 con 0x7f62b8100da0 2023-10-10T17:40:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.321+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b810bc20 con 0x7f62b8100da0 2023-10-10T17:40:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.321+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b810ba90 con 0x7f62b8100da0 2023-10-10T17:40:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.321+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b80fc330 con 0x7f62b8100da0 2023-10-10T17:40:41.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.322+0000 7f62a77fe700 1 -- 172.21.15.78:0/3306114666 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f62ac004750 con 0x7f62b8100da0 2023-10-10T17:40:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.323+0000 7f62a77fe700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62a00a1150 0x7f62a00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.323+0000 7f62b5d9b700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62a00a1150 0x7f62a00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.323+0000 7f62a77fe700 1 -- 172.21.15.78:0/3306114666 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f62ac0ccb00 con 0x7f62b8100da0 2023-10-10T17:40:41.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.323+0000 7f62b5d9b700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62a00a1150 0x7f62a00a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009bd0 tx=0x7f62a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:41.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.327+0000 7f62a77fe700 1 -- 172.21.15.78:0/3306114666 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62ac096400 con 0x7f62b8100da0 2023-10-10T17:40:41.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:41.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:40:41.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:41.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:41.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1859847754' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-10-10T17:40:41.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: osdmap e82: 12 total, 12 up, 12 in 2023-10-10T17:40:41.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: pgmap v254: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:41.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:41 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/275380693' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-10T17:40:41.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:41.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:40:41.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:41.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:41.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1859847754' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-10-10T17:40:41.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: osdmap e82: 12 total, 12 up, 12 in 2023-10-10T17:40:41.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: pgmap v254: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:41.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:41 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/275380693' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-10T17:40:41.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.572+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f62b80618d0 con 0x7f62b8100da0 2023-10-10T17:40:41.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.581+0000 7f62a77fe700 1 -- 172.21.15.78:0/3306114666 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f62ac0160c0 con 0x7f62b8100da0 2023-10-10T17:40:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.585+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62a00a1150 msgr2=0x7f62a00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.585+0000 7f62bcc99700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62a00a1150 0x7f62a00a3600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009bd0 tx=0x7f62a8009260 comp rx=0 tx=0).stop 2023-10-10T17:40:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.585+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8100da0 msgr2=0x7f62b8196140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.585+0000 7f62bcc99700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8100da0 0x7f62b8196140 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62ac00d980 tx=0x7f62ac00dc90 comp rx=0 tx=0).stop 2023-10-10T17:40:41.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.586+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 shutdown_connections 2023-10-10T17:40:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.586+0000 7f62bcc99700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62a00a1150 0x7f62a00a3600 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009bd0 tx=0x7f62a8009260 comp rx=0 tx=0).stop 2023-10-10T17:40:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.586+0000 7f62bcc99700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62b8105fe0 0x7f62b819d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.586+0000 7f62bcc99700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62b8100da0 0x7f62b8196140 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.586+0000 7f62bcc99700 1 --2- 172.21.15.78:0/3306114666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f62b81036c0 0x7f62b8196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.586+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 >> 172.21.15.78:0/3306114666 conn(0x7f62b80fa7c0 msgr2=0x7f62b8101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:41.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.602+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 shutdown_connections 2023-10-10T17:40:41.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.602+0000 7f62bcc99700 1 -- 172.21.15.78:0/3306114666 wait complete. 2023-10-10T17:40:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2023-10-10T17:40:41.623 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:41.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:40:41.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:41.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:41.624 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1859847754' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-10-10T17:40:41.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: osdmap e82: 12 total, 12 up, 12 in 2023-10-10T17:40:41.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: pgmap v254: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:40:41.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:41 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/275380693' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-10-10T17:40:41.626 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-10-10T17:40:41.626 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-10-10T17:40:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 -- 172.21.15.78:0/1946233776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904105500 msgr2=0x7f39041078e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 --2- 172.21.15.78:0/1946233776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904105500 0x7f39041078e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009990 tx=0x7f38f0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 -- 172.21.15.78:0/1946233776 shutdown_connections 2023-10-10T17:40:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 --2- 172.21.15.78:0/1946233776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904107e20 0x7f390410a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 --2- 172.21.15.78:0/1946233776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3904069ca0 0x7f3904104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 --2- 172.21.15.78:0/1946233776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904105500 0x7f39041078e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 -- 172.21.15.78:0/1946233776 >> 172.21.15.78:0/1946233776 conn(0x7f39040fa570 msgr2=0x7f39040fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 -- 172.21.15.78:0/1946233776 shutdown_connections 2023-10-10T17:40:41.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.757+0000 7f3909b9f700 1 -- 172.21.15.78:0/1946233776 wait complete. 2023-10-10T17:40:41.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 Processor -- start 2023-10-10T17:40:41.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 -- start start 2023-10-10T17:40:41.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904069ca0 0x7f3904197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3904105500 0x7f39041984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904107e20 0x7f390419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390419fb20 con 0x7f3904107e20 2023-10-10T17:40:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390419fc60 con 0x7f3904105500 2023-10-10T17:40:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3909b9f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390419fdf0 con 0x7f3904069ca0 2023-10-10T17:40:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904107e20 0x7f390419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904107e20 0x7f390419f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42154/0 (socket says 172.21.15.78:42154) 2023-10-10T17:40:41.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 -- 172.21.15.78:0/197780828 learned_addr learned my addr 172.21.15.78:0/197780828 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3903fff700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3904105500 0x7f39041984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f3908b9d700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904069ca0 0x7f3904197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 -- 172.21.15.78:0/197780828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904069ca0 msgr2=0x7f3904197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904069ca0 0x7f3904197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 -- 172.21.15.78:0/197780828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3904105500 msgr2=0x7f39041984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:41.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3904105500 0x7f39041984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:41.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.758+0000 7f390939e700 1 -- 172.21.15.78:0/197780828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f00095f0 con 0x7f3904107e20 2023-10-10T17:40:41.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.759+0000 7f390939e700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904107e20 0x7f390419f550 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00ada0 tx=0x7f38fc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:41.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.759+0000 7f3901ffb700 1 -- 172.21.15.78:0/197780828 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00f070 con 0x7f3904107e20 2023-10-10T17:40:41.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.759+0000 7f3901ffb700 1 -- 172.21.15.78:0/197780828 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38fc007b30 con 0x7f3904107e20 2023-10-10T17:40:41.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.759+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390410dbc0 con 0x7f3904107e20 2023-10-10T17:40:41.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.759+0000 7f3901ffb700 1 -- 172.21.15.78:0/197780828 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc005280 con 0x7f3904107e20 2023-10-10T17:40:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.759+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f390410da30 con 0x7f3904107e20 2023-10-10T17:40:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.760+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39040fc0e0 con 0x7f3904107e20 2023-10-10T17:40:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.762+0000 7f3901ffb700 1 -- 172.21.15.78:0/197780828 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f38fc00c8d0 con 0x7f3904107e20 2023-10-10T17:40:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.762+0000 7f3901ffb700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f38f40a1150 0x7f38f40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:41.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.762+0000 7f3901ffb700 1 -- 172.21.15.78:0/197780828 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f38fc0cbca0 con 0x7f3904107e20 2023-10-10T17:40:41.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.762+0000 7f3908b9d700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f38f40a1150 0x7f38f40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:41.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.763+0000 7f3908b9d700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f38f40a1150 0x7f38f40a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f39040699c0 tx=0x7f38f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:41.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.767+0000 7f3901ffb700 1 -- 172.21.15.78:0/197780828 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f38fc099060 con 0x7f3904107e20 2023-10-10T17:40:41.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:41.987+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f390402cfe0 con 0x7f3904107e20 2023-10-10T17:40:42.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/275380693' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-10-10T17:40:42.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: osdmap e83: 12 total, 12 up, 12 in 2023-10-10T17:40:42.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3306114666' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-10-10T17:40:42.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-10-10T17:40:42.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: osdmap e84: 12 total, 12 up, 12 in 2023-10-10T17:40:42.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3306114666' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-10-10T17:40:42.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-10-10T17:40:42.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-10-10T17:40:42.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:42 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/197780828' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-10T17:40:42.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/275380693' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-10-10T17:40:42.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: osdmap e83: 12 total, 12 up, 12 in 2023-10-10T17:40:42.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3306114666' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-10-10T17:40:42.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-10-10T17:40:42.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: osdmap e84: 12 total, 12 up, 12 in 2023-10-10T17:40:42.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3306114666' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-10-10T17:40:42.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-10-10T17:40:42.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-10-10T17:40:42.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:42 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/197780828' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-10T17:40:42.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.582+0000 7f3901ffb700 1 -- 172.21.15.78:0/197780828 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7f38fc098bf0 con 0x7f3904107e20 2023-10-10T17:40:42.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f38f40a1150 msgr2=0x7f38f40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:42.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f38f40a1150 0x7f38f40a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f39040699c0 tx=0x7f38f8009260 comp rx=0 tx=0).stop 2023-10-10T17:40:42.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904107e20 msgr2=0x7f390419f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:42.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904107e20 0x7f390419f550 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00ada0 tx=0x7f38fc00c330 comp rx=0 tx=0).stop 2023-10-10T17:40:42.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 shutdown_connections 2023-10-10T17:40:42.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f38f40a1150 0x7f38f40a3600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3904069ca0 0x7f3904197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3904107e20 0x7f390419f550 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 --2- 172.21.15.78:0/197780828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3904105500 0x7f39041984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 >> 172.21.15.78:0/197780828 conn(0x7f39040fa570 msgr2=0x7f3904106160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:42.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 shutdown_connections 2023-10-10T17:40:42.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.584+0000 7f3909b9f700 1 -- 172.21.15.78:0/197780828 wait complete. 2023-10-10T17:40:42.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/275380693' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-10-10T17:40:42.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: osdmap e83: 12 total, 12 up, 12 in 2023-10-10T17:40:42.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3306114666' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-10-10T17:40:42.598 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-10-10T17:40:42.599 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: osdmap e84: 12 total, 12 up, 12 in 2023-10-10T17:40:42.599 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3306114666' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-10-10T17:40:42.599 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-10-10T17:40:42.600 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-10-10T17:40:42.600 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/197780828' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-10-10T17:40:42.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-10-10T17:40:42.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 -- 172.21.15.78:0/3527895786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 msgr2=0x7fbb6c100700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:42.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 --2- 172.21.15.78:0/3527895786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c100700 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c009990 tx=0x7fbb5c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 -- 172.21.15.78:0/3527895786 shutdown_connections 2023-10-10T17:40:42.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 --2- 172.21.15.78:0/3527895786 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c101e70 0x7fbb6c108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 --2- 172.21.15.78:0/3527895786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c100700 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 --2- 172.21.15.78:0/3527895786 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb6c1014e0 0x7fbb6c101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 -- 172.21.15.78:0/3527895786 >> 172.21.15.78:0/3527895786 conn(0x7fbb6c0fb8d0 msgr2=0x7fbb6c0fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:42.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 -- 172.21.15.78:0/3527895786 shutdown_connections 2023-10-10T17:40:42.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.742+0000 7fbb7125e700 1 -- 172.21.15.78:0/3527895786 wait complete. 2023-10-10T17:40:42.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.743+0000 7fbb7125e700 1 Processor -- start 2023-10-10T17:40:42.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.743+0000 7fbb7125e700 1 -- start start 2023-10-10T17:40:42.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb7125e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c0748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:42.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb7125e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c1014e0 0x7fbb6c072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:42.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb7125e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb6c101e70 0x7fbb6c073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:42.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb7125e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c073bc0 con 0x7fbb6c1002f0 2023-10-10T17:40:42.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb7125e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c073d30 con 0x7fbb6c1014e0 2023-10-10T17:40:42.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb7125e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c1a0130 con 0x7fbb6c101e70 2023-10-10T17:40:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c0748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c0748a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42164/0 (socket says 172.21.15.78:42164) 2023-10-10T17:40:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 -- 172.21.15.78:0/1587788100 learned_addr learned my addr 172.21.15.78:0/1587788100 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6b59e700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb6c101e70 0x7fbb6c073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:42.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6a59c700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c1014e0 0x7fbb6c072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 -- 172.21.15.78:0/1587788100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb6c101e70 msgr2=0x7fbb6c073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb6c101e70 0x7fbb6c073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 -- 172.21.15.78:0/1587788100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c1014e0 msgr2=0x7fbb6c072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:42.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c1014e0 0x7fbb6c072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:42.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 -- 172.21.15.78:0/1587788100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c0095f0 con 0x7fbb6c1002f0 2023-10-10T17:40:42.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb6ad9d700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c0748a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c000c00 tx=0x7fbb5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb53fff700 1 -- 172.21.15.78:0/1587788100 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c01d070 con 0x7fbb6c1002f0 2023-10-10T17:40:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb53fff700 1 -- 172.21.15.78:0/1587788100 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb5c022470 con 0x7fbb6c1002f0 2023-10-10T17:40:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb53fff700 1 -- 172.21.15.78:0/1587788100 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c017650 con 0x7fbb6c1002f0 2023-10-10T17:40:42.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.744+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c10bce0 con 0x7fbb6c1002f0 2023-10-10T17:40:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.745+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c10bb50 con 0x7fbb6c1002f0 2023-10-10T17:40:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.746+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c1a0750 con 0x7fbb6c1002f0 2023-10-10T17:40:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.747+0000 7fbb53fff700 1 -- 172.21.15.78:0/1587788100 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbb5c0225e0 con 0x7fbb6c1002f0 2023-10-10T17:40:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.747+0000 7fbb53fff700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb540a11d0 0x7fbb540a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.747+0000 7fbb6a59c700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb540a11d0 0x7fbb540a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:42.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.747+0000 7fbb53fff700 1 -- 172.21.15.78:0/1587788100 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fbb5c0cddd0 con 0x7fbb6c1002f0 2023-10-10T17:40:42.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.747+0000 7fbb6a59c700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb540a11d0 0x7fbb540a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009c60 tx=0x7fbb60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:42.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.751+0000 7fbb53fff700 1 -- 172.21.15.78:0/1587788100 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbb5c097570 con 0x7fbb6c1002f0 2023-10-10T17:40:42.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:42.956+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbb6c1071c0 con 0x7fbb6c1002f0 2023-10-10T17:40:43.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: pgmap v257: 97 pgs: 20 creating+peering, 13 active+clean, 64 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2023-10-10T17:40:43.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] up:active 2023-10-10T17:40:43.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:43.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/197780828' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-10-10T17:40:43.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: osdmap e85: 12 total, 12 up, 12 in 2023-10-10T17:40:43.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:43.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:43.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:43.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:43 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1587788100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-10T17:40:43.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: pgmap v257: 97 pgs: 20 creating+peering, 13 active+clean, 64 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2023-10-10T17:40:43.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] up:active 2023-10-10T17:40:43.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:43.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/197780828' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-10-10T17:40:43.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: osdmap e85: 12 total, 12 up, 12 in 2023-10-10T17:40:43.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:43.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:43.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:43.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:43 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1587788100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-10T17:40:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: pgmap v257: 97 pgs: 20 creating+peering, 13 active+clean, 64 unknown; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 3 op/s 2023-10-10T17:40:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] up:active 2023-10-10T17:40:43.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:43.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/197780828' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-10-10T17:40:43.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: osdmap e85: 12 total, 12 up, 12 in 2023-10-10T17:40:43.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:43.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:40:43.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:40:43.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:40:43.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:43 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1587788100' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-10-10T17:40:44.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.694+0000 7fbb53fff700 1 -- 172.21.15.78:0/1587788100 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7fbb5c09ab90 con 0x7fbb6c1002f0 2023-10-10T17:40:44.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.700+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb540a11d0 msgr2=0x7fbb540a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:44.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.700+0000 7fbb7125e700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb540a11d0 0x7fbb540a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009c60 tx=0x7fbb60009260 comp rx=0 tx=0).stop 2023-10-10T17:40:44.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.700+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 msgr2=0x7fbb6c0748a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:44.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.700+0000 7fbb7125e700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c0748a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c000c00 tx=0x7fbb5c004730 comp rx=0 tx=0).stop 2023-10-10T17:40:44.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 shutdown_connections 2023-10-10T17:40:44.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb540a11d0 0x7fbb540a3680 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb6c101e70 0x7fbb6c073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb6c1002f0 0x7fbb6c0748a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 --2- 172.21.15.78:0/1587788100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb6c1014e0 0x7fbb6c072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 >> 172.21.15.78:0/1587788100 conn(0x7fbb6c0fb8d0 msgr2=0x7fbb6c0fdb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:44.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 shutdown_connections 2023-10-10T17:40:44.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.702+0000 7fbb7125e700 1 -- 172.21.15.78:0/1587788100 wait complete. 2023-10-10T17:40:44.704 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' created 2023-10-10T17:40:44.717 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-10-10T17:40:44.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.847+0000 7fbb15b88700 1 -- 172.21.15.78:0/2271946105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb10100d40 msgr2=0x7fbb10107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.847+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2271946105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb10100d40 0x7fbb10107600 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbb0800b0a0 tx=0x7fbb0800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.847+0000 7fbb15b88700 1 -- 172.21.15.78:0/2271946105 shutdown_connections 2023-10-10T17:40:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.847+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2271946105 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb100ff1c0 0x7fbb100ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.847+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2271946105 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb10100d40 0x7fbb10107600 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.847+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2271946105 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb101003b0 0x7fbb10100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.848+0000 7fbb15b88700 1 -- 172.21.15.78:0/2271946105 >> 172.21.15.78:0/2271946105 conn(0x7fbb100fa7c0 msgr2=0x7fbb100fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.848+0000 7fbb15b88700 1 -- 172.21.15.78:0/2271946105 shutdown_connections 2023-10-10T17:40:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.848+0000 7fbb15b88700 1 -- 172.21.15.78:0/2271946105 wait complete. 2023-10-10T17:40:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.848+0000 7fbb15b88700 1 Processor -- start 2023-10-10T17:40:44.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.848+0000 7fbb15b88700 1 -- start start 2023-10-10T17:40:44.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb100ff1c0 0x7fbb101962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:44.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb101003b0 0x7fbb10196800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:44.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb10100d40 0x7fbb1019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:44.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1019de40 con 0x7fbb100ff1c0 2023-10-10T17:40:44.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1019df80 con 0x7fbb10100d40 2023-10-10T17:40:44.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1019e110 con 0x7fbb101003b0 2023-10-10T17:40:44.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb100ff1c0 0x7fbb101962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:44.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb100ff1c0 0x7fbb101962c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42182/0 (socket says 172.21.15.78:42182) 2023-10-10T17:40:44.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 -- 172.21.15.78:0/2934788191 learned_addr learned my addr 172.21.15.78:0/2934788191 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0effd700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb101003b0 0x7fbb10196800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 -- 172.21.15.78:0/2934788191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb101003b0 msgr2=0x7fbb10196800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb101003b0 0x7fbb10196800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 -- 172.21.15.78:0/2934788191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb10100d40 msgr2=0x7fbb1019d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb10100d40 0x7fbb1019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 -- 172.21.15.78:0/2934788191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb08009d00 con 0x7fbb100ff1c0 2023-10-10T17:40:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0effd700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb101003b0 0x7fbb10196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0f7fe700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb100ff1c0 0x7fbb101962c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400d980 tx=0x7fbb0400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:44.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0cff9700 1 -- 172.21.15.78:0/2934788191 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0400de60 con 0x7fbb100ff1c0 2023-10-10T17:40:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1010bd50 con 0x7fbb100ff1c0 2023-10-10T17:40:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0cff9700 1 -- 172.21.15.78:0/2934788191 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb040044e0 con 0x7fbb100ff1c0 2023-10-10T17:40:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb0cff9700 1 -- 172.21.15.78:0/2934788191 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb04003be0 con 0x7fbb100ff1c0 2023-10-10T17:40:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.849+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1010bbc0 con 0x7fbb100ff1c0 2023-10-10T17:40:44.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.850+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb10106760 con 0x7fbb100ff1c0 2023-10-10T17:40:44.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.851+0000 7fbb0cff9700 1 -- 172.21.15.78:0/2934788191 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbb04009bb0 con 0x7fbb100ff1c0 2023-10-10T17:40:44.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.852+0000 7fbb0cff9700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbaf80a1150 0x7fbaf80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:44.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.852+0000 7fbb0effd700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbaf80a1150 0x7fbaf80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:44.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.852+0000 7fbb0cff9700 1 -- 172.21.15.78:0/2934788191 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8698+0+0 (secure 0 0 0) 0x7fbb040ccc30 con 0x7fbb100ff1c0 2023-10-10T17:40:44.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.852+0000 7fbb0effd700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbaf80a1150 0x7fbaf80a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbb000099d0 tx=0x7fbb00008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:44.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:44.855+0000 7fbb0cff9700 1 -- 172.21.15.78:0/2934788191 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbb040961b0 con 0x7fbb100ff1c0 2023-10-10T17:40:44.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:44 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:44.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:44 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:40:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:44 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:40:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:44 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:44.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:44 smithi078 ceph-mon[99494]: osdmap e86: 12 total, 12 up, 12 in 2023-10-10T17:40:44.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:44 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:44.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:44 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:40:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:44 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:40:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:44 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:44.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:44 smithi190 ceph-mon[101220]: osdmap e86: 12 total, 12 up, 12 in 2023-10-10T17:40:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:44 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:44 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:40:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:44 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:40:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:44 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:40:44.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:44 smithi157 ceph-mon[100896]: osdmap e86: 12 total, 12 up, 12 in 2023-10-10T17:40:45.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.045+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fbb1002cfe0 con 0x7fbb100ff1c0 2023-10-10T17:40:45.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.693+0000 7fbb0cff9700 1 -- 172.21.15.78:0/2934788191 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbb04009940 con 0x7fbb100ff1c0 2023-10-10T17:40:45.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.700+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbaf80a1150 msgr2=0x7fbaf80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.700+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbaf80a1150 0x7fbaf80a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbb000099d0 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2023-10-10T17:40:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.700+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb100ff1c0 msgr2=0x7fbb101962c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.700+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb100ff1c0 0x7fbb101962c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400d980 tx=0x7fbb0400dd40 comp rx=0 tx=0).stop 2023-10-10T17:40:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.702+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 shutdown_connections 2023-10-10T17:40:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.702+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbaf80a1150 0x7fbaf80a3600 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbb000099d0 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2023-10-10T17:40:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.702+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb10100d40 0x7fbb1019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.702+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb100ff1c0 0x7fbb101962c0 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbb0400d980 tx=0x7fbb0400dd40 comp rx=0 tx=0).stop 2023-10-10T17:40:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.702+0000 7fbb15b88700 1 --2- 172.21.15.78:0/2934788191 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb101003b0 0x7fbb10196800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.702+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 >> 172.21.15.78:0/2934788191 conn(0x7fbb100fa7c0 msgr2=0x7fbb101038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.710+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 shutdown_connections 2023-10-10T17:40:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.710+0000 7fbb15b88700 1 -- 172.21.15.78:0/2934788191 wait complete. 2023-10-10T17:40:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:set pool 4 allow_ec_overwrites to true 2023-10-10T17:40:45.726 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-10-10T17:40:45.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 -- 172.21.15.78:0/3258314765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 msgr2=0x7f3d900ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:45.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 --2- 172.21.15.78:0/3258314765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d900ff5d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009990 tx=0x7f3d78009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 -- 172.21.15.78:0/3258314765 shutdown_connections 2023-10-10T17:40:45.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 --2- 172.21.15.78:0/3258314765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d90100d40 0x7f3d90107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 --2- 172.21.15.78:0/3258314765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d900ff5d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 --2- 172.21.15.78:0/3258314765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d901003b0 0x7f3d90100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 -- 172.21.15.78:0/3258314765 >> 172.21.15.78:0/3258314765 conn(0x7f3d900fa7a0 msgr2=0x7f3d900fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:45.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 -- 172.21.15.78:0/3258314765 shutdown_connections 2023-10-10T17:40:45.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.850+0000 7f3d9486c700 1 -- 172.21.15.78:0/3258314765 wait complete. 2023-10-10T17:40:45.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 Processor -- start 2023-10-10T17:40:45.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 -- start start 2023-10-10T17:40:45.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d90194ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:45.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d901003b0 0x7f3d90195420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:45.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d90100d40 0x7f3d9019c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:45.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9019ca60 con 0x7f3d900ff1c0 2023-10-10T17:40:45.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9019cba0 con 0x7f3d90100d40 2023-10-10T17:40:45.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d9486c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9019cd30 con 0x7f3d901003b0 2023-10-10T17:40:45.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d8e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d90194ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:45.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d8e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d90194ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42204/0 (socket says 172.21.15.78:42204) 2023-10-10T17:40:45.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d8e59c700 1 -- 172.21.15.78:0/520087180 learned_addr learned my addr 172.21.15.78:0/520087180 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d8ed9d700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d90100d40 0x7f3d9019c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d8dd9b700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d901003b0 0x7f3d90195420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.851+0000 7f3d8e59c700 1 -- 172.21.15.78:0/520087180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d901003b0 msgr2=0x7f3d90195420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d8e59c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d901003b0 0x7f3d90195420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d8e59c700 1 -- 172.21.15.78:0/520087180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d90100d40 msgr2=0x7f3d9019c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d8e59c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d90100d40 0x7f3d9019c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d8e59c700 1 -- 172.21.15.78:0/520087180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d780095f0 con 0x7f3d900ff1c0 2023-10-10T17:40:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d8ed9d700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d90100d40 0x7f3d9019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d8dd9b700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d901003b0 0x7f3d90195420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d8e59c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d90194ee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3d780059d0 tx=0x7f3d78005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:45.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d877fe700 1 -- 172.21.15.78:0/520087180 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7801d070 con 0x7f3d900ff1c0 2023-10-10T17:40:45.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d877fe700 1 -- 172.21.15.78:0/520087180 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d78004120 con 0x7f3d900ff1c0 2023-10-10T17:40:45.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9010ab20 con 0x7f3d900ff1c0 2023-10-10T17:40:45.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d877fe700 1 -- 172.21.15.78:0/520087180 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78022de0 con 0x7f3d900ff1c0 2023-10-10T17:40:45.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.852+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9010a990 con 0x7f3d900ff1c0 2023-10-10T17:40:45.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.853+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d90069cb0 con 0x7f3d900ff1c0 2023-10-10T17:40:45.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.854+0000 7f3d877fe700 1 -- 172.21.15.78:0/520087180 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3d78022470 con 0x7f3d900ff1c0 2023-10-10T17:40:45.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.854+0000 7f3d877fe700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d7c0a1150 0x7f3d7c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:45.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.855+0000 7f3d8dd9b700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d7c0a1150 0x7f3d7c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:45.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.855+0000 7f3d877fe700 1 -- 172.21.15.78:0/520087180 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f3d780cdf80 con 0x7f3d900ff1c0 2023-10-10T17:40:45.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.855+0000 7f3d8dd9b700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d7c0a1150 0x7f3d7c0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3d80005fd0 tx=0x7f3d80005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:45.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:45.859+0000 7f3d877fe700 1 -- 172.21.15.78:0/520087180 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3d78097530 con 0x7f3d900ff1c0 2023-10-10T17:40:45.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:45 smithi078 ceph-mon[99494]: pgmap v260: 97 pgs: 20 creating+peering, 54 active+clean, 23 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2023-10-10T17:40:45.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:45 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1587788100' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-10-10T17:40:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:45 smithi078 ceph-mon[99494]: osdmap e87: 12 total, 12 up, 12 in 2023-10-10T17:40:45.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:45 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2934788191' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-10-10T17:40:45.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:45 smithi190 ceph-mon[101220]: pgmap v260: 97 pgs: 20 creating+peering, 54 active+clean, 23 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2023-10-10T17:40:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:45 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1587788100' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-10-10T17:40:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:45 smithi190 ceph-mon[101220]: osdmap e87: 12 total, 12 up, 12 in 2023-10-10T17:40:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:45 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2934788191' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-10-10T17:40:45.964 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:45 smithi157 ceph-mon[100896]: pgmap v260: 97 pgs: 20 creating+peering, 54 active+clean, 23 unknown; 579 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2023-10-10T17:40:45.964 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:45 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1587788100' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-10-10T17:40:45.964 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:45 smithi157 ceph-mon[100896]: osdmap e87: 12 total, 12 up, 12 in 2023-10-10T17:40:45.965 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:45 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2934788191' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-10-10T17:40:46.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.094+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f3d90069ee0 con 0x7f3d900ff1c0 2023-10-10T17:40:46.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.100+0000 7f3d877fe700 1 -- 172.21.15.78:0/520087180 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f3d7809ab50 con 0x7f3d900ff1c0 2023-10-10T17:40:46.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d7c0a1150 msgr2=0x7f3d7c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d7c0a1150 0x7f3d7c0a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3d80005fd0 tx=0x7f3d80005ee0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 msgr2=0x7f3d90194ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d90194ee0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3d780059d0 tx=0x7f3d78005ef0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 shutdown_connections 2023-10-10T17:40:46.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d7c0a1150 0x7f3d7c0a3600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d90100d40 0x7f3d9019c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d900ff1c0 0x7f3d90194ee0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 --2- 172.21.15.78:0/520087180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d901003b0 0x7f3d90195420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 >> 172.21.15.78:0/520087180 conn(0x7f3d900fa7a0 msgr2=0x7f3d90103870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:46.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 shutdown_connections 2023-10-10T17:40:46.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.102+0000 7f3d9486c700 1 -- 172.21.15.78:0/520087180 wait complete. 2023-10-10T17:40:46.106 INFO:teuthology.orchestra.run.smithi078.stderr:added data pool 4 to fsmap 2023-10-10T17:40:46.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:46.228 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:40:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: 2023-10-10T17:40:45.961+0000 7f85bbe16700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-10T17:40:46.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 -- 172.21.15.78:0/25745560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c105e50 msgr2=0x7f961c108300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 --2- 172.21.15.78:0/25745560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c105e50 0x7f961c108300 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f961400b0a0 tx=0x7f961400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 -- 172.21.15.78:0/25745560 shutdown_connections 2023-10-10T17:40:46.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 --2- 172.21.15.78:0/25745560 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c068b10 0x7f961c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 --2- 172.21.15.78:0/25745560 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c105e50 0x7f961c108300 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 --2- 172.21.15.78:0/25745560 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c103340 0x7f961c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 -- 172.21.15.78:0/25745560 >> 172.21.15.78:0/25745560 conn(0x7f961c0fa750 msgr2=0x7f961c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:46.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 -- 172.21.15.78:0/25745560 shutdown_connections 2023-10-10T17:40:46.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 -- 172.21.15.78:0/25745560 wait complete. 2023-10-10T17:40:46.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 Processor -- start 2023-10-10T17:40:46.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.237+0000 7f9622206700 1 -- start start 2023-10-10T17:40:46.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c068b10 0x7f961c100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c103340 0x7f961c0feda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c105e50 0x7f961c0ff2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0ffa80 con 0x7f961c068b10 2023-10-10T17:40:46.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c0ffbf0 con 0x7f961c105e50 2023-10-10T17:40:46.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961c1a00b0 con 0x7f961c103340 2023-10-10T17:40:46.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c068b10 0x7f961c100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c068b10 0x7f961c100760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42218/0 (socket says 172.21.15.78:42218) 2023-10-10T17:40:46.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 -- 172.21.15.78:0/769751908 learned_addr learned my addr 172.21.15.78:0/769751908 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:46.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961affd700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c103340 0x7f961c0feda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961bfff700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c105e50 0x7f961c0ff2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 -- 172.21.15.78:0/769751908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c103340 msgr2=0x7f961c0feda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c103340 0x7f961c0feda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 -- 172.21.15.78:0/769751908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c105e50 msgr2=0x7f961c0ff2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c105e50 0x7f961c0ff2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 -- 172.21.15.78:0/769751908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9614009d00 con 0x7f961c068b10 2023-10-10T17:40:46.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961bfff700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c105e50 0x7f961c0ff2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:46.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961affd700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c103340 0x7f961c0feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:46.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f961b7fe700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c068b10 0x7f961c100760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f961000e970 tx=0x7f961000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:46.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9618ff9700 1 -- 172.21.15.78:0/769751908 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610009790 con 0x7f961c068b10 2023-10-10T17:40:46.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9618ff9700 1 -- 172.21.15.78:0/769751908 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9610004cf0 con 0x7f961c068b10 2023-10-10T17:40:46.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961c10bbf0 con 0x7f961c068b10 2023-10-10T17:40:46.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9618ff9700 1 -- 172.21.15.78:0/769751908 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610005680 con 0x7f961c068b10 2023-10-10T17:40:46.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.238+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961c10ba60 con 0x7f961c068b10 2023-10-10T17:40:46.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.239+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f961c0fc2c0 con 0x7f961c068b10 2023-10-10T17:40:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.240+0000 7f9618ff9700 1 -- 172.21.15.78:0/769751908 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9610004750 con 0x7f961c068b10 2023-10-10T17:40:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.241+0000 7f9618ff9700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f96040a1250 0x7f96040a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.241+0000 7f961affd700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f96040a1250 0x7f96040a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.241+0000 7f9618ff9700 1 -- 172.21.15.78:0/769751908 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9610014070 con 0x7f961c068b10 2023-10-10T17:40:46.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.241+0000 7f961affd700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f96040a1250 0x7f96040a3700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f960c005950 tx=0x7f960c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:46.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.244+0000 7f9618ff9700 1 -- 172.21.15.78:0/769751908 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f96100964b0 con 0x7f961c068b10 2023-10-10T17:40:46.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.479+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f961c061700 con 0x7f961c068b10 2023-10-10T17:40:46.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.480+0000 7f9618ff9700 1 -- 172.21.15.78:0/769751908 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10747 (secure 0 0 0) 0x7f9610099ad0 con 0x7f961c068b10 2023-10-10T17:40:46.480 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:46.481 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:40:46.096182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-10T17:40:46.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f96040a1250 msgr2=0x7f96040a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f96040a1250 0x7f96040a3700 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f960c005950 tx=0x7f960c0058e0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c068b10 msgr2=0x7f961c100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c068b10 0x7f961c100760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f961000e970 tx=0x7f961000ed30 comp rx=0 tx=0).stop 2023-10-10T17:40:46.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 shutdown_connections 2023-10-10T17:40:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f96040a1250 0x7f96040a3700 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f961c105e50 0x7f961c0ff2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f961c068b10 0x7f961c100760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 --2- 172.21.15.78:0/769751908 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f961c103340 0x7f961c0feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 >> 172.21.15.78:0/769751908 conn(0x7f961c0fa750 msgr2=0x7f961c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 shutdown_connections 2023-10-10T17:40:46.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.482+0000 7f9622206700 1 -- 172.21.15.78:0/769751908 wait complete. 2023-10-10T17:40:46.486 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-10T17:40:46.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 -- 172.21.15.78:0/2410549227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100d10 msgr2=0x7f236c1075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 --2- 172.21.15.78:0/2410549227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100d10 0x7f236c1075d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f236400b0a0 tx=0x7f236400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 -- 172.21.15.78:0/2410549227 shutdown_connections 2023-10-10T17:40:46.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 --2- 172.21.15.78:0/2410549227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100d10 0x7f236c1075d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 --2- 172.21.15.78:0/2410549227 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f236c0ff190 0x7f236c0ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 --2- 172.21.15.78:0/2410549227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f236c100380 0x7f236c1007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 -- 172.21.15.78:0/2410549227 >> 172.21.15.78:0/2410549227 conn(0x7f236c0fa770 msgr2=0x7f236c0fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 -- 172.21.15.78:0/2410549227 shutdown_connections 2023-10-10T17:40:46.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.615+0000 7f237219a700 1 -- 172.21.15.78:0/2410549227 wait complete. 2023-10-10T17:40:46.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 Processor -- start 2023-10-10T17:40:46.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 -- start start 2023-10-10T17:40:46.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f236c0ff190 0x7f236c196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100380 0x7f236c1966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f236c100d10 0x7f236c19d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c19dce0 con 0x7f236c100380 2023-10-10T17:40:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c19de20 con 0x7f236c0ff190 2023-10-10T17:40:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f237219a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c19dfb0 con 0x7f236c100d10 2023-10-10T17:40:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f236affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100380 0x7f236c1966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f236affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100380 0x7f236c1966a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42234/0 (socket says 172.21.15.78:42234) 2023-10-10T17:40:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f236affd700 1 -- 172.21.15.78:0/1979170638 learned_addr learned my addr 172.21.15.78:0/1979170638 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f236bfff700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f236c100d10 0x7f236c19d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f236affd700 1 -- 172.21.15.78:0/1979170638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f236c100d10 msgr2=0x7f236c19d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.616+0000 7f236affd700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f236c100d10 0x7f236c19d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f236affd700 1 -- 172.21.15.78:0/1979170638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f236c0ff190 msgr2=0x7f236c196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f236b7fe700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f236c0ff190 0x7f236c196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f236affd700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f236c0ff190 0x7f236c196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f236affd700 1 -- 172.21.15.78:0/1979170638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009d00 con 0x7f236c100380 2023-10-10T17:40:46.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f236bfff700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f236c100d10 0x7f236c19d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:46.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f236b7fe700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f236c0ff190 0x7f236c196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:46.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f236affd700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100380 0x7f236c1966a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f236000b530 tx=0x7f236000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:46.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f2368ff9700 1 -- 172.21.15.78:0/1979170638 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2360011830 con 0x7f236c100380 2023-10-10T17:40:46.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c10ac10 con 0x7f236c100380 2023-10-10T17:40:46.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f2368ff9700 1 -- 172.21.15.78:0/1979170638 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2360011e50 con 0x7f236c100380 2023-10-10T17:40:46.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f2368ff9700 1 -- 172.21.15.78:0/1979170638 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236000f2e0 con 0x7f236c100380 2023-10-10T17:40:46.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.617+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c10aa80 con 0x7f236c100380 2023-10-10T17:40:46.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.618+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c19e7d0 con 0x7f236c100380 2023-10-10T17:40:46.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.619+0000 7f2368ff9700 1 -- 172.21.15.78:0/1979170638 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2360010580 con 0x7f236c100380 2023-10-10T17:40:46.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.620+0000 7f2368ff9700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23540a1150 0x7f23540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.620+0000 7f236b7fe700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23540a1150 0x7f23540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.620+0000 7f2368ff9700 1 -- 172.21.15.78:0/1979170638 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f23600ccb80 con 0x7f236c100380 2023-10-10T17:40:46.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.620+0000 7f236b7fe700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23540a1150 0x7f23540a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f236c1001e0 tx=0x7f235c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:46.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.623+0000 7f2368ff9700 1 -- 172.21.15.78:0/1979170638 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f23600960e0 con 0x7f236c100380 2023-10-10T17:40:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.817+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f236c19ed80 con 0x7f236c100380 2023-10-10T17:40:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.819+0000 7f2368ff9700 1 -- 172.21.15.78:0/1979170638 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f2360099700 con 0x7f236c100380 2023-10-10T17:40:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:46.820 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:46.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23540a1150 msgr2=0x7f23540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23540a1150 0x7f23540a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f236c1001e0 tx=0x7f235c008040 comp rx=0 tx=0).stop 2023-10-10T17:40:46.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100380 msgr2=0x7f236c1966a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100380 0x7f236c1966a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f236000b530 tx=0x7f236000b8f0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 shutdown_connections 2023-10-10T17:40:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23540a1150 0x7f23540a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f236c100d10 0x7f236c19d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f236c0ff190 0x7f236c196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 --2- 172.21.15.78:0/1979170638 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f236c100380 0x7f236c1966a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 >> 172.21.15.78:0/1979170638 conn(0x7f236c0fa770 msgr2=0x7f236c068e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 shutdown_connections 2023-10-10T17:40:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.821+0000 7f237219a700 1 -- 172.21.15.78:0/1979170638 wait complete. 2023-10-10T17:40:46.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2934788191' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-10-10T17:40:46.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: osdmap e88: 12 total, 12 up, 12 in 2023-10-10T17:40:46.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/520087180' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-10-10T17:40:46.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: osdmap e89: 12 total, 12 up, 12 in 2023-10-10T17:40:46.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/520087180' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-10-10T17:40:46.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:46.834 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: pgmap v264: 129 pgs: 8 creating+peering, 96 active+clean, 25 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 6 op/s 2023-10-10T17:40:46.835 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:46 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/769751908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:46.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:46.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 -- 172.21.15.78:0/3255258514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 msgr2=0x7fee280fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 --2- 172.21.15.78:0/3255258514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee280fe950 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fee24009990 tx=0x7fee24009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 -- 172.21.15.78:0/3255258514 shutdown_connections 2023-10-10T17:40:46.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 --2- 172.21.15.78:0/3255258514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee28101c40 0x7fee28108800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 --2- 172.21.15.78:0/3255258514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee280fe950 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 --2- 172.21.15.78:0/3255258514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee280fee90 0x7fee281015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 -- 172.21.15.78:0/3255258514 >> 172.21.15.78:0/3255258514 conn(0x7fee280f9ee0 msgr2=0x7fee280fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:46.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 -- 172.21.15.78:0/3255258514 shutdown_connections 2023-10-10T17:40:46.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.956+0000 7fee30164700 1 -- 172.21.15.78:0/3255258514 wait complete. 2023-10-10T17:40:46.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.957+0000 7fee30164700 1 Processor -- start 2023-10-10T17:40:46.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.957+0000 7fee30164700 1 -- start start 2023-10-10T17:40:46.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.957+0000 7fee30164700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee28195fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.957+0000 7fee30164700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee280fee90 0x7fee281964f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.957+0000 7fee30164700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee28101c40 0x7fee2819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee30164700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2819db30 con 0x7fee280fe540 2023-10-10T17:40:46.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee30164700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2819dc70 con 0x7fee28101c40 2023-10-10T17:40:46.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee30164700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2819de00 con 0x7fee280fee90 2023-10-10T17:40:46.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee28195fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee28195fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42246/0 (socket says 172.21.15.78:42246) 2023-10-10T17:40:46.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 -- 172.21.15.78:0/3839452595 learned_addr learned my addr 172.21.15.78:0/3839452595 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:46.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 -- 172.21.15.78:0/3839452595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee280fee90 msgr2=0x7fee281964f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee280fee90 0x7fee281964f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 -- 172.21.15.78:0/3839452595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee28101c40 msgr2=0x7fee2819d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:46.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee28101c40 0x7fee2819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:46.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 -- 172.21.15.78:0/3839452595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee240095f0 con 0x7fee280fe540 2023-10-10T17:40:46.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee2df00700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee28195fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fee24004760 tx=0x7fee24009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:46.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee1effd700 1 -- 172.21.15.78:0/3839452595 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2401d070 con 0x7fee280fe540 2023-10-10T17:40:46.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee1effd700 1 -- 172.21.15.78:0/3839452595 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee24004200 con 0x7fee280fe540 2023-10-10T17:40:46.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2810bc30 con 0x7fee280fe540 2023-10-10T17:40:46.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee1effd700 1 -- 172.21.15.78:0/3839452595 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee240049f0 con 0x7fee280fe540 2023-10-10T17:40:46.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.958+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2810baa0 con 0x7fee280fe540 2023-10-10T17:40:46.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.959+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2819e480 con 0x7fee280fe540 2023-10-10T17:40:46.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.960+0000 7fee1effd700 1 -- 172.21.15.78:0/3839452595 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fee24003d40 con 0x7fee280fe540 2023-10-10T17:40:46.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.961+0000 7fee1effd700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fee140a11d0 0x7fee140a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:46.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.961+0000 7fee2d6ff700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fee140a11d0 0x7fee140a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:46.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.961+0000 7fee1effd700 1 -- 172.21.15.78:0/3839452595 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fee240cda20 con 0x7fee280fe540 2023-10-10T17:40:46.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.961+0000 7fee2d6ff700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fee140a11d0 0x7fee140a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee28067a90 tx=0x7fee18005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:46.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:46.965+0000 7fee1effd700 1 -- 172.21.15.78:0/3839452595 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fee24096fb0 con 0x7fee280fe540 2023-10-10T17:40:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2934788191' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-10-10T17:40:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: osdmap e88: 12 total, 12 up, 12 in 2023-10-10T17:40:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/520087180' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-10-10T17:40:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: osdmap e89: 12 total, 12 up, 12 in 2023-10-10T17:40:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/520087180' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-10-10T17:40:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: pgmap v264: 129 pgs: 8 creating+peering, 96 active+clean, 25 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 6 op/s 2023-10-10T17:40:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:46 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/769751908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:47.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.156+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee281072f0 con 0x7fee280fe540 2023-10-10T17:40:47.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.157+0000 7fee1effd700 1 -- 172.21.15.78:0/3839452595 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7fee2409a5d0 con 0x7fee280fe540 2023-10-10T17:40:47.158 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:47.159 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:47.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.159+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fee140a11d0 msgr2=0x7fee140a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.159+0000 7fee30164700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fee140a11d0 0x7fee140a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fee28067a90 tx=0x7fee18005c80 comp rx=0 tx=0).stop 2023-10-10T17:40:47.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.159+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 msgr2=0x7fee28195fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.159+0000 7fee30164700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee28195fb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fee24004760 tx=0x7fee24009f90 comp rx=0 tx=0).stop 2023-10-10T17:40:47.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 shutdown_connections 2023-10-10T17:40:47.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fee140a11d0 0x7fee140a3680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee28101c40 0x7fee2819d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee280fe540 0x7fee28195fb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 --2- 172.21.15.78:0/3839452595 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee280fee90 0x7fee281964f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 >> 172.21.15.78:0/3839452595 conn(0x7fee280f9ee0 msgr2=0x7fee280fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:47.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 shutdown_connections 2023-10-10T17:40:47.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.160+0000 7fee30164700 1 -- 172.21.15.78:0/3839452595 wait complete. 2023-10-10T17:40:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2934788191' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-10-10T17:40:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: osdmap e88: 12 total, 12 up, 12 in 2023-10-10T17:40:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/520087180' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-10-10T17:40:47.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: osdmap e89: 12 total, 12 up, 12 in 2023-10-10T17:40:47.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/520087180' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-10-10T17:40:47.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:47.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: pgmap v264: 129 pgs: 8 creating+peering, 96 active+clean, 25 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s wr, 6 op/s 2023-10-10T17:40:47.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:46 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/769751908' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:47.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:40:47.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3542055494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 msgr2=0x7f6d3c100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3542055494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c100000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009990 tx=0x7f6d2c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3542055494 shutdown_connections 2023-10-10T17:40:47.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3542055494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d3c100540 0x7f6d3c106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3542055494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d3c0fe900 0x7f6d3c0fed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3542055494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c100000 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3542055494 >> 172.21.15.78:0/3542055494 conn(0x7f6d3c0f9ee0 msgr2=0x7f6d3c0fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:47.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3542055494 shutdown_connections 2023-10-10T17:40:47.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.310+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3542055494 wait complete. 2023-10-10T17:40:47.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 Processor -- start 2023-10-10T17:40:47.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 -- start start 2023-10-10T17:40:47.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d3c0fe900 0x7f6d3c10b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c19d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d3c100540 0x7f6d3c19f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c19fc30 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c19fda0 con 0x7f6d3c0fe900 2023-10-10T17:40:47.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c19ff60 con 0x7f6d3c100540 2023-10-10T17:40:47.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c19d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:47.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c19d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42262/0 (socket says 172.21.15.78:42262) 2023-10-10T17:40:47.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 -- 172.21.15.78:0/3097697392 learned_addr learned my addr 172.21.15.78:0/3097697392 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:47.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d42029700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d3c100540 0x7f6d3c19f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:47.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 -- 172.21.15.78:0/3097697392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d3c100540 msgr2=0x7f6d3c19f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d3c100540 0x7f6d3c19f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 -- 172.21.15.78:0/3097697392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d3c0fe900 msgr2=0x7f6d3c10b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d3c0fe900 0x7f6d3c10b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 -- 172.21.15.78:0/3097697392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d2c0095f0 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d41027700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c19d030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0098e0 tx=0x7f6d2c00f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:47.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d32ffd700 1 -- 172.21.15.78:0/3097697392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c01d070 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d32ffd700 1 -- 172.21.15.78:0/3097697392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6d2c00fb70 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3c10a260 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d32ffd700 1 -- 172.21.15.78:0/3097697392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c017650 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.311+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3c10a0d0 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.312+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3c105570 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.313+0000 7f6d32ffd700 1 -- 172.21.15.78:0/3097697392 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6d2c0050d0 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.314+0000 7f6d32ffd700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6d280a1150 0x7f6d280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.314+0000 7f6d41828700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6d280a1150 0x7f6d280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:47.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.314+0000 7f6d32ffd700 1 -- 172.21.15.78:0/3097697392 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6d2c0ce110 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.314+0000 7f6d41828700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6d280a1150 0x7f6d280a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d38005950 tx=0x7f6d380058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:47.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.317+0000 7f6d32ffd700 1 -- 172.21.15.78:0/3097697392 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6d2c097700 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.549+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d3c02cfe0 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.550+0000 7f6d32ffd700 1 -- 172.21.15.78:0/3097697392 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10747 (secure 0 0 0) 0x7f6d2c09ad20 con 0x7f6d3c0ffb90 2023-10-10T17:40:47.551 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:47.551 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24437,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24467,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24490,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:40:46.096182+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24500},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-10-10T17:40:47.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6d280a1150 msgr2=0x7f6d280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6d280a1150 0x7f6d280a3600 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6d38005950 tx=0x7f6d380058e0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 msgr2=0x7f6d3c19d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c19d030 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c0098e0 tx=0x7f6d2c00f740 comp rx=0 tx=0).stop 2023-10-10T17:40:47.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 shutdown_connections 2023-10-10T17:40:47.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6d280a1150 0x7f6d280a3600 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6d3c100540 0x7f6d3c19f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6d3c0fe900 0x7f6d3c10b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 --2- 172.21.15.78:0/3097697392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6d3c0ffb90 0x7f6d3c19d030 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 >> 172.21.15.78:0/3097697392 conn(0x7f6d3c0f9ee0 msgr2=0x7f6d3c0fb890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:47.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 shutdown_connections 2023-10-10T17:40:47.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.552+0000 7f6d43a8c700 1 -- 172.21.15.78:0/3097697392 wait complete. 2023-10-10T17:40:47.555 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-10-10T17:40:47.565 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:47.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 -- 172.21.15.78:0/1792761415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00100380 msgr2=0x7efe001007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 --2- 172.21.15.78:0/1792761415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00100380 0x7efe001007d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efdfc009990 tx=0x7efdfc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 -- 172.21.15.78:0/1792761415 shutdown_connections 2023-10-10T17:40:47.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 --2- 172.21.15.78:0/1792761415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe000ff190 0x7efe000ff5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 --2- 172.21.15.78:0/1792761415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe00100d10 0x7efe001075d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 --2- 172.21.15.78:0/1792761415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe00100380 0x7efe001007d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 -- 172.21.15.78:0/1792761415 >> 172.21.15.78:0/1792761415 conn(0x7efe000fa770 msgr2=0x7efe000fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:47.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 -- 172.21.15.78:0/1792761415 shutdown_connections 2023-10-10T17:40:47.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.685+0000 7efe08119700 1 -- 172.21.15.78:0/1792761415 wait complete. 2023-10-10T17:40:47.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 Processor -- start 2023-10-10T17:40:47.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 -- start start 2023-10-10T17:40:47.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe000ff190 0x7efe00195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe00100380 0x7efe00196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe00100d10 0x7efe0019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0019db40 con 0x7efe000ff190 2023-10-10T17:40:47.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0019dc80 con 0x7efe00100d10 2023-10-10T17:40:47.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe08119700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0019de10 con 0x7efe00100380 2023-10-10T17:40:47.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe05eb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe000ff190 0x7efe00195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:47.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe05eb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe000ff190 0x7efe00195fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42274/0 (socket says 172.21.15.78:42274) 2023-10-10T17:40:47.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe05eb5700 1 -- 172.21.15.78:0/3043355568 learned_addr learned my addr 172.21.15.78:0/3043355568 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:47.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe066b6700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe00100d10 0x7efe0019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:47.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.686+0000 7efe056b4700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe00100380 0x7efe00196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:47.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe05eb5700 1 -- 172.21.15.78:0/3043355568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe00100380 msgr2=0x7efe00196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe05eb5700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe00100380 0x7efe00196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe05eb5700 1 -- 172.21.15.78:0/3043355568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe00100d10 msgr2=0x7efe0019d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe05eb5700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe00100d10 0x7efe0019d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe05eb5700 1 -- 172.21.15.78:0/3043355568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc0095f0 con 0x7efe000ff190 2023-10-10T17:40:47.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe056b4700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe00100380 0x7efe00196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:47.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe066b6700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe00100d10 0x7efe0019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:47.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe05eb5700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe000ff190 0x7efe00195fc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efdf000e970 tx=0x7efdf000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:47.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efdf6ffd700 1 -- 172.21.15.78:0/3043355568 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0009790 con 0x7efe000ff190 2023-10-10T17:40:47.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0010ab50 con 0x7efe000ff190 2023-10-10T17:40:47.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efdf6ffd700 1 -- 172.21.15.78:0/3043355568 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efdf00044e0 con 0x7efe000ff190 2023-10-10T17:40:47.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efdf6ffd700 1 -- 172.21.15.78:0/3043355568 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0003e30 con 0x7efe000ff190 2023-10-10T17:40:47.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.687+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0010a9c0 con 0x7efe000ff190 2023-10-10T17:40:47.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.688+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0019e560 con 0x7efe000ff190 2023-10-10T17:40:47.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.689+0000 7efdf6ffd700 1 -- 172.21.15.78:0/3043355568 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efdf0004000 con 0x7efe000ff190 2023-10-10T17:40:47.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.689+0000 7efdf6ffd700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7efdec0a1150 0x7efdec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:47.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.689+0000 7efe056b4700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7efdec0a1150 0x7efdec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:47.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.689+0000 7efdf6ffd700 1 -- 172.21.15.78:0/3043355568 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efdf0014070 con 0x7efe000ff190 2023-10-10T17:40:47.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.690+0000 7efe056b4700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7efdec0a1150 0x7efdec0a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00b2e0 tx=0x7efdfc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:47.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.693+0000 7efdf6ffd700 1 -- 172.21.15.78:0/3043355568 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efdf00964d0 con 0x7efe000ff190 2023-10-10T17:40:47.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:47 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1979170638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:47 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3839452595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:47.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:47 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3097697392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:47.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.883+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe0019ec00 con 0x7efe000ff190 2023-10-10T17:40:47.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.884+0000 7efdf6ffd700 1 -- 172.21.15.78:0/3043355568 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7efdf0099af0 con 0x7efe000ff190 2023-10-10T17:40:47.886 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:47.886 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:47.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7efdec0a1150 msgr2=0x7efdec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7efdec0a1150 0x7efdec0a3600 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00b2e0 tx=0x7efdfc000bc0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe000ff190 msgr2=0x7efe00195fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:47.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe000ff190 0x7efe00195fc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7efdf000e970 tx=0x7efdf000ed30 comp rx=0 tx=0).stop 2023-10-10T17:40:47.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 shutdown_connections 2023-10-10T17:40:47.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7efdec0a1150 0x7efdec0a3600 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe00100d10 0x7efe0019d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe000ff190 0x7efe00195fc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 --2- 172.21.15.78:0/3043355568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe00100380 0x7efe00196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:47.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 >> 172.21.15.78:0/3043355568 conn(0x7efe000fa770 msgr2=0x7efe00068de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:47.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 shutdown_connections 2023-10-10T17:40:47.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:47.886+0000 7efe08119700 1 -- 172.21.15.78:0/3043355568 wait complete. 2023-10-10T17:40:47.899 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-10T17:40:41.574125+0000', 'modified': '2023-10-10T17:40:46.096182+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24500}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.157:6839/2554315829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 2554315829}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 2554315829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-10-10T17:40:47.900 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-10T17:40:47.900 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-10T17:40:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:47 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1979170638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:47 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3839452595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:47 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3097697392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:48.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.021+0000 7f540cb8f700 1 -- 172.21.15.78:0/2223768446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 msgr2=0x7f54081007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.021+0000 7f540cb8f700 1 --2- 172.21.15.78:0/2223768446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f54081007a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f53fc009990 tx=0x7f53fc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 -- 172.21.15.78:0/2223768446 shutdown_connections 2023-10-10T17:40:48.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 --2- 172.21.15.78:0/2223768446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5408100ce0 0x7f54081075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 --2- 172.21.15.78:0/2223768446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54080ff160 0x7f54080ff570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 --2- 172.21.15.78:0/2223768446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f54081007a0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 -- 172.21.15.78:0/2223768446 >> 172.21.15.78:0/2223768446 conn(0x7f54080fa780 msgr2=0x7f54080fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:48.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 -- 172.21.15.78:0/2223768446 shutdown_connections 2023-10-10T17:40:48.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 -- 172.21.15.78:0/2223768446 wait complete. 2023-10-10T17:40:48.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.022+0000 7f540cb8f700 1 Processor -- start 2023-10-10T17:40:48.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540cb8f700 1 -- start start 2023-10-10T17:40:48.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540cb8f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54080ff160 0x7f5408194e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540cb8f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f5408195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540cb8f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5408100ce0 0x7f540819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540cb8f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540819c9c0 con 0x7f5408100350 2023-10-10T17:40:48.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540cb8f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540819cb00 con 0x7f5408100ce0 2023-10-10T17:40:48.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540cb8f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540819cc90 con 0x7f54080ff160 2023-10-10T17:40:48.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f5408195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f5408195380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42290/0 (socket says 172.21.15.78:42290) 2023-10-10T17:40:48.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 -- 172.21.15.78:0/3325386676 learned_addr learned my addr 172.21.15.78:0/3325386676 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:48.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f540659c700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54080ff160 0x7f5408194e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5406d9d700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5408100ce0 0x7f540819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 -- 172.21.15.78:0/3325386676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54080ff160 msgr2=0x7f5408194e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54080ff160 0x7f5408194e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 -- 172.21.15.78:0/3325386676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5408100ce0 msgr2=0x7f540819c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5408100ce0 0x7f540819c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 -- 172.21.15.78:0/3325386676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc0095f0 con 0x7f5408100350 2023-10-10T17:40:48.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5406d9d700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5408100ce0 0x7f540819c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:48.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f5405d9b700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f5408195380 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f53fc004760 tx=0x7f53fc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:48.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.023+0000 7f53f77fe700 1 -- 172.21.15.78:0/3325386676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc01f070 con 0x7f5408100350 2023-10-10T17:40:48.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.024+0000 7f53f77fe700 1 -- 172.21.15.78:0/3325386676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53fc01ad70 con 0x7f5408100350 2023-10-10T17:40:48.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.024+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540810aac0 con 0x7f5408100350 2023-10-10T17:40:48.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.024+0000 7f53f77fe700 1 -- 172.21.15.78:0/3325386676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0049f0 con 0x7f5408100350 2023-10-10T17:40:48.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.024+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540810a930 con 0x7f5408100350 2023-10-10T17:40:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.025+0000 7f53f77fe700 1 -- 172.21.15.78:0/3325386676 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f53fc023070 con 0x7f5408100350 2023-10-10T17:40:48.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.026+0000 7f53f77fe700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53f00a1150 0x7f53f00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.026+0000 7f540659c700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53f00a1150 0x7f53f00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.026+0000 7f53f77fe700 1 -- 172.21.15.78:0/3325386676 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f53fc0cfb60 con 0x7f5408100350 2023-10-10T17:40:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.026+0000 7f540659c700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53f00a1150 0x7f53f00a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f53f8005950 tx=0x7f53f80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:48.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.027+0000 7f5418d1bb80 1 Processor -- start 2023-10-10T17:40:48.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.027+0000 7f5418d1bb80 1 -- start start 2023-10-10T17:40:48.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc2627fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2628ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2629530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc252d010 con 0x560bc2628500 2023-10-10T17:40:48.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc257f850 con 0x560bc2627bb0 2023-10-10T17:40:48.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc2578ce0 con 0x560bc26290c0 2023-10-10T17:40:48.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2628ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2628ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42298/0 (socket says 172.21.15.78:42298) 2023-10-10T17:40:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 -- 172.21.15.78:0/169159769 learned_addr learned my addr 172.21.15.78:0/169159769 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5406d9d700 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2629530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f540659c700 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc2627fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 -- 172.21.15.78:0/169159769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 msgr2=0x560bc2629530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2629530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 -- 172.21.15.78:0/169159769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 msgr2=0x560bc2627fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc2627fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 -- 172.21.15.78:0/169159769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560bc23fa5e0 con 0x560bc2628500 2023-10-10T17:40:48.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f540659c700 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc2627fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f5405d9b700 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2628ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f53fc099880 tx=0x7f53fc005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71999aee70b09136 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f53f4ff9700 1 -- 172.21.15.78:0/169159769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc09ce10 con 0x560bc2628500 2023-10-10T17:40:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f53f4ff9700 1 -- 172.21.15.78:0/169159769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53fc0a0e40 con 0x560bc2628500 2023-10-10T17:40:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.028+0000 7f53f4ff9700 1 -- 172.21.15.78:0/169159769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc09c500 con 0x560bc2628500 2023-10-10T17:40:48.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 -- 172.21.15.78:0/169159769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 msgr2=0x560bc2628ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2628ae0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f53fc099880 tx=0x7f53fc005490 comp rx=0 tx=0).stop 2023-10-10T17:40:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 -- 172.21.15.78:0/169159769 shutdown_connections 2023-10-10T17:40:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2629530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc2627fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/169159769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2628ae0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 -- 172.21.15.78:0/169159769 >> 172.21.15.78:0/169159769 conn(0x560bc261d310 msgr2=0x560bc2622620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 -- 172.21.15.78:0/169159769 shutdown_connections 2023-10-10T17:40:48.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 -- 172.21.15.78:0/169159769 wait complete. 2023-10-10T17:40:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.029+0000 7f5418d1bb80 1 -- start start 2023-10-10T17:40:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 Processor -- start 2023-10-10T17:40:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc26311e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2631720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2638760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc251a890 con 0x560bc2628500 2023-10-10T17:40:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc251a530 con 0x560bc2627bb0 2023-10-10T17:40:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc23fc8d0 con 0x560bc26290c0 2023-10-10T17:40:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 10 client.0 _ll_register_callbacks cb 0x560bc2538730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:40:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2631720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2631720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42304/0 (socket says 172.21.15.78:42304) 2023-10-10T17:40:48.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 -- 172.21.15.78:0/384846009 learned_addr learned my addr 172.21.15.78:0/384846009 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:48.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5406d9d700 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2638760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f540659c700 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc26311e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 -- 172.21.15.78:0/384846009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 msgr2=0x560bc2638760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2638760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 -- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 msgr2=0x560bc26311e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc26311e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 -- 172.21.15.78:0/384846009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540810a930 con 0x560bc2628500 2023-10-10T17:40:48.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5405d9b700 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2631720 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f53fc09cd50 tx=0x7f53fc005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e477f5846d5340a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:48.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f53edffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:40:48.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f53edffb700 1 -- 172.21.15.78:0/384846009 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53fc0bb1c0 con 0x560bc2628500 2023-10-10T17:40:48.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f53edffb700 1 -- 172.21.15.78:0/384846009 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f53fc0a4d40 con 0x560bc2628500 2023-10-10T17:40:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.030+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560bc261aad0 con 0x560bc2628500 2023-10-10T17:40:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f53edffb700 1 -- 172.21.15.78:0/384846009 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53fc0a4f00 con 0x560bc2628500 2023-10-10T17:40:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 10 client.14757 fetch_fsmap learned FSMap version 17 2023-10-10T17:40:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x560bc253f140 con 0x560bc2628500 2023-10-10T17:40:48.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f53edffb700 1 -- 172.21.15.78:0/384846009 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560bc261aad0 con 0x560bc2628500 2023-10-10T17:40:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f53edffb700 1 -- 172.21.15.78:0/384846009 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f53fc01a830 con 0x560bc2628500 2023-10-10T17:40:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 10 client.14757 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T17:40:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 10 client.14757 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T17:40:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 20 client.14757 populate_metadata read hostname 'smithi078' 2023-10-10T17:40:48.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc263bf70 0x560bc263b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 4 client.14757 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:40:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560bc253b050 con 0x560bc263bf70 2023-10-10T17:40:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.031+0000 7f540659c700 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc263bf70 0x560bc263b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.032+0000 7f53edffb700 10 client.14757 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:40:48.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.032+0000 7f540659c700 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc263bf70 0x560bc263b600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4270b100df62f33 server_cookie=305e0639185dac63 in_seq=0 out_seq=0 2023-10-10T17:40:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.032+0000 7f53edffb700 1 -- 172.21.15.78:0/384846009 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560bc253b050 con 0x560bc263bf70 2023-10-10T17:40:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.032+0000 7f53edffb700 10 client.14757 handle_command_reply: tid=0 2023-10-10T17:40:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.032+0000 7f5418d1bb80 1 client.14757 shutdown 2023-10-10T17:40:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 msgr2=0x560bc2631720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2631720 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f53fc09cd50 tx=0x7f53fc005390 comp rx=0 tx=0).stop 2023-10-10T17:40:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 shutdown_connections 2023-10-10T17:40:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc26290c0 0x560bc2638760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc263bf70 0x560bc263b600 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2627bb0 0x560bc26311e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f53edffb700 0 client.14757 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:40:48.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/384846009 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2628500 0x560bc2631720 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 >> 172.21.15.78:0/384846009 conn(0x560bc261d310 msgr2=0x560bc2621d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:48.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 shutdown_connections 2023-10-10T17:40:48.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 1 -- 172.21.15.78:0/384846009 wait complete. 2023-10-10T17:40:48.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.033+0000 7f5418d1bb80 20 client.14757 trim_cache size 0 max 16384 2023-10-10T17:40:48.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 Processor -- start 2023-10-10T17:40:48.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 -- start start 2023-10-10T17:40:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2627bb0 0x560bc2630f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2628500 0x560bc2638720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc26290c0 0x560bc26314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc263d410 con 0x560bc2627bb0 2023-10-10T17:40:48.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc257f850 con 0x560bc26290c0 2023-10-10T17:40:48.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc253b050 con 0x560bc2628500 2023-10-10T17:40:48.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.050+0000 7f540659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2627bb0 0x560bc2630f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2627bb0 0x560bc2630f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42310/0 (socket says 172.21.15.78:42310) 2023-10-10T17:40:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 -- 172.21.15.78:0/2822349956 learned_addr learned my addr 172.21.15.78:0/2822349956 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f5405d9b700 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2628500 0x560bc2638720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 -- 172.21.15.78:0/2822349956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2628500 msgr2=0x560bc2638720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2628500 0x560bc2638720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 -- 172.21.15.78:0/2822349956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc26290c0 msgr2=0x560bc26314d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc26290c0 0x560bc26314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 -- 172.21.15.78:0/2822349956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560bc253ee90 con 0x560bc2627bb0 2023-10-10T17:40:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f5405d9b700 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2628500 0x560bc2638720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:48.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f540659c700 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2627bb0 0x560bc2630f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x560bc2631f40 tx=0x7f53f8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71b5d2ace996cfa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f53f57fa700 1 -- 172.21.15.78:0/2822349956 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8011db0 con 0x560bc2627bb0 2023-10-10T17:40:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f53f57fa700 1 -- 172.21.15.78:0/2822349956 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53f8026d90 con 0x560bc2627bb0 2023-10-10T17:40:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f53f57fa700 1 -- 172.21.15.78:0/2822349956 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f801e790 con 0x560bc2627bb0 2023-10-10T17:40:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f5418d1bb80 1 -- 172.21.15.78:0/2822349956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2627bb0 msgr2=0x560bc2630f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.051+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2627bb0 0x560bc2630f90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x560bc2631f40 tx=0x7f53f8015000 comp rx=0 tx=0).stop 2023-10-10T17:40:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- 172.21.15.78:0/2822349956 shutdown_connections 2023-10-10T17:40:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc26290c0 0x560bc26314d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc2627bb0 0x560bc2630f90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/2822349956 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2628500 0x560bc2638720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- 172.21.15.78:0/2822349956 >> 172.21.15.78:0/2822349956 conn(0x560bc261d310 msgr2=0x560bc263b600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- 172.21.15.78:0/2822349956 shutdown_connections 2023-10-10T17:40:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- 172.21.15.78:0/2822349956 wait complete. 2023-10-10T17:40:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- start start 2023-10-10T17:40:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 Processor -- start 2023-10-10T17:40:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2627bb0 0x560bc26333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2628500 0x560bc2633910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc26290c0 0x560bc268fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc2638940 con 0x560bc26290c0 2023-10-10T17:40:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc2638ab0 con 0x560bc2628500 2023-10-10T17:40:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560bc2690310 con 0x560bc2627bb0 2023-10-10T17:40:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5418d1bb80 10 client.0 _ll_register_callbacks cb 0x560bc2538730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:40:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc26290c0 0x560bc268fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc26290c0 0x560bc268fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42320/0 (socket says 172.21.15.78:42320) 2023-10-10T17:40:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 -- 172.21.15.78:0/3400973904 learned_addr learned my addr 172.21.15.78:0/3400973904 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f540659c700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2627bb0 0x560bc26333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5405d9b700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2628500 0x560bc2633910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 -- 172.21.15.78:0/3400973904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2627bb0 msgr2=0x560bc26333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2627bb0 0x560bc26333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 -- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2628500 msgr2=0x560bc2633910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:40:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2628500 0x560bc2633910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f5406d9d700 1 -- 172.21.15.78:0/3400973904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560bc238fc10 con 0x560bc26290c0 2023-10-10T17:40:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.052+0000 7f540659c700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2627bb0 0x560bc26333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5406d9d700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc26290c0 0x560bc268fdd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5400007af0 tx=0x7f5400007eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e52593a677821c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f53edffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:40:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f53edffb700 1 -- 172.21.15.78:0/3400973904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54000149d0 con 0x560bc26290c0 2023-10-10T17:40:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f53edffb700 1 -- 172.21.15.78:0/3400973904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5400014b30 con 0x560bc26290c0 2023-10-10T17:40:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53fc0a0e40 con 0x560bc26290c0 2023-10-10T17:40:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f53edffb700 1 -- 172.21.15.78:0/3400973904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5400014cf0 con 0x560bc26290c0 2023-10-10T17:40:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 10 client.14763 fetch_fsmap learned FSMap version 17 2023-10-10T17:40:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x560bc23a6d90 con 0x560bc26290c0 2023-10-10T17:40:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f53edffb700 1 -- 172.21.15.78:0/3400973904 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f540002b080 con 0x560bc26290c0 2023-10-10T17:40:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f53edffb700 1 -- 172.21.15.78:0/3400973904 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f54000274c0 con 0x560bc26290c0 2023-10-10T17:40:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 10 client.14763 fetch_fsmap finished waiting for FSMap version 17 2023-10-10T17:40:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 10 client.14763 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T17:40:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 20 client.14763 populate_metadata read hostname 'smithi078' 2023-10-10T17:40:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc2695140 0x560bc2691a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 4 client.14763 mds_command: new command op to 24500 tid=0[{"prefix": "status"}] 2023-10-10T17:40:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.053+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560bc2547610 con 0x560bc2695140 2023-10-10T17:40:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f540659c700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc2695140 0x560bc2691a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f53edffb700 10 client.14763 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:40:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f540659c700 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc2695140 0x560bc2691a90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14be64c86c9c1263 server_cookie=c3e42b8ce240d589 in_seq=0 out_seq=0 2023-10-10T17:40:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f53edffb700 1 -- 172.21.15.78:0/3400973904 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x560bc2547610 con 0x560bc2695140 2023-10-10T17:40:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f53edffb700 10 client.14763 handle_command_reply: tid=0 2023-10-10T17:40:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f5418d1bb80 1 client.14763 shutdown 2023-10-10T17:40:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc26290c0 msgr2=0x560bc268fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.054+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc26290c0 0x560bc268fdd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5400007af0 tx=0x7f5400007eb0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 shutdown_connections 2023-10-10T17:40:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560bc2627bb0 0x560bc26333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560bc2695140 0x560bc2691a90 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560bc26290c0 0x560bc268fdd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f53edffb700 0 client.14763 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:40:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 --2- 172.21.15.78:0/3400973904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560bc2628500 0x560bc2633910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 >> 172.21.15.78:0/3400973904 conn(0x560bc261d310 msgr2=0x560bc2630f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 shutdown_connections 2023-10-10T17:40:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 1 -- 172.21.15.78:0/3400973904 wait complete. 2023-10-10T17:40:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f5418d1bb80 20 client.14763 trim_cache size 0 max 16384 2023-10-10T17:40:48.085 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:40:48.085 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:40:48.085 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-10T17:40:48.085 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24500, 2023-10-10T17:40:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-10T17:40:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-10T17:40:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-10T17:40:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 6.478230624, 2023-10-10T17:40:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-10-10T17:40:48.087 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 89, 2023-10-10T17:40:48.087 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 88, 2023-10-10T17:40:48.087 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 12.126760333 2023-10-10T17:40:48.087 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:40:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53f00a1150 msgr2=0x7f53f00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53f00a1150 0x7f53f00a3600 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f53f8005950 tx=0x7f53f80058e0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 msgr2=0x7f5408195380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f5408195380 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f53fc004760 tx=0x7f53fc009f90 comp rx=0 tx=0).stop 2023-10-10T17:40:48.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 shutdown_connections 2023-10-10T17:40:48.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53f00a1150 0x7f53f00a3600 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54080ff160 0x7f5408194e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5408100ce0 0x7f540819c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 --2- 172.21.15.78:0/3325386676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5408100350 0x7f5408195380 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:48.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 >> 172.21.15.78:0/3325386676 conn(0x7f54080fa780 msgr2=0x7f54080fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:48.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 shutdown_connections 2023-10-10T17:40:48.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:48.055+0000 7f540cb8f700 1 -- 172.21.15.78:0/3325386676 wait complete. 2023-10-10T17:40:48.091 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-10T17:40:48.092 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:48.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph fs ls 2023-10-10T17:40:48.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:47 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1979170638' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:48.258 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:47 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3839452595' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:48.259 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:47 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3097697392' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:40:48.729 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:40:48.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:48 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3043355568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:48.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:48 smithi078 ceph-mon[99494]: pgmap v265: 129 pgs: 110 active+clean, 19 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-10-10T17:40:48.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:48 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3043355568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:48 smithi157 ceph-mon[100896]: pgmap v265: 129 pgs: 110 active+clean, 19 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-10-10T17:40:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:48 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3043355568' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:49.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:48 smithi190 ceph-mon[101220]: pgmap v265: 129 pgs: 110 active+clean, 19 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2023-10-10T17:40:50.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 -- 172.21.15.78:0/2108511983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 msgr2=0x7faa680ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:50.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/2108511983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa680ff840 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0099c0 tx=0x7faa5c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 -- 172.21.15.78:0/2108511983 shutdown_connections 2023-10-10T17:40:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/2108511983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa68104510 0x7faa6810c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/2108511983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa680ff840 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/2108511983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa680ffe10 0x7faa68103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 -- 172.21.15.78:0/2108511983 >> 172.21.15.78:0/2108511983 conn(0x7faa680770d0 msgr2=0x7faa680774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:50.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 -- 172.21.15.78:0/2108511983 shutdown_connections 2023-10-10T17:40:50.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.023+0000 7faa6f0ea700 1 -- 172.21.15.78:0/2108511983 wait complete. 2023-10-10T17:40:50.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 Processor -- start 2023-10-10T17:40:50.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 -- start start 2023-10-10T17:40:50.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa68197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:50.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa680ffe10 0x7faa68197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:50.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa68104510 0x7faa681a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:50.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa681a27c0 con 0x7faa680ff470 2023-10-10T17:40:50.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa681a2900 con 0x7faa68104510 2023-10-10T17:40:50.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6f0ea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa681a2a40 con 0x7faa680ffe10 2023-10-10T17:40:50.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6ce86700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa68197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:50.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6ce86700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa68197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42328/0 (socket says 172.21.15.78:42328) 2023-10-10T17:40:50.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.024+0000 7faa6ce86700 1 -- 172.21.15.78:0/4294940364 learned_addr learned my addr 172.21.15.78:0/4294940364 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:50.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6d687700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa68104510 0x7faa681a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:50.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa67fff700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa680ffe10 0x7faa68197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6ce86700 1 -- 172.21.15.78:0/4294940364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa680ffe10 msgr2=0x7faa68197740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6ce86700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa680ffe10 0x7faa68197740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6ce86700 1 -- 172.21.15.78:0/4294940364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa68104510 msgr2=0x7faa681a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6ce86700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa68104510 0x7faa681a2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6ce86700 1 -- 172.21.15.78:0/4294940364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c0095f0 con 0x7faa680ff470 2023-10-10T17:40:50.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa67fff700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa680ffe10 0x7faa68197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6ce86700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa68197200 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7faa5c005910 tx=0x7faa5c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa65ffb700 1 -- 172.21.15.78:0/4294940364 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c01d070 con 0x7faa680ff470 2023-10-10T17:40:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa65ffb700 1 -- 172.21.15.78:0/4294940364 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa5c004200 con 0x7faa680ff470 2023-10-10T17:40:50.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa65ffb700 1 -- 172.21.15.78:0/4294940364 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa5c00f460 con 0x7faa680ff470 2023-10-10T17:40:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6810f250 con 0x7faa680ff470 2023-10-10T17:40:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.025+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6810f0c0 con 0x7faa680ff470 2023-10-10T17:40:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.026+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6810f3e0 con 0x7faa680ff470 2023-10-10T17:40:50.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.027+0000 7faa65ffb700 1 -- 172.21.15.78:0/4294940364 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faa5c0226c0 con 0x7faa680ff470 2023-10-10T17:40:50.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.028+0000 7faa65ffb700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:50.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.028+0000 7faa67fff700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:50.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.028+0000 7faa65ffb700 1 -- 172.21.15.78:0/4294940364 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7faa5c0ce130 con 0x7faa680ff470 2023-10-10T17:40:50.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.028+0000 7faa67fff700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa54005fd0 tx=0x7faa54005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:50.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.032+0000 7faa65ffb700 1 -- 172.21.15.78:0/4294940364 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faa5c0976c0 con 0x7faa680ff470 2023-10-10T17:40:50.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.264+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faa681a3340 con 0x7faa680ff470 2023-10-10T17:40:50.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.264+0000 7faa65ffb700 1 -- 172.21.15.78:0/4294940364 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7faa5c09ace0 con 0x7faa680ff470 2023-10-10T17:40:50.265 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-10T17:40:50.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 msgr2=0x7faa580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:50.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7faa54005fd0 tx=0x7faa54005dc0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 msgr2=0x7faa68197200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa68197200 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7faa5c005910 tx=0x7faa5c0036f0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 shutdown_connections 2023-10-10T17:40:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa68104510 0x7faa681a2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa680ff470 0x7faa68197200 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 --2- 172.21.15.78:0/4294940364 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa680ffe10 0x7faa68197740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 >> 172.21.15.78:0/4294940364 conn(0x7faa680770d0 msgr2=0x7faa68102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 shutdown_connections 2023-10-10T17:40:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:50.267+0000 7faa6f0ea700 1 -- 172.21.15.78:0/4294940364 wait complete. 2023-10-10T17:40:50.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:50 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4294940364' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-10T17:40:50.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:50 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4294940364' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-10T17:40:50.677 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-10T17:40:50.677 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-10T17:40:50.678 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2023-10-10T17:40:50.678 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-10-10T17:40:50.679 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:50.679 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-10T17:40:50.679 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-10T17:40:50.680 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-10T17:40:50.680 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-10-10T17:40:50.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:50.682 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2023-10-10T17:40:50.701 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-10T17:40:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-10T17:40:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2023-10-10T17:40:50.701 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-10T17:40:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2023-10-10T17:40:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-10T17:40:50.702 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T17:40:50.702 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2023-10-10T17:40:50.703 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T17:40:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2023-10-10T17:40:50.703 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T17:40:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2023-10-10T17:40:50.703 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-10T17:40:50.703 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2023-10-10T17:40:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-10T17:40:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 41886sec preferred_lft 41886sec 2023-10-10T17:40:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::d6a8:fe40:6759:a046/64 scope link noprefixroute 2023-10-10T17:40:50.704 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-10T17:40:50.705 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-10T17:40:50.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:50.707 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:40:50.708 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:40:50.708 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2023-10-10T17:40:50.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2023-10-10T17:40:50.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2023-10-10T17:40:50.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-10T17:40:50.710 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:40:50.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:50 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4294940364' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-10T17:40:50.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:50.875 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-10T17:40:50.903 INFO:teuthology.orchestra.run.smithi078.stdout:1 2023-10-10T17:40:50.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:50.905 DEBUG:teuthology.orchestra.run.smithi078:> route 2023-10-10T17:40:50.978 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2023-10-10T17:40:50.978 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-10T17:40:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-10T17:40:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-10T17:40:50.979 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-10T17:40:50.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:50.980 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:40:50.980 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:40:50.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-10T17:40:50.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-10T17:40:50.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-10T17:40:50.982 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:40:51.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:51.244 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-10-10T17:40:51.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:51.262 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2023-10-10T17:40:51.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:51.318 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:40:51.319 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:40:51.319 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-10T17:40:51.319 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-10-10T17:40:51.320 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:40:51.414 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-10-10T17:40:51.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:51.415 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:40:51.416 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:40:51.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-10-10T17:40:51.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-10T17:40:51.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-10-10T17:40:51.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-10-10T17:40:51.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-10-10T17:40:51.418 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:40:51.456 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:51 smithi078 ceph-mon[99494]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2023-10-10T17:40:51.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:51.640 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:40:51.641 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:40:51.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2023-10-10T17:40:51.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2023-10-10T17:40:51.642 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:40:51.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:51 smithi190 ceph-mon[101220]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2023-10-10T17:40:51.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:51 smithi157 ceph-mon[100896]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2023-10-10T17:40:51.778 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-10-10T17:40:51.778 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-10T17:40:51.779 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:51.796 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-10-10T17:40:51.796 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-10T17:40:51.796 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:51.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2023-10-10T17:40:51.921 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-10T17:40:51.921 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-10T17:40:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-10T17:40:51.991 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-10T17:40:51.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:51.991 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2023-10-10T17:40:52.048 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-10-10T17:40:52.049 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-10-10T17:40:52.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2023-10-10T17:40:52.117 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-10-10T17:40:52.118 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-10-10T17:40:52.184 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-10-10T17:40:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-10-10T17:40:52.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:52.186 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2023-10-10T17:40:52.193 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:2023-10-10T17:40:52.190+0000 7ff3d37ea5c0 -1 init, newargv = 0x55cf53c862b0 newargc=15 2023-10-10T17:40:52.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[136656]: starting ceph client 2023-10-10T17:40:52.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[136656]: starting fuse 2023-10-10T17:40:52.211 INFO:teuthology.orchestra.run.smithi078.stdout:124 2023-10-10T17:40:52.212 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-10-10T17:40:52.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-10-10T17:40:52.213 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2023-10-10T17:40:52.213 DEBUG:teuthology.orchestra.run.smithi078:> import re 2023-10-10T17:40:52.213 DEBUG:teuthology.orchestra.run.smithi078:> import os 2023-10-10T17:40:52.214 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2023-10-10T17:40:52.214 DEBUG:teuthology.orchestra.run.smithi078:> 2023-10-10T17:40:52.214 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2023-10-10T17:40:52.215 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-10-10T17:40:52.215 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2023-10-10T17:40:52.215 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-10-10T17:40:52.216 DEBUG:teuthology.orchestra.run.smithi078:> 2023-10-10T17:40:52.216 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2023-10-10T17:40:52.216 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2023-10-10T17:40:52.217 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2023-10-10T17:40:52.217 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2023-10-10T17:40:52.217 DEBUG:teuthology.orchestra.run.smithi078:> 2023-10-10T17:40:52.218 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2023-10-10T17:40:52.218 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-10-10T17:40:52.219 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2023-10-10T17:40:52.219 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-10-10T17:40:52.219 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2023-10-10T17:40:52.220 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2023-10-10T17:40:52.220 DEBUG:teuthology.orchestra.run.smithi078:> return f 2023-10-10T17:40:52.220 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-10-10T17:40:52.221 DEBUG:teuthology.orchestra.run.smithi078:> 2023-10-10T17:40:52.221 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.admin")) 2023-10-10T17:40:52.222 DEBUG:teuthology.orchestra.run.smithi078:> ' 2023-10-10T17:40:52.334 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.admin.136656.asok 2023-10-10T17:40:52.517 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.136656.asok 2023-10-10T17:40:52.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:52.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.136656.asok status 2023-10-10T17:40:52.674 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:40:52.674 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2023-10-10T17:40:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "05a43c8fb3988914c879a50970f9054a098e0612", 2023-10-10T17:40:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.0.0-3908-g05a43c8f (05a43c8fb3988914c879a50970f9054a098e0612) reef (dev)", 2023-10-10T17:40:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "admin", 2023-10-10T17:40:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2023-10-10T17:40:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-10-10T17:40:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "136656", 2023-10-10T17:40:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2023-10-10T17:40:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:40:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2023-10-10T17:40:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2023-10-10T17:40:52.677 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 14775, 2023-10-10T17:40:52.677 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2023-10-10T17:40:52.677 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2023-10-10T17:40:52.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2023-10-10T17:40:52.677 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 14775 2023-10-10T17:40:52.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:40:52.678 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2023-10-10T17:40:52.678 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2023-10-10T17:40:52.678 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2023-10-10T17:40:52.678 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2757956269 2023-10-10T17:40:52.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:40:52.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:40:52.679 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2023-10-10T17:40:52.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2023-10-10T17:40:52.680 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2023-10-10T17:40:52.680 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2757956269 2023-10-10T17:40:52.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:40:52.680 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.14775 192.168.0.1:0/2757956269", 2023-10-10T17:40:52.680 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/2757956269", 2023-10-10T17:40:52.681 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2023-10-10T17:40:52.681 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 17, 2023-10-10T17:40:52.681 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 89, 2023-10-10T17:40:52.682 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2023-10-10T17:40:52.682 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2023-10-10T17:40:52.682 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2023-10-10T17:40:52.682 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:40:52.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:52.685 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2023-10-10T17:40:52.685 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:52.703 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2023-10-10T17:40:52.703 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:52.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:52.704 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:52.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:52.771 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-10-10T17:40:52.874 INFO:teuthology.orchestra.run.smithi078.stdout:# file: . 2023-10-10T17:40:52.874 INFO:teuthology.orchestra.run.smithi078.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-10-10T17:40:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:52.875 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:40:52.875 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-10T17:40:52.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:53.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.053+0000 7ff459755700 1 -- 172.21.15.78:0/2552466687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff454100e00 msgr2=0x7ff4541076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.053+0000 7ff459755700 1 --2- 172.21.15.78:0/2552466687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff454100e00 0x7ff4541076a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff44c00b0a0 tx=0x7ff44c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 -- 172.21.15.78:0/2552466687 shutdown_connections 2023-10-10T17:40:53.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 --2- 172.21.15.78:0/2552466687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff454100e00 0x7ff4541076a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 --2- 172.21.15.78:0/2552466687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4540ff1c0 0x7ff4540ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 --2- 172.21.15.78:0/2552466687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff454100450 0x7ff4541008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 -- 172.21.15.78:0/2552466687 >> 172.21.15.78:0/2552466687 conn(0x7ff4540fa7c0 msgr2=0x7ff4540fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:53.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 -- 172.21.15.78:0/2552466687 shutdown_connections 2023-10-10T17:40:53.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 -- 172.21.15.78:0/2552466687 wait complete. 2023-10-10T17:40:53.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 Processor -- start 2023-10-10T17:40:53.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 -- start start 2023-10-10T17:40:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4540ff1c0 0x7ff4541031a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff454100450 0x7ff454101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff454100e00 0x7ff454101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454102510 con 0x7ff4540ff1c0 2023-10-10T17:40:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.054+0000 7ff459755700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff454102680 con 0x7ff454100450 2023-10-10T17:40:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff459755700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4541a00d0 con 0x7ff454100e00 2023-10-10T17:40:53.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4540ff1c0 0x7ff4541031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4540ff1c0 0x7ff4541031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42354/0 (socket says 172.21.15.78:42354) 2023-10-10T17:40:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 -- 172.21.15.78:0/334810025 learned_addr learned my addr 172.21.15.78:0/334810025 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff4537fe700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff454100e00 0x7ff454101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff4527fc700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff454100450 0x7ff454101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 -- 172.21.15.78:0/334810025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff454100e00 msgr2=0x7ff454101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff454100e00 0x7ff454101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 -- 172.21.15.78:0/334810025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff454100450 msgr2=0x7ff454101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff454100450 0x7ff454101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 -- 172.21.15.78:0/334810025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c009d00 con 0x7ff4540ff1c0 2023-10-10T17:40:53.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff452ffd700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4540ff1c0 0x7ff4541031a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff44400d950 tx=0x7ff44400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:53.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff43bfff700 1 -- 172.21.15.78:0/334810025 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4440098e0 con 0x7ff4540ff1c0 2023-10-10T17:40:53.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff43bfff700 1 -- 172.21.15.78:0/334810025 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff444010460 con 0x7ff4540ff1c0 2023-10-10T17:40:53.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff43bfff700 1 -- 172.21.15.78:0/334810025 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44400f620 con 0x7ff4540ff1c0 2023-10-10T17:40:53.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45410ab60 con 0x7ff4540ff1c0 2023-10-10T17:40:53.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.055+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45410a9d0 con 0x7ff4540ff1c0 2023-10-10T17:40:53.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.056+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4541a0770 con 0x7ff4540ff1c0 2023-10-10T17:40:53.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.058+0000 7ff43bfff700 1 -- 172.21.15.78:0/334810025 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff4440108d0 con 0x7ff4540ff1c0 2023-10-10T17:40:53.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.058+0000 7ff43bfff700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff43c0a11d0 0x7ff43c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.059+0000 7ff4527fc700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff43c0a11d0 0x7ff43c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.059+0000 7ff43bfff700 1 -- 172.21.15.78:0/334810025 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff4440cc8d0 con 0x7ff4540ff1c0 2023-10-10T17:40:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.059+0000 7ff4527fc700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff43c0a11d0 0x7ff43c0a3680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff448005950 tx=0x7ff4480058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:53.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.062+0000 7ff43bfff700 1 -- 172.21.15.78:0/334810025 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff444095dd0 con 0x7ff4540ff1c0 2023-10-10T17:40:53.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.255+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4541a0d20 con 0x7ff4540ff1c0 2023-10-10T17:40:53.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.256+0000 7ff43bfff700 1 -- 172.21.15.78:0/334810025 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7ff444020090 con 0x7ff4540ff1c0 2023-10-10T17:40:53.257 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:53.257 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:53.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff43c0a11d0 msgr2=0x7ff43c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff43c0a11d0 0x7ff43c0a3680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff448005950 tx=0x7ff4480058e0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4540ff1c0 msgr2=0x7ff4541031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4540ff1c0 0x7ff4541031a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff44400d950 tx=0x7ff44400dc60 comp rx=0 tx=0).stop 2023-10-10T17:40:53.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 shutdown_connections 2023-10-10T17:40:53.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff43c0a11d0 0x7ff43c0a3680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff454100e00 0x7ff454101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4540ff1c0 0x7ff4541031a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 --2- 172.21.15.78:0/334810025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff454100450 0x7ff454101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.258+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 >> 172.21.15.78:0/334810025 conn(0x7ff4540fa7c0 msgr2=0x7ff454069050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:53.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.259+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 shutdown_connections 2023-10-10T17:40:53.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.259+0000 7ff459755700 1 -- 172.21.15.78:0/334810025 wait complete. 2023-10-10T17:40:53.272 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:40:53.273 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:40:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 -- 172.21.15.78:0/218500806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7440100820 msgr2=0x7f7440100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 --2- 172.21.15.78:0/218500806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7440100820 0x7f7440100c30 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7430009990 tx=0x7f7430009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 -- 172.21.15.78:0/218500806 shutdown_connections 2023-10-10T17:40:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 --2- 172.21.15.78:0/218500806 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74400ffe20 0x7f74401076d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 --2- 172.21.15.78:0/218500806 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7440100820 0x7f7440100c30 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 --2- 172.21.15.78:0/218500806 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74400ff470 0x7f74400ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 -- 172.21.15.78:0/218500806 >> 172.21.15.78:0/218500806 conn(0x7f74400f9fb0 msgr2=0x7f74400fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 -- 172.21.15.78:0/218500806 shutdown_connections 2023-10-10T17:40:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.398+0000 7f744653b700 1 -- 172.21.15.78:0/218500806 wait complete. 2023-10-10T17:40:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 Processor -- start 2023-10-10T17:40:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 -- start start 2023-10-10T17:40:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74400ff470 0x7f7440103220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74400ffe20 0x7f7440101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7440100820 0x7f7440101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7440102540 con 0x7f74400ffe20 2023-10-10T17:40:53.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74401026b0 con 0x7f7440100820 2023-10-10T17:40:53.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f744653b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74401a00e0 con 0x7f74400ff470 2023-10-10T17:40:53.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74400ffe20 0x7f7440101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74400ffe20 0x7f7440101860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42368/0 (socket says 172.21.15.78:42368) 2023-10-10T17:40:53.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 -- 172.21.15.78:0/2447010034 learned_addr learned my addr 172.21.15.78:0/2447010034 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:53.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f743ffff700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74400ff470 0x7f7440103220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 -- 172.21.15.78:0/2447010034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74400ff470 msgr2=0x7f7440103220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74400ff470 0x7f7440103220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 -- 172.21.15.78:0/2447010034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7440100820 msgr2=0x7f7440101da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7440100820 0x7f7440101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 -- 172.21.15.78:0/2447010034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74300095f0 con 0x7f74400ffe20 2023-10-10T17:40:53.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.399+0000 7f7437fff700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74400ffe20 0x7f7440101860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f742800d950 tx=0x7f742800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:53.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.400+0000 7f743dffb700 1 -- 172.21.15.78:0/2447010034 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7428009b40 con 0x7f74400ffe20 2023-10-10T17:40:53.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.400+0000 7f743dffb700 1 -- 172.21.15.78:0/2447010034 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7428009ca0 con 0x7f74400ffe20 2023-10-10T17:40:53.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.400+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744010abe0 con 0x7f74400ffe20 2023-10-10T17:40:53.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.400+0000 7f743dffb700 1 -- 172.21.15.78:0/2447010034 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742800f620 con 0x7f74400ffe20 2023-10-10T17:40:53.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.400+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744010aa50 con 0x7f74400ffe20 2023-10-10T17:40:53.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.400+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74401a07d0 con 0x7f74400ffe20 2023-10-10T17:40:53.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.402+0000 7f743dffb700 1 -- 172.21.15.78:0/2447010034 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7428010460 con 0x7f74400ffe20 2023-10-10T17:40:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.402+0000 7f743dffb700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f742c0a1150 0x7f742c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.402+0000 7f743ffff700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f742c0a1150 0x7f742c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.402+0000 7f743dffb700 1 -- 172.21.15.78:0/2447010034 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f74280cce80 con 0x7f74400ffe20 2023-10-10T17:40:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.403+0000 7f743ffff700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f742c0a1150 0x7f742c0a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f743000b5c0 tx=0x7f7430005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.405+0000 7f743dffb700 1 -- 172.21.15.78:0/2447010034 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f74280963e0 con 0x7f74400ffe20 2023-10-10T17:40:53.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.597+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f74401a0d80 con 0x7f74400ffe20 2023-10-10T17:40:53.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.597+0000 7f743dffb700 1 -- 172.21.15.78:0/2447010034 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7428016020 con 0x7f74400ffe20 2023-10-10T17:40:53.598 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:40:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f742c0a1150 msgr2=0x7f742c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f742c0a1150 0x7f742c0a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f743000b5c0 tx=0x7f7430005fb0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74400ffe20 msgr2=0x7f7440101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74400ffe20 0x7f7440101860 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f742800d950 tx=0x7f742800dd10 comp rx=0 tx=0).stop 2023-10-10T17:40:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 shutdown_connections 2023-10-10T17:40:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f742c0a1150 0x7f742c0a3600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74400ff470 0x7f7440103220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7440100820 0x7f7440101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 --2- 172.21.15.78:0/2447010034 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74400ffe20 0x7f7440101860 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 >> 172.21.15.78:0/2447010034 conn(0x7f74400f9fb0 msgr2=0x7f74400fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 shutdown_connections 2023-10-10T17:40:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.599+0000 7f744653b700 1 -- 172.21.15.78:0/2447010034 wait complete. 2023-10-10T17:40:53.610 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:53 smithi078 ceph-mon[99494]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s wr, 0 op/s 2023-10-10T17:40:53.611 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:53 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/334810025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:53.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:40:53.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:53 smithi190 ceph-mon[101220]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s wr, 0 op/s 2023-10-10T17:40:53.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:53 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/334810025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:53.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:53 smithi157 ceph-mon[100896]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 272 B/s wr, 0 op/s 2023-10-10T17:40:53.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:53 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/334810025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 -- 172.21.15.78:0/2254531635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c100d60 msgr2=0x7f3d6c107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 --2- 172.21.15.78:0/2254531635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c100d60 0x7f3d6c107620 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400b0a0 tx=0x7f3d6400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 -- 172.21.15.78:0/2254531635 shutdown_connections 2023-10-10T17:40:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 --2- 172.21.15.78:0/2254531635 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d6c0ff1e0 0x7f3d6c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 --2- 172.21.15.78:0/2254531635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c100d60 0x7f3d6c107620 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 --2- 172.21.15.78:0/2254531635 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d6c1003d0 0x7f3d6c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 -- 172.21.15.78:0/2254531635 >> 172.21.15.78:0/2254531635 conn(0x7f3d6c0fa800 msgr2=0x7f3d6c0fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 -- 172.21.15.78:0/2254531635 shutdown_connections 2023-10-10T17:40:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.736+0000 7f3d71371700 1 -- 172.21.15.78:0/2254531635 wait complete. 2023-10-10T17:40:53.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 Processor -- start 2023-10-10T17:40:53.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 -- start start 2023-10-10T17:40:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c0ff1e0 0x7f3d6c069cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d6c1003d0 0x7f3d6c19ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d6c100d60 0x7f3d6c19d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c19dc40 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c19ddb0 con 0x7f3d6c100d60 2023-10-10T17:40:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d71371700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6c19df70 con 0x7f3d6c1003d0 2023-10-10T17:40:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c0ff1e0 0x7f3d6c069cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c0ff1e0 0x7f3d6c069cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42380/0 (socket says 172.21.15.78:42380) 2023-10-10T17:40:53.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6affd700 1 -- 172.21.15.78:0/4143436809 learned_addr learned my addr 172.21.15.78:0/4143436809 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6a7fc700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d6c1003d0 0x7f3d6c19ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6b7fe700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d6c100d60 0x7f3d6c19d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6affd700 1 -- 172.21.15.78:0/4143436809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d6c1003d0 msgr2=0x7f3d6c19ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6affd700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d6c1003d0 0x7f3d6c19ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.737+0000 7f3d6affd700 1 -- 172.21.15.78:0/4143436809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d6c100d60 msgr2=0x7f3d6c19d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d6affd700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d6c100d60 0x7f3d6c19d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d6affd700 1 -- 172.21.15.78:0/4143436809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d64009d00 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d6b7fe700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d6c100d60 0x7f3d6c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d6affd700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c0ff1e0 0x7f3d6c069cd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d980 tx=0x7f3d6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:53.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d53fff700 1 -- 172.21.15.78:0/4143436809 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60004020 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d53fff700 1 -- 172.21.15.78:0/4143436809 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d60009730 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d53fff700 1 -- 172.21.15.78:0/4143436809 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60003a80 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d6c10ab60 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.738+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c10a9d0 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.739+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6c19e670 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.740+0000 7f3d53fff700 1 -- 172.21.15.78:0/4143436809 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3d600098a0 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.740+0000 7f3d53fff700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d540a1150 0x7f3d540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:53.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.740+0000 7f3d53fff700 1 -- 172.21.15.78:0/4143436809 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3d600cc7d0 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.740+0000 7f3d6a7fc700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d540a1150 0x7f3d540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:53.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.741+0000 7f3d6a7fc700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d540a1150 0x7f3d540a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c069a40 tx=0x7f3d5c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:53.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.745+0000 7f3d53fff700 1 -- 172.21.15.78:0/4143436809 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3d60095d30 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.933+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3d6c105ff0 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.934+0000 7f3d53fff700 1 -- 172.21.15.78:0/4143436809 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3d60099350 con 0x7f3d6c0ff1e0 2023-10-10T17:40:53.935 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:40:53.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d540a1150 msgr2=0x7f3d540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d540a1150 0x7f3d540a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c069a40 tx=0x7f3d5c00b410 comp rx=0 tx=0).stop 2023-10-10T17:40:53.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c0ff1e0 msgr2=0x7f3d6c069cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:53.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c0ff1e0 0x7f3d6c069cd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000d980 tx=0x7f3d6000dc90 comp rx=0 tx=0).stop 2023-10-10T17:40:53.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 shutdown_connections 2023-10-10T17:40:53.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3d540a1150 0x7f3d540a3600 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d6c100d60 0x7f3d6c19d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d6c0ff1e0 0x7f3d6c069cd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 --2- 172.21.15.78:0/4143436809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d6c1003d0 0x7f3d6c19ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:53.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 >> 172.21.15.78:0/4143436809 conn(0x7f3d6c0fa800 msgr2=0x7f3d6c0fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:53.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 shutdown_connections 2023-10-10T17:40:53.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:53.936+0000 7f3d71371700 1 -- 172.21.15.78:0/4143436809 wait complete. 2023-10-10T17:40:53.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:40:54.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 -- 172.21.15.78:0/4263301938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c100d40 msgr2=0x7fcd4c107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 --2- 172.21.15.78:0/4263301938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c100d40 0x7fcd4c107600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcd4400b0a0 tx=0x7fcd4400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 -- 172.21.15.78:0/4263301938 shutdown_connections 2023-10-10T17:40:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 --2- 172.21.15.78:0/4263301938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd4c0ff1c0 0x7fcd4c0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 --2- 172.21.15.78:0/4263301938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c100d40 0x7fcd4c107600 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 --2- 172.21.15.78:0/4263301938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd4c1003b0 0x7fcd4c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 -- 172.21.15.78:0/4263301938 >> 172.21.15.78:0/4263301938 conn(0x7fcd4c0fa7a0 msgr2=0x7fcd4c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 -- 172.21.15.78:0/4263301938 shutdown_connections 2023-10-10T17:40:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.071+0000 7fcd5239f700 1 -- 172.21.15.78:0/4263301938 wait complete. 2023-10-10T17:40:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 Processor -- start 2023-10-10T17:40:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 -- start start 2023-10-10T17:40:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c0ff1c0 0x7fcd4c0699e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd4c1003b0 0x7fcd4c19d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd4c100d40 0x7fcd4c19f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c19fe90 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c1a0000 con 0x7fcd4c100d40 2023-10-10T17:40:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd5239f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4c1a01c0 con 0x7fcd4c1003b0 2023-10-10T17:40:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd4bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c0ff1c0 0x7fcd4c0699e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd4bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c0ff1c0 0x7fcd4c0699e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42394/0 (socket says 172.21.15.78:42394) 2023-10-10T17:40:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd4bfff700 1 -- 172.21.15.78:0/2551129657 learned_addr learned my addr 172.21.15.78:0/2551129657 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.072+0000 7fcd4b7fe700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd4c1003b0 0x7fcd4c19d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd5093c700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd4c100d40 0x7fcd4c19f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd4bfff700 1 -- 172.21.15.78:0/2551129657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd4c1003b0 msgr2=0x7fcd4c19d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd4bfff700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd4c1003b0 0x7fcd4c19d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd4bfff700 1 -- 172.21.15.78:0/2551129657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd4c100d40 msgr2=0x7fcd4c19f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd4bfff700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd4c100d40 0x7fcd4c19f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd4bfff700 1 -- 172.21.15.78:0/2551129657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd44009d00 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd5093c700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd4c100d40 0x7fcd4c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:54.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd4bfff700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c0ff1c0 0x7fcd4c0699e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000d980 tx=0x7fcd4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd497fa700 1 -- 172.21.15.78:0/2551129657 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40004020 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd497fa700 1 -- 172.21.15.78:0/2551129657 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcd40009730 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4c10ab80 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd497fa700 1 -- 172.21.15.78:0/2551129657 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40003a80 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.073+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd4c10a9f0 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.074+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4c1066b0 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.075+0000 7fcd497fa700 1 -- 172.21.15.78:0/2551129657 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcd400098a0 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.075+0000 7fcd497fa700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcd340a11d0 0x7fcd340a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.076+0000 7fcd497fa700 1 -- 172.21.15.78:0/2551129657 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcd400ccba0 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.076+0000 7fcd4b7fe700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcd340a11d0 0x7fcd340a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.076+0000 7fcd4b7fe700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcd340a11d0 0x7fcd340a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c005950 tx=0x7fcd3c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:54.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.079+0000 7fcd497fa700 1 -- 172.21.15.78:0/2551129657 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcd400960a0 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.269+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcd4c061700 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.269+0000 7fcd497fa700 1 -- 172.21.15.78:0/2551129657 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcd400996c0 con 0x7fcd4c0ff1c0 2023-10-10T17:40:54.270 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:40:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcd340a11d0 msgr2=0x7fcd340a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcd340a11d0 0x7fcd340a3680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c005950 tx=0x7fcd3c00b410 comp rx=0 tx=0).stop 2023-10-10T17:40:54.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c0ff1c0 msgr2=0x7fcd4c0699e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c0ff1c0 0x7fcd4c0699e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcd4000d980 tx=0x7fcd4000dc90 comp rx=0 tx=0).stop 2023-10-10T17:40:54.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 shutdown_connections 2023-10-10T17:40:54.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcd340a11d0 0x7fcd340a3680 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcd4c100d40 0x7fcd4c19f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcd4c0ff1c0 0x7fcd4c0699e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 --2- 172.21.15.78:0/2551129657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcd4c1003b0 0x7fcd4c19d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 >> 172.21.15.78:0/2551129657 conn(0x7fcd4c0fa7a0 msgr2=0x7fcd4c1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 shutdown_connections 2023-10-10T17:40:54.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.271+0000 7fcd5239f700 1 -- 172.21.15.78:0/2551129657 wait complete. 2023-10-10T17:40:54.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:40:54.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 -- 172.21.15.78:0/3598509662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2600ff1c0 msgr2=0x7ff2600ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 --2- 172.21.15.78:0/3598509662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2600ff1c0 0x7ff2600ff5d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff250009990 tx=0x7ff250009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 -- 172.21.15.78:0/3598509662 shutdown_connections 2023-10-10T17:40:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 --2- 172.21.15.78:0/3598509662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff260100d40 0x7ff260107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 --2- 172.21.15.78:0/3598509662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2600ff1c0 0x7ff2600ff5d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 --2- 172.21.15.78:0/3598509662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2601003b0 0x7ff260100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 -- 172.21.15.78:0/3598509662 >> 172.21.15.78:0/3598509662 conn(0x7ff2600fa7c0 msgr2=0x7ff2600fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 -- 172.21.15.78:0/3598509662 shutdown_connections 2023-10-10T17:40:54.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 -- 172.21.15.78:0/3598509662 wait complete. 2023-10-10T17:40:54.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 Processor -- start 2023-10-10T17:40:54.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.408+0000 7ff264eda700 1 -- start start 2023-10-10T17:40:54.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2600ff1c0 0x7ff260194fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2601003b0 0x7ff2601954e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff260100d40 0x7ff26019c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26019cb20 con 0x7ff260100d40 2023-10-10T17:40:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26019cc60 con 0x7ff2601003b0 2023-10-10T17:40:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26019cdf0 con 0x7ff2600ff1c0 2023-10-10T17:40:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff260100d40 0x7ff26019c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff260100d40 0x7ff26019c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42412/0 (socket says 172.21.15.78:42412) 2023-10-10T17:40:54.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 -- 172.21.15.78:0/45949483 learned_addr learned my addr 172.21.15.78:0/45949483 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25e59c700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2600ff1c0 0x7ff260194fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25dd9b700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2601003b0 0x7ff2601954e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 -- 172.21.15.78:0/45949483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2600ff1c0 msgr2=0x7ff260194fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2600ff1c0 0x7ff260194fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 -- 172.21.15.78:0/45949483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2601003b0 msgr2=0x7ff2601954e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2601003b0 0x7ff2601954e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 -- 172.21.15.78:0/45949483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2500095f0 con 0x7ff260100d40 2023-10-10T17:40:54.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff25ed9d700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff260100d40 0x7ff26019c550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff25800ef30 tx=0x7ff25800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:54.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff24f7fe700 1 -- 172.21.15.78:0/45949483 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258018070 con 0x7ff260100d40 2023-10-10T17:40:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff24f7fe700 1 -- 172.21.15.78:0/45949483 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff25800f040 con 0x7ff260100d40 2023-10-10T17:40:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26010ab20 con 0x7ff260100d40 2023-10-10T17:40:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff24f7fe700 1 -- 172.21.15.78:0/45949483 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25800cdb0 con 0x7ff260100d40 2023-10-10T17:40:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.409+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26010a990 con 0x7ff260100d40 2023-10-10T17:40:54.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.410+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2601067a0 con 0x7ff260100d40 2023-10-10T17:40:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.411+0000 7ff24f7fe700 1 -- 172.21.15.78:0/45949483 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff258003a00 con 0x7ff260100d40 2023-10-10T17:40:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.412+0000 7ff24f7fe700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff2480a11d0 0x7ff2480a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.412+0000 7ff25e59c700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff2480a11d0 0x7ff2480a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.412+0000 7ff24f7fe700 1 -- 172.21.15.78:0/45949483 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff2580cd040 con 0x7ff260100d40 2023-10-10T17:40:54.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.412+0000 7ff25e59c700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff2480a11d0 0x7ff2480a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff25000b2e0 tx=0x7ff25000b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:54.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.415+0000 7ff24f7fe700 1 -- 172.21.15.78:0/45949483 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff258096630 con 0x7ff260100d40 2023-10-10T17:40:54.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.605+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff260061700 con 0x7ff260100d40 2023-10-10T17:40:54.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.605+0000 7ff24f7fe700 1 -- 172.21.15.78:0/45949483 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff258099c50 con 0x7ff260100d40 2023-10-10T17:40:54.606 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:40:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff2480a11d0 msgr2=0x7ff2480a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff2480a11d0 0x7ff2480a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff25000b2e0 tx=0x7ff25000b5b0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff260100d40 msgr2=0x7ff26019c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff260100d40 0x7ff26019c550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff25800ef30 tx=0x7ff25800c3a0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 shutdown_connections 2023-10-10T17:40:54.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff2480a11d0 0x7ff2480a3680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2600ff1c0 0x7ff260194fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff260100d40 0x7ff26019c550 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 --2- 172.21.15.78:0/45949483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2601003b0 0x7ff2601954e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 >> 172.21.15.78:0/45949483 conn(0x7ff2600fa7c0 msgr2=0x7ff2601038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:54.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 shutdown_connections 2023-10-10T17:40:54.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.607+0000 7ff264eda700 1 -- 172.21.15.78:0/45949483 wait complete. 2023-10-10T17:40:54.620 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-10T17:40:54.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 -- 172.21.15.78:0/4013011883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70540ff110 msgr2=0x7f70540ff520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 --2- 172.21.15.78:0/4013011883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70540ff110 0x7f70540ff520 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7044009990 tx=0x7f7044009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 -- 172.21.15.78:0/4013011883 shutdown_connections 2023-10-10T17:40:54.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 --2- 172.21.15.78:0/4013011883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7054100c90 0x7f7054107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 --2- 172.21.15.78:0/4013011883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70540ff110 0x7f70540ff520 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 --2- 172.21.15.78:0/4013011883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7054100300 0x7f7054100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 -- 172.21.15.78:0/4013011883 >> 172.21.15.78:0/4013011883 conn(0x7f70540fa6f0 msgr2=0x7f70540fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:54.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 -- 172.21.15.78:0/4013011883 shutdown_connections 2023-10-10T17:40:54.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.742+0000 7f705a5b9700 1 -- 172.21.15.78:0/4013011883 wait complete. 2023-10-10T17:40:54.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 Processor -- start 2023-10-10T17:40:54.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 -- start start 2023-10-10T17:40:54.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70540ff110 0x7f7054195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7054100300 0x7f70541964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7054100c90 0x7f705419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705419db00 con 0x7f7054100c90 2023-10-10T17:40:54.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705419dc40 con 0x7f7054100300 2023-10-10T17:40:54.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f705a5b9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705419ddd0 con 0x7f70540ff110 2023-10-10T17:40:54.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7054100c90 0x7f705419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7054100c90 0x7f705419d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42426/0 (socket says 172.21.15.78:42426) 2023-10-10T17:40:54.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 -- 172.21.15.78:0/2825017617 learned_addr learned my addr 172.21.15.78:0/2825017617 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:54.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7053fff700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70540ff110 0x7f7054195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f70537fe700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7054100300 0x7f70541964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 -- 172.21.15.78:0/2825017617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70540ff110 msgr2=0x7f7054195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70540ff110 0x7f7054195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 -- 172.21.15.78:0/2825017617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7054100300 msgr2=0x7f70541964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7054100300 0x7f70541964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.743+0000 7f7058b56700 1 -- 172.21.15.78:0/2825017617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70440095f0 con 0x7f7054100c90 2023-10-10T17:40:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.744+0000 7f7058b56700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7054100c90 0x7f705419d530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f704c007ee0 tx=0x7f704c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.744+0000 7f70517fa700 1 -- 172.21.15.78:0/2825017617 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c017070 con 0x7f7054100c90 2023-10-10T17:40:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.744+0000 7f70517fa700 1 -- 172.21.15.78:0/2825017617 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f704c00f040 con 0x7f7054100c90 2023-10-10T17:40:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.744+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705410bc40 con 0x7f7054100c90 2023-10-10T17:40:54.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.744+0000 7f70517fa700 1 -- 172.21.15.78:0/2825017617 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704c005740 con 0x7f7054100c90 2023-10-10T17:40:54.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.744+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705410bab0 con 0x7f7054100c90 2023-10-10T17:40:54.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.745+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7054106060 con 0x7f7054100c90 2023-10-10T17:40:54.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.746+0000 7f70517fa700 1 -- 172.21.15.78:0/2825017617 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f704c00dc60 con 0x7f7054100c90 2023-10-10T17:40:54.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.746+0000 7f70517fa700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f703c0a1150 0x7f703c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:54.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.746+0000 7f70517fa700 1 -- 172.21.15.78:0/2825017617 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f704c0cc340 con 0x7f7054100c90 2023-10-10T17:40:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.746+0000 7f7053fff700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f703c0a1150 0x7f703c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.747+0000 7f7053fff700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f703c0a1150 0x7f703c0a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f704400b5c0 tx=0x7f7044005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:54.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.750+0000 7f70517fa700 1 -- 172.21.15.78:0/2825017617 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f704c0958a0 con 0x7f7054100c90 2023-10-10T17:40:54.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:54 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2447010034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:54 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4143436809' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:54 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2551129657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:54 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2447010034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:54 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4143436809' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:54 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2551129657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.958+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f705402cfe0 con 0x7f7054100c90 2023-10-10T17:40:54.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.958+0000 7f70517fa700 1 -- 172.21.15.78:0/2825017617 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+589 (secure 0 0 0) 0x7f704c095020 con 0x7f7054100c90 2023-10-10T17:40:54.959 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2447900312 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:54.960 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/942965343 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:54.960 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1662641434 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:54.960 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3346226185 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:54.960 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1720027577 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:54.960 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3346226185 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:54.961 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/843244395 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:54.961 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/667827327 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:54.961 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2252830072 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:54.961 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1662641434 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.960+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f703c0a1150 msgr2=0x7f703c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.960+0000 7f705a5b9700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f703c0a1150 0x7f703c0a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f704400b5c0 tx=0x7f7044005960 comp rx=0 tx=0).stop 2023-10-10T17:40:54.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.960+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7054100c90 msgr2=0x7f705419d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:54.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.960+0000 7f705a5b9700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7054100c90 0x7f705419d530 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f704c007ee0 tx=0x7f704c00d3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 shutdown_connections 2023-10-10T17:40:54.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f703c0a1150 0x7f703c0a3600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70540ff110 0x7f7054195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7054100c90 0x7f705419d530 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 --2- 172.21.15.78:0/2825017617 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7054100300 0x7f70541964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 >> 172.21.15.78:0/2825017617 conn(0x7f70540fa6f0 msgr2=0x7f70540fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:54.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 shutdown_connections 2023-10-10T17:40:54.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:54.961+0000 7f705a5b9700 1 -- 172.21.15.78:0/2825017617 wait complete. 2023-10-10T17:40:54.965 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2023-10-10T17:40:54.973 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:40:54.974 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-10T17:40:54.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:54 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2447010034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:54 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4143436809' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:54 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2551129657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:40:54.992 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:55.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 -- 172.21.15.78:0/4059411924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100db0 msgr2=0x7f02d4107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/4059411924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100db0 0x7f02d4107650 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00b0a0 tx=0x7f02cc00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 -- 172.21.15.78:0/4059411924 shutdown_connections 2023-10-10T17:40:55.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/4059411924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100db0 0x7f02d4107650 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/4059411924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02d40ff170 0x7f02d40ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/4059411924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d4100400 0x7f02d4100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 -- 172.21.15.78:0/4059411924 >> 172.21.15.78:0/4059411924 conn(0x7f02d40fa750 msgr2=0x7f02d40fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:55.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 -- 172.21.15.78:0/4059411924 shutdown_connections 2023-10-10T17:40:55.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.152+0000 7f02dc1f8700 1 -- 172.21.15.78:0/4059411924 wait complete. 2023-10-10T17:40:55.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 Processor -- start 2023-10-10T17:40:55.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 -- start start 2023-10-10T17:40:55.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d40ff170 0x7f02d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100400 0x7f02d419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02d4100db0 0x7f02d419d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d419db80 con 0x7f02d4100400 2023-10-10T17:40:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d419dcf0 con 0x7f02d4100db0 2023-10-10T17:40:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02dc1f8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d419deb0 con 0x7f02d40ff170 2023-10-10T17:40:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100400 0x7f02d419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100400 0x7f02d419ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42438/0 (socket says 172.21.15.78:42438) 2023-10-10T17:40:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 -- 172.21.15.78:0/3054365790 learned_addr learned my addr 172.21.15.78:0/3054365790 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9f94700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d40ff170 0x7f02d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02da795700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02d4100db0 0x7f02d419d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 -- 172.21.15.78:0/3054365790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d40ff170 msgr2=0x7f02d4068960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d40ff170 0x7f02d4068960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 -- 172.21.15.78:0/3054365790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02d4100db0 msgr2=0x7f02d419d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02d4100db0 0x7f02d419d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 -- 172.21.15.78:0/3054365790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc009d00 con 0x7f02d4100400 2023-10-10T17:40:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02da795700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02d4100db0 0x7f02d419d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:55.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02d9793700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100400 0x7f02d419ad60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f02d000d950 tx=0x7f02d000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:55.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.153+0000 7f02caffd700 1 -- 172.21.15.78:0/3054365790 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d0009b40 con 0x7f02d4100400 2023-10-10T17:40:55.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.154+0000 7f02caffd700 1 -- 172.21.15.78:0/3054365790 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f02d0009ca0 con 0x7f02d4100400 2023-10-10T17:40:55.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.154+0000 7f02caffd700 1 -- 172.21.15.78:0/3054365790 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d000f620 con 0x7f02d4100400 2023-10-10T17:40:55.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.154+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d410ab70 con 0x7f02d4100400 2023-10-10T17:40:55.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.154+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d410a9e0 con 0x7f02d4100400 2023-10-10T17:40:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.154+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d419e5a0 con 0x7f02d4100400 2023-10-10T17:40:55.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.156+0000 7f02caffd700 1 -- 172.21.15.78:0/3054365790 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f02d0010460 con 0x7f02d4100400 2023-10-10T17:40:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.156+0000 7f02caffd700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f02c00a11d0 0x7f02c00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.156+0000 7f02d9f94700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f02c00a11d0 0x7f02c00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.156+0000 7f02caffd700 1 -- 172.21.15.78:0/3054365790 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f02d00cce30 con 0x7f02d4100400 2023-10-10T17:40:55.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.157+0000 7f02d9f94700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f02c00a11d0 0x7f02c00a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f02c4005fd0 tx=0x7f02c400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:55.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.159+0000 7f02caffd700 1 -- 172.21.15.78:0/3054365790 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f02d0096390 con 0x7f02d4100400 2023-10-10T17:40:55.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.351+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02d41060c0 con 0x7f02d4100400 2023-10-10T17:40:55.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.352+0000 7f02caffd700 1 -- 172.21.15.78:0/3054365790 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f02d0016090 con 0x7f02d4100400 2023-10-10T17:40:55.353 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:55.354 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:55.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f02c00a11d0 msgr2=0x7f02c00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f02c00a11d0 0x7f02c00a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f02c4005fd0 tx=0x7f02c400a560 comp rx=0 tx=0).stop 2023-10-10T17:40:55.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100400 msgr2=0x7f02d419ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100400 0x7f02d419ad60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f02d000d950 tx=0x7f02d000dd10 comp rx=0 tx=0).stop 2023-10-10T17:40:55.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 shutdown_connections 2023-10-10T17:40:55.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f02c00a11d0 0x7f02c00a3680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f02d40ff170 0x7f02d4068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f02d4100db0 0x7f02d419d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 --2- 172.21.15.78:0/3054365790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f02d4100400 0x7f02d419ad60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 >> 172.21.15.78:0/3054365790 conn(0x7f02d40fa750 msgr2=0x7f02d40fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:55.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 shutdown_connections 2023-10-10T17:40:55.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.355+0000 7f02dc1f8700 1 -- 172.21.15.78:0/3054365790 wait complete. 2023-10-10T17:40:55.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:40:55.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:40:55.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 -- 172.21.15.78:0/1254820807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 msgr2=0x7f1788101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 --2- 172.21.15.78:0/1254820807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788101a90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1778009990 tx=0x7f1778009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 -- 172.21.15.78:0/1254820807 shutdown_connections 2023-10-10T17:40:55.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 --2- 172.21.15.78:0/1254820807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880fe5d0 0x7f17880fe9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 --2- 172.21.15.78:0/1254820807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1788101fd0 0x7f1788108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 --2- 172.21.15.78:0/1254820807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788101a90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 -- 172.21.15.78:0/1254820807 >> 172.21.15.78:0/1254820807 conn(0x7f17880f9fb0 msgr2=0x7f17880fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:55.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 -- 172.21.15.78:0/1254820807 shutdown_connections 2023-10-10T17:40:55.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.491+0000 7f178ff9a700 1 -- 172.21.15.78:0/1254820807 wait complete. 2023-10-10T17:40:55.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 Processor -- start 2023-10-10T17:40:55.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 -- start start 2023-10-10T17:40:55.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880fe5d0 0x7f1788195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1788101fd0 0x7f178819d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178819db60 con 0x7f1788101640 2023-10-10T17:40:55.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178819dca0 con 0x7f1788101fd0 2023-10-10T17:40:55.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178ff9a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178819de30 con 0x7f17880fe5d0 2023-10-10T17:40:55.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178d535700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178d535700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42450/0 (socket says 172.21.15.78:42450) 2023-10-10T17:40:55.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178d535700 1 -- 172.21.15.78:0/135996093 learned_addr learned my addr 172.21.15.78:0/135996093 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:55.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178e537700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1788101fd0 0x7f178819d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178dd36700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880fe5d0 0x7f1788195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.492+0000 7f178d535700 1 -- 172.21.15.78:0/135996093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880fe5d0 msgr2=0x7f1788195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178d535700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880fe5d0 0x7f1788195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178d535700 1 -- 172.21.15.78:0/135996093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1788101fd0 msgr2=0x7f178819d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178d535700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1788101fd0 0x7f178819d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178d535700 1 -- 172.21.15.78:0/135996093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17780095f0 con 0x7f1788101640 2023-10-10T17:40:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178e537700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1788101fd0 0x7f178819d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178dd36700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880fe5d0 0x7f1788195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178d535700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788196520 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1778000c00 tx=0x7f177800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:55.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f177effd700 1 -- 172.21.15.78:0/135996093 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177801d070 con 0x7f1788101640 2023-10-10T17:40:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f177effd700 1 -- 172.21.15.78:0/135996093 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f177800fc00 con 0x7f1788101640 2023-10-10T17:40:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178810bc50 con 0x7f1788101640 2023-10-10T17:40:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f177effd700 1 -- 172.21.15.78:0/135996093 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778017650 con 0x7f1788101640 2023-10-10T17:40:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.493+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178810bac0 con 0x7f1788101640 2023-10-10T17:40:55.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.494+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1788107380 con 0x7f1788101640 2023-10-10T17:40:55.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.495+0000 7f177effd700 1 -- 172.21.15.78:0/135996093 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1778022470 con 0x7f1788101640 2023-10-10T17:40:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.495+0000 7f177effd700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17740a1150 0x7f17740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.495+0000 7f178dd36700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17740a1150 0x7f17740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.495+0000 7f177effd700 1 -- 172.21.15.78:0/135996093 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f17780cdda0 con 0x7f1788101640 2023-10-10T17:40:55.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.496+0000 7f178dd36700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17740a1150 0x7f17740a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1784005fd0 tx=0x7f1784005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:55.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.500+0000 7f177effd700 1 -- 172.21.15.78:0/135996093 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1778097330 con 0x7f1788101640 2023-10-10T17:40:55.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:55 smithi078 ceph-mon[99494]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 600 B/s wr, 0 op/s 2023-10-10T17:40:55.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:55 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/45949483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:40:55.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:55 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2825017617' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:40:55.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:55 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3054365790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:55.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.688+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1788061c40 con 0x7f1788101640 2023-10-10T17:40:55.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.688+0000 7f177effd700 1 -- 172.21.15.78:0/135996093 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f177809a950 con 0x7f1788101640 2023-10-10T17:40:55.689 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:40:55.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17740a1150 msgr2=0x7f17740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17740a1150 0x7f17740a3600 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1784005fd0 tx=0x7f1784005dc0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 msgr2=0x7f1788196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788196520 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1778000c00 tx=0x7f177800f710 comp rx=0 tx=0).stop 2023-10-10T17:40:55.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 shutdown_connections 2023-10-10T17:40:55.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17740a1150 0x7f17740a3600 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17880fe5d0 0x7f1788195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1788101fd0 0x7f178819d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 --2- 172.21.15.78:0/135996093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1788101640 0x7f1788196520 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 >> 172.21.15.78:0/135996093 conn(0x7f17880f9fb0 msgr2=0x7f17880fc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:55.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 shutdown_connections 2023-10-10T17:40:55.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.690+0000 7f178ff9a700 1 -- 172.21.15.78:0/135996093 wait complete. 2023-10-10T17:40:55.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:40:55.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 -- 172.21.15.78:0/1608783398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 msgr2=0x7f36dc105aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 --2- 172.21.15.78:0/1608783398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc105aa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009990 tx=0x7f36cc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 -- 172.21.15.78:0/1608783398 shutdown_connections 2023-10-10T17:40:55.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 --2- 172.21.15.78:0/1608783398 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36dc105fe0 0x7f36dc1083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 --2- 172.21.15.78:0/1608783398 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36dc100da0 0x7f36dc103180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 --2- 172.21.15.78:0/1608783398 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc105aa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 -- 172.21.15.78:0/1608783398 >> 172.21.15.78:0/1608783398 conn(0x7f36dc0fa7a0 msgr2=0x7f36dc0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 -- 172.21.15.78:0/1608783398 shutdown_connections 2023-10-10T17:40:55.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.827+0000 7f36e09fc700 1 -- 172.21.15.78:0/1608783398 wait complete. 2023-10-10T17:40:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 Processor -- start 2023-10-10T17:40:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 -- start start 2023-10-10T17:40:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36dc100da0 0x7f36dc195f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc1964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36dc105fe0 0x7f36dc19d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc19db00 con 0x7f36dc1036c0 2023-10-10T17:40:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc19dc40 con 0x7f36dc105fe0 2023-10-10T17:40:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36e09fc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36dc19ddd0 con 0x7f36dc100da0 2023-10-10T17:40:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc1964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc1964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42464/0 (socket says 172.21.15.78:42464) 2023-10-10T17:40:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 -- 172.21.15.78:0/3762643026 learned_addr learned my addr 172.21.15.78:0/3762643026 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36da59c700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36dc100da0 0x7f36dc195f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36dad9d700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36dc105fe0 0x7f36dc19d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 -- 172.21.15.78:0/3762643026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36dc100da0 msgr2=0x7f36dc195f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36dc100da0 0x7f36dc195f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 -- 172.21.15.78:0/3762643026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36dc105fe0 msgr2=0x7f36dc19d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36dc105fe0 0x7f36dc19d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 -- 172.21.15.78:0/3762643026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc0095f0 con 0x7f36dc1036c0 2023-10-10T17:40:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.828+0000 7f36d9d9b700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc1964c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f36cc000c00 tx=0x7f36cc00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:55.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.829+0000 7f36cb7fe700 1 -- 172.21.15.78:0/3762643026 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc01d070 con 0x7f36dc1036c0 2023-10-10T17:40:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.829+0000 7f36cb7fe700 1 -- 172.21.15.78:0/3762643026 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36cc00fb50 con 0x7f36dc1036c0 2023-10-10T17:40:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.829+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36dc10bc40 con 0x7f36dc1036c0 2023-10-10T17:40:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.829+0000 7f36cb7fe700 1 -- 172.21.15.78:0/3762643026 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc017650 con 0x7f36dc1036c0 2023-10-10T17:40:55.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.829+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36dc10bab0 con 0x7f36dc1036c0 2023-10-10T17:40:55.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.830+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36dc0fc310 con 0x7f36dc1036c0 2023-10-10T17:40:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.830+0000 7f36cb7fe700 1 -- 172.21.15.78:0/3762643026 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f36cc00fcc0 con 0x7f36dc1036c0 2023-10-10T17:40:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.831+0000 7f36cb7fe700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36c40a1150 0x7f36c40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:55.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.831+0000 7f36da59c700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36c40a1150 0x7f36c40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.831+0000 7f36cb7fe700 1 -- 172.21.15.78:0/3762643026 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f36cc0cde50 con 0x7f36dc1036c0 2023-10-10T17:40:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.832+0000 7f36da59c700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36c40a1150 0x7f36c40a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f36d000ac50 tx=0x7f36d000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:55.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:55.836+0000 7f36cb7fe700 1 -- 172.21.15.78:0/3762643026 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f36cc097440 con 0x7f36dc1036c0 2023-10-10T17:40:55.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:55 smithi190 ceph-mon[101220]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 600 B/s wr, 0 op/s 2023-10-10T17:40:55.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:55 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/45949483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:40:55.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:55 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2825017617' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:40:55.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:55 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3054365790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:55.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:55 smithi157 ceph-mon[100896]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 600 B/s wr, 0 op/s 2023-10-10T17:40:55.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:55 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/45949483' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:40:55.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:55 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2825017617' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:40:55.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:55 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3054365790' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:56.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.022+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f36dc061700 con 0x7f36dc1036c0 2023-10-10T17:40:56.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.022+0000 7f36cb7fe700 1 -- 172.21.15.78:0/3762643026 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f36cc09aa60 con 0x7f36dc1036c0 2023-10-10T17:40:56.024 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:40:56.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36c40a1150 msgr2=0x7f36c40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36c40a1150 0x7f36c40a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f36d000ac50 tx=0x7f36d000a380 comp rx=0 tx=0).stop 2023-10-10T17:40:56.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 msgr2=0x7f36dc1964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc1964c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f36cc000c00 tx=0x7f36cc00f710 comp rx=0 tx=0).stop 2023-10-10T17:40:56.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 shutdown_connections 2023-10-10T17:40:56.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36c40a1150 0x7f36c40a3600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36dc100da0 0x7f36dc195f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36dc105fe0 0x7f36dc19d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.024+0000 7f36e09fc700 1 --2- 172.21.15.78:0/3762643026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36dc1036c0 0x7f36dc1964c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.025+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 >> 172.21.15.78:0/3762643026 conn(0x7f36dc0fa7a0 msgr2=0x7f36dc101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:56.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.025+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 shutdown_connections 2023-10-10T17:40:56.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.025+0000 7f36e09fc700 1 -- 172.21.15.78:0/3762643026 wait complete. 2023-10-10T17:40:56.037 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:40:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 -- 172.21.15.78:0/4119014604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe33c100d40 msgr2=0x7fe33c107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 --2- 172.21.15.78:0/4119014604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe33c100d40 0x7fe33c107600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe33400b0a0 tx=0x7fe33400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 -- 172.21.15.78:0/4119014604 shutdown_connections 2023-10-10T17:40:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 --2- 172.21.15.78:0/4119014604 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe33c100d40 0x7fe33c107600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 --2- 172.21.15.78:0/4119014604 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe33c0ff1c0 0x7fe33c0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 --2- 172.21.15.78:0/4119014604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c1003b0 0x7fe33c100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 -- 172.21.15.78:0/4119014604 >> 172.21.15.78:0/4119014604 conn(0x7fe33c0fa7c0 msgr2=0x7fe33c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:56.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 -- 172.21.15.78:0/4119014604 shutdown_connections 2023-10-10T17:40:56.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.162+0000 7fe342d19700 1 -- 172.21.15.78:0/4119014604 wait complete. 2023-10-10T17:40:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 Processor -- start 2023-10-10T17:40:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 -- start start 2023-10-10T17:40:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c0ff1c0 0x7fe33c195f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe33c1003b0 0x7fe33c196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe33c100d40 0x7fe33c19d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c19dad0 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c19dc10 con 0x7fe33c1003b0 2023-10-10T17:40:56.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe342d19700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33c19dda0 con 0x7fe33c100d40 2023-10-10T17:40:56.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c0ff1c0 0x7fe33c195f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c0ff1c0 0x7fe33c195f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42476/0 (socket says 172.21.15.78:42476) 2023-10-10T17:40:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 -- 172.21.15.78:0/1486552499 learned_addr learned my addr 172.21.15.78:0/1486552499 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe33bfff700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe33c1003b0 0x7fe33c196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 -- 172.21.15.78:0/1486552499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe33c100d40 msgr2=0x7fe33c19d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe33c100d40 0x7fe33c19d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 -- 172.21.15.78:0/1486552499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe33c1003b0 msgr2=0x7fe33c196490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe33c1003b0 0x7fe33c196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 -- 172.21.15.78:0/1486552499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe334009d00 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe340ab5700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c0ff1c0 0x7fe33c195f50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe33000d980 tx=0x7fe33000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:56.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.163+0000 7fe339ffb700 1 -- 172.21.15.78:0/1486552499 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33000de60 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.164+0000 7fe339ffb700 1 -- 172.21.15.78:0/1486552499 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3300044e0 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.164+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33c10aa80 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.164+0000 7fe339ffb700 1 -- 172.21.15.78:0/1486552499 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330003be0 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.164+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33c10a8f0 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.165+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe33c107250 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.166+0000 7fe339ffb700 1 -- 172.21.15.78:0/1486552499 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe330009bb0 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.166+0000 7fe339ffb700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe32c0a1150 0x7fe32c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.166+0000 7fe339ffb700 1 -- 172.21.15.78:0/1486552499 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe3300cc8b0 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.166+0000 7fe33bfff700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe32c0a1150 0x7fe32c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.167+0000 7fe33bfff700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe32c0a1150 0x7fe32c0a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe328009b30 tx=0x7fe328009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:56.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.170+0000 7fe339ffb700 1 -- 172.21.15.78:0/1486552499 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe330095e10 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.359+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe33c061930 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.360+0000 7fe339ffb700 1 -- 172.21.15.78:0/1486552499 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe330009940 con 0x7fe33c0ff1c0 2023-10-10T17:40:56.361 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:40:56.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.361+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe32c0a1150 msgr2=0x7fe32c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.361+0000 7fe342d19700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe32c0a1150 0x7fe32c0a3600 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe328009b30 tx=0x7fe328009260 comp rx=0 tx=0).stop 2023-10-10T17:40:56.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.361+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c0ff1c0 msgr2=0x7fe33c195f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.361+0000 7fe342d19700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c0ff1c0 0x7fe33c195f50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe33000d980 tx=0x7fe33000dd40 comp rx=0 tx=0).stop 2023-10-10T17:40:56.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 shutdown_connections 2023-10-10T17:40:56.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe32c0a1150 0x7fe32c0a3600 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe33c100d40 0x7fe33c19d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe33c0ff1c0 0x7fe33c195f50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 --2- 172.21.15.78:0/1486552499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe33c1003b0 0x7fe33c196490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 >> 172.21.15.78:0/1486552499 conn(0x7fe33c0fa7c0 msgr2=0x7fe33c0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:56.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 shutdown_connections 2023-10-10T17:40:56.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.362+0000 7fe342d19700 1 -- 172.21.15.78:0/1486552499 wait complete. 2023-10-10T17:40:56.374 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:40:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 -- 172.21.15.78:0/3235158697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea40fe590 msgr2=0x7f6ea40fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 --2- 172.21.15.78:0/3235158697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea40fe590 0x7f6ea40fe9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6e94009990 tx=0x7f6e94009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 -- 172.21.15.78:0/3235158697 shutdown_connections 2023-10-10T17:40:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 --2- 172.21.15.78:0/3235158697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea4101f90 0x7f6ea4108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 --2- 172.21.15.78:0/3235158697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea40fe590 0x7f6ea40fe9a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 --2- 172.21.15.78:0/3235158697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ea4101600 0x7f6ea4101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 -- 172.21.15.78:0/3235158697 >> 172.21.15.78:0/3235158697 conn(0x7f6ea40f9f30 msgr2=0x7f6ea40fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 -- 172.21.15.78:0/3235158697 shutdown_connections 2023-10-10T17:40:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.497+0000 7f6ea900f700 1 -- 172.21.15.78:0/3235158697 wait complete. 2023-10-10T17:40:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 Processor -- start 2023-10-10T17:40:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 -- start start 2023-10-10T17:40:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ea40fe590 0x7f6ea4196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea4101600 0x7f6ea4196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea4101f90 0x7f6ea419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea419dbb0 con 0x7f6ea4101600 2023-10-10T17:40:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea419dcf0 con 0x7f6ea40fe590 2023-10-10T17:40:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea900f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea419de80 con 0x7f6ea4101f90 2023-10-10T17:40:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea4101600 0x7f6ea4196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea4101600 0x7f6ea4196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42492/0 (socket says 172.21.15.78:42492) 2023-10-10T17:40:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea259c700 1 -- 172.21.15.78:0/616521804 learned_addr learned my addr 172.21.15.78:0/616521804 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.498+0000 7f6ea2d9d700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ea40fe590 0x7f6ea4196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea359e700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea4101f90 0x7f6ea419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea259c700 1 -- 172.21.15.78:0/616521804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea4101f90 msgr2=0x7f6ea419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea259c700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea4101f90 0x7f6ea419d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea259c700 1 -- 172.21.15.78:0/616521804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ea40fe590 msgr2=0x7f6ea4196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea259c700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ea40fe590 0x7f6ea4196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea259c700 1 -- 172.21.15.78:0/616521804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e940095f0 con 0x7f6ea4101600 2023-10-10T17:40:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea259c700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea4101600 0x7f6ea4196570 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d950 tx=0x7f6e9800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6e8bfff700 1 -- 172.21.15.78:0/616521804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98009940 con 0x7f6ea4101600 2023-10-10T17:40:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6e8bfff700 1 -- 172.21.15.78:0/616521804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6e98010460 con 0x7f6ea4101600 2023-10-10T17:40:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6e8bfff700 1 -- 172.21.15.78:0/616521804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800f620 con 0x7f6ea4101600 2023-10-10T17:40:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea410bc00 con 0x7f6ea4101600 2023-10-10T17:40:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.499+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea410ba70 con 0x7f6ea4101600 2023-10-10T17:40:56.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.500+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea41079a0 con 0x7f6ea4101600 2023-10-10T17:40:56.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.501+0000 7f6e8bfff700 1 -- 172.21.15.78:0/616521804 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6e980108d0 con 0x7f6ea4101600 2023-10-10T17:40:56.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.501+0000 7f6e8bfff700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e8c0a1150 0x7f6e8c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.502+0000 7f6ea2d9d700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e8c0a1150 0x7f6e8c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.502+0000 7f6e8bfff700 1 -- 172.21.15.78:0/616521804 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6e980cc990 con 0x7f6ea4101600 2023-10-10T17:40:56.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.502+0000 7f6ea2d9d700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e8c0a1150 0x7f6e8c0a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400b5c0 tx=0x7f6e94005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:56.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.505+0000 7f6e8bfff700 1 -- 172.21.15.78:0/616521804 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6e98095ef0 con 0x7f6ea4101600 2023-10-10T17:40:56.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:56 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/135996093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:56 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3762643026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:56 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1486552499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.697+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6ea4061700 con 0x7f6ea4101600 2023-10-10T17:40:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.697+0000 7f6e8bfff700 1 -- 172.21.15.78:0/616521804 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6e98020020 con 0x7f6ea4101600 2023-10-10T17:40:56.700 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:40:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e8c0a1150 msgr2=0x7f6e8c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e8c0a1150 0x7f6e8c0a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6e9400b5c0 tx=0x7f6e94005fd0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea4101600 msgr2=0x7f6ea4196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea4101600 0x7f6ea4196570 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800d950 tx=0x7f6e9800dc60 comp rx=0 tx=0).stop 2023-10-10T17:40:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 shutdown_connections 2023-10-10T17:40:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6e8c0a1150 0x7f6e8c0a3600 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ea4101f90 0x7f6ea419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ea40fe590 0x7f6ea4196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 --2- 172.21.15.78:0/616521804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ea4101600 0x7f6ea4196570 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 >> 172.21.15.78:0/616521804 conn(0x7f6ea40f9f30 msgr2=0x7f6ea4104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:56.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 shutdown_connections 2023-10-10T17:40:56.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.699+0000 7f6ea900f700 1 -- 172.21.15.78:0/616521804 wait complete. 2023-10-10T17:40:56.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-10T17:40:56.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/4104128344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100300 msgr2=0x7f1ccc100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/4104128344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100300 0x7f1ccc100750 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009990 tx=0x7f1cb8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/4104128344 shutdown_connections 2023-10-10T17:40:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/4104128344 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ccc0ff110 0x7f1ccc0ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/4104128344 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ccc100c90 0x7f1ccc107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/4104128344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100300 0x7f1ccc100750 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/4104128344 >> 172.21.15.78:0/4104128344 conn(0x7f1ccc0fa6f0 msgr2=0x7f1ccc0fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/4104128344 shutdown_connections 2023-10-10T17:40:56.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.834+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/4104128344 wait complete. 2023-10-10T17:40:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 Processor -- start 2023-10-10T17:40:56.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 -- start start 2023-10-10T17:40:56.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ccc0ff110 0x7f1ccc195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ccc100300 0x7f1ccc1964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100c90 0x7f1ccc19d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc19db20 con 0x7f1ccc100c90 2023-10-10T17:40:56.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc19dc60 con 0x7f1ccc100300 2023-10-10T17:40:56.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd2ca7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ccc19ddf0 con 0x7f1ccc0ff110 2023-10-10T17:40:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100c90 0x7f1ccc19d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100c90 0x7f1ccc19d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42508/0 (socket says 172.21.15.78:42508) 2023-10-10T17:40:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 -- 172.21.15.78:0/3501739020 learned_addr learned my addr 172.21.15.78:0/3501739020 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:56.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd0a43700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ccc0ff110 0x7f1ccc195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1ccbfff700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ccc100300 0x7f1ccc1964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 -- 172.21.15.78:0/3501739020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ccc0ff110 msgr2=0x7f1ccc195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ccc0ff110 0x7f1ccc195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 -- 172.21.15.78:0/3501739020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ccc100300 msgr2=0x7f1ccc1964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:56.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ccc100300 0x7f1ccc1964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:56.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.835+0000 7f1cd1244700 1 -- 172.21.15.78:0/3501739020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb80095f0 con 0x7f1ccc100c90 2023-10-10T17:40:56.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.836+0000 7f1cd1244700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100c90 0x7f1ccc19d550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400ada0 tx=0x7f1cc400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:56.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.836+0000 7f1cc9ffb700 1 -- 172.21.15.78:0/3501739020 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc400f070 con 0x7f1ccc100c90 2023-10-10T17:40:56.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.836+0000 7f1cc9ffb700 1 -- 172.21.15.78:0/3501739020 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cc4007b30 con 0x7f1ccc100c90 2023-10-10T17:40:56.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.836+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ccc10bc40 con 0x7f1ccc100c90 2023-10-10T17:40:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.836+0000 7f1cc9ffb700 1 -- 172.21.15.78:0/3501739020 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc40053e0 con 0x7f1ccc100c90 2023-10-10T17:40:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.836+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ccc10bab0 con 0x7f1ccc100c90 2023-10-10T17:40:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.837+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ccc106680 con 0x7f1ccc100c90 2023-10-10T17:40:56.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.838+0000 7f1cc9ffb700 1 -- 172.21.15.78:0/3501739020 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1cc400c980 con 0x7f1ccc100c90 2023-10-10T17:40:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.839+0000 7f1cc9ffb700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1cbc0a1150 0x7f1cbc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.839+0000 7f1cd0a43700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1cbc0a1150 0x7f1cbc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:56.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.839+0000 7f1cc9ffb700 1 -- 172.21.15.78:0/3501739020 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1cc40cc130 con 0x7f1ccc100c90 2023-10-10T17:40:56.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.839+0000 7f1cd0a43700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1cbc0a1150 0x7f1cbc0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc100160 tx=0x7f1cc0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:56.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:56.844+0000 7f1cc9ffb700 1 -- 172.21.15.78:0/3501739020 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1cc40956c0 con 0x7f1ccc100c90 2023-10-10T17:40:56.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:56 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/135996093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:56 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3762643026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:56 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1486552499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:56 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/135996093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:56 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3762643026' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:40:56.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:56 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1486552499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:40:57.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.048+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1ccc02cfe0 con 0x7f1ccc100c90 2023-10-10T17:40:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.049+0000 7f1cc9ffb700 1 -- 172.21.15.78:0/3501739020 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+589 (secure 0 0 0) 0x7f1cc4098ce0 con 0x7f1ccc100c90 2023-10-10T17:40:57.051 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2447900312 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:57.051 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/942965343 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:57.051 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1662641434 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3346226185 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1720027577 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:57.052 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3346226185 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/843244395 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:57.053 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/667827327 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:57.054 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2252830072 2023-10-11T17:34:13.659771+0000 2023-10-10T17:40:57.054 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1662641434 2023-10-11T17:34:36.143528+0000 2023-10-10T17:40:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1cbc0a1150 msgr2=0x7f1cbc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1cbc0a1150 0x7f1cbc0a3600 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc100160 tx=0x7f1cc0009260 comp rx=0 tx=0).stop 2023-10-10T17:40:57.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100c90 msgr2=0x7f1ccc19d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100c90 0x7f1ccc19d550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f1cc400ada0 tx=0x7f1cc400c330 comp rx=0 tx=0).stop 2023-10-10T17:40:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 shutdown_connections 2023-10-10T17:40:57.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1cbc0a1150 0x7f1cbc0a3600 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ccc0ff110 0x7f1ccc195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ccc100c90 0x7f1ccc19d550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 --2- 172.21.15.78:0/3501739020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ccc100300 0x7f1ccc1964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 >> 172.21.15.78:0/3501739020 conn(0x7f1ccc0fa6f0 msgr2=0x7f1ccc1037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 shutdown_connections 2023-10-10T17:40:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.051+0000 7f1cd2ca7700 1 -- 172.21.15.78:0/3501739020 wait complete. 2023-10-10T17:40:57.059 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2023-10-10T17:40:57.064 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2023-10-10T17:40:57.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:57.065 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-10-10T17:40:57.102 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[136656]: fuse finished with error 0 and tester_r 0 2023-10-10T17:40:57.103 INFO:teuthology.orchestra.run:waiting for 300 2023-10-10T17:40:57.180 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:40:57.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:40:57.181 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-10-10T17:40:57.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 -- 172.21.15.78:0/3986385943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49640ffb30 msgr2=0x7f4964107700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 --2- 172.21.15.78:0/3986385943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49640ffb30 0x7f4964107700 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f495c00b0a0 tx=0x7f495c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 -- 172.21.15.78:0/3986385943 shutdown_connections 2023-10-10T17:40:57.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 --2- 172.21.15.78:0/3986385943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49640ffb30 0x7f4964107700 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 --2- 172.21.15.78:0/3986385943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4964100760 0x7f4964100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 --2- 172.21.15.78:0/3986385943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49641010b0 0x7f49640ff4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 -- 172.21.15.78:0/3986385943 >> 172.21.15.78:0/3986385943 conn(0x7f49640f9ee0 msgr2=0x7f49640fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 -- 172.21.15.78:0/3986385943 shutdown_connections 2023-10-10T17:40:57.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.372+0000 7f496b619700 1 -- 172.21.15.78:0/3986385943 wait complete. 2023-10-10T17:40:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 Processor -- start 2023-10-10T17:40:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 -- start start 2023-10-10T17:40:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49640ffb30 0x7f49640fef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4964100760 0x7f496419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49641010b0 0x7f496419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496419dba0 con 0x7f4964100760 2023-10-10T17:40:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496419dd10 con 0x7f49641010b0 2023-10-10T17:40:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496419ded0 con 0x7f49640ffb30 2023-10-10T17:40:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4964100760 0x7f496419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4964100760 0x7f496419adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42524/0 (socket says 172.21.15.78:42524) 2023-10-10T17:40:57.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 -- 172.21.15.78:0/3360874423 learned_addr learned my addr 172.21.15.78:0/3360874423 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f49693b5700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49640ffb30 0x7f49640fef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4969bb6700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49641010b0 0x7f496419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 -- 172.21.15.78:0/3360874423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49640ffb30 msgr2=0x7f49640fef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49640ffb30 0x7f49640fef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 -- 172.21.15.78:0/3360874423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49641010b0 msgr2=0x7f496419d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49641010b0 0x7f496419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 -- 172.21.15.78:0/3360874423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495c009d00 con 0x7f4964100760 2023-10-10T17:40:57.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f4968bb4700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4964100760 0x7f496419adc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f496000d980 tx=0x7f496000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:57.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f495a7fc700 1 -- 172.21.15.78:0/3360874423 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4960009930 con 0x7f4964100760 2023-10-10T17:40:57.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f495a7fc700 1 -- 172.21.15.78:0/3360874423 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4960010460 con 0x7f4964100760 2023-10-10T17:40:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496410ab30 con 0x7f4964100760 2023-10-10T17:40:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.373+0000 7f495a7fc700 1 -- 172.21.15.78:0/3360874423 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496000f620 con 0x7f4964100760 2023-10-10T17:40:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.374+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496410a9a0 con 0x7f4964100760 2023-10-10T17:40:57.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.374+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496419e5c0 con 0x7f4964100760 2023-10-10T17:40:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.375+0000 7f495a7fc700 1 -- 172.21.15.78:0/3360874423 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f49600108d0 con 0x7f4964100760 2023-10-10T17:40:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.376+0000 7f495a7fc700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49500a1150 0x7f49500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.376+0000 7f495a7fc700 1 -- 172.21.15.78:0/3360874423 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f49600cc5b0 con 0x7f4964100760 2023-10-10T17:40:57.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.376+0000 7f49693b5700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49500a1150 0x7f49500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.376+0000 7f49693b5700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49500a1150 0x7f49500a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4954005950 tx=0x7f495400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:57.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.380+0000 7f495a7fc700 1 -- 172.21.15.78:0/3360874423 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4960095b10 con 0x7f4964100760 2023-10-10T17:40:57.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.569+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4964106140 con 0x7f4964100760 2023-10-10T17:40:57.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.571+0000 7f495a7fc700 1 -- 172.21.15.78:0/3360874423 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f4960099130 con 0x7f4964100760 2023-10-10T17:40:57.572 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:57.573 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:57.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49500a1150 msgr2=0x7f49500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49500a1150 0x7f49500a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4954005950 tx=0x7f495400a300 comp rx=0 tx=0).stop 2023-10-10T17:40:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4964100760 msgr2=0x7f496419adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4964100760 0x7f496419adc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f496000d980 tx=0x7f496000dc90 comp rx=0 tx=0).stop 2023-10-10T17:40:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 shutdown_connections 2023-10-10T17:40:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49500a1150 0x7f49500a3600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49640ffb30 0x7f49640fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49641010b0 0x7f496419d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 --2- 172.21.15.78:0/3360874423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4964100760 0x7f496419adc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 >> 172.21.15.78:0/3360874423 conn(0x7f49640f9ee0 msgr2=0x7f49640fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:57.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 shutdown_connections 2023-10-10T17:40:57.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.573+0000 7f496b619700 1 -- 172.21.15.78:0/3360874423 wait complete. 2023-10-10T17:40:57.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:40:57.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1330375731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100400 msgr2=0x7f7fc8100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1330375731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100400 0x7f7fc8100870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f7fb4009990 tx=0x7f7fb4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1330375731 shutdown_connections 2023-10-10T17:40:57.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1330375731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc80ff170 0x7f7fc80ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1330375731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fc8100db0 0x7f7fc8107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1330375731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100400 0x7f7fc8100870 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1330375731 >> 172.21.15.78:0/1330375731 conn(0x7f7fc80fa750 msgr2=0x7f7fc80fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1330375731 shutdown_connections 2023-10-10T17:40:57.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1330375731 wait complete. 2023-10-10T17:40:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 Processor -- start 2023-10-10T17:40:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 -- start start 2023-10-10T17:40:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.710+0000 7f7fceb6e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fc80ff170 0x7f7fc8194e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fceb6e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc8100400 0x7f7fc8195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fceb6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100db0 0x7f7fc819c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fceb6e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc819c9c0 con 0x7f7fc8100db0 2023-10-10T17:40:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fceb6e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc819cb00 con 0x7f7fc80ff170 2023-10-10T17:40:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fceb6e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc819cc90 con 0x7f7fc8100400 2023-10-10T17:40:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100db0 0x7f7fc819c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100db0 0x7f7fc819c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42542/0 (socket says 172.21.15.78:42542) 2023-10-10T17:40:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 -- 172.21.15.78:0/1183369392 learned_addr learned my addr 172.21.15.78:0/1183369392 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fc7fff700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc8100400 0x7f7fc8195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcc90a700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fc80ff170 0x7f7fc8194e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 -- 172.21.15.78:0/1183369392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc8100400 msgr2=0x7f7fc8195380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc8100400 0x7f7fc8195380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 -- 172.21.15.78:0/1183369392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fc80ff170 msgr2=0x7f7fc8194e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fc80ff170 0x7f7fc8194e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 -- 172.21.15.78:0/1183369392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40095f0 con 0x7f7fc8100db0 2023-10-10T17:40:57.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fcd10b700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100db0 0x7f7fc819c3f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0007ee0 tx=0x7f7fc000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:57.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fc5ffb700 1 -- 172.21.15.78:0/1183369392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0017070 con 0x7f7fc8100db0 2023-10-10T17:40:57.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc810ab10 con 0x7f7fc8100db0 2023-10-10T17:40:57.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fc5ffb700 1 -- 172.21.15.78:0/1183369392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fc000f040 con 0x7f7fc8100db0 2023-10-10T17:40:57.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.711+0000 7f7fc5ffb700 1 -- 172.21.15.78:0/1183369392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0005740 con 0x7f7fc8100db0 2023-10-10T17:40:57.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.712+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc810a980 con 0x7f7fc8100db0 2023-10-10T17:40:57.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.712+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc8106190 con 0x7f7fc8100db0 2023-10-10T17:40:57.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.713+0000 7f7fc5ffb700 1 -- 172.21.15.78:0/1183369392 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7fc000dc60 con 0x7f7fc8100db0 2023-10-10T17:40:57.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.714+0000 7f7fc5ffb700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7fb80a1150 0x7f7fb80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:57.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.714+0000 7f7fc5ffb700 1 -- 172.21.15.78:0/1183369392 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7fc00cc210 con 0x7f7fc8100db0 2023-10-10T17:40:57.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.714+0000 7f7fcc90a700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7fb80a1150 0x7f7fb80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:57.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.714+0000 7f7fcc90a700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7fb80a1150 0x7f7fb80a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00bbd0 tx=0x7f7fbc00b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:57.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.718+0000 7f7fc5ffb700 1 -- 172.21.15.78:0/1183369392 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7fc0095770 con 0x7f7fc8100db0 2023-10-10T17:40:57.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:57 smithi078 ceph-mon[99494]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 505 B/s wr, 0 op/s 2023-10-10T17:40:57.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:57 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/616521804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:40:57.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:57 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3501739020' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:40:57.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.909+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fc802cfe0 con 0x7f7fc8100db0 2023-10-10T17:40:57.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.911+0000 7f7fc5ffb700 1 -- 172.21.15.78:0/1183369392 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f7fc0098d90 con 0x7f7fc8100db0 2023-10-10T17:40:57.911 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:40:57.912 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:40:57.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7fb80a1150 msgr2=0x7f7fb80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7fb80a1150 0x7f7fb80a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc00bbd0 tx=0x7f7fbc00b5b0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100db0 msgr2=0x7f7fc819c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:57.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100db0 0x7f7fc819c3f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7fc0007ee0 tx=0x7f7fc000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 shutdown_connections 2023-10-10T17:40:57.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7fb80a1150 0x7f7fb80a3600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7fc8100db0 0x7f7fc819c3f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7fc80ff170 0x7f7fc8194e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 --2- 172.21.15.78:0/1183369392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7fc8100400 0x7f7fc8195380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:57.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 >> 172.21.15.78:0/1183369392 conn(0x7f7fc80fa750 msgr2=0x7f7fc80fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:57.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 shutdown_connections 2023-10-10T17:40:57.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:57.913+0000 7f7fceb6e700 1 -- 172.21.15.78:0/1183369392 wait complete. 2023-10-10T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:57 smithi190 ceph-mon[101220]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 505 B/s wr, 0 op/s 2023-10-10T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:57 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/616521804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:40:57.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:57 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3501739020' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:40:57.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-10T17:40:57.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:57 smithi157 ceph-mon[100896]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 505 B/s wr, 0 op/s 2023-10-10T17:40:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:57 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/616521804' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:40:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:57 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3501739020' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:40:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 -- 172.21.15.78:0/3382342281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a601003b0 msgr2=0x7f5a60100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3382342281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a601003b0 0x7f5a60100800 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5a50009990 tx=0x7f5a50009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 -- 172.21.15.78:0/3382342281 shutdown_connections 2023-10-10T17:40:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3382342281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a600ff1c0 0x7f5a600ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3382342281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a60100d40 0x7f5a60107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3382342281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a601003b0 0x7f5a60100800 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 -- 172.21.15.78:0/3382342281 >> 172.21.15.78:0/3382342281 conn(0x7f5a600fa7c0 msgr2=0x7f5a600fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 -- 172.21.15.78:0/3382342281 shutdown_connections 2023-10-10T17:40:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.049+0000 7f5a65df8700 1 -- 172.21.15.78:0/3382342281 wait complete. 2023-10-10T17:40:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 Processor -- start 2023-10-10T17:40:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 -- start start 2023-10-10T17:40:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a600ff1c0 0x7f5a60103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a601003b0 0x7f5a60101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a60100d40 0x7f5a60101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a60102470 con 0x7f5a60100d40 2023-10-10T17:40:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601025e0 con 0x7f5a600ff1c0 2023-10-10T17:40:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a65df8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a601a00e0 con 0x7f5a601003b0 2023-10-10T17:40:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a5ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a60100d40 0x7f5a60101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a5ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a60100d40 0x7f5a60101cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42554/0 (socket says 172.21.15.78:42554) 2023-10-10T17:40:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.050+0000 7f5a5ffff700 1 -- 172.21.15.78:0/3692923171 learned_addr learned my addr 172.21.15.78:0/3692923171 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5f7fe700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a600ff1c0 0x7f5a60103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5effd700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a601003b0 0x7f5a60101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5ffff700 1 -- 172.21.15.78:0/3692923171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a601003b0 msgr2=0x7f5a60101790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5ffff700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a601003b0 0x7f5a60101790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5ffff700 1 -- 172.21.15.78:0/3692923171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a600ff1c0 msgr2=0x7f5a60103150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5ffff700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a600ff1c0 0x7f5a60103150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5ffff700 1 -- 172.21.15.78:0/3692923171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a500095f0 con 0x7f5a60100d40 2023-10-10T17:40:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5f7fe700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a600ff1c0 0x7f5a60103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:40:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5ffff700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a60100d40 0x7f5a60101cd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a58007ee0 tx=0x7f5a5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5cff9700 1 -- 172.21.15.78:0/3692923171 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58017070 con 0x7f5a60100d40 2023-10-10T17:40:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6010ab30 con 0x7f5a60100d40 2023-10-10T17:40:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5cff9700 1 -- 172.21.15.78:0/3692923171 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a5800f040 con 0x7f5a60100d40 2023-10-10T17:40:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a5cff9700 1 -- 172.21.15.78:0/3692923171 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a58005740 con 0x7f5a60100d40 2023-10-10T17:40:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.051+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6010a9a0 con 0x7f5a60100d40 2023-10-10T17:40:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.052+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a60067b00 con 0x7f5a60100d40 2023-10-10T17:40:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.053+0000 7f5a5cff9700 1 -- 172.21.15.78:0/3692923171 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5a580058a0 con 0x7f5a60100d40 2023-10-10T17:40:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.054+0000 7f5a5cff9700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5a480a1150 0x7f5a480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.054+0000 7f5a5cff9700 1 -- 172.21.15.78:0/3692923171 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5a580cca00 con 0x7f5a60100d40 2023-10-10T17:40:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.054+0000 7f5a5f7fe700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5a480a1150 0x7f5a480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.054+0000 7f5a5f7fe700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5a480a1150 0x7f5a480a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a60100210 tx=0x7f5a5400b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:58.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.057+0000 7f5a5cff9700 1 -- 172.21.15.78:0/3692923171 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5a58095f00 con 0x7f5a60100d40 2023-10-10T17:40:58.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.292+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5a601a0bf0 con 0x7f5a60100d40 2023-10-10T17:40:58.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.571+0000 7f5a5cff9700 1 -- 172.21.15.78:0/3692923171 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5a58099520 con 0x7f5a60100d40 2023-10-10T17:40:58.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5a480a1150 msgr2=0x7f5a480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5a480a1150 0x7f5a480a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5a60100210 tx=0x7f5a5400b670 comp rx=0 tx=0).stop 2023-10-10T17:40:58.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a60100d40 msgr2=0x7f5a60101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a60100d40 0x7f5a60101cd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5a58007ee0 tx=0x7f5a5800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 shutdown_connections 2023-10-10T17:40:58.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5a480a1150 0x7f5a480a3600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a60100d40 0x7f5a60101cd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a600ff1c0 0x7f5a60103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 --2- 172.21.15.78:0/3692923171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a601003b0 0x7f5a60101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 >> 172.21.15.78:0/3692923171 conn(0x7f5a600fa7c0 msgr2=0x7f5a60067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:58.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 shutdown_connections 2023-10-10T17:40:58.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.574+0000 7f5a65df8700 1 -- 172.21.15.78:0/3692923171 wait complete. 2023-10-10T17:40:58.586 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True} 2023-10-10T17:40:58.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-10T17:40:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 -- 172.21.15.78:0/3905046620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26741003b0 msgr2=0x7f2674100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 --2- 172.21.15.78:0/3905046620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26741003b0 0x7f2674100800 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2664009990 tx=0x7f2664009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 -- 172.21.15.78:0/3905046620 shutdown_connections 2023-10-10T17:40:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 --2- 172.21.15.78:0/3905046620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26740ff1c0 0x7f26740ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 --2- 172.21.15.78:0/3905046620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2674100d40 0x7f2674107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 --2- 172.21.15.78:0/3905046620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26741003b0 0x7f2674100800 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 -- 172.21.15.78:0/3905046620 >> 172.21.15.78:0/3905046620 conn(0x7f26740fa7c0 msgr2=0x7f26740fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:58.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 -- 172.21.15.78:0/3905046620 shutdown_connections 2023-10-10T17:40:58.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.709+0000 7f2678b16700 1 -- 172.21.15.78:0/3905046620 wait complete. 2023-10-10T17:40:58.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 Processor -- start 2023-10-10T17:40:58.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 -- start start 2023-10-10T17:40:58.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26740ff1c0 0x7f2674068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26741003b0 0x7f267419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2674100d40 0x7f267419d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267419db80 con 0x7f26740ff1c0 2023-10-10T17:40:58.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267419dcf0 con 0x7f26741003b0 2023-10-10T17:40:58.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f2678b16700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267419deb0 con 0x7f2674100d40 2023-10-10T17:40:58.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f267259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26740ff1c0 0x7f2674068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:58.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f267259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26740ff1c0 0x7f2674068960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42564/0 (socket says 172.21.15.78:42564) 2023-10-10T17:40:58.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.710+0000 7f267259c700 1 -- 172.21.15.78:0/2582323815 learned_addr learned my addr 172.21.15.78:0/2582323815 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:58.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f2671d9b700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26741003b0 0x7f267419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:58.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f267259c700 1 -- 172.21.15.78:0/2582323815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2674100d40 msgr2=0x7f267419d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f267259c700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2674100d40 0x7f267419d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f267259c700 1 -- 172.21.15.78:0/2582323815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26741003b0 msgr2=0x7f267419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:58.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f267259c700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26741003b0 0x7f267419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f267259c700 1 -- 172.21.15.78:0/2582323815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26640095f0 con 0x7f26740ff1c0 2023-10-10T17:40:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f2671d9b700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26741003b0 0x7f267419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f267259c700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26740ff1c0 0x7f2674068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f266800d980 tx=0x7f266800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f26637fe700 1 -- 172.21.15.78:0/2582323815 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266800de60 con 0x7f26740ff1c0 2023-10-10T17:40:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f26637fe700 1 -- 172.21.15.78:0/2582323815 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26680044e0 con 0x7f26740ff1c0 2023-10-10T17:40:58.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267410ab70 con 0x7f26740ff1c0 2023-10-10T17:40:58.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f26637fe700 1 -- 172.21.15.78:0/2582323815 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2668003be0 con 0x7f26740ff1c0 2023-10-10T17:40:58.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.711+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267410a9e0 con 0x7f26740ff1c0 2023-10-10T17:40:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.712+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26741060f0 con 0x7f26740ff1c0 2023-10-10T17:40:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.713+0000 7f26637fe700 1 -- 172.21.15.78:0/2582323815 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2668009bb0 con 0x7f26740ff1c0 2023-10-10T17:40:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.714+0000 7f26637fe700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f265c0a1150 0x7f265c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.714+0000 7f2671d9b700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f265c0a1150 0x7f265c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.714+0000 7f26637fe700 1 -- 172.21.15.78:0/2582323815 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f26680cc910 con 0x7f26740ff1c0 2023-10-10T17:40:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.714+0000 7f2671d9b700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f265c0a1150 0x7f265c0a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f266400b5c0 tx=0x7f2664003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.718+0000 7f26637fe700 1 -- 172.21.15.78:0/2582323815 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2668095e10 con 0x7f26740ff1c0 2023-10-10T17:40:58.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:58 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3360874423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:58.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:58 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1183369392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:58.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:58 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3692923171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-10-10T17:40:58.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:58 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3360874423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:58.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:58 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1183369392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:58.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:58 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3692923171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-10-10T17:40:58.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:58.950+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f2674061900 con 0x7f26740ff1c0 2023-10-10T17:40:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:58 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3360874423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:58 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1183369392' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:40:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:58 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3692923171' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-10-10T17:40:59.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.574+0000 7f26637fe700 1 -- 172.21.15.78:0/2582323815 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f2668009940 con 0x7f26740ff1c0 2023-10-10T17:40:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f265c0a1150 msgr2=0x7f265c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f265c0a1150 0x7f265c0a3600 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f266400b5c0 tx=0x7f2664003680 comp rx=0 tx=0).stop 2023-10-10T17:40:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26740ff1c0 msgr2=0x7f2674068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26740ff1c0 0x7f2674068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f266800d980 tx=0x7f266800dd40 comp rx=0 tx=0).stop 2023-10-10T17:40:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 shutdown_connections 2023-10-10T17:40:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f265c0a1150 0x7f265c0a3600 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2674100d40 0x7f267419d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26740ff1c0 0x7f2674068960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 --2- 172.21.15.78:0/2582323815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26741003b0 0x7f267419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 >> 172.21.15.78:0/2582323815 conn(0x7f26740fa7c0 msgr2=0x7f26740fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 shutdown_connections 2023-10-10T17:40:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.577+0000 7f2678b16700 1 -- 172.21.15.78:0/2582323815 wait complete. 2023-10-10T17:40:59.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-10-10T17:40:59.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 -- 172.21.15.78:0/1923678233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23181014e0 msgr2=0x7f2318101930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:59.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 --2- 172.21.15.78:0/1923678233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23181014e0 0x7f2318101930 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2308009990 tx=0x7f2308009ca0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 -- 172.21.15.78:0/1923678233 shutdown_connections 2023-10-10T17:40:59.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 --2- 172.21.15.78:0/1923678233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2318101e70 0x7f2318108730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 --2- 172.21.15.78:0/1923678233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23181002f0 0x7f2318100700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 --2- 172.21.15.78:0/1923678233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23181014e0 0x7f2318101930 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 -- 172.21.15.78:0/1923678233 >> 172.21.15.78:0/1923678233 conn(0x7f23180fb930 msgr2=0x7f23180fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:40:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 -- 172.21.15.78:0/1923678233 shutdown_connections 2023-10-10T17:40:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.723+0000 7f231da35700 1 -- 172.21.15.78:0/1923678233 wait complete. 2023-10-10T17:40:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 Processor -- start 2023-10-10T17:40:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 -- start start 2023-10-10T17:40:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23181002f0 0x7f2318196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23181014e0 0x7f2318196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:59.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318101e70 0x7f231819d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:59.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231819dcc0 con 0x7f2318101e70 2023-10-10T17:40:59.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231819de00 con 0x7f23181014e0 2023-10-10T17:40:59.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f231da35700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231819df90 con 0x7f23181002f0 2023-10-10T17:40:59.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f23177fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318101e70 0x7f231819d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:59.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f23177fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318101e70 0x7f231819d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42582/0 (socket says 172.21.15.78:42582) 2023-10-10T17:40:59.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f23177fe700 1 -- 172.21.15.78:0/3723476070 learned_addr learned my addr 172.21.15.78:0/3723476070 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:40:59.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f2316ffd700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23181002f0 0x7f2318196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:59.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f23167fc700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23181014e0 0x7f2318196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f23177fe700 1 -- 172.21.15.78:0/3723476070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23181002f0 msgr2=0x7f2318196140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.724+0000 7f23177fe700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23181002f0 0x7f2318196140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f23177fe700 1 -- 172.21.15.78:0/3723476070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23181014e0 msgr2=0x7f2318196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:40:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f23177fe700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23181014e0 0x7f2318196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:40:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f23177fe700 1 -- 172.21.15.78:0/3723476070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080095f0 con 0x7f2318101e70 2023-10-10T17:40:59.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f23167fc700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23181014e0 0x7f2318196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:40:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f23177fe700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318101e70 0x7f231819d6f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f231000ada0 tx=0x7f231000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f231ca33700 1 -- 172.21.15.78:0/3723476070 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231000f070 con 0x7f2318101e70 2023-10-10T17:40:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f231ca33700 1 -- 172.21.15.78:0/3723476070 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2310007b30 con 0x7f2318101e70 2023-10-10T17:40:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231810bd40 con 0x7f2318101e70 2023-10-10T17:40:59.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f231ca33700 1 -- 172.21.15.78:0/3723476070 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23100053e0 con 0x7f2318101e70 2023-10-10T17:40:59.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.725+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231810bbb0 con 0x7f2318101e70 2023-10-10T17:40:59.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.726+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2318107880 con 0x7f2318101e70 2023-10-10T17:40:59.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.727+0000 7f231ca33700 1 -- 172.21.15.78:0/3723476070 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f231000c980 con 0x7f2318101e70 2023-10-10T17:40:59.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.727+0000 7f231ca33700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23000a1150 0x7f23000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:40:59.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.728+0000 7f2316ffd700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23000a1150 0x7f23000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:40:59.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.728+0000 7f231ca33700 1 -- 172.21.15.78:0/3723476070 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f23100cc0d0 con 0x7f2318101e70 2023-10-10T17:40:59.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.728+0000 7f2316ffd700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23000a1150 0x7f23000a3600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2318101340 tx=0x7f230c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:40:59.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.732+0000 7f231ca33700 1 -- 172.21.15.78:0/3723476070 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f23100956c0 con 0x7f2318101e70 2023-10-10T17:40:59.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:59 smithi078 ceph-mon[99494]: pgmap v270: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-10-10T17:40:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:59 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3692923171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-10-10T17:40:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:59 smithi078 ceph-mon[99494]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:59.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:40:59 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2582323815' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-10T17:40:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:59 smithi190 ceph-mon[101220]: pgmap v270: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-10-10T17:40:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:59 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3692923171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-10-10T17:40:59.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:59 smithi190 ceph-mon[101220]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:59.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:40:59 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2582323815' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-10T17:40:59.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:40:59.964+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f231802cfe0 con 0x7f2318101e70 2023-10-10T17:40:59.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:59 smithi157 ceph-mon[100896]: pgmap v270: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-10-10T17:40:59.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:59 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3692923171' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-10-10T17:40:59.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:59 smithi157 ceph-mon[100896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:40:59.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:40:59 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2582323815' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-10-10T17:41:00.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.584+0000 7f231ca33700 1 -- 172.21.15.78:0/3723476070 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f2310098ce0 con 0x7f2318101e70 2023-10-10T17:41:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23000a1150 msgr2=0x7f23000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23000a1150 0x7f23000a3600 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2318101340 tx=0x7f230c009190 comp rx=0 tx=0).stop 2023-10-10T17:41:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318101e70 msgr2=0x7f231819d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318101e70 0x7f231819d6f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f231000ada0 tx=0x7f231000c330 comp rx=0 tx=0).stop 2023-10-10T17:41:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 shutdown_connections 2023-10-10T17:41:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23000a1150 0x7f23000a3600 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23181002f0 0x7f2318196140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2318101e70 0x7f231819d6f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 --2- 172.21.15.78:0/3723476070 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f23181014e0 0x7f2318196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.587+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 >> 172.21.15.78:0/3723476070 conn(0x7f23180fb930 msgr2=0x7f23181049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.588+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 shutdown_connections 2023-10-10T17:41:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.588+0000 7f231da35700 1 -- 172.21.15.78:0/3723476070 wait complete. 2023-10-10T17:41:00.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-10T17:41:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.736+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/3536178599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 msgr2=0x7f2c28107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.736+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/3536178599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c28107600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000b0a0 tx=0x7f2c2000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/3536178599 shutdown_connections 2023-10-10T17:41:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/3536178599 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c28107600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/3536178599 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c280ff1c0 0x7f2c280ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/3536178599 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c281003b0 0x7f2c28100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/3536178599 >> 172.21.15.78:0/3536178599 conn(0x7f2c280fa7c0 msgr2=0x7f2c280fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/3536178599 shutdown_connections 2023-10-10T17:41:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/3536178599 wait complete. 2023-10-10T17:41:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 Processor -- start 2023-10-10T17:41:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- start start 2023-10-10T17:41:00.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c280ff1c0 0x7f2c28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c281003b0 0x7f2c2819adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c2819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2819dc10 con 0x7f2c28100d40 2023-10-10T17:41:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2819dd80 con 0x7f2c280ff1c0 2023-10-10T17:41:00.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2ee0c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2819df40 con 0x7f2c281003b0 2023-10-10T17:41:00.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2d3a9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c2819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:00.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2d3a9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c2819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42596/0 (socket says 172.21.15.78:42596) 2023-10-10T17:41:00.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.737+0000 7f2c2d3a9700 1 -- 172.21.15.78:0/1661221494 learned_addr learned my addr 172.21.15.78:0/1661221494 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:00.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2cba8700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c280ff1c0 0x7f2c28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:00.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c27fff700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c281003b0 0x7f2c2819adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:00.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2d3a9700 1 -- 172.21.15.78:0/1661221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c281003b0 msgr2=0x7f2c2819adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:00.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2d3a9700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c281003b0 0x7f2c2819adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2d3a9700 1 -- 172.21.15.78:0/1661221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c280ff1c0 msgr2=0x7f2c28068960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:00.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2d3a9700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c280ff1c0 0x7f2c28068960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:00.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2d3a9700 1 -- 172.21.15.78:0/1661221494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20009d00 con 0x7f2c28100d40 2023-10-10T17:41:00.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c27fff700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c281003b0 0x7f2c2819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:00.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2d3a9700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c2819d440 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000d010 tx=0x7f2c20004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:00.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c25ffb700 1 -- 172.21.15.78:0/1661221494 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000e040 con 0x7f2c28100d40 2023-10-10T17:41:00.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c25ffb700 1 -- 172.21.15.78:0/1661221494 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c2001db50 con 0x7f2c28100d40 2023-10-10T17:41:00.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2810ab40 con 0x7f2c28100d40 2023-10-10T17:41:00.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c25ffb700 1 -- 172.21.15.78:0/1661221494 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2001bc00 con 0x7f2c28100d40 2023-10-10T17:41:00.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.738+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2810a9b0 con 0x7f2c28100d40 2023-10-10T17:41:00.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.739+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c28067b00 con 0x7f2c28100d40 2023-10-10T17:41:00.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.740+0000 7f2c25ffb700 1 -- 172.21.15.78:0/1661221494 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2c2001d690 con 0x7f2c28100d40 2023-10-10T17:41:00.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.741+0000 7f2c25ffb700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f2c180a1150 0x7f2c180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:00.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.741+0000 7f2c2cba8700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f2c180a1150 0x7f2c180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:00.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.741+0000 7f2c25ffb700 1 -- 172.21.15.78:0/1661221494 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2c200cef00 con 0x7f2c28100d40 2023-10-10T17:41:00.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.741+0000 7f2c2cba8700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f2c180a1150 0x7f2c180a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c005fd0 tx=0x7f2c1c005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:00.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.744+0000 7f2c25ffb700 1 -- 172.21.15.78:0/1661221494 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2c20098490 con 0x7f2c28100d40 2023-10-10T17:41:00.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:00 smithi078 ceph-mon[99494]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-10-10T17:41:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:00 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2582323815' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-10-10T17:41:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:00 smithi078 ceph-mon[99494]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-10-10T17:41:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:00 smithi078 ceph-mon[99494]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:41:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:00 smithi078 ceph-mon[99494]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-10-10T17:41:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:00 smithi078 ceph-mon[99494]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-10-10T17:41:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:00 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3723476070' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-10-10T17:41:00.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:00 smithi190 ceph-mon[101220]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-10-10T17:41:00.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:00 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2582323815' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-10-10T17:41:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:00 smithi190 ceph-mon[101220]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-10-10T17:41:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:00 smithi190 ceph-mon[101220]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:41:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:00 smithi190 ceph-mon[101220]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-10-10T17:41:00.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:00 smithi190 ceph-mon[101220]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-10-10T17:41:00.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:00 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3723476070' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-10-10T17:41:00.964 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-mon[100896]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-10-10T17:41:00.964 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2582323815' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-10-10T17:41:00.965 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-mon[100896]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-10-10T17:41:00.965 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-mon[100896]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-10-10T17:41:00.965 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-mon[100896]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-10-10T17:41:00.965 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-mon[100896]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-10-10T17:41:00.965 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3723476070' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-10-10T17:41:00.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:00.985+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f2c28106020 con 0x7f2c28100d40 2023-10-10T17:41:01.227 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:41:00 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: 2023-10-10T17:41:00.963+0000 7f85bbe16700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-10T17:41:01.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.589+0000 7f2c25ffb700 1 -- 172.21.15.78:0/1661221494 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2c2009bab0 con 0x7f2c28100d40 2023-10-10T17:41:01.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f2c180a1150 msgr2=0x7f2c180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:01.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f2c180a1150 0x7f2c180a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c005fd0 tx=0x7f2c1c005e50 comp rx=0 tx=0).stop 2023-10-10T17:41:01.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 msgr2=0x7f2c2819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:01.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c2819d440 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2c2000d010 tx=0x7f2c20004db0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 shutdown_connections 2023-10-10T17:41:01.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f2c180a1150 0x7f2c180a3600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c28100d40 0x7f2c2819d440 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c280ff1c0 0x7f2c28068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 --2- 172.21.15.78:0/1661221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c281003b0 0x7f2c2819adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 >> 172.21.15.78:0/1661221494 conn(0x7f2c280fa7c0 msgr2=0x7f2c280fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 shutdown_connections 2023-10-10T17:41:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.592+0000 7f2c2ee0c700 1 -- 172.21.15.78:0/1661221494 wait complete. 2023-10-10T17:41:01.608 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-10T17:41:01.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:41:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.739+0000 7f8b754af700 1 -- 172.21.15.78:0/2324114952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 msgr2=0x7f8b701008d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:01.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.739+0000 7f8b754af700 1 --2- 172.21.15.78:0/2324114952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b701008d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009990 tx=0x7f8b64009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.739+0000 7f8b754af700 1 -- 172.21.15.78:0/2324114952 shutdown_connections 2023-10-10T17:41:01.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.739+0000 7f8b754af700 1 --2- 172.21.15.78:0/2324114952 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b700ff1d0 0x7f8b700ff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.739+0000 7f8b754af700 1 --2- 172.21.15.78:0/2324114952 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b70100e10 0x7f8b701076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.739+0000 7f8b754af700 1 --2- 172.21.15.78:0/2324114952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b701008d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.739+0000 7f8b754af700 1 -- 172.21.15.78:0/2324114952 >> 172.21.15.78:0/2324114952 conn(0x7f8b700fa7d0 msgr2=0x7f8b700fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 -- 172.21.15.78:0/2324114952 shutdown_connections 2023-10-10T17:41:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 -- 172.21.15.78:0/2324114952 wait complete. 2023-10-10T17:41:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 Processor -- start 2023-10-10T17:41:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 -- start start 2023-10-10T17:41:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b700ff1d0 0x7f8b70103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b70101840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b70100e10 0x7f8b70101d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:01.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70102520 con 0x7f8b70100460 2023-10-10T17:41:01.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b70102690 con 0x7f8b70100e10 2023-10-10T17:41:01.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b754af700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b701a0020 con 0x7f8b700ff1d0 2023-10-10T17:41:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b6e7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b70101840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.740+0000 7f8b6e7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b70101840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42608/0 (socket says 172.21.15.78:42608) 2023-10-10T17:41:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6e7fc700 1 -- 172.21.15.78:0/3463984637 learned_addr learned my addr 172.21.15.78:0/3463984637 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6effd700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b700ff1d0 0x7f8b70103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6e7fc700 1 -- 172.21.15.78:0/3463984637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b700ff1d0 msgr2=0x7f8b70103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6e7fc700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b700ff1d0 0x7f8b70103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6e7fc700 1 -- 172.21.15.78:0/3463984637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b70100e10 msgr2=0x7f8b70101d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6f7fe700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b70100e10 0x7f8b70101d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6e7fc700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b70100e10 0x7f8b70101d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6e7fc700 1 -- 172.21.15.78:0/3463984637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640095f0 con 0x7f8b70100460 2023-10-10T17:41:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6f7fe700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b70100e10 0x7f8b70101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b6e7fc700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b70101840 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b64000c00 tx=0x7f8b6400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b57fff700 1 -- 172.21.15.78:0/3463984637 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6401d070 con 0x7f8b70100460 2023-10-10T17:41:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b57fff700 1 -- 172.21.15.78:0/3463984637 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b6400fc00 con 0x7f8b70100460 2023-10-10T17:41:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7010abd0 con 0x7f8b70100460 2023-10-10T17:41:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b57fff700 1 -- 172.21.15.78:0/3463984637 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b64017650 con 0x7f8b70100460 2023-10-10T17:41:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.741+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7010aa40 con 0x7f8b70100460 2023-10-10T17:41:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.742+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b701a0650 con 0x7f8b70100460 2023-10-10T17:41:01.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.743+0000 7f8b57fff700 1 -- 172.21.15.78:0/3463984637 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8b64022470 con 0x7f8b70100460 2023-10-10T17:41:01.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.744+0000 7f8b57fff700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8b580a1150 0x7f8b580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:01.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.744+0000 7f8b57fff700 1 -- 172.21.15.78:0/3463984637 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8b640cddd0 con 0x7f8b70100460 2023-10-10T17:41:01.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.744+0000 7f8b6effd700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8b580a1150 0x7f8b580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:01.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.744+0000 7f8b6effd700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8b580a1150 0x7f8b580a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8b60005950 tx=0x7f8b600058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:01.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.748+0000 7f8b57fff700 1 -- 172.21.15.78:0/3463984637 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8b640973c0 con 0x7f8b70100460 2023-10-10T17:41:01.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: pgmap v271: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-10-10T17:41:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] up:active 2023-10-10T17:41:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3723476070' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-10-10T17:41:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-10-10T17:41:01.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-10-10T17:41:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: Cluster is now healthy 2023-10-10T17:41:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-10-10T17:41:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-10-10T17:41:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-10-10T17:41:01.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:01 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1661221494' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-10-10T17:41:01.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: pgmap v271: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-10-10T17:41:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] up:active 2023-10-10T17:41:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3723476070' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-10-10T17:41:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-10-10T17:41:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-10-10T17:41:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: Cluster is now healthy 2023-10-10T17:41:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-10-10T17:41:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-10-10T17:41:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-10-10T17:41:01.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:01 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1661221494' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-10-10T17:41:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: pgmap v271: 129 pgs: 129 active+clean; 581 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-10-10T17:41:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] up:active 2023-10-10T17:41:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3723476070' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-10-10T17:41:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-10-10T17:41:01.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-10-10T17:41:01.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: Cluster is now healthy 2023-10-10T17:41:01.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-10-10T17:41:01.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2023-10-10T17:41:01.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: daemon mds.c is now active in filesystem cephfs as rank 2 2023-10-10T17:41:01.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:01 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1661221494' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-10-10T17:41:01.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.979+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b701a0c00 con 0x7f8b70100460 2023-10-10T17:41:01.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.979+0000 7f8b57fff700 1 -- 172.21.15.78:0/3463984637 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7f8b6409a9e0 con 0x7f8b70100460 2023-10-10T17:41:01.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:01.981 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:41:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8b580a1150 msgr2=0x7f8b580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8b580a1150 0x7f8b580a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8b60005950 tx=0x7f8b600058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 msgr2=0x7f8b70101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b70101840 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8b64000c00 tx=0x7f8b6400f710 comp rx=0 tx=0).stop 2023-10-10T17:41:01.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 shutdown_connections 2023-10-10T17:41:01.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8b580a1150 0x7f8b580a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8b700ff1d0 0x7f8b70103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8b70100e10 0x7f8b70101d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 --2- 172.21.15.78:0/3463984637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b70100460 0x7f8b70101840 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:01.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.981+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 >> 172.21.15.78:0/3463984637 conn(0x7f8b700fa7d0 msgr2=0x7f8b70069070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:01.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.982+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 shutdown_connections 2023-10-10T17:41:01.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:01.982+0000 7f8b754af700 1 -- 172.21.15.78:0/3463984637 wait complete. 2023-10-10T17:41:01.985 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:41:01.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:41:02.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- 172.21.15.78:0/3254056355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 msgr2=0x7f74580ff4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 --2- 172.21.15.78:0/3254056355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f74580ff4c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7454009990 tx=0x7f7454009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- 172.21.15.78:0/3254056355 shutdown_connections 2023-10-10T17:41:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 --2- 172.21.15.78:0/3254056355 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7458100780 0x7f7458100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 --2- 172.21.15.78:0/3254056355 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f74580ffb30 0x7f7458107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 --2- 172.21.15.78:0/3254056355 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f74580ff4c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- 172.21.15.78:0/3254056355 >> 172.21.15.78:0/3254056355 conn(0x7f74580f9ee0 msgr2=0x7f74580fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:02.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- 172.21.15.78:0/3254056355 shutdown_connections 2023-10-10T17:41:02.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- 172.21.15.78:0/3254056355 wait complete. 2023-10-10T17:41:02.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 Processor -- start 2023-10-10T17:41:02.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- start start 2023-10-10T17:41:02.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74580ffb30 0x7f7458103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7458100780 0x7f74581018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f7458101df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458102590 con 0x7f74581010d0 2023-10-10T17:41:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7458102700 con 0x7f7458100780 2023-10-10T17:41:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.115+0000 7f745f423700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74581a0030 con 0x7f74580ffb30 2023-10-10T17:41:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f7458101df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f7458101df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42624/0 (socket says 172.21.15.78:42624) 2023-10-10T17:41:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 -- 172.21.15.78:0/630832904 learned_addr learned my addr 172.21.15.78:0/630832904 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745c9be700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7458100780 0x7f74581018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d1bf700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74580ffb30 0x7f7458103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 -- 172.21.15.78:0/630832904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74580ffb30 msgr2=0x7f7458103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74580ffb30 0x7f7458103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 -- 172.21.15.78:0/630832904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7458100780 msgr2=0x7f74581018b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7458100780 0x7f74581018b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 -- 172.21.15.78:0/630832904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74540095f0 con 0x7f74581010d0 2023-10-10T17:41:02.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d1bf700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74580ffb30 0x7f7458103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:02.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745d9c0700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f7458101df0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7450007ee0 tx=0x7f745000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f744e7fc700 1 -- 172.21.15.78:0/630832904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450017070 con 0x7f74581010d0 2023-10-10T17:41:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f745810ab50 con 0x7f74581010d0 2023-10-10T17:41:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f744e7fc700 1 -- 172.21.15.78:0/630832904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f745000f040 con 0x7f74581010d0 2023-10-10T17:41:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f744e7fc700 1 -- 172.21.15.78:0/630832904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450005740 con 0x7f74581010d0 2023-10-10T17:41:02.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.116+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f745810a9c0 con 0x7f74581010d0 2023-10-10T17:41:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.117+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74581a0720 con 0x7f74581010d0 2023-10-10T17:41:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.118+0000 7f744e7fc700 1 -- 172.21.15.78:0/630832904 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f745000dc60 con 0x7f74581010d0 2023-10-10T17:41:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.119+0000 7f744e7fc700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f74440a1150 0x7f74440a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.119+0000 7f744e7fc700 1 -- 172.21.15.78:0/630832904 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f74500cc9b0 con 0x7f74581010d0 2023-10-10T17:41:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.119+0000 7f745d1bf700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f74440a1150 0x7f74440a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.119+0000 7f745d1bf700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f74440a1150 0x7f74440a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7448005950 tx=0x7f74480058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.122+0000 7f744e7fc700 1 -- 172.21.15.78:0/630832904 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7450095f10 con 0x7f74581010d0 2023-10-10T17:41:02.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.313+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74581a0cd0 con 0x7f74581010d0 2023-10-10T17:41:02.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.314+0000 7f744e7fc700 1 -- 172.21.15.78:0/630832904 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f7450099530 con 0x7f74581010d0 2023-10-10T17:41:02.315 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:02.315 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:41:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f74440a1150 msgr2=0x7f74440a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f74440a1150 0x7f74440a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7448005950 tx=0x7f74480058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 msgr2=0x7f7458101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f7458101df0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7450007ee0 tx=0x7f745000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 shutdown_connections 2023-10-10T17:41:02.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f74440a1150 0x7f74440a3600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f74580ffb30 0x7f7458103270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74581010d0 0x7f7458101df0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 --2- 172.21.15.78:0/630832904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7458100780 0x7f74581018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 >> 172.21.15.78:0/630832904 conn(0x7f74580f9ee0 msgr2=0x7f74580fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 shutdown_connections 2023-10-10T17:41:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.316+0000 7f745f423700 1 -- 172.21.15.78:0/630832904 wait complete. 2023-10-10T17:41:02.329 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-10T17:41:02.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T17:41:02.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 -- 172.21.15.78:0/2826746621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb80100400 msgr2=0x7fbb80100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 --2- 172.21.15.78:0/2826746621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb80100400 0x7fbb80100870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb74009990 tx=0x7fbb74009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 -- 172.21.15.78:0/2826746621 shutdown_connections 2023-10-10T17:41:02.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 --2- 172.21.15.78:0/2826746621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80100db0 0x7fbb80107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 --2- 172.21.15.78:0/2826746621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb800ff170 0x7fbb800ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 --2- 172.21.15.78:0/2826746621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb80100400 0x7fbb80100870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 -- 172.21.15.78:0/2826746621 >> 172.21.15.78:0/2826746621 conn(0x7fbb800fa750 msgr2=0x7fbb800fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:02.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 -- 172.21.15.78:0/2826746621 shutdown_connections 2023-10-10T17:41:02.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 -- 172.21.15.78:0/2826746621 wait complete. 2023-10-10T17:41:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.450+0000 7fbb85608700 1 Processor -- start 2023-10-10T17:41:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 -- start start 2023-10-10T17:41:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb800ff170 0x7fbb80195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb80100400 0x7fbb801964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80100db0 0x7fbb8019d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019daf0 con 0x7fbb800ff170 2023-10-10T17:41:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019dc30 con 0x7fbb80100400 2023-10-10T17:41:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8019ddc0 con 0x7fbb80100db0 2023-10-10T17:41:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb800ff170 0x7fbb80195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb800ff170 0x7fbb80195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42634/0 (socket says 172.21.15.78:42634) 2023-10-10T17:41:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 -- 172.21.15.78:0/1450867452 learned_addr learned my addr 172.21.15.78:0/1450867452 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7f7fe700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80100db0 0x7fbb8019d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 -- 172.21.15.78:0/1450867452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80100db0 msgr2=0x7fbb8019d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7e7fc700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb80100400 0x7fbb801964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80100db0 0x7fbb8019d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 -- 172.21.15.78:0/1450867452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb80100400 msgr2=0x7fbb801964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb80100400 0x7fbb801964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 -- 172.21.15.78:0/1450867452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb740095f0 con 0x7fbb800ff170 2023-10-10T17:41:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7f7fe700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80100db0 0x7fbb8019d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7e7fc700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb80100400 0x7fbb801964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:02.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb7effd700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb800ff170 0x7fbb80195f70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000d950 tx=0x7fbb7000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb67fff700 1 -- 172.21.15.78:0/1450867452 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70009b40 con 0x7fbb800ff170 2023-10-10T17:41:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb67fff700 1 -- 172.21.15.78:0/1450867452 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb70009ca0 con 0x7fbb800ff170 2023-10-10T17:41:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8010abc0 con 0x7fbb800ff170 2023-10-10T17:41:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb67fff700 1 -- 172.21.15.78:0/1450867452 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7000f620 con 0x7fbb800ff170 2023-10-10T17:41:02.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.451+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8010aa30 con 0x7fbb800ff170 2023-10-10T17:41:02.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.452+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb80067a60 con 0x7fbb800ff170 2023-10-10T17:41:02.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.453+0000 7fbb67fff700 1 -- 172.21.15.78:0/1450867452 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbb70010460 con 0x7fbb800ff170 2023-10-10T17:41:02.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.454+0000 7fbb67fff700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb680a1150 0x7fbb680a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.454+0000 7fbb7e7fc700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb680a1150 0x7fbb680a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.454+0000 7fbb67fff700 1 -- 172.21.15.78:0/1450867452 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbb700ccee0 con 0x7fbb800ff170 2023-10-10T17:41:02.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.454+0000 7fbb7e7fc700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb680a1150 0x7fbb680a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400b5c0 tx=0x7fbb74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:02.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.457+0000 7fbb67fff700 1 -- 172.21.15.78:0/1450867452 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbb700963e0 con 0x7fbb800ff170 2023-10-10T17:41:02.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.644+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbb8019e990 con 0x7fbb680a1150 2023-10-10T17:41:02.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.656+0000 7fbb67fff700 1 -- 172.21.15.78:0/1450867452 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418715 (secure 0 0 0) 0x7fbb8019e990 con 0x7fbb680a1150 2023-10-10T17:41:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb680a1150 msgr2=0x7fbb680a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb680a1150 0x7fbb680a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbb7400b5c0 tx=0x7fbb74005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb800ff170 msgr2=0x7fbb80195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb800ff170 0x7fbb80195f70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000d950 tx=0x7fbb7000dd10 comp rx=0 tx=0).stop 2023-10-10T17:41:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 shutdown_connections 2023-10-10T17:41:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbb680a1150 0x7fbb680a3600 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbb80100db0 0x7fbb8019d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbb800ff170 0x7fbb80195f70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 --2- 172.21.15.78:0/1450867452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbb80100400 0x7fbb801964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.660+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 >> 172.21.15.78:0/1450867452 conn(0x7fbb800fa750 msgr2=0x7fbb80067dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.661+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 shutdown_connections 2023-10-10T17:41:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.661+0000 7fbb85608700 1 -- 172.21.15.78:0/1450867452 wait complete. 2023-10-10T17:41:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-10T17:41:02.673 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2023-10-10T17:41:02.220427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596340,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":141,"num_read_kb":124,"num_write":254,"num_write_kb":2970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79748,"kb_used_data":11492,"kb_used_omap":0,"kb_used_meta":68160,"kb_avail":1124862076,"statfs":{"total":1151940427776,"available":1151858765824,"internally_reserved":0,"allocated":11767808,"data_stored":7457835,"data_compressed":248493,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":69795840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3610,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":28,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003116"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277491+0000","last_change":"2023-10-10T17:40:41.604268+0000","last_active":"2023-10-10T17:40:46.277491+0000","last_peered":"2023-10-10T17:40:46.277491+0000","last_clean":"2023-10-10T17:40:46.277491+0000","last_became_active":"2023-10-10T17:40:41.603839+0000","last_became_peered":"2023-10-10T17:40:41.603839+0000","last_unstale":"2023-10-10T17:40:46.277491+0000","last_undegraded":"2023-10-10T17:40:46.277491+0000","last_fullsized":"2023-10-10T17:40:46.277491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:06:30.271619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275640+0000","last_change":"2023-10-10T17:40:41.603642+0000","last_active":"2023-10-10T17:40:46.275640+0000","last_peered":"2023-10-10T17:40:46.275640+0000","last_clean":"2023-10-10T17:40:46.275640+0000","last_became_active":"2023-10-10T17:40:41.603091+0000","last_became_peered":"2023-10-10T17:40:41.603091+0000","last_unstale":"2023-10-10T17:40:46.275640+0000","last_undegraded":"2023-10-10T17:40:46.275640+0000","last_fullsized":"2023-10-10T17:40:46.275640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:45:53.398893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276386+0000","last_change":"2023-10-10T17:40:41.610647+0000","last_active":"2023-10-10T17:40:46.276386+0000","last_peered":"2023-10-10T17:40:46.276386+0000","last_clean":"2023-10-10T17:40:46.276386+0000","last_became_active":"2023-10-10T17:40:41.610338+0000","last_became_peered":"2023-10-10T17:40:41.610338+0000","last_unstale":"2023-10-10T17:40:46.276386+0000","last_undegraded":"2023-10-10T17:40:46.276386+0000","last_fullsized":"2023-10-10T17:40:46.276386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:17:10.849937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276163+0000","last_change":"2023-10-10T17:40:41.607832+0000","last_active":"2023-10-10T17:40:46.276163+0000","last_peered":"2023-10-10T17:40:46.276163+0000","last_clean":"2023-10-10T17:40:46.276163+0000","last_became_active":"2023-10-10T17:40:41.607492+0000","last_became_peered":"2023-10-10T17:40:41.607492+0000","last_unstale":"2023-10-10T17:40:46.276163+0000","last_undegraded":"2023-10-10T17:40:46.276163+0000","last_fullsized":"2023-10-10T17:40:46.276163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:38:19.839774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275756+0000","last_change":"2023-10-10T17:40:41.601975+0000","last_active":"2023-10-10T17:40:46.275756+0000","last_peered":"2023-10-10T17:40:46.275756+0000","last_clean":"2023-10-10T17:40:46.275756+0000","last_became_active":"2023-10-10T17:40:41.601668+0000","last_became_peered":"2023-10-10T17:40:41.601668+0000","last_unstale":"2023-10-10T17:40:46.275756+0000","last_undegraded":"2023-10-10T17:40:46.275756+0000","last_fullsized":"2023-10-10T17:40:46.275756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:13:04.900016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100867+0000","last_change":"2023-10-10T17:40:41.594445+0000","last_active":"2023-10-10T17:40:46.100867+0000","last_peered":"2023-10-10T17:40:46.100867+0000","last_clean":"2023-10-10T17:40:46.100867+0000","last_became_active":"2023-10-10T17:40:41.594123+0000","last_became_peered":"2023-10-10T17:40:41.594123+0000","last_unstale":"2023-10-10T17:40:46.100867+0000","last_undegraded":"2023-10-10T17:40:46.100867+0000","last_fullsized":"2023-10-10T17:40:46.100867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:50:57.458496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.456896+0000","last_change":"2023-10-10T17:40:41.594628+0000","last_active":"2023-10-10T17:40:46.456896+0000","last_peered":"2023-10-10T17:40:46.456896+0000","last_clean":"2023-10-10T17:40:46.456896+0000","last_became_active":"2023-10-10T17:40:41.594325+0000","last_became_peered":"2023-10-10T17:40:41.594325+0000","last_unstale":"2023-10-10T17:40:46.456896+0000","last_undegraded":"2023-10-10T17:40:46.456896+0000","last_fullsized":"2023-10-10T17:40:46.456896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:41:16.119478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275229+0000","last_change":"2023-10-10T17:40:41.598032+0000","last_active":"2023-10-10T17:40:46.275229+0000","last_peered":"2023-10-10T17:40:46.275229+0000","last_clean":"2023-10-10T17:40:46.275229+0000","last_became_active":"2023-10-10T17:40:41.597671+0000","last_became_peered":"2023-10-10T17:40:41.597671+0000","last_unstale":"2023-10-10T17:40:46.275229+0000","last_undegraded":"2023-10-10T17:40:46.275229+0000","last_fullsized":"2023-10-10T17:40:46.275229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:45:42.730985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276566+0000","last_change":"2023-10-10T17:40:41.599770+0000","last_active":"2023-10-10T17:40:46.276566+0000","last_peered":"2023-10-10T17:40:46.276566+0000","last_clean":"2023-10-10T17:40:46.276566+0000","last_became_active":"2023-10-10T17:40:41.599449+0000","last_became_peered":"2023-10-10T17:40:41.599449+0000","last_unstale":"2023-10-10T17:40:46.276566+0000","last_undegraded":"2023-10-10T17:40:46.276566+0000","last_fullsized":"2023-10-10T17:40:46.276566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:49:34.537216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.101227+0000","last_change":"2023-10-10T17:40:41.610655+0000","last_active":"2023-10-10T17:40:46.101227+0000","last_peered":"2023-10-10T17:40:46.101227+0000","last_clean":"2023-10-10T17:40:46.101227+0000","last_became_active":"2023-10-10T17:40:41.610133+0000","last_became_peered":"2023-10-10T17:40:41.610133+0000","last_unstale":"2023-10-10T17:40:46.101227+0000","last_undegraded":"2023-10-10T17:40:46.101227+0000","last_fullsized":"2023-10-10T17:40:46.101227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:28:16.307704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457571+0000","last_change":"2023-10-10T17:40:41.601065+0000","last_active":"2023-10-10T17:40:46.457571+0000","last_peered":"2023-10-10T17:40:46.457571+0000","last_clean":"2023-10-10T17:40:46.457571+0000","last_became_active":"2023-10-10T17:40:41.600530+0000","last_became_peered":"2023-10-10T17:40:41.600530+0000","last_unstale":"2023-10-10T17:40:46.457571+0000","last_undegraded":"2023-10-10T17:40:46.457571+0000","last_fullsized":"2023-10-10T17:40:46.457571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:40:11.842872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275953+0000","last_change":"2023-10-10T17:40:41.596067+0000","last_active":"2023-10-10T17:40:46.275953+0000","last_peered":"2023-10-10T17:40:46.275953+0000","last_clean":"2023-10-10T17:40:46.275953+0000","last_became_active":"2023-10-10T17:40:41.595703+0000","last_became_peered":"2023-10-10T17:40:41.595703+0000","last_unstale":"2023-10-10T17:40:46.275953+0000","last_undegraded":"2023-10-10T17:40:46.275953+0000","last_fullsized":"2023-10-10T17:40:46.275953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:32:24.161847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213247+0000","last_change":"2023-10-10T17:40:41.593296+0000","last_active":"2023-10-10T17:40:46.213247+0000","last_peered":"2023-10-10T17:40:46.213247+0000","last_clean":"2023-10-10T17:40:46.213247+0000","last_became_active":"2023-10-10T17:40:41.592965+0000","last_became_peered":"2023-10-10T17:40:41.592965+0000","last_unstale":"2023-10-10T17:40:46.213247+0000","last_undegraded":"2023-10-10T17:40:46.213247+0000","last_fullsized":"2023-10-10T17:40:46.213247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:34:33.800195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276227+0000","last_change":"2023-10-10T17:40:41.604542+0000","last_active":"2023-10-10T17:40:46.276227+0000","last_peered":"2023-10-10T17:40:46.276227+0000","last_clean":"2023-10-10T17:40:46.276227+0000","last_became_active":"2023-10-10T17:40:41.604000+0000","last_became_peered":"2023-10-10T17:40:41.604000+0000","last_unstale":"2023-10-10T17:40:46.276227+0000","last_undegraded":"2023-10-10T17:40:46.276227+0000","last_fullsized":"2023-10-10T17:40:46.276227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:46:21.329180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275082+0000","last_change":"2023-10-10T17:40:41.610608+0000","last_active":"2023-10-10T17:40:46.275082+0000","last_peered":"2023-10-10T17:40:46.275082+0000","last_clean":"2023-10-10T17:40:46.275082+0000","last_became_active":"2023-10-10T17:40:41.610304+0000","last_became_peered":"2023-10-10T17:40:41.610304+0000","last_unstale":"2023-10-10T17:40:46.275082+0000","last_undegraded":"2023-10-10T17:40:46.275082+0000","last_fullsized":"2023-10-10T17:40:46.275082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:07:45.639458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275063+0000","last_change":"2023-10-10T17:40:41.606659+0000","last_active":"2023-10-10T17:40:46.275063+0000","last_peered":"2023-10-10T17:40:46.275063+0000","last_clean":"2023-10-10T17:40:46.275063+0000","last_became_active":"2023-10-10T17:40:41.606061+0000","last_became_peered":"2023-10-10T17:40:41.606061+0000","last_unstale":"2023-10-10T17:40:46.275063+0000","last_undegraded":"2023-10-10T17:40:46.275063+0000","last_fullsized":"2023-10-10T17:40:46.275063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:10:47.643441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276918+0000","last_change":"2023-10-10T17:40:41.607630+0000","last_active":"2023-10-10T17:40:46.276918+0000","last_peered":"2023-10-10T17:40:46.276918+0000","last_clean":"2023-10-10T17:40:46.276918+0000","last_became_active":"2023-10-10T17:40:41.607297+0000","last_became_peered":"2023-10-10T17:40:41.607297+0000","last_unstale":"2023-10-10T17:40:46.276918+0000","last_undegraded":"2023-10-10T17:40:46.276918+0000","last_fullsized":"2023-10-10T17:40:46.276918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:18:11.997842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099994+0000","last_change":"2023-10-10T17:40:41.594082+0000","last_active":"2023-10-10T17:40:46.099994+0000","last_peered":"2023-10-10T17:40:46.099994+0000","last_clean":"2023-10-10T17:40:46.099994+0000","last_became_active":"2023-10-10T17:40:41.593787+0000","last_became_peered":"2023-10-10T17:40:41.593787+0000","last_unstale":"2023-10-10T17:40:46.099994+0000","last_undegraded":"2023-10-10T17:40:46.099994+0000","last_fullsized":"2023-10-10T17:40:46.099994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:46:51.860019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100994+0000","last_change":"2023-10-10T17:40:41.608529+0000","last_active":"2023-10-10T17:40:46.100994+0000","last_peered":"2023-10-10T17:40:46.100994+0000","last_clean":"2023-10-10T17:40:46.100994+0000","last_became_active":"2023-10-10T17:40:41.608247+0000","last_became_peered":"2023-10-10T17:40:41.608247+0000","last_unstale":"2023-10-10T17:40:46.100994+0000","last_undegraded":"2023-10-10T17:40:46.100994+0000","last_fullsized":"2023-10-10T17:40:46.100994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:05:15.777617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100685+0000","last_change":"2023-10-10T17:40:45.707455+0000","last_active":"2023-10-10T17:40:46.100685+0000","last_peered":"2023-10-10T17:40:46.100685+0000","last_clean":"2023-10-10T17:40:46.100685+0000","last_became_active":"2023-10-10T17:40:45.707024+0000","last_became_peered":"2023-10-10T17:40:45.707024+0000","last_unstale":"2023-10-10T17:40:46.100685+0000","last_undegraded":"2023-10-10T17:40:46.100685+0000","last_fullsized":"2023-10-10T17:40:46.100685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:12:19.029228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100021+0000","last_change":"2023-10-10T17:40:41.165983+0000","last_active":"2023-10-10T17:40:46.100021+0000","last_peered":"2023-10-10T17:40:46.100021+0000","last_clean":"2023-10-10T17:40:46.100021+0000","last_became_active":"2023-10-10T17:40:41.165487+0000","last_became_peered":"2023-10-10T17:40:41.165487+0000","last_unstale":"2023-10-10T17:40:46.100021+0000","last_undegraded":"2023-10-10T17:40:46.100021+0000","last_fullsized":"2023-10-10T17:40:46.100021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:17.388327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275769+0000","last_change":"2023-10-10T17:40:41.614258+0000","last_active":"2023-10-10T17:40:46.275769+0000","last_peered":"2023-10-10T17:40:46.275769+0000","last_clean":"2023-10-10T17:40:46.275769+0000","last_became_active":"2023-10-10T17:40:41.613951+0000","last_became_peered":"2023-10-10T17:40:41.613951+0000","last_unstale":"2023-10-10T17:40:46.275769+0000","last_undegraded":"2023-10-10T17:40:46.275769+0000","last_fullsized":"2023-10-10T17:40:46.275769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:40:16.468408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276150+0000","last_change":"2023-10-10T17:40:45.708568+0000","last_active":"2023-10-10T17:40:46.276150+0000","last_peered":"2023-10-10T17:40:46.276150+0000","last_clean":"2023-10-10T17:40:46.276150+0000","last_became_active":"2023-10-10T17:40:45.708108+0000","last_became_peered":"2023-10-10T17:40:45.708108+0000","last_unstale":"2023-10-10T17:40:46.276150+0000","last_undegraded":"2023-10-10T17:40:46.276150+0000","last_fullsized":"2023-10-10T17:40:46.276150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:55:48.792350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276022+0000","last_change":"2023-10-10T17:40:41.167744+0000","last_active":"2023-10-10T17:40:46.276022+0000","last_peered":"2023-10-10T17:40:46.276022+0000","last_clean":"2023-10-10T17:40:46.276022+0000","last_became_active":"2023-10-10T17:40:41.167189+0000","last_became_peered":"2023-10-10T17:40:41.167189+0000","last_unstale":"2023-10-10T17:40:46.276022+0000","last_undegraded":"2023-10-10T17:40:46.276022+0000","last_fullsized":"2023-10-10T17:40:46.276022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:11.349794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276388+0000","last_change":"2023-10-10T17:40:41.608514+0000","last_active":"2023-10-10T17:40:46.276388+0000","last_peered":"2023-10-10T17:40:46.276388+0000","last_clean":"2023-10-10T17:40:46.276388+0000","last_became_active":"2023-10-10T17:40:41.608208+0000","last_became_peered":"2023-10-10T17:40:41.608208+0000","last_unstale":"2023-10-10T17:40:46.276388+0000","last_undegraded":"2023-10-10T17:40:46.276388+0000","last_fullsized":"2023-10-10T17:40:46.276388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:05.744048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275854+0000","last_change":"2023-10-10T17:40:45.710094+0000","last_active":"2023-10-10T17:40:46.275854+0000","last_peered":"2023-10-10T17:40:46.275854+0000","last_clean":"2023-10-10T17:40:46.275854+0000","last_became_active":"2023-10-10T17:40:45.709520+0000","last_became_peered":"2023-10-10T17:40:45.709520+0000","last_unstale":"2023-10-10T17:40:46.275854+0000","last_undegraded":"2023-10-10T17:40:46.275854+0000","last_fullsized":"2023-10-10T17:40:46.275854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:03:32.151300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:41:00.593651+0000","last_change":"2023-10-10T17:40:41.169821+0000","last_active":"2023-10-10T17:41:00.593651+0000","last_peered":"2023-10-10T17:41:00.593651+0000","last_clean":"2023-10-10T17:41:00.593651+0000","last_became_active":"2023-10-10T17:40:41.169502+0000","last_became_peered":"2023-10-10T17:40:41.169502+0000","last_unstale":"2023-10-10T17:41:00.593651+0000","last_undegraded":"2023-10-10T17:41:00.593651+0000","last_fullsized":"2023-10-10T17:41:00.593651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:47:41.083180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275369+0000","last_change":"2023-10-10T17:40:41.602720+0000","last_active":"2023-10-10T17:40:46.275369+0000","last_peered":"2023-10-10T17:40:46.275369+0000","last_clean":"2023-10-10T17:40:46.275369+0000","last_became_active":"2023-10-10T17:40:41.602187+0000","last_became_peered":"2023-10-10T17:40:41.602187+0000","last_unstale":"2023-10-10T17:40:46.275369+0000","last_undegraded":"2023-10-10T17:40:46.275369+0000","last_fullsized":"2023-10-10T17:40:46.275369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:12:04.205253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275449+0000","last_change":"2023-10-10T17:40:45.706946+0000","last_active":"2023-10-10T17:40:46.275449+0000","last_peered":"2023-10-10T17:40:46.275449+0000","last_clean":"2023-10-10T17:40:46.275449+0000","last_became_active":"2023-10-10T17:40:45.706495+0000","last_became_peered":"2023-10-10T17:40:45.706495+0000","last_unstale":"2023-10-10T17:40:46.275449+0000","last_undegraded":"2023-10-10T17:40:46.275449+0000","last_fullsized":"2023-10-10T17:40:46.275449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:24:40.545583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277100+0000","last_change":"2023-10-10T17:40:41.171855+0000","last_active":"2023-10-10T17:40:46.277100+0000","last_peered":"2023-10-10T17:40:46.277100+0000","last_clean":"2023-10-10T17:40:46.277100+0000","last_became_active":"2023-10-10T17:40:41.171515+0000","last_became_peered":"2023-10-10T17:40:41.171515+0000","last_unstale":"2023-10-10T17:40:46.277100+0000","last_undegraded":"2023-10-10T17:40:46.277100+0000","last_fullsized":"2023-10-10T17:40:46.277100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:31:10.288552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276592+0000","last_change":"2023-10-10T17:40:41.587083+0000","last_active":"2023-10-10T17:40:46.276592+0000","last_peered":"2023-10-10T17:40:46.276592+0000","last_clean":"2023-10-10T17:40:46.276592+0000","last_became_active":"2023-10-10T17:40:41.586758+0000","last_became_peered":"2023-10-10T17:40:41.586758+0000","last_unstale":"2023-10-10T17:40:46.276592+0000","last_undegraded":"2023-10-10T17:40:46.276592+0000","last_fullsized":"2023-10-10T17:40:46.276592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:04:36.358639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100878+0000","last_change":"2023-10-10T17:40:45.710492+0000","last_active":"2023-10-10T17:40:46.100878+0000","last_peered":"2023-10-10T17:40:46.100878+0000","last_clean":"2023-10-10T17:40:46.100878+0000","last_became_active":"2023-10-10T17:40:45.710069+0000","last_became_peered":"2023-10-10T17:40:45.710069+0000","last_unstale":"2023-10-10T17:40:46.100878+0000","last_undegraded":"2023-10-10T17:40:46.100878+0000","last_fullsized":"2023-10-10T17:40:46.100878+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:38:14.020875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.590663+0000","last_change":"2023-10-10T17:40:41.167449+0000","last_active":"2023-10-10T17:40:59.590663+0000","last_peered":"2023-10-10T17:40:59.590663+0000","last_clean":"2023-10-10T17:40:59.590663+0000","last_became_active":"2023-10-10T17:40:41.167038+0000","last_became_peered":"2023-10-10T17:40:41.167038+0000","last_unstale":"2023-10-10T17:40:59.590663+0000","last_undegraded":"2023-10-10T17:40:59.590663+0000","last_fullsized":"2023-10-10T17:40:59.590663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:11:53.872618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276574+0000","last_change":"2023-10-10T17:40:41.594327+0000","last_active":"2023-10-10T17:40:46.276574+0000","last_peered":"2023-10-10T17:40:46.276574+0000","last_clean":"2023-10-10T17:40:46.276574+0000","last_became_active":"2023-10-10T17:40:41.594008+0000","last_became_peered":"2023-10-10T17:40:41.594008+0000","last_unstale":"2023-10-10T17:40:46.276574+0000","last_undegraded":"2023-10-10T17:40:46.276574+0000","last_fullsized":"2023-10-10T17:40:46.276574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:09:06.415005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100713+0000","last_change":"2023-10-10T17:40:45.710630+0000","last_active":"2023-10-10T17:40:46.100713+0000","last_peered":"2023-10-10T17:40:46.100713+0000","last_clean":"2023-10-10T17:40:46.100713+0000","last_became_active":"2023-10-10T17:40:45.710202+0000","last_became_peered":"2023-10-10T17:40:45.710202+0000","last_unstale":"2023-10-10T17:40:46.100713+0000","last_undegraded":"2023-10-10T17:40:46.100713+0000","last_fullsized":"2023-10-10T17:40:46.100713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:01.973715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100754+0000","last_change":"2023-10-10T17:40:41.171926+0000","last_active":"2023-10-10T17:40:46.100754+0000","last_peered":"2023-10-10T17:40:46.100754+0000","last_clean":"2023-10-10T17:40:46.100754+0000","last_became_active":"2023-10-10T17:40:41.171627+0000","last_became_peered":"2023-10-10T17:40:41.171627+0000","last_unstale":"2023-10-10T17:40:46.100754+0000","last_undegraded":"2023-10-10T17:40:46.100754+0000","last_fullsized":"2023-10-10T17:40:46.100754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:10:23.967086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276470+0000","last_change":"2023-10-10T17:40:41.593135+0000","last_active":"2023-10-10T17:40:46.276470+0000","last_peered":"2023-10-10T17:40:46.276470+0000","last_clean":"2023-10-10T17:40:46.276470+0000","last_became_active":"2023-10-10T17:40:41.592871+0000","last_became_peered":"2023-10-10T17:40:41.592871+0000","last_unstale":"2023-10-10T17:40:46.276470+0000","last_undegraded":"2023-10-10T17:40:46.276470+0000","last_fullsized":"2023-10-10T17:40:46.276470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:42:32.503283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275747+0000","last_change":"2023-10-10T17:40:45.703557+0000","last_active":"2023-10-10T17:40:46.275747+0000","last_peered":"2023-10-10T17:40:46.275747+0000","last_clean":"2023-10-10T17:40:46.275747+0000","last_became_active":"2023-10-10T17:40:45.703085+0000","last_became_peered":"2023-10-10T17:40:45.703085+0000","last_unstale":"2023-10-10T17:40:46.275747+0000","last_undegraded":"2023-10-10T17:40:46.275747+0000","last_fullsized":"2023-10-10T17:40:46.275747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:34:18.186700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213122+0000","last_change":"2023-10-10T17:40:41.162829+0000","last_active":"2023-10-10T17:40:46.213122+0000","last_peered":"2023-10-10T17:40:46.213122+0000","last_clean":"2023-10-10T17:40:46.213122+0000","last_became_active":"2023-10-10T17:40:41.162516+0000","last_became_peered":"2023-10-10T17:40:41.162516+0000","last_unstale":"2023-10-10T17:40:46.213122+0000","last_undegraded":"2023-10-10T17:40:46.213122+0000","last_fullsized":"2023-10-10T17:40:46.213122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:45:47.035059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275578+0000","last_change":"2023-10-10T17:40:41.606908+0000","last_active":"2023-10-10T17:40:46.275578+0000","last_peered":"2023-10-10T17:40:46.275578+0000","last_clean":"2023-10-10T17:40:46.275578+0000","last_became_active":"2023-10-10T17:40:41.606492+0000","last_became_peered":"2023-10-10T17:40:41.606492+0000","last_unstale":"2023-10-10T17:40:46.275578+0000","last_undegraded":"2023-10-10T17:40:46.275578+0000","last_fullsized":"2023-10-10T17:40:46.275578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:08:48.702270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275681+0000","last_change":"2023-10-10T17:40:45.707139+0000","last_active":"2023-10-10T17:40:46.275681+0000","last_peered":"2023-10-10T17:40:46.275681+0000","last_clean":"2023-10-10T17:40:46.275681+0000","last_became_active":"2023-10-10T17:40:45.706708+0000","last_became_peered":"2023-10-10T17:40:45.706708+0000","last_unstale":"2023-10-10T17:40:46.275681+0000","last_undegraded":"2023-10-10T17:40:46.275681+0000","last_fullsized":"2023-10-10T17:40:46.275681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:09:37.435925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276991+0000","last_change":"2023-10-10T17:40:41.170512+0000","last_active":"2023-10-10T17:40:46.276991+0000","last_peered":"2023-10-10T17:40:46.276991+0000","last_clean":"2023-10-10T17:40:46.276991+0000","last_became_active":"2023-10-10T17:40:41.170264+0000","last_became_peered":"2023-10-10T17:40:41.170264+0000","last_unstale":"2023-10-10T17:40:46.276991+0000","last_undegraded":"2023-10-10T17:40:46.276991+0000","last_fullsized":"2023-10-10T17:40:46.276991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:22.102865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099516+0000","last_change":"2023-10-10T17:40:41.597647+0000","last_active":"2023-10-10T17:40:46.099516+0000","last_peered":"2023-10-10T17:40:46.099516+0000","last_clean":"2023-10-10T17:40:46.099516+0000","last_became_active":"2023-10-10T17:40:41.597245+0000","last_became_peered":"2023-10-10T17:40:41.597245+0000","last_unstale":"2023-10-10T17:40:46.099516+0000","last_undegraded":"2023-10-10T17:40:46.099516+0000","last_fullsized":"2023-10-10T17:40:46.099516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:53:56.653913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275791+0000","last_change":"2023-10-10T17:40:45.705089+0000","last_active":"2023-10-10T17:40:46.275791+0000","last_peered":"2023-10-10T17:40:46.275791+0000","last_clean":"2023-10-10T17:40:46.275791+0000","last_became_active":"2023-10-10T17:40:45.704475+0000","last_became_peered":"2023-10-10T17:40:45.704475+0000","last_unstale":"2023-10-10T17:40:46.275791+0000","last_undegraded":"2023-10-10T17:40:46.275791+0000","last_fullsized":"2023-10-10T17:40:46.275791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:51:15.851183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100943+0000","last_change":"2023-10-10T17:40:41.167409+0000","last_active":"2023-10-10T17:40:46.100943+0000","last_peered":"2023-10-10T17:40:46.100943+0000","last_clean":"2023-10-10T17:40:46.100943+0000","last_became_active":"2023-10-10T17:40:41.167125+0000","last_became_peered":"2023-10-10T17:40:41.167125+0000","last_unstale":"2023-10-10T17:40:46.100943+0000","last_undegraded":"2023-10-10T17:40:46.100943+0000","last_fullsized":"2023-10-10T17:40:46.100943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:01:37.543896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276398+0000","last_change":"2023-10-10T17:40:41.606216+0000","last_active":"2023-10-10T17:40:46.276398+0000","last_peered":"2023-10-10T17:40:46.276398+0000","last_clean":"2023-10-10T17:40:46.276398+0000","last_became_active":"2023-10-10T17:40:41.603258+0000","last_became_peered":"2023-10-10T17:40:41.603258+0000","last_unstale":"2023-10-10T17:40:46.276398+0000","last_undegraded":"2023-10-10T17:40:46.276398+0000","last_fullsized":"2023-10-10T17:40:46.276398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:50:11.631129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099501+0000","last_change":"2023-10-10T17:40:45.705623+0000","last_active":"2023-10-10T17:40:46.099501+0000","last_peered":"2023-10-10T17:40:46.099501+0000","last_clean":"2023-10-10T17:40:46.099501+0000","last_became_active":"2023-10-10T17:40:45.705242+0000","last_became_peered":"2023-10-10T17:40:45.705242+0000","last_unstale":"2023-10-10T17:40:46.099501+0000","last_undegraded":"2023-10-10T17:40:46.099501+0000","last_fullsized":"2023-10-10T17:40:46.099501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:07:11.500067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457530+0000","last_change":"2023-10-10T17:40:41.171552+0000","last_active":"2023-10-10T17:40:46.457530+0000","last_peered":"2023-10-10T17:40:46.457530+0000","last_clean":"2023-10-10T17:40:46.457530+0000","last_became_active":"2023-10-10T17:40:41.171207+0000","last_became_peered":"2023-10-10T17:40:41.171207+0000","last_unstale":"2023-10-10T17:40:46.457530+0000","last_undegraded":"2023-10-10T17:40:46.457530+0000","last_fullsized":"2023-10-10T17:40:46.457530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:39:37.128599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276523+0000","last_change":"2023-10-10T17:40:41.609655+0000","last_active":"2023-10-10T17:40:46.276523+0000","last_peered":"2023-10-10T17:40:46.276523+0000","last_clean":"2023-10-10T17:40:46.276523+0000","last_became_active":"2023-10-10T17:40:41.609318+0000","last_became_peered":"2023-10-10T17:40:41.609318+0000","last_unstale":"2023-10-10T17:40:46.276523+0000","last_undegraded":"2023-10-10T17:40:46.276523+0000","last_fullsized":"2023-10-10T17:40:46.276523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:24:46.334636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276360+0000","last_change":"2023-10-10T17:40:45.709513+0000","last_active":"2023-10-10T17:40:46.276360+0000","last_peered":"2023-10-10T17:40:46.276360+0000","last_clean":"2023-10-10T17:40:46.276360+0000","last_became_active":"2023-10-10T17:40:45.708963+0000","last_became_peered":"2023-10-10T17:40:45.708963+0000","last_unstale":"2023-10-10T17:40:46.276360+0000","last_undegraded":"2023-10-10T17:40:46.276360+0000","last_fullsized":"2023-10-10T17:40:46.276360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:11:59.727814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276467+0000","last_change":"2023-10-10T17:40:41.163234+0000","last_active":"2023-10-10T17:40:46.276467+0000","last_peered":"2023-10-10T17:40:46.276467+0000","last_clean":"2023-10-10T17:40:46.276467+0000","last_became_active":"2023-10-10T17:40:41.162916+0000","last_became_peered":"2023-10-10T17:40:41.162916+0000","last_unstale":"2023-10-10T17:40:46.276467+0000","last_undegraded":"2023-10-10T17:40:46.276467+0000","last_fullsized":"2023-10-10T17:40:46.276467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:54.672510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275767+0000","last_change":"2023-10-10T17:40:41.602852+0000","last_active":"2023-10-10T17:40:46.275767+0000","last_peered":"2023-10-10T17:40:46.275767+0000","last_clean":"2023-10-10T17:40:46.275767+0000","last_became_active":"2023-10-10T17:40:41.602530+0000","last_became_peered":"2023-10-10T17:40:41.602530+0000","last_unstale":"2023-10-10T17:40:46.275767+0000","last_undegraded":"2023-10-10T17:40:46.275767+0000","last_fullsized":"2023-10-10T17:40:46.275767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:55:10.460505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099787+0000","last_change":"2023-10-10T17:40:45.713714+0000","last_active":"2023-10-10T17:40:46.099787+0000","last_peered":"2023-10-10T17:40:46.099787+0000","last_clean":"2023-10-10T17:40:46.099787+0000","last_became_active":"2023-10-10T17:40:45.712953+0000","last_became_peered":"2023-10-10T17:40:45.712953+0000","last_unstale":"2023-10-10T17:40:46.099787+0000","last_undegraded":"2023-10-10T17:40:46.099787+0000","last_fullsized":"2023-10-10T17:40:46.099787+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:43:25.326564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:57.105298+0000","last_change":"2023-10-10T17:40:41.168328+0000","last_active":"2023-10-10T17:40:57.105298+0000","last_peered":"2023-10-10T17:40:57.105298+0000","last_clean":"2023-10-10T17:40:57.105298+0000","last_became_active":"2023-10-10T17:40:41.168044+0000","last_became_peered":"2023-10-10T17:40:41.168044+0000","last_unstale":"2023-10-10T17:40:57.105298+0000","last_undegraded":"2023-10-10T17:40:57.105298+0000","last_fullsized":"2023-10-10T17:40:57.105298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:19:58.468515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275980+0000","last_change":"2023-10-10T17:40:41.597795+0000","last_active":"2023-10-10T17:40:46.275980+0000","last_peered":"2023-10-10T17:40:46.275980+0000","last_clean":"2023-10-10T17:40:46.275980+0000","last_became_active":"2023-10-10T17:40:41.597537+0000","last_became_peered":"2023-10-10T17:40:41.597537+0000","last_unstale":"2023-10-10T17:40:46.275980+0000","last_undegraded":"2023-10-10T17:40:46.275980+0000","last_fullsized":"2023-10-10T17:40:46.275980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:08:46.240820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277087+0000","last_change":"2023-10-10T17:40:45.709827+0000","last_active":"2023-10-10T17:40:46.277087+0000","last_peered":"2023-10-10T17:40:46.277087+0000","last_clean":"2023-10-10T17:40:46.277087+0000","last_became_active":"2023-10-10T17:40:45.709373+0000","last_became_peered":"2023-10-10T17:40:45.709373+0000","last_unstale":"2023-10-10T17:40:46.277087+0000","last_undegraded":"2023-10-10T17:40:46.277087+0000","last_fullsized":"2023-10-10T17:40:46.277087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:56:06.379501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100574+0000","last_change":"2023-10-10T17:40:41.160804+0000","last_active":"2023-10-10T17:40:46.100574+0000","last_peered":"2023-10-10T17:40:46.100574+0000","last_clean":"2023-10-10T17:40:46.100574+0000","last_became_active":"2023-10-10T17:40:41.160493+0000","last_became_peered":"2023-10-10T17:40:41.160493+0000","last_unstale":"2023-10-10T17:40:46.100574+0000","last_undegraded":"2023-10-10T17:40:46.100574+0000","last_fullsized":"2023-10-10T17:40:46.100574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:32:18.782209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213141+0000","last_change":"2023-10-10T17:40:41.601357+0000","last_active":"2023-10-10T17:40:46.213141+0000","last_peered":"2023-10-10T17:40:46.213141+0000","last_clean":"2023-10-10T17:40:46.213141+0000","last_became_active":"2023-10-10T17:40:41.601057+0000","last_became_peered":"2023-10-10T17:40:41.601057+0000","last_unstale":"2023-10-10T17:40:46.213141+0000","last_undegraded":"2023-10-10T17:40:46.213141+0000","last_fullsized":"2023-10-10T17:40:46.213141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:03:02.150396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275966+0000","last_change":"2023-10-10T17:40:45.708067+0000","last_active":"2023-10-10T17:40:46.275966+0000","last_peered":"2023-10-10T17:40:46.275966+0000","last_clean":"2023-10-10T17:40:46.275966+0000","last_became_active":"2023-10-10T17:40:45.707686+0000","last_became_peered":"2023-10-10T17:40:45.707686+0000","last_unstale":"2023-10-10T17:40:46.275966+0000","last_undegraded":"2023-10-10T17:40:46.275966+0000","last_fullsized":"2023-10-10T17:40:46.275966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:00:14.708537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.591858+0000","last_change":"2023-10-10T17:40:41.168006+0000","last_active":"2023-10-10T17:40:59.591858+0000","last_peered":"2023-10-10T17:40:59.591858+0000","last_clean":"2023-10-10T17:40:59.591858+0000","last_became_active":"2023-10-10T17:40:41.167777+0000","last_became_peered":"2023-10-10T17:40:41.167777+0000","last_unstale":"2023-10-10T17:40:59.591858+0000","last_undegraded":"2023-10-10T17:40:59.591858+0000","last_fullsized":"2023-10-10T17:40:59.591858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:38:44.110762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457271+0000","last_change":"2023-10-10T17:40:41.591215+0000","last_active":"2023-10-10T17:40:46.457271+0000","last_peered":"2023-10-10T17:40:46.457271+0000","last_clean":"2023-10-10T17:40:46.457271+0000","last_became_active":"2023-10-10T17:40:41.590929+0000","last_became_peered":"2023-10-10T17:40:41.590929+0000","last_unstale":"2023-10-10T17:40:46.457271+0000","last_undegraded":"2023-10-10T17:40:46.457271+0000","last_fullsized":"2023-10-10T17:40:46.457271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:34:40.010647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099959+0000","last_change":"2023-10-10T17:40:45.709038+0000","last_active":"2023-10-10T17:40:46.099959+0000","last_peered":"2023-10-10T17:40:46.099959+0000","last_clean":"2023-10-10T17:40:46.099959+0000","last_became_active":"2023-10-10T17:40:45.708683+0000","last_became_peered":"2023-10-10T17:40:45.708683+0000","last_unstale":"2023-10-10T17:40:46.099959+0000","last_undegraded":"2023-10-10T17:40:46.099959+0000","last_fullsized":"2023-10-10T17:40:46.099959+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:25:44.015116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276240+0000","last_change":"2023-10-10T17:40:41.163817+0000","last_active":"2023-10-10T17:40:46.276240+0000","last_peered":"2023-10-10T17:40:46.276240+0000","last_clean":"2023-10-10T17:40:46.276240+0000","last_became_active":"2023-10-10T17:40:41.163497+0000","last_became_peered":"2023-10-10T17:40:41.163497+0000","last_unstale":"2023-10-10T17:40:46.276240+0000","last_undegraded":"2023-10-10T17:40:46.276240+0000","last_fullsized":"2023-10-10T17:40:46.276240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:44:23.887489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457017+0000","last_change":"2023-10-10T17:40:41.593843+0000","last_active":"2023-10-10T17:40:46.457017+0000","last_peered":"2023-10-10T17:40:46.457017+0000","last_clean":"2023-10-10T17:40:46.457017+0000","last_became_active":"2023-10-10T17:40:41.593585+0000","last_became_peered":"2023-10-10T17:40:41.593585+0000","last_unstale":"2023-10-10T17:40:46.457017+0000","last_undegraded":"2023-10-10T17:40:46.457017+0000","last_fullsized":"2023-10-10T17:40:46.457017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:03.463052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213888+0000","last_change":"2023-10-10T17:40:45.714900+0000","last_active":"2023-10-10T17:40:46.213888+0000","last_peered":"2023-10-10T17:40:46.213888+0000","last_clean":"2023-10-10T17:40:46.213888+0000","last_became_active":"2023-10-10T17:40:45.714525+0000","last_became_peered":"2023-10-10T17:40:45.714525+0000","last_unstale":"2023-10-10T17:40:46.213888+0000","last_undegraded":"2023-10-10T17:40:46.213888+0000","last_fullsized":"2023-10-10T17:40:46.213888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:43:57.654613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:41:00.596892+0000","last_change":"2023-10-10T17:40:41.164365+0000","last_active":"2023-10-10T17:41:00.596892+0000","last_peered":"2023-10-10T17:41:00.596892+0000","last_clean":"2023-10-10T17:41:00.596892+0000","last_became_active":"2023-10-10T17:40:41.164046+0000","last_became_peered":"2023-10-10T17:40:41.164046+0000","last_unstale":"2023-10-10T17:41:00.596892+0000","last_undegraded":"2023-10-10T17:41:00.596892+0000","last_fullsized":"2023-10-10T17:41:00.596892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:26:07.378218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277194+0000","last_change":"2023-10-10T17:40:41.586591+0000","last_active":"2023-10-10T17:40:46.277194+0000","last_peered":"2023-10-10T17:40:46.277194+0000","last_clean":"2023-10-10T17:40:46.277194+0000","last_became_active":"2023-10-10T17:40:41.586293+0000","last_became_peered":"2023-10-10T17:40:41.586293+0000","last_unstale":"2023-10-10T17:40:46.277194+0000","last_undegraded":"2023-10-10T17:40:46.277194+0000","last_fullsized":"2023-10-10T17:40:46.277194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:10.920114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277050+0000","last_change":"2023-10-10T17:40:45.708348+0000","last_active":"2023-10-10T17:40:46.277050+0000","last_peered":"2023-10-10T17:40:46.277050+0000","last_clean":"2023-10-10T17:40:46.277050+0000","last_became_active":"2023-10-10T17:40:45.707899+0000","last_became_peered":"2023-10-10T17:40:45.707899+0000","last_unstale":"2023-10-10T17:40:46.277050+0000","last_undegraded":"2023-10-10T17:40:46.277050+0000","last_fullsized":"2023-10-10T17:40:46.277050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:00:08.005331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275906+0000","last_change":"2023-10-10T17:40:41.168209+0000","last_active":"2023-10-10T17:40:46.275906+0000","last_peered":"2023-10-10T17:40:46.275906+0000","last_clean":"2023-10-10T17:40:46.275906+0000","last_became_active":"2023-10-10T17:40:41.167907+0000","last_became_peered":"2023-10-10T17:40:41.167907+0000","last_unstale":"2023-10-10T17:40:46.275906+0000","last_undegraded":"2023-10-10T17:40:46.275906+0000","last_fullsized":"2023-10-10T17:40:46.275906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:43:42.334992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276368+0000","last_change":"2023-10-10T17:40:41.594848+0000","last_active":"2023-10-10T17:40:46.276368+0000","last_peered":"2023-10-10T17:40:46.276368+0000","last_clean":"2023-10-10T17:40:46.276368+0000","last_became_active":"2023-10-10T17:40:41.594562+0000","last_became_peered":"2023-10-10T17:40:41.594562+0000","last_unstale":"2023-10-10T17:40:46.276368+0000","last_undegraded":"2023-10-10T17:40:46.276368+0000","last_fullsized":"2023-10-10T17:40:46.276368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:58:54.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276721+0000","last_change":"2023-10-10T17:40:45.711043+0000","last_active":"2023-10-10T17:40:46.276721+0000","last_peered":"2023-10-10T17:40:46.276721+0000","last_clean":"2023-10-10T17:40:46.276721+0000","last_became_active":"2023-10-10T17:40:45.710617+0000","last_became_peered":"2023-10-10T17:40:45.710617+0000","last_unstale":"2023-10-10T17:40:46.276721+0000","last_undegraded":"2023-10-10T17:40:46.276721+0000","last_fullsized":"2023-10-10T17:40:46.276721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:48.626413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099764+0000","last_change":"2023-10-10T17:40:41.163238+0000","last_active":"2023-10-10T17:40:46.099764+0000","last_peered":"2023-10-10T17:40:46.099764+0000","last_clean":"2023-10-10T17:40:46.099764+0000","last_became_active":"2023-10-10T17:40:41.162934+0000","last_became_peered":"2023-10-10T17:40:41.162934+0000","last_unstale":"2023-10-10T17:40:46.099764+0000","last_undegraded":"2023-10-10T17:40:46.099764+0000","last_fullsized":"2023-10-10T17:40:46.099764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:37:54.019115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.101058+0000","last_change":"2023-10-10T17:40:41.597026+0000","last_active":"2023-10-10T17:40:46.101058+0000","last_peered":"2023-10-10T17:40:46.101058+0000","last_clean":"2023-10-10T17:40:46.101058+0000","last_became_active":"2023-10-10T17:40:41.596747+0000","last_became_peered":"2023-10-10T17:40:41.596747+0000","last_unstale":"2023-10-10T17:40:46.101058+0000","last_undegraded":"2023-10-10T17:40:46.101058+0000","last_fullsized":"2023-10-10T17:40:46.101058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:26:07.488204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213403+0000","last_change":"2023-10-10T17:40:45.707218+0000","last_active":"2023-10-10T17:40:46.213403+0000","last_peered":"2023-10-10T17:40:46.213403+0000","last_clean":"2023-10-10T17:40:46.213403+0000","last_became_active":"2023-10-10T17:40:45.706850+0000","last_became_peered":"2023-10-10T17:40:45.706850+0000","last_unstale":"2023-10-10T17:40:46.213403+0000","last_undegraded":"2023-10-10T17:40:46.213403+0000","last_fullsized":"2023-10-10T17:40:46.213403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:15:20.838041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457087+0000","last_change":"2023-10-10T17:40:41.166713+0000","last_active":"2023-10-10T17:40:46.457087+0000","last_peered":"2023-10-10T17:40:46.457087+0000","last_clean":"2023-10-10T17:40:46.457087+0000","last_became_active":"2023-10-10T17:40:41.165951+0000","last_became_peered":"2023-10-10T17:40:41.165951+0000","last_unstale":"2023-10-10T17:40:46.457087+0000","last_undegraded":"2023-10-10T17:40:46.457087+0000","last_fullsized":"2023-10-10T17:40:46.457087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:16:54.814748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100908+0000","last_change":"2023-10-10T17:40:41.605860+0000","last_active":"2023-10-10T17:40:46.100908+0000","last_peered":"2023-10-10T17:40:46.100908+0000","last_clean":"2023-10-10T17:40:46.100908+0000","last_became_active":"2023-10-10T17:40:41.605556+0000","last_became_peered":"2023-10-10T17:40:41.605556+0000","last_unstale":"2023-10-10T17:40:46.100908+0000","last_undegraded":"2023-10-10T17:40:46.100908+0000","last_fullsized":"2023-10-10T17:40:46.100908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:33:10.684987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213514+0000","last_change":"2023-10-10T17:40:45.711844+0000","last_active":"2023-10-10T17:40:46.213514+0000","last_peered":"2023-10-10T17:40:46.213514+0000","last_clean":"2023-10-10T17:40:46.213514+0000","last_became_active":"2023-10-10T17:40:45.711413+0000","last_became_peered":"2023-10-10T17:40:45.711413+0000","last_unstale":"2023-10-10T17:40:46.213514+0000","last_undegraded":"2023-10-10T17:40:46.213514+0000","last_fullsized":"2023-10-10T17:40:46.213514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:58:41.509326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277401+0000","last_change":"2023-10-10T17:40:41.608625+0000","last_active":"2023-10-10T17:40:46.277401+0000","last_peered":"2023-10-10T17:40:46.277401+0000","last_clean":"2023-10-10T17:40:46.277401+0000","last_became_active":"2023-10-10T17:40:41.608332+0000","last_became_peered":"2023-10-10T17:40:41.608332+0000","last_unstale":"2023-10-10T17:40:46.277401+0000","last_undegraded":"2023-10-10T17:40:46.277401+0000","last_fullsized":"2023-10-10T17:40:46.277401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:17:17.274726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.212988+0000","last_change":"2023-10-10T17:40:41.169807+0000","last_active":"2023-10-10T17:40:46.212988+0000","last_peered":"2023-10-10T17:40:46.212988+0000","last_clean":"2023-10-10T17:40:46.212988+0000","last_became_active":"2023-10-10T17:40:41.169475+0000","last_became_peered":"2023-10-10T17:40:41.169475+0000","last_unstale":"2023-10-10T17:40:46.212988+0000","last_undegraded":"2023-10-10T17:40:46.212988+0000","last_fullsized":"2023-10-10T17:40:46.212988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:14.806814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457445+0000","last_change":"2023-10-10T17:40:45.710544+0000","last_active":"2023-10-10T17:40:46.457445+0000","last_peered":"2023-10-10T17:40:46.457445+0000","last_clean":"2023-10-10T17:40:46.457445+0000","last_became_active":"2023-10-10T17:40:45.710084+0000","last_became_peered":"2023-10-10T17:40:45.710084+0000","last_unstale":"2023-10-10T17:40:46.457445+0000","last_undegraded":"2023-10-10T17:40:46.457445+0000","last_fullsized":"2023-10-10T17:40:46.457445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:22.877276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275269+0000","last_change":"2023-10-10T17:40:41.599028+0000","last_active":"2023-10-10T17:40:46.275269+0000","last_peered":"2023-10-10T17:40:46.275269+0000","last_clean":"2023-10-10T17:40:46.275269+0000","last_became_active":"2023-10-10T17:40:41.598684+0000","last_became_peered":"2023-10-10T17:40:41.598684+0000","last_unstale":"2023-10-10T17:40:46.275269+0000","last_undegraded":"2023-10-10T17:40:46.275269+0000","last_fullsized":"2023-10-10T17:40:46.275269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:30:10.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275985+0000","last_change":"2023-10-10T17:40:41.165756+0000","last_active":"2023-10-10T17:40:46.275985+0000","last_peered":"2023-10-10T17:40:46.275985+0000","last_clean":"2023-10-10T17:40:46.275985+0000","last_became_active":"2023-10-10T17:40:41.165368+0000","last_became_peered":"2023-10-10T17:40:41.165368+0000","last_unstale":"2023-10-10T17:40:46.275985+0000","last_undegraded":"2023-10-10T17:40:46.275985+0000","last_fullsized":"2023-10-10T17:40:46.275985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:29:16.457611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.456850+0000","last_change":"2023-10-10T17:39:42.508828+0000","last_active":"2023-10-10T17:40:46.456850+0000","last_peered":"2023-10-10T17:40:46.456850+0000","last_clean":"2023-10-10T17:40:46.456850+0000","last_became_active":"2023-10-10T17:39:42.508475+0000","last_became_peered":"2023-10-10T17:39:42.508475+0000","last_unstale":"2023-10-10T17:40:46.456850+0000","last_undegraded":"2023-10-10T17:40:46.456850+0000","last_fullsized":"2023-10-10T17:40:46.456850+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_clean_scrub_stamp":"2023-10-10T17:36:46.553699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:52:08.434364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277319+0000","last_change":"2023-10-10T17:40:45.714604+0000","last_active":"2023-10-10T17:40:46.277319+0000","last_peered":"2023-10-10T17:40:46.277319+0000","last_clean":"2023-10-10T17:40:46.277319+0000","last_became_active":"2023-10-10T17:40:45.714185+0000","last_became_peered":"2023-10-10T17:40:45.714185+0000","last_unstale":"2023-10-10T17:40:46.277319+0000","last_undegraded":"2023-10-10T17:40:46.277319+0000","last_fullsized":"2023-10-10T17:40:46.277319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:36.214433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.212973+0000","last_change":"2023-10-10T17:40:41.610002+0000","last_active":"2023-10-10T17:40:46.212973+0000","last_peered":"2023-10-10T17:40:46.212973+0000","last_clean":"2023-10-10T17:40:46.212973+0000","last_became_active":"2023-10-10T17:40:41.609708+0000","last_became_peered":"2023-10-10T17:40:41.609708+0000","last_unstale":"2023-10-10T17:40:46.212973+0000","last_undegraded":"2023-10-10T17:40:46.212973+0000","last_fullsized":"2023-10-10T17:40:46.212973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:16:26.799522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.456827+0000","last_change":"2023-10-10T17:40:41.167715+0000","last_active":"2023-10-10T17:40:46.456827+0000","last_peered":"2023-10-10T17:40:46.456827+0000","last_clean":"2023-10-10T17:40:46.456827+0000","last_became_active":"2023-10-10T17:40:41.166385+0000","last_became_peered":"2023-10-10T17:40:41.166385+0000","last_unstale":"2023-10-10T17:40:46.456827+0000","last_undegraded":"2023-10-10T17:40:46.456827+0000","last_fullsized":"2023-10-10T17:40:46.456827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:50:45.606203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276936+0000","last_change":"2023-10-10T17:40:45.707236+0000","last_active":"2023-10-10T17:40:46.276936+0000","last_peered":"2023-10-10T17:40:46.276936+0000","last_clean":"2023-10-10T17:40:46.276936+0000","last_became_active":"2023-10-10T17:40:45.706827+0000","last_became_peered":"2023-10-10T17:40:45.706827+0000","last_unstale":"2023-10-10T17:40:46.276936+0000","last_undegraded":"2023-10-10T17:40:46.276936+0000","last_fullsized":"2023-10-10T17:40:46.276936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:38:34.181039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.591891+0000","last_change":"2023-10-10T17:40:41.170523+0000","last_active":"2023-10-10T17:40:59.591891+0000","last_peered":"2023-10-10T17:40:59.591891+0000","last_clean":"2023-10-10T17:40:59.591891+0000","last_became_active":"2023-10-10T17:40:41.170243+0000","last_became_peered":"2023-10-10T17:40:41.170243+0000","last_unstale":"2023-10-10T17:40:59.591891+0000","last_undegraded":"2023-10-10T17:40:59.591891+0000","last_fullsized":"2023-10-10T17:40:59.591891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:21:25.745722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213475+0000","last_change":"2023-10-10T17:40:41.610472+0000","last_active":"2023-10-10T17:40:46.213475+0000","last_peered":"2023-10-10T17:40:46.213475+0000","last_clean":"2023-10-10T17:40:46.213475+0000","last_became_active":"2023-10-10T17:40:41.610139+0000","last_became_peered":"2023-10-10T17:40:41.610139+0000","last_unstale":"2023-10-10T17:40:46.213475+0000","last_undegraded":"2023-10-10T17:40:46.213475+0000","last_fullsized":"2023-10-10T17:40:46.213475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:38.378204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275493+0000","last_change":"2023-10-10T17:40:45.712179+0000","last_active":"2023-10-10T17:40:46.275493+0000","last_peered":"2023-10-10T17:40:46.275493+0000","last_clean":"2023-10-10T17:40:46.275493+0000","last_became_active":"2023-10-10T17:40:45.711708+0000","last_became_peered":"2023-10-10T17:40:45.711708+0000","last_unstale":"2023-10-10T17:40:46.275493+0000","last_undegraded":"2023-10-10T17:40:46.275493+0000","last_fullsized":"2023-10-10T17:40:46.275493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:49:03.281192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099962+0000","last_change":"2023-10-10T17:40:41.164882+0000","last_active":"2023-10-10T17:40:46.099962+0000","last_peered":"2023-10-10T17:40:46.099962+0000","last_clean":"2023-10-10T17:40:46.099962+0000","last_became_active":"2023-10-10T17:40:41.164536+0000","last_became_peered":"2023-10-10T17:40:41.164536+0000","last_unstale":"2023-10-10T17:40:46.099962+0000","last_undegraded":"2023-10-10T17:40:46.099962+0000","last_fullsized":"2023-10-10T17:40:46.099962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:57:53.185248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276385+0000","last_change":"2023-10-10T17:40:41.595043+0000","last_active":"2023-10-10T17:40:46.276385+0000","last_peered":"2023-10-10T17:40:46.276385+0000","last_clean":"2023-10-10T17:40:46.276385+0000","last_became_active":"2023-10-10T17:40:41.594703+0000","last_became_peered":"2023-10-10T17:40:41.594703+0000","last_unstale":"2023-10-10T17:40:46.276385+0000","last_undegraded":"2023-10-10T17:40:46.276385+0000","last_fullsized":"2023-10-10T17:40:46.276385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:30:41.976969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100408+0000","last_change":"2023-10-10T17:40:45.705232+0000","last_active":"2023-10-10T17:40:46.100408+0000","last_peered":"2023-10-10T17:40:46.100408+0000","last_clean":"2023-10-10T17:40:46.100408+0000","last_became_active":"2023-10-10T17:40:45.704811+0000","last_became_peered":"2023-10-10T17:40:45.704811+0000","last_unstale":"2023-10-10T17:40:46.100408+0000","last_undegraded":"2023-10-10T17:40:46.100408+0000","last_fullsized":"2023-10-10T17:40:46.100408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:11.555315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276046+0000","last_change":"2023-10-10T17:40:41.165620+0000","last_active":"2023-10-10T17:40:46.276046+0000","last_peered":"2023-10-10T17:40:46.276046+0000","last_clean":"2023-10-10T17:40:46.276046+0000","last_became_active":"2023-10-10T17:40:41.165262+0000","last_became_peered":"2023-10-10T17:40:41.165262+0000","last_unstale":"2023-10-10T17:40:46.276046+0000","last_undegraded":"2023-10-10T17:40:46.276046+0000","last_fullsized":"2023-10-10T17:40:46.276046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:34:17.999550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275768+0000","last_change":"2023-10-10T17:40:41.603549+0000","last_active":"2023-10-10T17:40:46.275768+0000","last_peered":"2023-10-10T17:40:46.275768+0000","last_clean":"2023-10-10T17:40:46.275768+0000","last_became_active":"2023-10-10T17:40:41.602991+0000","last_became_peered":"2023-10-10T17:40:41.602991+0000","last_unstale":"2023-10-10T17:40:46.275768+0000","last_undegraded":"2023-10-10T17:40:46.275768+0000","last_fullsized":"2023-10-10T17:40:46.275768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:24:55.680074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275593+0000","last_change":"2023-10-10T17:40:45.714764+0000","last_active":"2023-10-10T17:40:46.275593+0000","last_peered":"2023-10-10T17:40:46.275593+0000","last_clean":"2023-10-10T17:40:46.275593+0000","last_became_active":"2023-10-10T17:40:45.714173+0000","last_became_peered":"2023-10-10T17:40:45.714173+0000","last_unstale":"2023-10-10T17:40:46.275593+0000","last_undegraded":"2023-10-10T17:40:46.275593+0000","last_fullsized":"2023-10-10T17:40:46.275593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:53.882859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275175+0000","last_change":"2023-10-10T17:40:41.169859+0000","last_active":"2023-10-10T17:40:46.275175+0000","last_peered":"2023-10-10T17:40:46.275175+0000","last_clean":"2023-10-10T17:40:46.275175+0000","last_became_active":"2023-10-10T17:40:41.169507+0000","last_became_peered":"2023-10-10T17:40:41.169507+0000","last_unstale":"2023-10-10T17:40:46.275175+0000","last_undegraded":"2023-10-10T17:40:46.275175+0000","last_fullsized":"2023-10-10T17:40:46.275175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:01:40.153696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275512+0000","last_change":"2023-10-10T17:40:41.593929+0000","last_active":"2023-10-10T17:40:46.275512+0000","last_peered":"2023-10-10T17:40:46.275512+0000","last_clean":"2023-10-10T17:40:46.275512+0000","last_became_active":"2023-10-10T17:40:41.593565+0000","last_became_peered":"2023-10-10T17:40:41.593565+0000","last_unstale":"2023-10-10T17:40:46.275512+0000","last_undegraded":"2023-10-10T17:40:46.275512+0000","last_fullsized":"2023-10-10T17:40:46.275512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:11.833671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100095+0000","last_change":"2023-10-10T17:40:45.710759+0000","last_active":"2023-10-10T17:40:46.100095+0000","last_peered":"2023-10-10T17:40:46.100095+0000","last_clean":"2023-10-10T17:40:46.100095+0000","last_became_active":"2023-10-10T17:40:45.710118+0000","last_became_peered":"2023-10-10T17:40:45.710118+0000","last_unstale":"2023-10-10T17:40:46.100095+0000","last_undegraded":"2023-10-10T17:40:46.100095+0000","last_fullsized":"2023-10-10T17:40:46.100095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:23:06.946107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.590803+0000","last_change":"2023-10-10T17:40:41.165988+0000","last_active":"2023-10-10T17:40:59.590803+0000","last_peered":"2023-10-10T17:40:59.590803+0000","last_clean":"2023-10-10T17:40:59.590803+0000","last_became_active":"2023-10-10T17:40:41.165407+0000","last_became_peered":"2023-10-10T17:40:41.165407+0000","last_unstale":"2023-10-10T17:40:59.590803+0000","last_undegraded":"2023-10-10T17:40:59.590803+0000","last_fullsized":"2023-10-10T17:40:59.590803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:54.001805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275819+0000","last_change":"2023-10-10T17:40:41.609834+0000","last_active":"2023-10-10T17:40:46.275819+0000","last_peered":"2023-10-10T17:40:46.275819+0000","last_clean":"2023-10-10T17:40:46.275819+0000","last_became_active":"2023-10-10T17:40:41.609509+0000","last_became_peered":"2023-10-10T17:40:41.609509+0000","last_unstale":"2023-10-10T17:40:46.275819+0000","last_undegraded":"2023-10-10T17:40:46.275819+0000","last_fullsized":"2023-10-10T17:40:46.275819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:42:46.393277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275681+0000","last_change":"2023-10-10T17:40:45.709003+0000","last_active":"2023-10-10T17:40:46.275681+0000","last_peered":"2023-10-10T17:40:46.275681+0000","last_clean":"2023-10-10T17:40:46.275681+0000","last_became_active":"2023-10-10T17:40:45.708610+0000","last_became_peered":"2023-10-10T17:40:45.708610+0000","last_unstale":"2023-10-10T17:40:46.275681+0000","last_undegraded":"2023-10-10T17:40:46.275681+0000","last_fullsized":"2023-10-10T17:40:46.275681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:47.748938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100416+0000","last_change":"2023-10-10T17:40:41.168104+0000","last_active":"2023-10-10T17:40:46.100416+0000","last_peered":"2023-10-10T17:40:46.100416+0000","last_clean":"2023-10-10T17:40:46.100416+0000","last_became_active":"2023-10-10T17:40:41.167802+0000","last_became_peered":"2023-10-10T17:40:41.167802+0000","last_unstale":"2023-10-10T17:40:46.100416+0000","last_undegraded":"2023-10-10T17:40:46.100416+0000","last_fullsized":"2023-10-10T17:40:46.100416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:52.597542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276168+0000","last_change":"2023-10-10T17:40:41.606553+0000","last_active":"2023-10-10T17:40:46.276168+0000","last_peered":"2023-10-10T17:40:46.276168+0000","last_clean":"2023-10-10T17:40:46.276168+0000","last_became_active":"2023-10-10T17:40:41.606087+0000","last_became_peered":"2023-10-10T17:40:41.606087+0000","last_unstale":"2023-10-10T17:40:46.276168+0000","last_undegraded":"2023-10-10T17:40:46.276168+0000","last_fullsized":"2023-10-10T17:40:46.276168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:13:31.697453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275876+0000","last_change":"2023-10-10T17:40:45.712028+0000","last_active":"2023-10-10T17:40:46.275876+0000","last_peered":"2023-10-10T17:40:46.275876+0000","last_clean":"2023-10-10T17:40:46.275876+0000","last_became_active":"2023-10-10T17:40:45.711534+0000","last_became_peered":"2023-10-10T17:40:45.711534+0000","last_unstale":"2023-10-10T17:40:46.275876+0000","last_undegraded":"2023-10-10T17:40:46.275876+0000","last_fullsized":"2023-10-10T17:40:46.275876+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:45:49.387605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.592441+0000","last_change":"2023-10-10T17:40:41.164456+0000","last_active":"2023-10-10T17:40:59.592441+0000","last_peered":"2023-10-10T17:40:59.592441+0000","last_clean":"2023-10-10T17:40:59.592441+0000","last_became_active":"2023-10-10T17:40:41.164018+0000","last_became_peered":"2023-10-10T17:40:41.164018+0000","last_unstale":"2023-10-10T17:40:59.592441+0000","last_undegraded":"2023-10-10T17:40:59.592441+0000","last_fullsized":"2023-10-10T17:40:59.592441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:26:56.502719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457221+0000","last_change":"2023-10-10T17:40:41.599724+0000","last_active":"2023-10-10T17:40:46.457221+0000","last_peered":"2023-10-10T17:40:46.457221+0000","last_clean":"2023-10-10T17:40:46.457221+0000","last_became_active":"2023-10-10T17:40:41.599356+0000","last_became_peered":"2023-10-10T17:40:41.599356+0000","last_unstale":"2023-10-10T17:40:46.457221+0000","last_undegraded":"2023-10-10T17:40:46.457221+0000","last_fullsized":"2023-10-10T17:40:46.457221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:16.776490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275891+0000","last_change":"2023-10-10T17:40:45.704911+0000","last_active":"2023-10-10T17:40:46.275891+0000","last_peered":"2023-10-10T17:40:46.275891+0000","last_clean":"2023-10-10T17:40:46.275891+0000","last_became_active":"2023-10-10T17:40:45.704299+0000","last_became_peered":"2023-10-10T17:40:45.704299+0000","last_unstale":"2023-10-10T17:40:46.275891+0000","last_undegraded":"2023-10-10T17:40:46.275891+0000","last_fullsized":"2023-10-10T17:40:46.275891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:18:41.829018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100891+0000","last_change":"2023-10-10T17:40:41.166091+0000","last_active":"2023-10-10T17:40:46.100891+0000","last_peered":"2023-10-10T17:40:46.100891+0000","last_clean":"2023-10-10T17:40:46.100891+0000","last_became_active":"2023-10-10T17:40:41.165610+0000","last_became_peered":"2023-10-10T17:40:41.165610+0000","last_unstale":"2023-10-10T17:40:46.100891+0000","last_undegraded":"2023-10-10T17:40:46.100891+0000","last_fullsized":"2023-10-10T17:40:46.100891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:41:34.118170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457584+0000","last_change":"2023-10-10T17:40:41.600837+0000","last_active":"2023-10-10T17:40:46.457584+0000","last_peered":"2023-10-10T17:40:46.457584+0000","last_clean":"2023-10-10T17:40:46.457584+0000","last_became_active":"2023-10-10T17:40:41.600090+0000","last_became_peered":"2023-10-10T17:40:41.600090+0000","last_unstale":"2023-10-10T17:40:46.457584+0000","last_undegraded":"2023-10-10T17:40:46.457584+0000","last_fullsized":"2023-10-10T17:40:46.457584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:02:54.300749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275138+0000","last_change":"2023-10-10T17:40:45.705712+0000","last_active":"2023-10-10T17:40:46.275138+0000","last_peered":"2023-10-10T17:40:46.275138+0000","last_clean":"2023-10-10T17:40:46.275138+0000","last_became_active":"2023-10-10T17:40:45.705301+0000","last_became_peered":"2023-10-10T17:40:45.705301+0000","last_unstale":"2023-10-10T17:40:46.275138+0000","last_undegraded":"2023-10-10T17:40:46.275138+0000","last_fullsized":"2023-10-10T17:40:46.275138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:59:20.860596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276742+0000","last_change":"2023-10-10T17:40:41.169820+0000","last_active":"2023-10-10T17:40:46.276742+0000","last_peered":"2023-10-10T17:40:46.276742+0000","last_clean":"2023-10-10T17:40:46.276742+0000","last_became_active":"2023-10-10T17:40:41.168767+0000","last_became_peered":"2023-10-10T17:40:41.168767+0000","last_unstale":"2023-10-10T17:40:46.276742+0000","last_undegraded":"2023-10-10T17:40:46.276742+0000","last_fullsized":"2023-10-10T17:40:46.276742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:55:51.881314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457344+0000","last_change":"2023-10-10T17:40:41.597482+0000","last_active":"2023-10-10T17:40:46.457344+0000","last_peered":"2023-10-10T17:40:46.457344+0000","last_clean":"2023-10-10T17:40:46.457344+0000","last_became_active":"2023-10-10T17:40:41.597166+0000","last_became_peered":"2023-10-10T17:40:41.597166+0000","last_unstale":"2023-10-10T17:40:46.457344+0000","last_undegraded":"2023-10-10T17:40:46.457344+0000","last_fullsized":"2023-10-10T17:40:46.457344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:12:25.007888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276549+0000","last_change":"2023-10-10T17:40:45.707175+0000","last_active":"2023-10-10T17:40:46.276549+0000","last_peered":"2023-10-10T17:40:46.276549+0000","last_clean":"2023-10-10T17:40:46.276549+0000","last_became_active":"2023-10-10T17:40:45.706448+0000","last_became_peered":"2023-10-10T17:40:45.706448+0000","last_unstale":"2023-10-10T17:40:46.276549+0000","last_undegraded":"2023-10-10T17:40:46.276549+0000","last_fullsized":"2023-10-10T17:40:46.276549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:59:16.405321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213980+0000","last_change":"2023-10-10T17:40:41.169806+0000","last_active":"2023-10-10T17:40:46.213980+0000","last_peered":"2023-10-10T17:40:46.213980+0000","last_clean":"2023-10-10T17:40:46.213980+0000","last_became_active":"2023-10-10T17:40:41.169496+0000","last_became_peered":"2023-10-10T17:40:41.169496+0000","last_unstale":"2023-10-10T17:40:46.213980+0000","last_undegraded":"2023-10-10T17:40:46.213980+0000","last_fullsized":"2023-10-10T17:40:46.213980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:42:39.410394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457671+0000","last_change":"2023-10-10T17:40:41.598366+0000","last_active":"2023-10-10T17:40:46.457671+0000","last_peered":"2023-10-10T17:40:46.457671+0000","last_clean":"2023-10-10T17:40:46.457671+0000","last_became_active":"2023-10-10T17:40:41.598036+0000","last_became_peered":"2023-10-10T17:40:41.598036+0000","last_unstale":"2023-10-10T17:40:46.457671+0000","last_undegraded":"2023-10-10T17:40:46.457671+0000","last_fullsized":"2023-10-10T17:40:46.457671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:32.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275953+0000","last_change":"2023-10-10T17:40:41.609923+0000","last_active":"2023-10-10T17:40:46.275953+0000","last_peered":"2023-10-10T17:40:46.275953+0000","last_clean":"2023-10-10T17:40:46.275953+0000","last_became_active":"2023-10-10T17:40:41.609575+0000","last_became_peered":"2023-10-10T17:40:41.609575+0000","last_unstale":"2023-10-10T17:40:46.275953+0000","last_undegraded":"2023-10-10T17:40:46.275953+0000","last_fullsized":"2023-10-10T17:40:46.275953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:51:30.575597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100629+0000","last_change":"2023-10-10T17:40:41.609012+0000","last_active":"2023-10-10T17:40:46.100629+0000","last_peered":"2023-10-10T17:40:46.100629+0000","last_clean":"2023-10-10T17:40:46.100629+0000","last_became_active":"2023-10-10T17:40:41.608738+0000","last_became_peered":"2023-10-10T17:40:41.608738+0000","last_unstale":"2023-10-10T17:40:46.100629+0000","last_undegraded":"2023-10-10T17:40:46.100629+0000","last_fullsized":"2023-10-10T17:40:46.100629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:20:01.313649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100540+0000","last_change":"2023-10-10T17:40:41.608801+0000","last_active":"2023-10-10T17:40:46.100540+0000","last_peered":"2023-10-10T17:40:46.100540+0000","last_clean":"2023-10-10T17:40:46.100540+0000","last_became_active":"2023-10-10T17:40:41.608510+0000","last_became_peered":"2023-10-10T17:40:41.608510+0000","last_unstale":"2023-10-10T17:40:46.100540+0000","last_undegraded":"2023-10-10T17:40:46.100540+0000","last_fullsized":"2023-10-10T17:40:46.100540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:25:21.531109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275330+0000","last_change":"2023-10-10T17:40:41.603516+0000","last_active":"2023-10-10T17:40:46.275330+0000","last_peered":"2023-10-10T17:40:46.275330+0000","last_clean":"2023-10-10T17:40:46.275330+0000","last_became_active":"2023-10-10T17:40:41.602948+0000","last_became_peered":"2023-10-10T17:40:41.602948+0000","last_unstale":"2023-10-10T17:40:46.275330+0000","last_undegraded":"2023-10-10T17:40:46.275330+0000","last_fullsized":"2023-10-10T17:40:46.275330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:59:40.345468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213549+0000","last_change":"2023-10-10T17:40:41.597594+0000","last_active":"2023-10-10T17:40:46.213549+0000","last_peered":"2023-10-10T17:40:46.213549+0000","last_clean":"2023-10-10T17:40:46.213549+0000","last_became_active":"2023-10-10T17:40:41.597286+0000","last_became_peered":"2023-10-10T17:40:41.597286+0000","last_unstale":"2023-10-10T17:40:46.213549+0000","last_undegraded":"2023-10-10T17:40:46.213549+0000","last_fullsized":"2023-10-10T17:40:46.213549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:04:44.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276363+0000","last_change":"2023-10-10T17:40:41.597595+0000","last_active":"2023-10-10T17:40:46.276363+0000","last_peered":"2023-10-10T17:40:46.276363+0000","last_clean":"2023-10-10T17:40:46.276363+0000","last_became_active":"2023-10-10T17:40:41.597301+0000","last_became_peered":"2023-10-10T17:40:41.597301+0000","last_unstale":"2023-10-10T17:40:46.276363+0000","last_undegraded":"2023-10-10T17:40:46.276363+0000","last_fullsized":"2023-10-10T17:40:46.276363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:14.213403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.101071+0000","last_change":"2023-10-10T17:40:41.611841+0000","last_active":"2023-10-10T17:40:46.101071+0000","last_peered":"2023-10-10T17:40:46.101071+0000","last_clean":"2023-10-10T17:40:46.101071+0000","last_became_active":"2023-10-10T17:40:41.611438+0000","last_became_peered":"2023-10-10T17:40:41.611438+0000","last_unstale":"2023-10-10T17:40:46.101071+0000","last_undegraded":"2023-10-10T17:40:46.101071+0000","last_fullsized":"2023-10-10T17:40:46.101071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:56:25.508210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276423+0000","last_change":"2023-10-10T17:40:41.607707+0000","last_active":"2023-10-10T17:40:46.276423+0000","last_peered":"2023-10-10T17:40:46.276423+0000","last_clean":"2023-10-10T17:40:46.276423+0000","last_became_active":"2023-10-10T17:40:41.607412+0000","last_became_peered":"2023-10-10T17:40:41.607412+0000","last_unstale":"2023-10-10T17:40:46.276423+0000","last_undegraded":"2023-10-10T17:40:46.276423+0000","last_fullsized":"2023-10-10T17:40:46.276423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:46:36.504686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277088+0000","last_change":"2023-10-10T17:40:41.593674+0000","last_active":"2023-10-10T17:40:46.277088+0000","last_peered":"2023-10-10T17:40:46.277088+0000","last_clean":"2023-10-10T17:40:46.277088+0000","last_became_active":"2023-10-10T17:40:41.593339+0000","last_became_peered":"2023-10-10T17:40:41.593339+0000","last_unstale":"2023-10-10T17:40:46.277088+0000","last_undegraded":"2023-10-10T17:40:46.277088+0000","last_fullsized":"2023-10-10T17:40:46.277088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:41:56.490267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276071+0000","last_change":"2023-10-10T17:40:41.610075+0000","last_active":"2023-10-10T17:40:46.276071+0000","last_peered":"2023-10-10T17:40:46.276071+0000","last_clean":"2023-10-10T17:40:46.276071+0000","last_became_active":"2023-10-10T17:40:41.609669+0000","last_became_peered":"2023-10-10T17:40:41.609669+0000","last_unstale":"2023-10-10T17:40:46.276071+0000","last_undegraded":"2023-10-10T17:40:46.276071+0000","last_fullsized":"2023-10-10T17:40:46.276071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:28:56.002362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276200+0000","last_change":"2023-10-10T17:40:41.596780+0000","last_active":"2023-10-10T17:40:46.276200+0000","last_peered":"2023-10-10T17:40:46.276200+0000","last_clean":"2023-10-10T17:40:46.276200+0000","last_became_active":"2023-10-10T17:40:41.596477+0000","last_became_peered":"2023-10-10T17:40:41.596477+0000","last_unstale":"2023-10-10T17:40:46.276200+0000","last_undegraded":"2023-10-10T17:40:46.276200+0000","last_fullsized":"2023-10-10T17:40:46.276200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:02:51.280429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213297+0000","last_change":"2023-10-10T17:40:41.606226+0000","last_active":"2023-10-10T17:40:46.213297+0000","last_peered":"2023-10-10T17:40:46.213297+0000","last_clean":"2023-10-10T17:40:46.213297+0000","last_became_active":"2023-10-10T17:40:41.605843+0000","last_became_peered":"2023-10-10T17:40:41.605843+0000","last_unstale":"2023-10-10T17:40:46.213297+0000","last_undegraded":"2023-10-10T17:40:46.213297+0000","last_fullsized":"2023-10-10T17:40:46.213297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:37:26.232295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276695+0000","last_change":"2023-10-10T17:40:41.596122+0000","last_active":"2023-10-10T17:40:46.276695+0000","last_peered":"2023-10-10T17:40:46.276695+0000","last_clean":"2023-10-10T17:40:46.276695+0000","last_became_active":"2023-10-10T17:40:41.595818+0000","last_became_peered":"2023-10-10T17:40:41.595818+0000","last_unstale":"2023-10-10T17:40:46.276695+0000","last_undegraded":"2023-10-10T17:40:46.276695+0000","last_fullsized":"2023-10-10T17:40:46.276695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:46:33.542672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250871,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5972,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":73,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":15918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6068,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739084,"statfs":{"total":95995035648,"available":95988822016,"internally_reserved":0,"allocated":962560,"data_stored":473302,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]}]},{"osd":10,"up_from":73,"seq":313532612625,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6324,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738828,"statfs":{"total":95995035648,"available":95988559872,"internally_reserved":0,"allocated":1028096,"data_stored":1066745,"data_compressed":47944,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.751},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.63200000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44900000000000001}]},{"osd":4,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68899999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61199999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.70899999999999996}]},{"osd":8,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.71299999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.53100000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6260,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738892,"statfs":{"total":95995035648,"available":95988625408,"internally_reserved":0,"allocated":962560,"data_stored":473302,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.318}]},{"osd":1,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.43099999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.51900000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.41799999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.66900000000000004}]},{"osd":6,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.63300000000000001}]},{"osd":8,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.60899999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:40:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.57399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6404,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738748,"statfs":{"total":95995035648,"available":95988477952,"internally_reserved":0,"allocated":978944,"data_stored":473458,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.84199999999999997}]},{"osd":1,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.47199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76000000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.438,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92900000000000005,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.877}]},{"osd":2,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39200000000000002},"min":{"1min":0.32400000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61499999999999999,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.45200000000000001,"15min":0.44400000000000001},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63700000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.60499999999999998}]},{"osd":3,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72899999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.47799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82899999999999996,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60199999999999998,"15min":0.59399999999999997},"min":{"1min":0.52100000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80900000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.753}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.52000000000000002},"min":{"1min":0.36099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.5,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81499999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96999999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.91700000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69199999999999995,"15min":0.67900000000000005},"min":{"1min":0.57999999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0029999999999999,"5min":1.022,"15min":1.022},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.59999999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86799999999999999,"5min":1.101,"15min":1.101},"last":0.85699999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.96999999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:40:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.90500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":7,"up_from":52,"seq":223338299423,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6496,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738656,"statfs":{"total":95995035648,"available":95988383744,"internally_reserved":0,"allocated":1007616,"data_stored":1063397,"data_compressed":47944,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72099999999999997,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.374}]},{"osd":1,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.39700000000000002,"15min":0.38900000000000001},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47199999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.76300000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56799999999999995}]},{"osd":4,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.34599999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.504}]},{"osd":5,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.46700000000000003,"5min":0.438,"15min":0.438},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.44800000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.871,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.71299999999999997}]},{"osd":6,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66000000000000003,"15min":0.65600000000000003},"min":{"1min":0.54100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60899999999999999}]},{"osd":8,"last update":"Tue Oct 10 17:40:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.79000000000000004}]},{"osd":9,"last update":"Tue Oct 10 17:40:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.70699999999999996}]},{"osd":10,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.83899999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462948,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":958464,"data_stored":472720,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35199999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.437},{"interface":"front","average":{"1min":0.371,"5min":0.375,"15min":0.375},"min":{"1min":0.27000000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.29699999999999999}]},{"osd":1,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52000000000000002}]},{"osd":2,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.38700000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.501}]},{"osd":3,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63800000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.36899999999999999}]},{"osd":4,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.55000000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56299999999999994,"15min":0.55700000000000005},"min":{"1min":0.39900000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.72699999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.46400000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39800000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:40:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.44,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.59599999999999997,"15min":0.58099999999999996},"min":{"1min":0.53000000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64000000000000001}]},{"osd":8,"last update":"Tue Oct 10 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46500000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.40899999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.751}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7292,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737860,"statfs":{"total":95995035648,"available":95987568640,"internally_reserved":0,"allocated":970752,"data_stored":476514,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.51200000000000001,"15min":0.48499999999999999},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61699999999999999,"15min":0.68899999999999995},"min":{"1min":0.41199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.88300000000000001,"5min":5.859,"15min":5.859},"last":0.61799999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:40:53 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.746},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.433,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79000000000000004,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.86899999999999999}]},{"osd":3,"last update":"Tue Oct 10 17:40:11 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57499999999999996,"15min":0.55700000000000005},"min":{"1min":0.44,"5min":0.379,"15min":0.379},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.79300000000000004}]},{"osd":4,"last update":"Tue Oct 10 17:40:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.41699999999999998,"15min":0.39100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71899999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.42999999999999999,"15min":0.40200000000000002},"min":{"1min":0.34300000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.76400000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88700000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.81599999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:40:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72899999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.91800000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85299999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83299999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":1}]},{"osd":8,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":1.0169999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:40:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.90200000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:40:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.98499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7284,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737868,"statfs":{"total":95995035648,"available":95987576832,"internally_reserved":0,"allocated":962560,"data_stored":472810,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 17:40:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.39300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.433},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.59899999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.56599999999999995,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75900000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:39:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27700000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63800000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:40:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.35699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.498},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.39900000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.50800000000000001},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70499999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.47499999999999998,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.48899999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.626,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.624},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.53100000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.75}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.61099999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.68600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7356,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737796,"statfs":{"total":95995035648,"available":95987503104,"internally_reserved":0,"allocated":970752,"data_stored":475929,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.31,"5min":0.252,"15min":0.252},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48799999999999999,"15min":0.48099999999999998},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.61699999999999999}]},{"osd":1,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.56799999999999995,"15min":0.52000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.484,"5min":5.484,"15min":5.484},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68999999999999995}]},{"osd":3,"last update":"Tue Oct 10 17:40:12 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53700000000000003,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.753},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.41199999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51900000000000002,"15min":0.502},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71999999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52100000000000002,"15min":0.501},"min":{"1min":0.42199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.77700000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.41999999999999998,"15min":0.40100000000000002},"min":{"1min":0.33900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.46500000000000002,"15min":0.44900000000000001},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.79300000000000004}]},{"osd":6,"last update":"Tue Oct 10 17:40:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.83799999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77900000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.93400000000000005}]},{"osd":8,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.86899999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:40:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.98899999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.92400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":970752,"data_stored":472673,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.35099999999999998,"15min":0.34699999999999998},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.49399999999999999,"5min":0.752,"15min":0.752},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.379,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55000000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.35399999999999998}]},{"osd":1,"last update":"Tue Oct 10 17:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.55200000000000005,"15min":0.53500000000000003},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.36299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58799999999999997}]},{"osd":2,"last update":"Tue Oct 10 17:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.442,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55200000000000005,"15min":0.53500000000000003},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.61199999999999999}]},{"osd":4,"last update":"Tue Oct 10 17:40:31 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.91400000000000003,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.41499999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95999999999999996,"5min":1.214,"15min":1.214},"last":0.46000000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.48099999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72099999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Tue Oct 10 17:40:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.89500000000000002,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67500000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.372}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73899999999999999,"5min":0.88,"15min":0.88},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86699999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.52400000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:39:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53700000000000003}]},{"osd":9,"last update":"Tue Oct 10 17:40:18 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.41499999999999998}]},{"osd":10,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]}]},{"osd":4,"up_from":34,"seq":146028888109,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":962560,"data_stored":472742,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.432,"5min":0.372,"15min":0.372},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.372,"15min":0.35899999999999999},"min":{"1min":0.30499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57299999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.38300000000000001,"15min":0.36399999999999999},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.372}]},{"osd":2,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.58299999999999996,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82299999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.59299999999999997,"15min":0.56100000000000005},"min":{"1min":0.495,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53500000000000003,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78100000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.40300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.57799999999999996},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86799999999999999,"5min":1.589,"15min":1.589},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.74099999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:40:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.433},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.48599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.497}]},{"osd":7,"last update":"Tue Oct 10 17:40:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.53100000000000003,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.629},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46899999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.74399999999999999,"15min":0.71199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":6.1639999999999997,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.73199999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53200000000000003}]},{"osd":10,"last update":"Tue Oct 10 17:40:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724585,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6648,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738504,"statfs":{"total":95995035648,"available":95988228096,"internally_reserved":0,"allocated":1032192,"data_stored":1064243,"data_compressed":47944,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.41099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.41099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53400000000000003}]},{"osd":2,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.54000000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.58699999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54700000000000004}]},{"osd":4,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.44600000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72999999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:40:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.622,"15min":0.624},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80400000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.623,"15min":0.624},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84199999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.745}]},{"osd":8,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.64700000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:40:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63500000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:40:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T17:41:02.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T17:41:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 -- 172.21.15.78:0/821754196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c80ffde0 msgr2=0x7fc2c8107690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/821754196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c80ffde0 0x7fc2c8107690 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000b0a0 tx=0x7fc2c000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 -- 172.21.15.78:0/821754196 shutdown_connections 2023-10-10T17:41:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/821754196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c8100870 0x7fc2c8100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/821754196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c80ffde0 0x7fc2c8107690 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/821754196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c80ff430 0x7fc2c80ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 -- 172.21.15.78:0/821754196 >> 172.21.15.78:0/821754196 conn(0x7fc2c80f9fb0 msgr2=0x7fc2c80fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 -- 172.21.15.78:0/821754196 shutdown_connections 2023-10-10T17:41:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.834+0000 7fc2cecf9700 1 -- 172.21.15.78:0/821754196 wait complete. 2023-10-10T17:41:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 Processor -- start 2023-10-10T17:41:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 -- start start 2023-10-10T17:41:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c80ff430 0x7fc2c8195000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c80ffde0 0x7fc2c8195540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c8100870 0x7fc2c819c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c819cb80 con 0x7fc2c8100870 2023-10-10T17:41:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c819ccc0 con 0x7fc2c80ffde0 2023-10-10T17:41:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cecf9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c819ce50 con 0x7fc2c80ff430 2023-10-10T17:41:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c8100870 0x7fc2c819c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c8100870 0x7fc2c819c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42652/0 (socket says 172.21.15.78:42652) 2023-10-10T17:41:02.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 -- 172.21.15.78:0/2696539116 learned_addr learned my addr 172.21.15.78:0/2696539116 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:02.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cca95700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c80ff430 0x7fc2c8195000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2c7fff700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c80ffde0 0x7fc2c8195540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 -- 172.21.15.78:0/2696539116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c80ff430 msgr2=0x7fc2c8195000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c80ff430 0x7fc2c8195000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 -- 172.21.15.78:0/2696539116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c80ffde0 msgr2=0x7fc2c8195540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c80ffde0 0x7fc2c8195540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:02.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 -- 172.21.15.78:0/2696539116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c0009d00 con 0x7fc2c8100870 2023-10-10T17:41:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2c7fff700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c80ffde0 0x7fc2c8195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.835+0000 7fc2cd296700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c8100870 0x7fc2c819c5b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000b070 tx=0x7fc2c0004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:02.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.836+0000 7fc2c5ffb700 1 -- 172.21.15.78:0/2696539116 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c000e040 con 0x7fc2c8100870 2023-10-10T17:41:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.836+0000 7fc2c5ffb700 1 -- 172.21.15.78:0/2696539116 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc2c0004e80 con 0x7fc2c8100870 2023-10-10T17:41:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.836+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c810ac10 con 0x7fc2c8100870 2023-10-10T17:41:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.836+0000 7fc2c5ffb700 1 -- 172.21.15.78:0/2696539116 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2c001b6b0 con 0x7fc2c8100870 2023-10-10T17:41:02.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.836+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c810aa80 con 0x7fc2c8100870 2023-10-10T17:41:02.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.837+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c8106750 con 0x7fc2c8100870 2023-10-10T17:41:02.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.838+0000 7fc2c5ffb700 1 -- 172.21.15.78:0/2696539116 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc2c0003a00 con 0x7fc2c8100870 2023-10-10T17:41:02.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.838+0000 7fc2c5ffb700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc2b80a1150 0x7fc2b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.838+0000 7fc2c5ffb700 1 -- 172.21.15.78:0/2696539116 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc2c00cd910 con 0x7fc2c8100870 2023-10-10T17:41:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.838+0000 7fc2cca95700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc2b80a1150 0x7fc2b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:02.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.839+0000 7fc2cca95700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc2b80a1150 0x7fc2b80a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005950 tx=0x7fc2bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:02.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:02.843+0000 7fc2c5ffb700 1 -- 172.21.15.78:0/2696539116 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc2c0096f00 con 0x7fc2c8100870 2023-10-10T17:41:02.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:02 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] up:active 2023-10-10T17:41:02.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:02 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1661221494' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-10-10T17:41:02.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:02 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2023-10-10T17:41:02.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:02 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:41:02.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:02 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3463984637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:02.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:02 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/630832904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:02.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:02 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] up:active 2023-10-10T17:41:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:02 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1661221494' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-10-10T17:41:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:02 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2023-10-10T17:41:02.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:02 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:41:02.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:02 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3463984637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:02.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:02 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/630832904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:02 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] up:active 2023-10-10T17:41:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:02 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1661221494' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-10-10T17:41:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:02 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2023-10-10T17:41:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:02 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:41:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:02 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3463984637' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:02 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/630832904' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:03.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.028+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2c8061a50 con 0x7fc2b80a1150 2023-10-10T17:41:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.040+0000 7fc2c5ffb700 1 -- 172.21.15.78:0/2696539116 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418715 (secure 0 0 0) 0x7fc2c8061a50 con 0x7fc2b80a1150 2023-10-10T17:41:03.050 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:03.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc2b80a1150 msgr2=0x7fc2b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:03.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc2b80a1150 0x7fc2b80a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005950 tx=0x7fc2bc00b410 comp rx=0 tx=0).stop 2023-10-10T17:41:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c8100870 msgr2=0x7fc2c819c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c8100870 0x7fc2c819c5b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc2c000b070 tx=0x7fc2c0004a50 comp rx=0 tx=0).stop 2023-10-10T17:41:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 shutdown_connections 2023-10-10T17:41:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc2b80a1150 0x7fc2b80a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc2c80ff430 0x7fc2c8195000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc2c8100870 0x7fc2c819c5b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 --2- 172.21.15.78:0/2696539116 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc2c80ffde0 0x7fc2c8195540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 >> 172.21.15.78:0/2696539116 conn(0x7fc2c80f9fb0 msgr2=0x7fc2c8103890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 shutdown_connections 2023-10-10T17:41:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:03.044+0000 7fc2cecf9700 1 -- 172.21.15.78:0/2696539116 wait complete. 2023-10-10T17:41:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-10T17:41:03.063 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2023-10-10T17:41:02.220427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596340,"num_objects":37,"num_object_clones":0,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":0,"num_read":141,"num_read_kb":124,"num_write":254,"num_write_kb":2970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":79748,"kb_used_data":11492,"kb_used_omap":0,"kb_used_meta":68160,"kb_avail":1124862076,"statfs":{"total":1151940427776,"available":1151858765824,"internally_reserved":0,"allocated":11767808,"data_stored":7457835,"data_compressed":248493,"data_compressed_allocated":331776,"data_compressed_original":2162688,"omap_allocated":0,"internal_metadata":69795840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3610,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":28,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003116"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277491+0000","last_change":"2023-10-10T17:40:41.604268+0000","last_active":"2023-10-10T17:40:46.277491+0000","last_peered":"2023-10-10T17:40:46.277491+0000","last_clean":"2023-10-10T17:40:46.277491+0000","last_became_active":"2023-10-10T17:40:41.603839+0000","last_became_peered":"2023-10-10T17:40:41.603839+0000","last_unstale":"2023-10-10T17:40:46.277491+0000","last_undegraded":"2023-10-10T17:40:46.277491+0000","last_fullsized":"2023-10-10T17:40:46.277491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:06:30.271619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275640+0000","last_change":"2023-10-10T17:40:41.603642+0000","last_active":"2023-10-10T17:40:46.275640+0000","last_peered":"2023-10-10T17:40:46.275640+0000","last_clean":"2023-10-10T17:40:46.275640+0000","last_became_active":"2023-10-10T17:40:41.603091+0000","last_became_peered":"2023-10-10T17:40:41.603091+0000","last_unstale":"2023-10-10T17:40:46.275640+0000","last_undegraded":"2023-10-10T17:40:46.275640+0000","last_fullsized":"2023-10-10T17:40:46.275640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:45:53.398893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276386+0000","last_change":"2023-10-10T17:40:41.610647+0000","last_active":"2023-10-10T17:40:46.276386+0000","last_peered":"2023-10-10T17:40:46.276386+0000","last_clean":"2023-10-10T17:40:46.276386+0000","last_became_active":"2023-10-10T17:40:41.610338+0000","last_became_peered":"2023-10-10T17:40:41.610338+0000","last_unstale":"2023-10-10T17:40:46.276386+0000","last_undegraded":"2023-10-10T17:40:46.276386+0000","last_fullsized":"2023-10-10T17:40:46.276386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:17:10.849937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276163+0000","last_change":"2023-10-10T17:40:41.607832+0000","last_active":"2023-10-10T17:40:46.276163+0000","last_peered":"2023-10-10T17:40:46.276163+0000","last_clean":"2023-10-10T17:40:46.276163+0000","last_became_active":"2023-10-10T17:40:41.607492+0000","last_became_peered":"2023-10-10T17:40:41.607492+0000","last_unstale":"2023-10-10T17:40:46.276163+0000","last_undegraded":"2023-10-10T17:40:46.276163+0000","last_fullsized":"2023-10-10T17:40:46.276163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:38:19.839774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275756+0000","last_change":"2023-10-10T17:40:41.601975+0000","last_active":"2023-10-10T17:40:46.275756+0000","last_peered":"2023-10-10T17:40:46.275756+0000","last_clean":"2023-10-10T17:40:46.275756+0000","last_became_active":"2023-10-10T17:40:41.601668+0000","last_became_peered":"2023-10-10T17:40:41.601668+0000","last_unstale":"2023-10-10T17:40:46.275756+0000","last_undegraded":"2023-10-10T17:40:46.275756+0000","last_fullsized":"2023-10-10T17:40:46.275756+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:13:04.900016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100867+0000","last_change":"2023-10-10T17:40:41.594445+0000","last_active":"2023-10-10T17:40:46.100867+0000","last_peered":"2023-10-10T17:40:46.100867+0000","last_clean":"2023-10-10T17:40:46.100867+0000","last_became_active":"2023-10-10T17:40:41.594123+0000","last_became_peered":"2023-10-10T17:40:41.594123+0000","last_unstale":"2023-10-10T17:40:46.100867+0000","last_undegraded":"2023-10-10T17:40:46.100867+0000","last_fullsized":"2023-10-10T17:40:46.100867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:50:57.458496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.456896+0000","last_change":"2023-10-10T17:40:41.594628+0000","last_active":"2023-10-10T17:40:46.456896+0000","last_peered":"2023-10-10T17:40:46.456896+0000","last_clean":"2023-10-10T17:40:46.456896+0000","last_became_active":"2023-10-10T17:40:41.594325+0000","last_became_peered":"2023-10-10T17:40:41.594325+0000","last_unstale":"2023-10-10T17:40:46.456896+0000","last_undegraded":"2023-10-10T17:40:46.456896+0000","last_fullsized":"2023-10-10T17:40:46.456896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:41:16.119478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275229+0000","last_change":"2023-10-10T17:40:41.598032+0000","last_active":"2023-10-10T17:40:46.275229+0000","last_peered":"2023-10-10T17:40:46.275229+0000","last_clean":"2023-10-10T17:40:46.275229+0000","last_became_active":"2023-10-10T17:40:41.597671+0000","last_became_peered":"2023-10-10T17:40:41.597671+0000","last_unstale":"2023-10-10T17:40:46.275229+0000","last_undegraded":"2023-10-10T17:40:46.275229+0000","last_fullsized":"2023-10-10T17:40:46.275229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:45:42.730985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276566+0000","last_change":"2023-10-10T17:40:41.599770+0000","last_active":"2023-10-10T17:40:46.276566+0000","last_peered":"2023-10-10T17:40:46.276566+0000","last_clean":"2023-10-10T17:40:46.276566+0000","last_became_active":"2023-10-10T17:40:41.599449+0000","last_became_peered":"2023-10-10T17:40:41.599449+0000","last_unstale":"2023-10-10T17:40:46.276566+0000","last_undegraded":"2023-10-10T17:40:46.276566+0000","last_fullsized":"2023-10-10T17:40:46.276566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:49:34.537216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.101227+0000","last_change":"2023-10-10T17:40:41.610655+0000","last_active":"2023-10-10T17:40:46.101227+0000","last_peered":"2023-10-10T17:40:46.101227+0000","last_clean":"2023-10-10T17:40:46.101227+0000","last_became_active":"2023-10-10T17:40:41.610133+0000","last_became_peered":"2023-10-10T17:40:41.610133+0000","last_unstale":"2023-10-10T17:40:46.101227+0000","last_undegraded":"2023-10-10T17:40:46.101227+0000","last_fullsized":"2023-10-10T17:40:46.101227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:28:16.307704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457571+0000","last_change":"2023-10-10T17:40:41.601065+0000","last_active":"2023-10-10T17:40:46.457571+0000","last_peered":"2023-10-10T17:40:46.457571+0000","last_clean":"2023-10-10T17:40:46.457571+0000","last_became_active":"2023-10-10T17:40:41.600530+0000","last_became_peered":"2023-10-10T17:40:41.600530+0000","last_unstale":"2023-10-10T17:40:46.457571+0000","last_undegraded":"2023-10-10T17:40:46.457571+0000","last_fullsized":"2023-10-10T17:40:46.457571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:40:11.842872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275953+0000","last_change":"2023-10-10T17:40:41.596067+0000","last_active":"2023-10-10T17:40:46.275953+0000","last_peered":"2023-10-10T17:40:46.275953+0000","last_clean":"2023-10-10T17:40:46.275953+0000","last_became_active":"2023-10-10T17:40:41.595703+0000","last_became_peered":"2023-10-10T17:40:41.595703+0000","last_unstale":"2023-10-10T17:40:46.275953+0000","last_undegraded":"2023-10-10T17:40:46.275953+0000","last_fullsized":"2023-10-10T17:40:46.275953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:32:24.161847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213247+0000","last_change":"2023-10-10T17:40:41.593296+0000","last_active":"2023-10-10T17:40:46.213247+0000","last_peered":"2023-10-10T17:40:46.213247+0000","last_clean":"2023-10-10T17:40:46.213247+0000","last_became_active":"2023-10-10T17:40:41.592965+0000","last_became_peered":"2023-10-10T17:40:41.592965+0000","last_unstale":"2023-10-10T17:40:46.213247+0000","last_undegraded":"2023-10-10T17:40:46.213247+0000","last_fullsized":"2023-10-10T17:40:46.213247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:34:33.800195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276227+0000","last_change":"2023-10-10T17:40:41.604542+0000","last_active":"2023-10-10T17:40:46.276227+0000","last_peered":"2023-10-10T17:40:46.276227+0000","last_clean":"2023-10-10T17:40:46.276227+0000","last_became_active":"2023-10-10T17:40:41.604000+0000","last_became_peered":"2023-10-10T17:40:41.604000+0000","last_unstale":"2023-10-10T17:40:46.276227+0000","last_undegraded":"2023-10-10T17:40:46.276227+0000","last_fullsized":"2023-10-10T17:40:46.276227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:46:21.329180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275082+0000","last_change":"2023-10-10T17:40:41.610608+0000","last_active":"2023-10-10T17:40:46.275082+0000","last_peered":"2023-10-10T17:40:46.275082+0000","last_clean":"2023-10-10T17:40:46.275082+0000","last_became_active":"2023-10-10T17:40:41.610304+0000","last_became_peered":"2023-10-10T17:40:41.610304+0000","last_unstale":"2023-10-10T17:40:46.275082+0000","last_undegraded":"2023-10-10T17:40:46.275082+0000","last_fullsized":"2023-10-10T17:40:46.275082+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:07:45.639458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275063+0000","last_change":"2023-10-10T17:40:41.606659+0000","last_active":"2023-10-10T17:40:46.275063+0000","last_peered":"2023-10-10T17:40:46.275063+0000","last_clean":"2023-10-10T17:40:46.275063+0000","last_became_active":"2023-10-10T17:40:41.606061+0000","last_became_peered":"2023-10-10T17:40:41.606061+0000","last_unstale":"2023-10-10T17:40:46.275063+0000","last_undegraded":"2023-10-10T17:40:46.275063+0000","last_fullsized":"2023-10-10T17:40:46.275063+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:10:47.643441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276918+0000","last_change":"2023-10-10T17:40:41.607630+0000","last_active":"2023-10-10T17:40:46.276918+0000","last_peered":"2023-10-10T17:40:46.276918+0000","last_clean":"2023-10-10T17:40:46.276918+0000","last_became_active":"2023-10-10T17:40:41.607297+0000","last_became_peered":"2023-10-10T17:40:41.607297+0000","last_unstale":"2023-10-10T17:40:46.276918+0000","last_undegraded":"2023-10-10T17:40:46.276918+0000","last_fullsized":"2023-10-10T17:40:46.276918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:18:11.997842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099994+0000","last_change":"2023-10-10T17:40:41.594082+0000","last_active":"2023-10-10T17:40:46.099994+0000","last_peered":"2023-10-10T17:40:46.099994+0000","last_clean":"2023-10-10T17:40:46.099994+0000","last_became_active":"2023-10-10T17:40:41.593787+0000","last_became_peered":"2023-10-10T17:40:41.593787+0000","last_unstale":"2023-10-10T17:40:46.099994+0000","last_undegraded":"2023-10-10T17:40:46.099994+0000","last_fullsized":"2023-10-10T17:40:46.099994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:46:51.860019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100994+0000","last_change":"2023-10-10T17:40:41.608529+0000","last_active":"2023-10-10T17:40:46.100994+0000","last_peered":"2023-10-10T17:40:46.100994+0000","last_clean":"2023-10-10T17:40:46.100994+0000","last_became_active":"2023-10-10T17:40:41.608247+0000","last_became_peered":"2023-10-10T17:40:41.608247+0000","last_unstale":"2023-10-10T17:40:46.100994+0000","last_undegraded":"2023-10-10T17:40:46.100994+0000","last_fullsized":"2023-10-10T17:40:46.100994+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:05:15.777617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100685+0000","last_change":"2023-10-10T17:40:45.707455+0000","last_active":"2023-10-10T17:40:46.100685+0000","last_peered":"2023-10-10T17:40:46.100685+0000","last_clean":"2023-10-10T17:40:46.100685+0000","last_became_active":"2023-10-10T17:40:45.707024+0000","last_became_peered":"2023-10-10T17:40:45.707024+0000","last_unstale":"2023-10-10T17:40:46.100685+0000","last_undegraded":"2023-10-10T17:40:46.100685+0000","last_fullsized":"2023-10-10T17:40:46.100685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:12:19.029228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100021+0000","last_change":"2023-10-10T17:40:41.165983+0000","last_active":"2023-10-10T17:40:46.100021+0000","last_peered":"2023-10-10T17:40:46.100021+0000","last_clean":"2023-10-10T17:40:46.100021+0000","last_became_active":"2023-10-10T17:40:41.165487+0000","last_became_peered":"2023-10-10T17:40:41.165487+0000","last_unstale":"2023-10-10T17:40:46.100021+0000","last_undegraded":"2023-10-10T17:40:46.100021+0000","last_fullsized":"2023-10-10T17:40:46.100021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:06:17.388327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275769+0000","last_change":"2023-10-10T17:40:41.614258+0000","last_active":"2023-10-10T17:40:46.275769+0000","last_peered":"2023-10-10T17:40:46.275769+0000","last_clean":"2023-10-10T17:40:46.275769+0000","last_became_active":"2023-10-10T17:40:41.613951+0000","last_became_peered":"2023-10-10T17:40:41.613951+0000","last_unstale":"2023-10-10T17:40:46.275769+0000","last_undegraded":"2023-10-10T17:40:46.275769+0000","last_fullsized":"2023-10-10T17:40:46.275769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:40:16.468408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276150+0000","last_change":"2023-10-10T17:40:45.708568+0000","last_active":"2023-10-10T17:40:46.276150+0000","last_peered":"2023-10-10T17:40:46.276150+0000","last_clean":"2023-10-10T17:40:46.276150+0000","last_became_active":"2023-10-10T17:40:45.708108+0000","last_became_peered":"2023-10-10T17:40:45.708108+0000","last_unstale":"2023-10-10T17:40:46.276150+0000","last_undegraded":"2023-10-10T17:40:46.276150+0000","last_fullsized":"2023-10-10T17:40:46.276150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:55:48.792350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276022+0000","last_change":"2023-10-10T17:40:41.167744+0000","last_active":"2023-10-10T17:40:46.276022+0000","last_peered":"2023-10-10T17:40:46.276022+0000","last_clean":"2023-10-10T17:40:46.276022+0000","last_became_active":"2023-10-10T17:40:41.167189+0000","last_became_peered":"2023-10-10T17:40:41.167189+0000","last_unstale":"2023-10-10T17:40:46.276022+0000","last_undegraded":"2023-10-10T17:40:46.276022+0000","last_fullsized":"2023-10-10T17:40:46.276022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:11.349794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276388+0000","last_change":"2023-10-10T17:40:41.608514+0000","last_active":"2023-10-10T17:40:46.276388+0000","last_peered":"2023-10-10T17:40:46.276388+0000","last_clean":"2023-10-10T17:40:46.276388+0000","last_became_active":"2023-10-10T17:40:41.608208+0000","last_became_peered":"2023-10-10T17:40:41.608208+0000","last_unstale":"2023-10-10T17:40:46.276388+0000","last_undegraded":"2023-10-10T17:40:46.276388+0000","last_fullsized":"2023-10-10T17:40:46.276388+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:05.744048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275854+0000","last_change":"2023-10-10T17:40:45.710094+0000","last_active":"2023-10-10T17:40:46.275854+0000","last_peered":"2023-10-10T17:40:46.275854+0000","last_clean":"2023-10-10T17:40:46.275854+0000","last_became_active":"2023-10-10T17:40:45.709520+0000","last_became_peered":"2023-10-10T17:40:45.709520+0000","last_unstale":"2023-10-10T17:40:46.275854+0000","last_undegraded":"2023-10-10T17:40:46.275854+0000","last_fullsized":"2023-10-10T17:40:46.275854+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:03:32.151300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:41:00.593651+0000","last_change":"2023-10-10T17:40:41.169821+0000","last_active":"2023-10-10T17:41:00.593651+0000","last_peered":"2023-10-10T17:41:00.593651+0000","last_clean":"2023-10-10T17:41:00.593651+0000","last_became_active":"2023-10-10T17:40:41.169502+0000","last_became_peered":"2023-10-10T17:40:41.169502+0000","last_unstale":"2023-10-10T17:41:00.593651+0000","last_undegraded":"2023-10-10T17:41:00.593651+0000","last_fullsized":"2023-10-10T17:41:00.593651+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:47:41.083180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275369+0000","last_change":"2023-10-10T17:40:41.602720+0000","last_active":"2023-10-10T17:40:46.275369+0000","last_peered":"2023-10-10T17:40:46.275369+0000","last_clean":"2023-10-10T17:40:46.275369+0000","last_became_active":"2023-10-10T17:40:41.602187+0000","last_became_peered":"2023-10-10T17:40:41.602187+0000","last_unstale":"2023-10-10T17:40:46.275369+0000","last_undegraded":"2023-10-10T17:40:46.275369+0000","last_fullsized":"2023-10-10T17:40:46.275369+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:12:04.205253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275449+0000","last_change":"2023-10-10T17:40:45.706946+0000","last_active":"2023-10-10T17:40:46.275449+0000","last_peered":"2023-10-10T17:40:46.275449+0000","last_clean":"2023-10-10T17:40:46.275449+0000","last_became_active":"2023-10-10T17:40:45.706495+0000","last_became_peered":"2023-10-10T17:40:45.706495+0000","last_unstale":"2023-10-10T17:40:46.275449+0000","last_undegraded":"2023-10-10T17:40:46.275449+0000","last_fullsized":"2023-10-10T17:40:46.275449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:24:40.545583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277100+0000","last_change":"2023-10-10T17:40:41.171855+0000","last_active":"2023-10-10T17:40:46.277100+0000","last_peered":"2023-10-10T17:40:46.277100+0000","last_clean":"2023-10-10T17:40:46.277100+0000","last_became_active":"2023-10-10T17:40:41.171515+0000","last_became_peered":"2023-10-10T17:40:41.171515+0000","last_unstale":"2023-10-10T17:40:46.277100+0000","last_undegraded":"2023-10-10T17:40:46.277100+0000","last_fullsized":"2023-10-10T17:40:46.277100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:31:10.288552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276592+0000","last_change":"2023-10-10T17:40:41.587083+0000","last_active":"2023-10-10T17:40:46.276592+0000","last_peered":"2023-10-10T17:40:46.276592+0000","last_clean":"2023-10-10T17:40:46.276592+0000","last_became_active":"2023-10-10T17:40:41.586758+0000","last_became_peered":"2023-10-10T17:40:41.586758+0000","last_unstale":"2023-10-10T17:40:46.276592+0000","last_undegraded":"2023-10-10T17:40:46.276592+0000","last_fullsized":"2023-10-10T17:40:46.276592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:04:36.358639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100878+0000","last_change":"2023-10-10T17:40:45.710492+0000","last_active":"2023-10-10T17:40:46.100878+0000","last_peered":"2023-10-10T17:40:46.100878+0000","last_clean":"2023-10-10T17:40:46.100878+0000","last_became_active":"2023-10-10T17:40:45.710069+0000","last_became_peered":"2023-10-10T17:40:45.710069+0000","last_unstale":"2023-10-10T17:40:46.100878+0000","last_undegraded":"2023-10-10T17:40:46.100878+0000","last_fullsized":"2023-10-10T17:40:46.100878+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:38:14.020875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.590663+0000","last_change":"2023-10-10T17:40:41.167449+0000","last_active":"2023-10-10T17:40:59.590663+0000","last_peered":"2023-10-10T17:40:59.590663+0000","last_clean":"2023-10-10T17:40:59.590663+0000","last_became_active":"2023-10-10T17:40:41.167038+0000","last_became_peered":"2023-10-10T17:40:41.167038+0000","last_unstale":"2023-10-10T17:40:59.590663+0000","last_undegraded":"2023-10-10T17:40:59.590663+0000","last_fullsized":"2023-10-10T17:40:59.590663+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:11:53.872618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276574+0000","last_change":"2023-10-10T17:40:41.594327+0000","last_active":"2023-10-10T17:40:46.276574+0000","last_peered":"2023-10-10T17:40:46.276574+0000","last_clean":"2023-10-10T17:40:46.276574+0000","last_became_active":"2023-10-10T17:40:41.594008+0000","last_became_peered":"2023-10-10T17:40:41.594008+0000","last_unstale":"2023-10-10T17:40:46.276574+0000","last_undegraded":"2023-10-10T17:40:46.276574+0000","last_fullsized":"2023-10-10T17:40:46.276574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:09:06.415005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100713+0000","last_change":"2023-10-10T17:40:45.710630+0000","last_active":"2023-10-10T17:40:46.100713+0000","last_peered":"2023-10-10T17:40:46.100713+0000","last_clean":"2023-10-10T17:40:46.100713+0000","last_became_active":"2023-10-10T17:40:45.710202+0000","last_became_peered":"2023-10-10T17:40:45.710202+0000","last_unstale":"2023-10-10T17:40:46.100713+0000","last_undegraded":"2023-10-10T17:40:46.100713+0000","last_fullsized":"2023-10-10T17:40:46.100713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:01.973715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100754+0000","last_change":"2023-10-10T17:40:41.171926+0000","last_active":"2023-10-10T17:40:46.100754+0000","last_peered":"2023-10-10T17:40:46.100754+0000","last_clean":"2023-10-10T17:40:46.100754+0000","last_became_active":"2023-10-10T17:40:41.171627+0000","last_became_peered":"2023-10-10T17:40:41.171627+0000","last_unstale":"2023-10-10T17:40:46.100754+0000","last_undegraded":"2023-10-10T17:40:46.100754+0000","last_fullsized":"2023-10-10T17:40:46.100754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:10:23.967086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276470+0000","last_change":"2023-10-10T17:40:41.593135+0000","last_active":"2023-10-10T17:40:46.276470+0000","last_peered":"2023-10-10T17:40:46.276470+0000","last_clean":"2023-10-10T17:40:46.276470+0000","last_became_active":"2023-10-10T17:40:41.592871+0000","last_became_peered":"2023-10-10T17:40:41.592871+0000","last_unstale":"2023-10-10T17:40:46.276470+0000","last_undegraded":"2023-10-10T17:40:46.276470+0000","last_fullsized":"2023-10-10T17:40:46.276470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:42:32.503283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275747+0000","last_change":"2023-10-10T17:40:45.703557+0000","last_active":"2023-10-10T17:40:46.275747+0000","last_peered":"2023-10-10T17:40:46.275747+0000","last_clean":"2023-10-10T17:40:46.275747+0000","last_became_active":"2023-10-10T17:40:45.703085+0000","last_became_peered":"2023-10-10T17:40:45.703085+0000","last_unstale":"2023-10-10T17:40:46.275747+0000","last_undegraded":"2023-10-10T17:40:46.275747+0000","last_fullsized":"2023-10-10T17:40:46.275747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:34:18.186700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213122+0000","last_change":"2023-10-10T17:40:41.162829+0000","last_active":"2023-10-10T17:40:46.213122+0000","last_peered":"2023-10-10T17:40:46.213122+0000","last_clean":"2023-10-10T17:40:46.213122+0000","last_became_active":"2023-10-10T17:40:41.162516+0000","last_became_peered":"2023-10-10T17:40:41.162516+0000","last_unstale":"2023-10-10T17:40:46.213122+0000","last_undegraded":"2023-10-10T17:40:46.213122+0000","last_fullsized":"2023-10-10T17:40:46.213122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:45:47.035059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275578+0000","last_change":"2023-10-10T17:40:41.606908+0000","last_active":"2023-10-10T17:40:46.275578+0000","last_peered":"2023-10-10T17:40:46.275578+0000","last_clean":"2023-10-10T17:40:46.275578+0000","last_became_active":"2023-10-10T17:40:41.606492+0000","last_became_peered":"2023-10-10T17:40:41.606492+0000","last_unstale":"2023-10-10T17:40:46.275578+0000","last_undegraded":"2023-10-10T17:40:46.275578+0000","last_fullsized":"2023-10-10T17:40:46.275578+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:08:48.702270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275681+0000","last_change":"2023-10-10T17:40:45.707139+0000","last_active":"2023-10-10T17:40:46.275681+0000","last_peered":"2023-10-10T17:40:46.275681+0000","last_clean":"2023-10-10T17:40:46.275681+0000","last_became_active":"2023-10-10T17:40:45.706708+0000","last_became_peered":"2023-10-10T17:40:45.706708+0000","last_unstale":"2023-10-10T17:40:46.275681+0000","last_undegraded":"2023-10-10T17:40:46.275681+0000","last_fullsized":"2023-10-10T17:40:46.275681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:09:37.435925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276991+0000","last_change":"2023-10-10T17:40:41.170512+0000","last_active":"2023-10-10T17:40:46.276991+0000","last_peered":"2023-10-10T17:40:46.276991+0000","last_clean":"2023-10-10T17:40:46.276991+0000","last_became_active":"2023-10-10T17:40:41.170264+0000","last_became_peered":"2023-10-10T17:40:41.170264+0000","last_unstale":"2023-10-10T17:40:46.276991+0000","last_undegraded":"2023-10-10T17:40:46.276991+0000","last_fullsized":"2023-10-10T17:40:46.276991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:22.102865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099516+0000","last_change":"2023-10-10T17:40:41.597647+0000","last_active":"2023-10-10T17:40:46.099516+0000","last_peered":"2023-10-10T17:40:46.099516+0000","last_clean":"2023-10-10T17:40:46.099516+0000","last_became_active":"2023-10-10T17:40:41.597245+0000","last_became_peered":"2023-10-10T17:40:41.597245+0000","last_unstale":"2023-10-10T17:40:46.099516+0000","last_undegraded":"2023-10-10T17:40:46.099516+0000","last_fullsized":"2023-10-10T17:40:46.099516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:53:56.653913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275791+0000","last_change":"2023-10-10T17:40:45.705089+0000","last_active":"2023-10-10T17:40:46.275791+0000","last_peered":"2023-10-10T17:40:46.275791+0000","last_clean":"2023-10-10T17:40:46.275791+0000","last_became_active":"2023-10-10T17:40:45.704475+0000","last_became_peered":"2023-10-10T17:40:45.704475+0000","last_unstale":"2023-10-10T17:40:46.275791+0000","last_undegraded":"2023-10-10T17:40:46.275791+0000","last_fullsized":"2023-10-10T17:40:46.275791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:51:15.851183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100943+0000","last_change":"2023-10-10T17:40:41.167409+0000","last_active":"2023-10-10T17:40:46.100943+0000","last_peered":"2023-10-10T17:40:46.100943+0000","last_clean":"2023-10-10T17:40:46.100943+0000","last_became_active":"2023-10-10T17:40:41.167125+0000","last_became_peered":"2023-10-10T17:40:41.167125+0000","last_unstale":"2023-10-10T17:40:46.100943+0000","last_undegraded":"2023-10-10T17:40:46.100943+0000","last_fullsized":"2023-10-10T17:40:46.100943+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:01:37.543896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276398+0000","last_change":"2023-10-10T17:40:41.606216+0000","last_active":"2023-10-10T17:40:46.276398+0000","last_peered":"2023-10-10T17:40:46.276398+0000","last_clean":"2023-10-10T17:40:46.276398+0000","last_became_active":"2023-10-10T17:40:41.603258+0000","last_became_peered":"2023-10-10T17:40:41.603258+0000","last_unstale":"2023-10-10T17:40:46.276398+0000","last_undegraded":"2023-10-10T17:40:46.276398+0000","last_fullsized":"2023-10-10T17:40:46.276398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:50:11.631129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099501+0000","last_change":"2023-10-10T17:40:45.705623+0000","last_active":"2023-10-10T17:40:46.099501+0000","last_peered":"2023-10-10T17:40:46.099501+0000","last_clean":"2023-10-10T17:40:46.099501+0000","last_became_active":"2023-10-10T17:40:45.705242+0000","last_became_peered":"2023-10-10T17:40:45.705242+0000","last_unstale":"2023-10-10T17:40:46.099501+0000","last_undegraded":"2023-10-10T17:40:46.099501+0000","last_fullsized":"2023-10-10T17:40:46.099501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:07:11.500067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457530+0000","last_change":"2023-10-10T17:40:41.171552+0000","last_active":"2023-10-10T17:40:46.457530+0000","last_peered":"2023-10-10T17:40:46.457530+0000","last_clean":"2023-10-10T17:40:46.457530+0000","last_became_active":"2023-10-10T17:40:41.171207+0000","last_became_peered":"2023-10-10T17:40:41.171207+0000","last_unstale":"2023-10-10T17:40:46.457530+0000","last_undegraded":"2023-10-10T17:40:46.457530+0000","last_fullsized":"2023-10-10T17:40:46.457530+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:39:37.128599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276523+0000","last_change":"2023-10-10T17:40:41.609655+0000","last_active":"2023-10-10T17:40:46.276523+0000","last_peered":"2023-10-10T17:40:46.276523+0000","last_clean":"2023-10-10T17:40:46.276523+0000","last_became_active":"2023-10-10T17:40:41.609318+0000","last_became_peered":"2023-10-10T17:40:41.609318+0000","last_unstale":"2023-10-10T17:40:46.276523+0000","last_undegraded":"2023-10-10T17:40:46.276523+0000","last_fullsized":"2023-10-10T17:40:46.276523+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:24:46.334636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276360+0000","last_change":"2023-10-10T17:40:45.709513+0000","last_active":"2023-10-10T17:40:46.276360+0000","last_peered":"2023-10-10T17:40:46.276360+0000","last_clean":"2023-10-10T17:40:46.276360+0000","last_became_active":"2023-10-10T17:40:45.708963+0000","last_became_peered":"2023-10-10T17:40:45.708963+0000","last_unstale":"2023-10-10T17:40:46.276360+0000","last_undegraded":"2023-10-10T17:40:46.276360+0000","last_fullsized":"2023-10-10T17:40:46.276360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:11:59.727814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276467+0000","last_change":"2023-10-10T17:40:41.163234+0000","last_active":"2023-10-10T17:40:46.276467+0000","last_peered":"2023-10-10T17:40:46.276467+0000","last_clean":"2023-10-10T17:40:46.276467+0000","last_became_active":"2023-10-10T17:40:41.162916+0000","last_became_peered":"2023-10-10T17:40:41.162916+0000","last_unstale":"2023-10-10T17:40:46.276467+0000","last_undegraded":"2023-10-10T17:40:46.276467+0000","last_fullsized":"2023-10-10T17:40:46.276467+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:54.672510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275767+0000","last_change":"2023-10-10T17:40:41.602852+0000","last_active":"2023-10-10T17:40:46.275767+0000","last_peered":"2023-10-10T17:40:46.275767+0000","last_clean":"2023-10-10T17:40:46.275767+0000","last_became_active":"2023-10-10T17:40:41.602530+0000","last_became_peered":"2023-10-10T17:40:41.602530+0000","last_unstale":"2023-10-10T17:40:46.275767+0000","last_undegraded":"2023-10-10T17:40:46.275767+0000","last_fullsized":"2023-10-10T17:40:46.275767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:55:10.460505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099787+0000","last_change":"2023-10-10T17:40:45.713714+0000","last_active":"2023-10-10T17:40:46.099787+0000","last_peered":"2023-10-10T17:40:46.099787+0000","last_clean":"2023-10-10T17:40:46.099787+0000","last_became_active":"2023-10-10T17:40:45.712953+0000","last_became_peered":"2023-10-10T17:40:45.712953+0000","last_unstale":"2023-10-10T17:40:46.099787+0000","last_undegraded":"2023-10-10T17:40:46.099787+0000","last_fullsized":"2023-10-10T17:40:46.099787+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:43:25.326564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:57.105298+0000","last_change":"2023-10-10T17:40:41.168328+0000","last_active":"2023-10-10T17:40:57.105298+0000","last_peered":"2023-10-10T17:40:57.105298+0000","last_clean":"2023-10-10T17:40:57.105298+0000","last_became_active":"2023-10-10T17:40:41.168044+0000","last_became_peered":"2023-10-10T17:40:41.168044+0000","last_unstale":"2023-10-10T17:40:57.105298+0000","last_undegraded":"2023-10-10T17:40:57.105298+0000","last_fullsized":"2023-10-10T17:40:57.105298+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:19:58.468515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275980+0000","last_change":"2023-10-10T17:40:41.597795+0000","last_active":"2023-10-10T17:40:46.275980+0000","last_peered":"2023-10-10T17:40:46.275980+0000","last_clean":"2023-10-10T17:40:46.275980+0000","last_became_active":"2023-10-10T17:40:41.597537+0000","last_became_peered":"2023-10-10T17:40:41.597537+0000","last_unstale":"2023-10-10T17:40:46.275980+0000","last_undegraded":"2023-10-10T17:40:46.275980+0000","last_fullsized":"2023-10-10T17:40:46.275980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:08:46.240820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277087+0000","last_change":"2023-10-10T17:40:45.709827+0000","last_active":"2023-10-10T17:40:46.277087+0000","last_peered":"2023-10-10T17:40:46.277087+0000","last_clean":"2023-10-10T17:40:46.277087+0000","last_became_active":"2023-10-10T17:40:45.709373+0000","last_became_peered":"2023-10-10T17:40:45.709373+0000","last_unstale":"2023-10-10T17:40:46.277087+0000","last_undegraded":"2023-10-10T17:40:46.277087+0000","last_fullsized":"2023-10-10T17:40:46.277087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:56:06.379501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100574+0000","last_change":"2023-10-10T17:40:41.160804+0000","last_active":"2023-10-10T17:40:46.100574+0000","last_peered":"2023-10-10T17:40:46.100574+0000","last_clean":"2023-10-10T17:40:46.100574+0000","last_became_active":"2023-10-10T17:40:41.160493+0000","last_became_peered":"2023-10-10T17:40:41.160493+0000","last_unstale":"2023-10-10T17:40:46.100574+0000","last_undegraded":"2023-10-10T17:40:46.100574+0000","last_fullsized":"2023-10-10T17:40:46.100574+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:32:18.782209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213141+0000","last_change":"2023-10-10T17:40:41.601357+0000","last_active":"2023-10-10T17:40:46.213141+0000","last_peered":"2023-10-10T17:40:46.213141+0000","last_clean":"2023-10-10T17:40:46.213141+0000","last_became_active":"2023-10-10T17:40:41.601057+0000","last_became_peered":"2023-10-10T17:40:41.601057+0000","last_unstale":"2023-10-10T17:40:46.213141+0000","last_undegraded":"2023-10-10T17:40:46.213141+0000","last_fullsized":"2023-10-10T17:40:46.213141+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:03:02.150396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275966+0000","last_change":"2023-10-10T17:40:45.708067+0000","last_active":"2023-10-10T17:40:46.275966+0000","last_peered":"2023-10-10T17:40:46.275966+0000","last_clean":"2023-10-10T17:40:46.275966+0000","last_became_active":"2023-10-10T17:40:45.707686+0000","last_became_peered":"2023-10-10T17:40:45.707686+0000","last_unstale":"2023-10-10T17:40:46.275966+0000","last_undegraded":"2023-10-10T17:40:46.275966+0000","last_fullsized":"2023-10-10T17:40:46.275966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:00:14.708537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.591858+0000","last_change":"2023-10-10T17:40:41.168006+0000","last_active":"2023-10-10T17:40:59.591858+0000","last_peered":"2023-10-10T17:40:59.591858+0000","last_clean":"2023-10-10T17:40:59.591858+0000","last_became_active":"2023-10-10T17:40:41.167777+0000","last_became_peered":"2023-10-10T17:40:41.167777+0000","last_unstale":"2023-10-10T17:40:59.591858+0000","last_undegraded":"2023-10-10T17:40:59.591858+0000","last_fullsized":"2023-10-10T17:40:59.591858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:38:44.110762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457271+0000","last_change":"2023-10-10T17:40:41.591215+0000","last_active":"2023-10-10T17:40:46.457271+0000","last_peered":"2023-10-10T17:40:46.457271+0000","last_clean":"2023-10-10T17:40:46.457271+0000","last_became_active":"2023-10-10T17:40:41.590929+0000","last_became_peered":"2023-10-10T17:40:41.590929+0000","last_unstale":"2023-10-10T17:40:46.457271+0000","last_undegraded":"2023-10-10T17:40:46.457271+0000","last_fullsized":"2023-10-10T17:40:46.457271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:34:40.010647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099959+0000","last_change":"2023-10-10T17:40:45.709038+0000","last_active":"2023-10-10T17:40:46.099959+0000","last_peered":"2023-10-10T17:40:46.099959+0000","last_clean":"2023-10-10T17:40:46.099959+0000","last_became_active":"2023-10-10T17:40:45.708683+0000","last_became_peered":"2023-10-10T17:40:45.708683+0000","last_unstale":"2023-10-10T17:40:46.099959+0000","last_undegraded":"2023-10-10T17:40:46.099959+0000","last_fullsized":"2023-10-10T17:40:46.099959+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:25:44.015116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276240+0000","last_change":"2023-10-10T17:40:41.163817+0000","last_active":"2023-10-10T17:40:46.276240+0000","last_peered":"2023-10-10T17:40:46.276240+0000","last_clean":"2023-10-10T17:40:46.276240+0000","last_became_active":"2023-10-10T17:40:41.163497+0000","last_became_peered":"2023-10-10T17:40:41.163497+0000","last_unstale":"2023-10-10T17:40:46.276240+0000","last_undegraded":"2023-10-10T17:40:46.276240+0000","last_fullsized":"2023-10-10T17:40:46.276240+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:44:23.887489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457017+0000","last_change":"2023-10-10T17:40:41.593843+0000","last_active":"2023-10-10T17:40:46.457017+0000","last_peered":"2023-10-10T17:40:46.457017+0000","last_clean":"2023-10-10T17:40:46.457017+0000","last_became_active":"2023-10-10T17:40:41.593585+0000","last_became_peered":"2023-10-10T17:40:41.593585+0000","last_unstale":"2023-10-10T17:40:46.457017+0000","last_undegraded":"2023-10-10T17:40:46.457017+0000","last_fullsized":"2023-10-10T17:40:46.457017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:13:03.463052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213888+0000","last_change":"2023-10-10T17:40:45.714900+0000","last_active":"2023-10-10T17:40:46.213888+0000","last_peered":"2023-10-10T17:40:46.213888+0000","last_clean":"2023-10-10T17:40:46.213888+0000","last_became_active":"2023-10-10T17:40:45.714525+0000","last_became_peered":"2023-10-10T17:40:45.714525+0000","last_unstale":"2023-10-10T17:40:46.213888+0000","last_undegraded":"2023-10-10T17:40:46.213888+0000","last_fullsized":"2023-10-10T17:40:46.213888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:43:57.654613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:41:00.596892+0000","last_change":"2023-10-10T17:40:41.164365+0000","last_active":"2023-10-10T17:41:00.596892+0000","last_peered":"2023-10-10T17:41:00.596892+0000","last_clean":"2023-10-10T17:41:00.596892+0000","last_became_active":"2023-10-10T17:40:41.164046+0000","last_became_peered":"2023-10-10T17:40:41.164046+0000","last_unstale":"2023-10-10T17:41:00.596892+0000","last_undegraded":"2023-10-10T17:41:00.596892+0000","last_fullsized":"2023-10-10T17:41:00.596892+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:26:07.378218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277194+0000","last_change":"2023-10-10T17:40:41.586591+0000","last_active":"2023-10-10T17:40:46.277194+0000","last_peered":"2023-10-10T17:40:46.277194+0000","last_clean":"2023-10-10T17:40:46.277194+0000","last_became_active":"2023-10-10T17:40:41.586293+0000","last_became_peered":"2023-10-10T17:40:41.586293+0000","last_unstale":"2023-10-10T17:40:46.277194+0000","last_undegraded":"2023-10-10T17:40:46.277194+0000","last_fullsized":"2023-10-10T17:40:46.277194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:10.920114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277050+0000","last_change":"2023-10-10T17:40:45.708348+0000","last_active":"2023-10-10T17:40:46.277050+0000","last_peered":"2023-10-10T17:40:46.277050+0000","last_clean":"2023-10-10T17:40:46.277050+0000","last_became_active":"2023-10-10T17:40:45.707899+0000","last_became_peered":"2023-10-10T17:40:45.707899+0000","last_unstale":"2023-10-10T17:40:46.277050+0000","last_undegraded":"2023-10-10T17:40:46.277050+0000","last_fullsized":"2023-10-10T17:40:46.277050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:00:08.005331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275906+0000","last_change":"2023-10-10T17:40:41.168209+0000","last_active":"2023-10-10T17:40:46.275906+0000","last_peered":"2023-10-10T17:40:46.275906+0000","last_clean":"2023-10-10T17:40:46.275906+0000","last_became_active":"2023-10-10T17:40:41.167907+0000","last_became_peered":"2023-10-10T17:40:41.167907+0000","last_unstale":"2023-10-10T17:40:46.275906+0000","last_undegraded":"2023-10-10T17:40:46.275906+0000","last_fullsized":"2023-10-10T17:40:46.275906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:43:42.334992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276368+0000","last_change":"2023-10-10T17:40:41.594848+0000","last_active":"2023-10-10T17:40:46.276368+0000","last_peered":"2023-10-10T17:40:46.276368+0000","last_clean":"2023-10-10T17:40:46.276368+0000","last_became_active":"2023-10-10T17:40:41.594562+0000","last_became_peered":"2023-10-10T17:40:41.594562+0000","last_unstale":"2023-10-10T17:40:46.276368+0000","last_undegraded":"2023-10-10T17:40:46.276368+0000","last_fullsized":"2023-10-10T17:40:46.276368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:58:54.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276721+0000","last_change":"2023-10-10T17:40:45.711043+0000","last_active":"2023-10-10T17:40:46.276721+0000","last_peered":"2023-10-10T17:40:46.276721+0000","last_clean":"2023-10-10T17:40:46.276721+0000","last_became_active":"2023-10-10T17:40:45.710617+0000","last_became_peered":"2023-10-10T17:40:45.710617+0000","last_unstale":"2023-10-10T17:40:46.276721+0000","last_undegraded":"2023-10-10T17:40:46.276721+0000","last_fullsized":"2023-10-10T17:40:46.276721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:57:48.626413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099764+0000","last_change":"2023-10-10T17:40:41.163238+0000","last_active":"2023-10-10T17:40:46.099764+0000","last_peered":"2023-10-10T17:40:46.099764+0000","last_clean":"2023-10-10T17:40:46.099764+0000","last_became_active":"2023-10-10T17:40:41.162934+0000","last_became_peered":"2023-10-10T17:40:41.162934+0000","last_unstale":"2023-10-10T17:40:46.099764+0000","last_undegraded":"2023-10-10T17:40:46.099764+0000","last_fullsized":"2023-10-10T17:40:46.099764+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:37:54.019115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.101058+0000","last_change":"2023-10-10T17:40:41.597026+0000","last_active":"2023-10-10T17:40:46.101058+0000","last_peered":"2023-10-10T17:40:46.101058+0000","last_clean":"2023-10-10T17:40:46.101058+0000","last_became_active":"2023-10-10T17:40:41.596747+0000","last_became_peered":"2023-10-10T17:40:41.596747+0000","last_unstale":"2023-10-10T17:40:46.101058+0000","last_undegraded":"2023-10-10T17:40:46.101058+0000","last_fullsized":"2023-10-10T17:40:46.101058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:26:07.488204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213403+0000","last_change":"2023-10-10T17:40:45.707218+0000","last_active":"2023-10-10T17:40:46.213403+0000","last_peered":"2023-10-10T17:40:46.213403+0000","last_clean":"2023-10-10T17:40:46.213403+0000","last_became_active":"2023-10-10T17:40:45.706850+0000","last_became_peered":"2023-10-10T17:40:45.706850+0000","last_unstale":"2023-10-10T17:40:46.213403+0000","last_undegraded":"2023-10-10T17:40:46.213403+0000","last_fullsized":"2023-10-10T17:40:46.213403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:15:20.838041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457087+0000","last_change":"2023-10-10T17:40:41.166713+0000","last_active":"2023-10-10T17:40:46.457087+0000","last_peered":"2023-10-10T17:40:46.457087+0000","last_clean":"2023-10-10T17:40:46.457087+0000","last_became_active":"2023-10-10T17:40:41.165951+0000","last_became_peered":"2023-10-10T17:40:41.165951+0000","last_unstale":"2023-10-10T17:40:46.457087+0000","last_undegraded":"2023-10-10T17:40:46.457087+0000","last_fullsized":"2023-10-10T17:40:46.457087+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:16:54.814748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100908+0000","last_change":"2023-10-10T17:40:41.605860+0000","last_active":"2023-10-10T17:40:46.100908+0000","last_peered":"2023-10-10T17:40:46.100908+0000","last_clean":"2023-10-10T17:40:46.100908+0000","last_became_active":"2023-10-10T17:40:41.605556+0000","last_became_peered":"2023-10-10T17:40:41.605556+0000","last_unstale":"2023-10-10T17:40:46.100908+0000","last_undegraded":"2023-10-10T17:40:46.100908+0000","last_fullsized":"2023-10-10T17:40:46.100908+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:33:10.684987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213514+0000","last_change":"2023-10-10T17:40:45.711844+0000","last_active":"2023-10-10T17:40:46.213514+0000","last_peered":"2023-10-10T17:40:46.213514+0000","last_clean":"2023-10-10T17:40:46.213514+0000","last_became_active":"2023-10-10T17:40:45.711413+0000","last_became_peered":"2023-10-10T17:40:45.711413+0000","last_unstale":"2023-10-10T17:40:46.213514+0000","last_undegraded":"2023-10-10T17:40:46.213514+0000","last_fullsized":"2023-10-10T17:40:46.213514+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:58:41.509326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277401+0000","last_change":"2023-10-10T17:40:41.608625+0000","last_active":"2023-10-10T17:40:46.277401+0000","last_peered":"2023-10-10T17:40:46.277401+0000","last_clean":"2023-10-10T17:40:46.277401+0000","last_became_active":"2023-10-10T17:40:41.608332+0000","last_became_peered":"2023-10-10T17:40:41.608332+0000","last_unstale":"2023-10-10T17:40:46.277401+0000","last_undegraded":"2023-10-10T17:40:46.277401+0000","last_fullsized":"2023-10-10T17:40:46.277401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:17:17.274726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.212988+0000","last_change":"2023-10-10T17:40:41.169807+0000","last_active":"2023-10-10T17:40:46.212988+0000","last_peered":"2023-10-10T17:40:46.212988+0000","last_clean":"2023-10-10T17:40:46.212988+0000","last_became_active":"2023-10-10T17:40:41.169475+0000","last_became_peered":"2023-10-10T17:40:41.169475+0000","last_unstale":"2023-10-10T17:40:46.212988+0000","last_undegraded":"2023-10-10T17:40:46.212988+0000","last_fullsized":"2023-10-10T17:40:46.212988+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:14.806814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457445+0000","last_change":"2023-10-10T17:40:45.710544+0000","last_active":"2023-10-10T17:40:46.457445+0000","last_peered":"2023-10-10T17:40:46.457445+0000","last_clean":"2023-10-10T17:40:46.457445+0000","last_became_active":"2023-10-10T17:40:45.710084+0000","last_became_peered":"2023-10-10T17:40:45.710084+0000","last_unstale":"2023-10-10T17:40:46.457445+0000","last_undegraded":"2023-10-10T17:40:46.457445+0000","last_fullsized":"2023-10-10T17:40:46.457445+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:22.877276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275269+0000","last_change":"2023-10-10T17:40:41.599028+0000","last_active":"2023-10-10T17:40:46.275269+0000","last_peered":"2023-10-10T17:40:46.275269+0000","last_clean":"2023-10-10T17:40:46.275269+0000","last_became_active":"2023-10-10T17:40:41.598684+0000","last_became_peered":"2023-10-10T17:40:41.598684+0000","last_unstale":"2023-10-10T17:40:46.275269+0000","last_undegraded":"2023-10-10T17:40:46.275269+0000","last_fullsized":"2023-10-10T17:40:46.275269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:30:10.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275985+0000","last_change":"2023-10-10T17:40:41.165756+0000","last_active":"2023-10-10T17:40:46.275985+0000","last_peered":"2023-10-10T17:40:46.275985+0000","last_clean":"2023-10-10T17:40:46.275985+0000","last_became_active":"2023-10-10T17:40:41.165368+0000","last_became_peered":"2023-10-10T17:40:41.165368+0000","last_unstale":"2023-10-10T17:40:46.275985+0000","last_undegraded":"2023-10-10T17:40:46.275985+0000","last_fullsized":"2023-10-10T17:40:46.275985+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:29:16.457611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":354,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.456850+0000","last_change":"2023-10-10T17:39:42.508828+0000","last_active":"2023-10-10T17:40:46.456850+0000","last_peered":"2023-10-10T17:40:46.456850+0000","last_clean":"2023-10-10T17:40:46.456850+0000","last_became_active":"2023-10-10T17:39:42.508475+0000","last_became_peered":"2023-10-10T17:39:42.508475+0000","last_unstale":"2023-10-10T17:40:46.456850+0000","last_undegraded":"2023-10-10T17:40:46.456850+0000","last_fullsized":"2023-10-10T17:40:46.456850+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_clean_scrub_stamp":"2023-10-10T17:36:46.553699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:52:08.434364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277319+0000","last_change":"2023-10-10T17:40:45.714604+0000","last_active":"2023-10-10T17:40:46.277319+0000","last_peered":"2023-10-10T17:40:46.277319+0000","last_clean":"2023-10-10T17:40:46.277319+0000","last_became_active":"2023-10-10T17:40:45.714185+0000","last_became_peered":"2023-10-10T17:40:45.714185+0000","last_unstale":"2023-10-10T17:40:46.277319+0000","last_undegraded":"2023-10-10T17:40:46.277319+0000","last_fullsized":"2023-10-10T17:40:46.277319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:36.214433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.212973+0000","last_change":"2023-10-10T17:40:41.610002+0000","last_active":"2023-10-10T17:40:46.212973+0000","last_peered":"2023-10-10T17:40:46.212973+0000","last_clean":"2023-10-10T17:40:46.212973+0000","last_became_active":"2023-10-10T17:40:41.609708+0000","last_became_peered":"2023-10-10T17:40:41.609708+0000","last_unstale":"2023-10-10T17:40:46.212973+0000","last_undegraded":"2023-10-10T17:40:46.212973+0000","last_fullsized":"2023-10-10T17:40:46.212973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:16:26.799522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.456827+0000","last_change":"2023-10-10T17:40:41.167715+0000","last_active":"2023-10-10T17:40:46.456827+0000","last_peered":"2023-10-10T17:40:46.456827+0000","last_clean":"2023-10-10T17:40:46.456827+0000","last_became_active":"2023-10-10T17:40:41.166385+0000","last_became_peered":"2023-10-10T17:40:41.166385+0000","last_unstale":"2023-10-10T17:40:46.456827+0000","last_undegraded":"2023-10-10T17:40:46.456827+0000","last_fullsized":"2023-10-10T17:40:46.456827+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:50:45.606203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276936+0000","last_change":"2023-10-10T17:40:45.707236+0000","last_active":"2023-10-10T17:40:46.276936+0000","last_peered":"2023-10-10T17:40:46.276936+0000","last_clean":"2023-10-10T17:40:46.276936+0000","last_became_active":"2023-10-10T17:40:45.706827+0000","last_became_peered":"2023-10-10T17:40:45.706827+0000","last_unstale":"2023-10-10T17:40:46.276936+0000","last_undegraded":"2023-10-10T17:40:46.276936+0000","last_fullsized":"2023-10-10T17:40:46.276936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:38:34.181039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.591891+0000","last_change":"2023-10-10T17:40:41.170523+0000","last_active":"2023-10-10T17:40:59.591891+0000","last_peered":"2023-10-10T17:40:59.591891+0000","last_clean":"2023-10-10T17:40:59.591891+0000","last_became_active":"2023-10-10T17:40:41.170243+0000","last_became_peered":"2023-10-10T17:40:41.170243+0000","last_unstale":"2023-10-10T17:40:59.591891+0000","last_undegraded":"2023-10-10T17:40:59.591891+0000","last_fullsized":"2023-10-10T17:40:59.591891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:21:25.745722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213475+0000","last_change":"2023-10-10T17:40:41.610472+0000","last_active":"2023-10-10T17:40:46.213475+0000","last_peered":"2023-10-10T17:40:46.213475+0000","last_clean":"2023-10-10T17:40:46.213475+0000","last_became_active":"2023-10-10T17:40:41.610139+0000","last_became_peered":"2023-10-10T17:40:41.610139+0000","last_unstale":"2023-10-10T17:40:46.213475+0000","last_undegraded":"2023-10-10T17:40:46.213475+0000","last_fullsized":"2023-10-10T17:40:46.213475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:39:38.378204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275493+0000","last_change":"2023-10-10T17:40:45.712179+0000","last_active":"2023-10-10T17:40:46.275493+0000","last_peered":"2023-10-10T17:40:46.275493+0000","last_clean":"2023-10-10T17:40:46.275493+0000","last_became_active":"2023-10-10T17:40:45.711708+0000","last_became_peered":"2023-10-10T17:40:45.711708+0000","last_unstale":"2023-10-10T17:40:46.275493+0000","last_undegraded":"2023-10-10T17:40:46.275493+0000","last_fullsized":"2023-10-10T17:40:46.275493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:49:03.281192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"88'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.099962+0000","last_change":"2023-10-10T17:40:41.164882+0000","last_active":"2023-10-10T17:40:46.099962+0000","last_peered":"2023-10-10T17:40:46.099962+0000","last_clean":"2023-10-10T17:40:46.099962+0000","last_became_active":"2023-10-10T17:40:41.164536+0000","last_became_peered":"2023-10-10T17:40:41.164536+0000","last_unstale":"2023-10-10T17:40:46.099962+0000","last_undegraded":"2023-10-10T17:40:46.099962+0000","last_fullsized":"2023-10-10T17:40:46.099962+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:57:53.185248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276385+0000","last_change":"2023-10-10T17:40:41.595043+0000","last_active":"2023-10-10T17:40:46.276385+0000","last_peered":"2023-10-10T17:40:46.276385+0000","last_clean":"2023-10-10T17:40:46.276385+0000","last_became_active":"2023-10-10T17:40:41.594703+0000","last_became_peered":"2023-10-10T17:40:41.594703+0000","last_unstale":"2023-10-10T17:40:46.276385+0000","last_undegraded":"2023-10-10T17:40:46.276385+0000","last_fullsized":"2023-10-10T17:40:46.276385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:30:41.976969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100408+0000","last_change":"2023-10-10T17:40:45.705232+0000","last_active":"2023-10-10T17:40:46.100408+0000","last_peered":"2023-10-10T17:40:46.100408+0000","last_clean":"2023-10-10T17:40:46.100408+0000","last_became_active":"2023-10-10T17:40:45.704811+0000","last_became_peered":"2023-10-10T17:40:45.704811+0000","last_unstale":"2023-10-10T17:40:46.100408+0000","last_undegraded":"2023-10-10T17:40:46.100408+0000","last_fullsized":"2023-10-10T17:40:46.100408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:47:11.555315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276046+0000","last_change":"2023-10-10T17:40:41.165620+0000","last_active":"2023-10-10T17:40:46.276046+0000","last_peered":"2023-10-10T17:40:46.276046+0000","last_clean":"2023-10-10T17:40:46.276046+0000","last_became_active":"2023-10-10T17:40:41.165262+0000","last_became_peered":"2023-10-10T17:40:41.165262+0000","last_unstale":"2023-10-10T17:40:46.276046+0000","last_undegraded":"2023-10-10T17:40:46.276046+0000","last_fullsized":"2023-10-10T17:40:46.276046+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:34:17.999550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275768+0000","last_change":"2023-10-10T17:40:41.603549+0000","last_active":"2023-10-10T17:40:46.275768+0000","last_peered":"2023-10-10T17:40:46.275768+0000","last_clean":"2023-10-10T17:40:46.275768+0000","last_became_active":"2023-10-10T17:40:41.602991+0000","last_became_peered":"2023-10-10T17:40:41.602991+0000","last_unstale":"2023-10-10T17:40:46.275768+0000","last_undegraded":"2023-10-10T17:40:46.275768+0000","last_fullsized":"2023-10-10T17:40:46.275768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:24:55.680074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275593+0000","last_change":"2023-10-10T17:40:45.714764+0000","last_active":"2023-10-10T17:40:46.275593+0000","last_peered":"2023-10-10T17:40:46.275593+0000","last_clean":"2023-10-10T17:40:46.275593+0000","last_became_active":"2023-10-10T17:40:45.714173+0000","last_became_peered":"2023-10-10T17:40:45.714173+0000","last_unstale":"2023-10-10T17:40:46.275593+0000","last_undegraded":"2023-10-10T17:40:46.275593+0000","last_fullsized":"2023-10-10T17:40:46.275593+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:53.882859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275175+0000","last_change":"2023-10-10T17:40:41.169859+0000","last_active":"2023-10-10T17:40:46.275175+0000","last_peered":"2023-10-10T17:40:46.275175+0000","last_clean":"2023-10-10T17:40:46.275175+0000","last_became_active":"2023-10-10T17:40:41.169507+0000","last_became_peered":"2023-10-10T17:40:41.169507+0000","last_unstale":"2023-10-10T17:40:46.275175+0000","last_undegraded":"2023-10-10T17:40:46.275175+0000","last_fullsized":"2023-10-10T17:40:46.275175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:01:40.153696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275512+0000","last_change":"2023-10-10T17:40:41.593929+0000","last_active":"2023-10-10T17:40:46.275512+0000","last_peered":"2023-10-10T17:40:46.275512+0000","last_clean":"2023-10-10T17:40:46.275512+0000","last_became_active":"2023-10-10T17:40:41.593565+0000","last_became_peered":"2023-10-10T17:40:41.593565+0000","last_unstale":"2023-10-10T17:40:46.275512+0000","last_undegraded":"2023-10-10T17:40:46.275512+0000","last_fullsized":"2023-10-10T17:40:46.275512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:11.833671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100095+0000","last_change":"2023-10-10T17:40:45.710759+0000","last_active":"2023-10-10T17:40:46.100095+0000","last_peered":"2023-10-10T17:40:46.100095+0000","last_clean":"2023-10-10T17:40:46.100095+0000","last_became_active":"2023-10-10T17:40:45.710118+0000","last_became_peered":"2023-10-10T17:40:45.710118+0000","last_unstale":"2023-10-10T17:40:46.100095+0000","last_undegraded":"2023-10-10T17:40:46.100095+0000","last_fullsized":"2023-10-10T17:40:46.100095+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:23:06.946107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.590803+0000","last_change":"2023-10-10T17:40:41.165988+0000","last_active":"2023-10-10T17:40:59.590803+0000","last_peered":"2023-10-10T17:40:59.590803+0000","last_clean":"2023-10-10T17:40:59.590803+0000","last_became_active":"2023-10-10T17:40:41.165407+0000","last_became_peered":"2023-10-10T17:40:41.165407+0000","last_unstale":"2023-10-10T17:40:59.590803+0000","last_undegraded":"2023-10-10T17:40:59.590803+0000","last_fullsized":"2023-10-10T17:40:59.590803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:54.001805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275819+0000","last_change":"2023-10-10T17:40:41.609834+0000","last_active":"2023-10-10T17:40:46.275819+0000","last_peered":"2023-10-10T17:40:46.275819+0000","last_clean":"2023-10-10T17:40:46.275819+0000","last_became_active":"2023-10-10T17:40:41.609509+0000","last_became_peered":"2023-10-10T17:40:41.609509+0000","last_unstale":"2023-10-10T17:40:46.275819+0000","last_undegraded":"2023-10-10T17:40:46.275819+0000","last_fullsized":"2023-10-10T17:40:46.275819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:42:46.393277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275681+0000","last_change":"2023-10-10T17:40:45.709003+0000","last_active":"2023-10-10T17:40:46.275681+0000","last_peered":"2023-10-10T17:40:46.275681+0000","last_clean":"2023-10-10T17:40:46.275681+0000","last_became_active":"2023-10-10T17:40:45.708610+0000","last_became_peered":"2023-10-10T17:40:45.708610+0000","last_unstale":"2023-10-10T17:40:46.275681+0000","last_undegraded":"2023-10-10T17:40:46.275681+0000","last_fullsized":"2023-10-10T17:40:46.275681+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:03:47.748938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100416+0000","last_change":"2023-10-10T17:40:41.168104+0000","last_active":"2023-10-10T17:40:46.100416+0000","last_peered":"2023-10-10T17:40:46.100416+0000","last_clean":"2023-10-10T17:40:46.100416+0000","last_became_active":"2023-10-10T17:40:41.167802+0000","last_became_peered":"2023-10-10T17:40:41.167802+0000","last_unstale":"2023-10-10T17:40:46.100416+0000","last_undegraded":"2023-10-10T17:40:46.100416+0000","last_fullsized":"2023-10-10T17:40:46.100416+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:52.597542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276168+0000","last_change":"2023-10-10T17:40:41.606553+0000","last_active":"2023-10-10T17:40:46.276168+0000","last_peered":"2023-10-10T17:40:46.276168+0000","last_clean":"2023-10-10T17:40:46.276168+0000","last_became_active":"2023-10-10T17:40:41.606087+0000","last_became_peered":"2023-10-10T17:40:41.606087+0000","last_unstale":"2023-10-10T17:40:46.276168+0000","last_undegraded":"2023-10-10T17:40:46.276168+0000","last_fullsized":"2023-10-10T17:40:46.276168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:13:31.697453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275876+0000","last_change":"2023-10-10T17:40:45.712028+0000","last_active":"2023-10-10T17:40:46.275876+0000","last_peered":"2023-10-10T17:40:46.275876+0000","last_clean":"2023-10-10T17:40:46.275876+0000","last_became_active":"2023-10-10T17:40:45.711534+0000","last_became_peered":"2023-10-10T17:40:45.711534+0000","last_unstale":"2023-10-10T17:40:46.275876+0000","last_undegraded":"2023-10-10T17:40:46.275876+0000","last_fullsized":"2023-10-10T17:40:46.275876+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:45:49.387605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.592441+0000","last_change":"2023-10-10T17:40:41.164456+0000","last_active":"2023-10-10T17:40:59.592441+0000","last_peered":"2023-10-10T17:40:59.592441+0000","last_clean":"2023-10-10T17:40:59.592441+0000","last_became_active":"2023-10-10T17:40:41.164018+0000","last_became_peered":"2023-10-10T17:40:41.164018+0000","last_unstale":"2023-10-10T17:40:59.592441+0000","last_undegraded":"2023-10-10T17:40:59.592441+0000","last_fullsized":"2023-10-10T17:40:59.592441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:26:56.502719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457221+0000","last_change":"2023-10-10T17:40:41.599724+0000","last_active":"2023-10-10T17:40:46.457221+0000","last_peered":"2023-10-10T17:40:46.457221+0000","last_clean":"2023-10-10T17:40:46.457221+0000","last_became_active":"2023-10-10T17:40:41.599356+0000","last_became_peered":"2023-10-10T17:40:41.599356+0000","last_unstale":"2023-10-10T17:40:46.457221+0000","last_undegraded":"2023-10-10T17:40:46.457221+0000","last_fullsized":"2023-10-10T17:40:46.457221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:16.776490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275891+0000","last_change":"2023-10-10T17:40:45.704911+0000","last_active":"2023-10-10T17:40:46.275891+0000","last_peered":"2023-10-10T17:40:46.275891+0000","last_clean":"2023-10-10T17:40:46.275891+0000","last_became_active":"2023-10-10T17:40:45.704299+0000","last_became_peered":"2023-10-10T17:40:45.704299+0000","last_unstale":"2023-10-10T17:40:46.275891+0000","last_undegraded":"2023-10-10T17:40:46.275891+0000","last_fullsized":"2023-10-10T17:40:46.275891+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:18:41.829018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100891+0000","last_change":"2023-10-10T17:40:41.166091+0000","last_active":"2023-10-10T17:40:46.100891+0000","last_peered":"2023-10-10T17:40:46.100891+0000","last_clean":"2023-10-10T17:40:46.100891+0000","last_became_active":"2023-10-10T17:40:41.165610+0000","last_became_peered":"2023-10-10T17:40:41.165610+0000","last_unstale":"2023-10-10T17:40:46.100891+0000","last_undegraded":"2023-10-10T17:40:46.100891+0000","last_fullsized":"2023-10-10T17:40:46.100891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:41:34.118170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457584+0000","last_change":"2023-10-10T17:40:41.600837+0000","last_active":"2023-10-10T17:40:46.457584+0000","last_peered":"2023-10-10T17:40:46.457584+0000","last_clean":"2023-10-10T17:40:46.457584+0000","last_became_active":"2023-10-10T17:40:41.600090+0000","last_became_peered":"2023-10-10T17:40:41.600090+0000","last_unstale":"2023-10-10T17:40:46.457584+0000","last_undegraded":"2023-10-10T17:40:46.457584+0000","last_fullsized":"2023-10-10T17:40:46.457584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:02:54.300749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275138+0000","last_change":"2023-10-10T17:40:45.705712+0000","last_active":"2023-10-10T17:40:46.275138+0000","last_peered":"2023-10-10T17:40:46.275138+0000","last_clean":"2023-10-10T17:40:46.275138+0000","last_became_active":"2023-10-10T17:40:45.705301+0000","last_became_peered":"2023-10-10T17:40:45.705301+0000","last_unstale":"2023-10-10T17:40:46.275138+0000","last_undegraded":"2023-10-10T17:40:46.275138+0000","last_fullsized":"2023-10-10T17:40:46.275138+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:59:20.860596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276742+0000","last_change":"2023-10-10T17:40:41.169820+0000","last_active":"2023-10-10T17:40:46.276742+0000","last_peered":"2023-10-10T17:40:46.276742+0000","last_clean":"2023-10-10T17:40:46.276742+0000","last_became_active":"2023-10-10T17:40:41.168767+0000","last_became_peered":"2023-10-10T17:40:41.168767+0000","last_unstale":"2023-10-10T17:40:46.276742+0000","last_undegraded":"2023-10-10T17:40:46.276742+0000","last_fullsized":"2023-10-10T17:40:46.276742+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:55:51.881314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457344+0000","last_change":"2023-10-10T17:40:41.597482+0000","last_active":"2023-10-10T17:40:46.457344+0000","last_peered":"2023-10-10T17:40:46.457344+0000","last_clean":"2023-10-10T17:40:46.457344+0000","last_became_active":"2023-10-10T17:40:41.597166+0000","last_became_peered":"2023-10-10T17:40:41.597166+0000","last_unstale":"2023-10-10T17:40:46.457344+0000","last_undegraded":"2023-10-10T17:40:46.457344+0000","last_fullsized":"2023-10-10T17:40:46.457344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:12:25.007888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276549+0000","last_change":"2023-10-10T17:40:45.707175+0000","last_active":"2023-10-10T17:40:46.276549+0000","last_peered":"2023-10-10T17:40:46.276549+0000","last_clean":"2023-10-10T17:40:46.276549+0000","last_became_active":"2023-10-10T17:40:45.706448+0000","last_became_peered":"2023-10-10T17:40:45.706448+0000","last_unstale":"2023-10-10T17:40:46.276549+0000","last_undegraded":"2023-10-10T17:40:46.276549+0000","last_fullsized":"2023-10-10T17:40:46.276549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:59:16.405321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213980+0000","last_change":"2023-10-10T17:40:41.169806+0000","last_active":"2023-10-10T17:40:46.213980+0000","last_peered":"2023-10-10T17:40:46.213980+0000","last_clean":"2023-10-10T17:40:46.213980+0000","last_became_active":"2023-10-10T17:40:41.169496+0000","last_became_peered":"2023-10-10T17:40:41.169496+0000","last_unstale":"2023-10-10T17:40:46.213980+0000","last_undegraded":"2023-10-10T17:40:46.213980+0000","last_fullsized":"2023-10-10T17:40:46.213980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:42:39.410394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.457671+0000","last_change":"2023-10-10T17:40:41.598366+0000","last_active":"2023-10-10T17:40:46.457671+0000","last_peered":"2023-10-10T17:40:46.457671+0000","last_clean":"2023-10-10T17:40:46.457671+0000","last_became_active":"2023-10-10T17:40:41.598036+0000","last_became_peered":"2023-10-10T17:40:41.598036+0000","last_unstale":"2023-10-10T17:40:46.457671+0000","last_undegraded":"2023-10-10T17:40:46.457671+0000","last_fullsized":"2023-10-10T17:40:46.457671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:32.266809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275953+0000","last_change":"2023-10-10T17:40:41.609923+0000","last_active":"2023-10-10T17:40:46.275953+0000","last_peered":"2023-10-10T17:40:46.275953+0000","last_clean":"2023-10-10T17:40:46.275953+0000","last_became_active":"2023-10-10T17:40:41.609575+0000","last_became_peered":"2023-10-10T17:40:41.609575+0000","last_unstale":"2023-10-10T17:40:46.275953+0000","last_undegraded":"2023-10-10T17:40:46.275953+0000","last_fullsized":"2023-10-10T17:40:46.275953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:51:30.575597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100629+0000","last_change":"2023-10-10T17:40:41.609012+0000","last_active":"2023-10-10T17:40:46.100629+0000","last_peered":"2023-10-10T17:40:46.100629+0000","last_clean":"2023-10-10T17:40:46.100629+0000","last_became_active":"2023-10-10T17:40:41.608738+0000","last_became_peered":"2023-10-10T17:40:41.608738+0000","last_unstale":"2023-10-10T17:40:46.100629+0000","last_undegraded":"2023-10-10T17:40:46.100629+0000","last_fullsized":"2023-10-10T17:40:46.100629+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:20:01.313649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.100540+0000","last_change":"2023-10-10T17:40:41.608801+0000","last_active":"2023-10-10T17:40:46.100540+0000","last_peered":"2023-10-10T17:40:46.100540+0000","last_clean":"2023-10-10T17:40:46.100540+0000","last_became_active":"2023-10-10T17:40:41.608510+0000","last_became_peered":"2023-10-10T17:40:41.608510+0000","last_unstale":"2023-10-10T17:40:46.100540+0000","last_undegraded":"2023-10-10T17:40:46.100540+0000","last_fullsized":"2023-10-10T17:40:46.100540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:25:21.531109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275330+0000","last_change":"2023-10-10T17:40:41.603516+0000","last_active":"2023-10-10T17:40:46.275330+0000","last_peered":"2023-10-10T17:40:46.275330+0000","last_clean":"2023-10-10T17:40:46.275330+0000","last_became_active":"2023-10-10T17:40:41.602948+0000","last_became_peered":"2023-10-10T17:40:41.602948+0000","last_unstale":"2023-10-10T17:40:46.275330+0000","last_undegraded":"2023-10-10T17:40:46.275330+0000","last_fullsized":"2023-10-10T17:40:46.275330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:59:40.345468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213549+0000","last_change":"2023-10-10T17:40:41.597594+0000","last_active":"2023-10-10T17:40:46.213549+0000","last_peered":"2023-10-10T17:40:46.213549+0000","last_clean":"2023-10-10T17:40:46.213549+0000","last_became_active":"2023-10-10T17:40:41.597286+0000","last_became_peered":"2023-10-10T17:40:41.597286+0000","last_unstale":"2023-10-10T17:40:46.213549+0000","last_undegraded":"2023-10-10T17:40:46.213549+0000","last_fullsized":"2023-10-10T17:40:46.213549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:04:44.083728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276363+0000","last_change":"2023-10-10T17:40:41.597595+0000","last_active":"2023-10-10T17:40:46.276363+0000","last_peered":"2023-10-10T17:40:46.276363+0000","last_clean":"2023-10-10T17:40:46.276363+0000","last_became_active":"2023-10-10T17:40:41.597301+0000","last_became_peered":"2023-10-10T17:40:41.597301+0000","last_unstale":"2023-10-10T17:40:46.276363+0000","last_undegraded":"2023-10-10T17:40:46.276363+0000","last_fullsized":"2023-10-10T17:40:46.276363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:14.213403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.101071+0000","last_change":"2023-10-10T17:40:41.611841+0000","last_active":"2023-10-10T17:40:46.101071+0000","last_peered":"2023-10-10T17:40:46.101071+0000","last_clean":"2023-10-10T17:40:46.101071+0000","last_became_active":"2023-10-10T17:40:41.611438+0000","last_became_peered":"2023-10-10T17:40:41.611438+0000","last_unstale":"2023-10-10T17:40:46.101071+0000","last_undegraded":"2023-10-10T17:40:46.101071+0000","last_fullsized":"2023-10-10T17:40:46.101071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:56:25.508210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276423+0000","last_change":"2023-10-10T17:40:41.607707+0000","last_active":"2023-10-10T17:40:46.276423+0000","last_peered":"2023-10-10T17:40:46.276423+0000","last_clean":"2023-10-10T17:40:46.276423+0000","last_became_active":"2023-10-10T17:40:41.607412+0000","last_became_peered":"2023-10-10T17:40:41.607412+0000","last_unstale":"2023-10-10T17:40:46.276423+0000","last_undegraded":"2023-10-10T17:40:46.276423+0000","last_fullsized":"2023-10-10T17:40:46.276423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:46:36.504686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277088+0000","last_change":"2023-10-10T17:40:41.593674+0000","last_active":"2023-10-10T17:40:46.277088+0000","last_peered":"2023-10-10T17:40:46.277088+0000","last_clean":"2023-10-10T17:40:46.277088+0000","last_became_active":"2023-10-10T17:40:41.593339+0000","last_became_peered":"2023-10-10T17:40:41.593339+0000","last_unstale":"2023-10-10T17:40:46.277088+0000","last_undegraded":"2023-10-10T17:40:46.277088+0000","last_fullsized":"2023-10-10T17:40:46.277088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:41:56.490267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276071+0000","last_change":"2023-10-10T17:40:41.610075+0000","last_active":"2023-10-10T17:40:46.276071+0000","last_peered":"2023-10-10T17:40:46.276071+0000","last_clean":"2023-10-10T17:40:46.276071+0000","last_became_active":"2023-10-10T17:40:41.609669+0000","last_became_peered":"2023-10-10T17:40:41.609669+0000","last_unstale":"2023-10-10T17:40:46.276071+0000","last_undegraded":"2023-10-10T17:40:46.276071+0000","last_fullsized":"2023-10-10T17:40:46.276071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:28:56.002362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276200+0000","last_change":"2023-10-10T17:40:41.596780+0000","last_active":"2023-10-10T17:40:46.276200+0000","last_peered":"2023-10-10T17:40:46.276200+0000","last_clean":"2023-10-10T17:40:46.276200+0000","last_became_active":"2023-10-10T17:40:41.596477+0000","last_became_peered":"2023-10-10T17:40:41.596477+0000","last_unstale":"2023-10-10T17:40:46.276200+0000","last_undegraded":"2023-10-10T17:40:46.276200+0000","last_fullsized":"2023-10-10T17:40:46.276200+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:02:51.280429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.213297+0000","last_change":"2023-10-10T17:40:41.606226+0000","last_active":"2023-10-10T17:40:46.213297+0000","last_peered":"2023-10-10T17:40:46.213297+0000","last_clean":"2023-10-10T17:40:46.213297+0000","last_became_active":"2023-10-10T17:40:41.605843+0000","last_became_peered":"2023-10-10T17:40:41.605843+0000","last_unstale":"2023-10-10T17:40:46.213297+0000","last_undegraded":"2023-10-10T17:40:46.213297+0000","last_fullsized":"2023-10-10T17:40:46.213297+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:37:26.232295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276695+0000","last_change":"2023-10-10T17:40:41.596122+0000","last_active":"2023-10-10T17:40:46.276695+0000","last_peered":"2023-10-10T17:40:46.276695+0000","last_clean":"2023-10-10T17:40:46.276695+0000","last_became_active":"2023-10-10T17:40:41.595818+0000","last_became_peered":"2023-10-10T17:40:41.595818+0000","last_unstale":"2023-10-10T17:40:46.276695+0000","last_undegraded":"2023-10-10T17:40:46.276695+0000","last_fullsized":"2023-10-10T17:40:46.276695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:46:33.542672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250871,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5972,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":73,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":20,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":15918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383695,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6068,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739084,"statfs":{"total":95995035648,"available":95988822016,"internally_reserved":0,"allocated":962560,"data_stored":473302,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]}]},{"osd":10,"up_from":73,"seq":313532612625,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6324,"kb_used_data":1004,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738828,"statfs":{"total":95995035648,"available":95988559872,"internally_reserved":0,"allocated":1028096,"data_stored":1066745,"data_compressed":47944,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.751},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.63200000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.44900000000000001}]},{"osd":4,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.68899999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.61199999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.70899999999999996}]},{"osd":8,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.71299999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.53100000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]}]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6260,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738892,"statfs":{"total":95995035648,"available":95988625408,"internally_reserved":0,"allocated":962560,"data_stored":473302,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.318}]},{"osd":1,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.43099999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.51900000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.41799999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.66900000000000004}]},{"osd":6,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.63300000000000001}]},{"osd":8,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.60899999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:40:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.57399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6404,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738748,"statfs":{"total":95995035648,"available":95988477952,"internally_reserved":0,"allocated":978944,"data_stored":473458,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.84199999999999997}]},{"osd":1,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.47199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76000000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.438,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92900000000000005,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.877}]},{"osd":2,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39200000000000002},"min":{"1min":0.32400000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61499999999999999,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.45200000000000001,"15min":0.44400000000000001},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63700000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.60499999999999998}]},{"osd":3,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72899999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.71899999999999997}]},{"osd":4,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.47799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82899999999999996,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60199999999999998,"15min":0.59399999999999997},"min":{"1min":0.52100000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80900000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.753}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.52000000000000002},"min":{"1min":0.36099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.5,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81499999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96999999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.91700000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69199999999999995,"15min":0.67900000000000005},"min":{"1min":0.57999999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0029999999999999,"5min":1.022,"15min":1.022},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.59999999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86799999999999999,"5min":1.101,"15min":1.101},"last":0.85699999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.96999999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:40:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.90500000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]}]},{"osd":7,"up_from":52,"seq":223338299423,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6496,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738656,"statfs":{"total":95995035648,"available":95988383744,"internally_reserved":0,"allocated":1007616,"data_stored":1063397,"data_compressed":47944,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72099999999999997,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.374}]},{"osd":1,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.39700000000000002,"15min":0.38900000000000001},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47199999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.76300000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56799999999999995}]},{"osd":4,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.34599999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.504}]},{"osd":5,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.46700000000000003,"5min":0.438,"15min":0.438},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.44800000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.871,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.71299999999999997}]},{"osd":6,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66000000000000003,"15min":0.65600000000000003},"min":{"1min":0.54100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60899999999999999}]},{"osd":8,"last update":"Tue Oct 10 17:40:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.79000000000000004}]},{"osd":9,"last update":"Tue Oct 10 17:40:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.70699999999999996}]},{"osd":10,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.83899999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462948,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":958464,"data_stored":472720,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.35799999999999998,"15min":0.35199999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.437},{"interface":"front","average":{"1min":0.371,"5min":0.375,"15min":0.375},"min":{"1min":0.27000000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.29699999999999999}]},{"osd":1,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.53200000000000003,"15min":0.52400000000000002},"min":{"1min":0.34699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.52000000000000002}]},{"osd":2,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55000000000000004,"15min":0.54600000000000004},"min":{"1min":0.38700000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.54100000000000004,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.501}]},{"osd":3,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.63800000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.36899999999999999}]},{"osd":4,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54900000000000004,"15min":0.55000000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.56299999999999994,"15min":0.55700000000000005},"min":{"1min":0.39900000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.72699999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.58099999999999996,"15min":0.57299999999999995},"min":{"1min":0.46400000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.40799999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.39800000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:40:36 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.44,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.59599999999999997,"15min":0.58099999999999996},"min":{"1min":0.53000000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64000000000000001}]},{"osd":8,"last update":"Tue Oct 10 17:39:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.46500000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.40899999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.751}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7292,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737860,"statfs":{"total":95995035648,"available":95987568640,"internally_reserved":0,"allocated":970752,"data_stored":476514,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.51200000000000001,"15min":0.48499999999999999},"min":{"1min":0.38500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61699999999999999,"15min":0.68899999999999995},"min":{"1min":0.41199999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.88300000000000001,"5min":5.859,"15min":5.859},"last":0.61799999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:40:53 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.746},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.433,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79000000000000004,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.86899999999999999}]},{"osd":3,"last update":"Tue Oct 10 17:40:11 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.56999999999999995,"15min":0.55800000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57499999999999996,"15min":0.55700000000000005},"min":{"1min":0.44,"5min":0.379,"15min":0.379},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.79300000000000004}]},{"osd":4,"last update":"Tue Oct 10 17:40:34 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.41699999999999998,"15min":0.39100000000000001},"min":{"1min":0.36499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71899999999999997,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.42999999999999999,"15min":0.40200000000000002},"min":{"1min":0.34300000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.76400000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88700000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.81599999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:40:16 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58399999999999996,"15min":0.58599999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72899999999999998,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.91800000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85299999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83299999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":1}]},{"osd":8,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":1.0169999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:40:19 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.90200000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:40:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.98499999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.05}]}]},{"osd":0,"up_from":8,"seq":34359738429,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7284,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737868,"statfs":{"total":95995035648,"available":95987576832,"internally_reserved":0,"allocated":962560,"data_stored":472810,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 17:40:27 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.39300000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.433},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.54000000000000004,"15min":0.53100000000000003},"min":{"1min":0.40899999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.59899999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.56599999999999995,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75900000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:39:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27700000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63800000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41299999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:40:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.53500000000000003,"15min":0.50900000000000001},"min":{"1min":0.35699999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.498},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53500000000000003,"15min":0.51500000000000001},"min":{"1min":0.39900000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.50800000000000001},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.70499999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.47499999999999998,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.48899999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.626,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.624},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.53100000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:39:57 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.75}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.61099999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.68600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7356,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737796,"statfs":{"total":95995035648,"available":95987503104,"internally_reserved":0,"allocated":970752,"data_stored":475929,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.31,"5min":0.252,"15min":0.252},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48799999999999999,"15min":0.48099999999999998},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.61699999999999999}]},{"osd":1,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.56799999999999995,"15min":0.52000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.484,"5min":5.484,"15min":5.484},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68999999999999995}]},{"osd":3,"last update":"Tue Oct 10 17:40:12 2023","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53700000000000003,"15min":0.51400000000000001},"min":{"1min":0.33600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.753},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53300000000000003},"min":{"1min":0.41199999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.71499999999999997}]},{"osd":4,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51900000000000002,"15min":0.502},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71999999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52100000000000002,"15min":0.501},"min":{"1min":0.42199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.77700000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.41999999999999998,"15min":0.40100000000000002},"min":{"1min":0.33900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.46500000000000002,"15min":0.44900000000000001},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.79300000000000004}]},{"osd":6,"last update":"Tue Oct 10 17:40:12 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.623,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.83799999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77900000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.93400000000000005}]},{"osd":8,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.86899999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:40:18 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.98899999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.92400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96499999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149744,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":970752,"data_stored":472673,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.35099999999999998,"15min":0.34699999999999998},"min":{"1min":0.22500000000000001,"5min":0.217,"15min":0.217},"max":{"1min":0.49399999999999999,"5min":0.752,"15min":0.752},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.379,"5min":0.36399999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55000000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.35399999999999998}]},{"osd":1,"last update":"Tue Oct 10 17:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.55200000000000005,"15min":0.53500000000000003},"min":{"1min":0.317,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.501},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.36299999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.58799999999999997}]},{"osd":2,"last update":"Tue Oct 10 17:40:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.53300000000000003},"min":{"1min":0.442,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.55200000000000005,"15min":0.53500000000000003},"min":{"1min":0.48299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.876,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.61199999999999999}]},{"osd":4,"last update":"Tue Oct 10 17:40:31 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.40600000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.91400000000000003,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.41499999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.95999999999999996,"5min":1.214,"15min":1.214},"last":0.46000000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.48099999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72099999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.56200000000000006}]},{"osd":6,"last update":"Tue Oct 10 17:40:16 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.308,"15min":0.308},"max":{"1min":0.89500000000000002,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48299999999999998,"15min":0.48599999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67500000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.372}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73899999999999999,"5min":0.88,"15min":0.88},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86699999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.52400000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:39:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53700000000000003}]},{"osd":9,"last update":"Tue Oct 10 17:40:18 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.41499999999999998}]},{"osd":10,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60099999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69099999999999995}]}]},{"osd":4,"up_from":34,"seq":146028888109,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":962560,"data_stored":472742,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.432,"5min":0.372,"15min":0.372},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.372,"15min":0.35899999999999999},"min":{"1min":0.30499999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.57299999999999995,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.38300000000000001,"15min":0.36399999999999999},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.372}]},{"osd":2,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.58299999999999996,"15min":0.56999999999999995},"min":{"1min":0.38800000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82299999999999995,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.59299999999999997,"15min":0.56100000000000005},"min":{"1min":0.495,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.53500000000000003,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78100000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.54100000000000004,"15min":0.53800000000000003},"min":{"1min":0.40300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.57799999999999996},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86799999999999999,"5min":1.589,"15min":1.589},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.74099999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:40:16 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.41699999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85299999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.433},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.64300000000000002,"15min":0.63500000000000001},"min":{"1min":0.48599999999999999,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.497}]},{"osd":7,"last update":"Tue Oct 10 17:40:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.53100000000000003,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.629},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.46899999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.74399999999999999,"15min":0.71199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":6.1639999999999997,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.73199999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:40:20 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53200000000000003}]},{"osd":10,"last update":"Tue Oct 10 17:40:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.56100000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724585,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6648,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738504,"statfs":{"total":95995035648,"available":95988228096,"internally_reserved":0,"allocated":1032192,"data_stored":1064243,"data_compressed":47944,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50700000000000001,"15min":0.499},"min":{"1min":0.41099999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52300000000000002,"15min":0.51800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.41099999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53400000000000003}]},{"osd":2,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.29299999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.54000000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40400000000000003,"15min":0.40600000000000003},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.58699999999999997,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.52500000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.55600000000000005,"15min":0.55200000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54700000000000004}]},{"osd":4,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.44600000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.60599999999999998,"15min":0.60199999999999998},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.72999999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:40:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.622,"15min":0.624},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80400000000000005,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.623,"15min":0.624},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84199999999999997,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59099999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.745}]},{"osd":8,"last update":"Tue Oct 10 17:39:59 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.64700000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:40:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63500000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:40:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.77900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T17:41:03.082 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-10T17:41:03.083 INFO:tasks.cephadm:Setting up client nodes... 2023-10-10T17:41:03.084 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-10-10T17:41:03.743 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:03.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:03 smithi078 ceph-mon[99494]: pgmap v272: 129 pgs: 129 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-10-10T17:41:03.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:03 smithi190 ceph-mon[101220]: pgmap v272: 129 pgs: 129 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-10-10T17:41:03.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:03 smithi157 ceph-mon[100896]: pgmap v272: 129 pgs: 129 active+clean; 582 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 1.5 KiB/s wr, 2 op/s 2023-10-10T17:41:04.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:04 smithi078 ceph-mon[99494]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:04.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:04 smithi078 ceph-mon[99494]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:04.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:04 smithi190 ceph-mon[101220]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:04.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:04 smithi190 ceph-mon[101220]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:04.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 -- 172.21.15.78:0/2234705322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480101f10 msgr2=0x7f14801087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:04.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 --2- 172.21.15.78:0/2234705322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480101f10 0x7f14801087d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f147800b0a0 tx=0x7f147800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:04.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 -- 172.21.15.78:0/2234705322 shutdown_connections 2023-10-10T17:41:04.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 --2- 172.21.15.78:0/2234705322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480101f10 0x7f14801087d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:04.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 --2- 172.21.15.78:0/2234705322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1480100380 0x7f1480100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:04.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 --2- 172.21.15.78:0/2234705322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480101580 0x7f14801019d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:04.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 -- 172.21.15.78:0/2234705322 >> 172.21.15.78:0/2234705322 conn(0x7f14800fb9d0 msgr2=0x7f14800fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:04.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 -- 172.21.15.78:0/2234705322 shutdown_connections 2023-10-10T17:41:04.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.937+0000 7f1487516700 1 -- 172.21.15.78:0/2234705322 wait complete. 2023-10-10T17:41:04.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 Processor -- start 2023-10-10T17:41:04.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 -- start start 2023-10-10T17:41:04.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480100380 0x7f14801960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:04.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1480101580 0x7f1480196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:04.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480101f10 0x7f148019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:04.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148019dc50 con 0x7f1480100380 2023-10-10T17:41:04.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148019dd90 con 0x7f1480101580 2023-10-10T17:41:04.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.938+0000 7f1487516700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148019df20 con 0x7f1480101f10 2023-10-10T17:41:04.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480100380 0x7f14801960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:04.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480100380 0x7f14801960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42660/0 (socket says 172.21.15.78:42660) 2023-10-10T17:41:04.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 -- 172.21.15.78:0/2807047920 learned_addr learned my addr 172.21.15.78:0/2807047920 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:04.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f1485ab3700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480101f10 0x7f148019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:04.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f1484ab1700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1480101580 0x7f1480196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:04.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 -- 172.21.15.78:0/2807047920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480101f10 msgr2=0x7f148019d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:04.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480101f10 0x7f148019d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:04.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 -- 172.21.15.78:0/2807047920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1480101580 msgr2=0x7f1480196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:04.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1480101580 0x7f1480196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:04.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 -- 172.21.15.78:0/2807047920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1478009d00 con 0x7f1480100380 2023-10-10T17:41:04.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f1484ab1700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1480101580 0x7f1480196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:04.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14852b2700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480100380 0x7f14801960c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f147000c9c0 tx=0x7f147000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:04.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14767fc700 1 -- 172.21.15.78:0/2807047920 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470007ac0 con 0x7f1480100380 2023-10-10T17:41:04.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14767fc700 1 -- 172.21.15.78:0/2807047920 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1470007c20 con 0x7f1480100380 2023-10-10T17:41:04.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148010bd60 con 0x7f1480100380 2023-10-10T17:41:04.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f14767fc700 1 -- 172.21.15.78:0/2807047920 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470018660 con 0x7f1480100380 2023-10-10T17:41:04.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.939+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f148010bbd0 con 0x7f1480100380 2023-10-10T17:41:04.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.940+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148019e6e0 con 0x7f1480100380 2023-10-10T17:41:04.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.941+0000 7f14767fc700 1 -- 172.21.15.78:0/2807047920 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1470004020 con 0x7f1480100380 2023-10-10T17:41:04.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.942+0000 7f14767fc700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f146c0a1150 0x7f146c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:04.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.942+0000 7f1484ab1700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f146c0a1150 0x7f146c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:04.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.942+0000 7f14767fc700 1 -- 172.21.15.78:0/2807047920 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f14700ccc80 con 0x7f1480100380 2023-10-10T17:41:04.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.942+0000 7f1484ab1700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f146c0a1150 0x7f146c0a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f147c007900 tx=0x7f147c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:04.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:04.946+0000 7f14767fc700 1 -- 172.21.15.78:0/2807047920 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1470096270 con 0x7f1480100380 2023-10-10T17:41:04.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:04 smithi157 ceph-mon[100896]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:04.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:04 smithi157 ceph-mon[100896]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:05.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.211+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f14801071f0 con 0x7f1480100380 2023-10-10T17:41:05.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.215+0000 7f14767fc700 1 -- 172.21.15.78:0/2807047920 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1470099890 con 0x7f1480100380 2023-10-10T17:41:05.216 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2023-10-10T17:41:05.216 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQCxjCVl0lukDBAA1kr4L1D8n2nyBsQm6OpaxQ== 2023-10-10T17:41:05.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f146c0a1150 msgr2=0x7f146c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:05.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f146c0a1150 0x7f146c0a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f147c007900 tx=0x7f147c008040 comp rx=0 tx=0).stop 2023-10-10T17:41:05.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480100380 msgr2=0x7f14801960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:05.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480100380 0x7f14801960c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f147000c9c0 tx=0x7f147000cd80 comp rx=0 tx=0).stop 2023-10-10T17:41:05.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 shutdown_connections 2023-10-10T17:41:05.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f146c0a1150 0x7f146c0a3600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:05.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1480101f10 0x7f148019d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:05.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1480100380 0x7f14801960c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:05.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 --2- 172.21.15.78:0/2807047920 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1480101580 0x7f1480196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:05.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 >> 172.21.15.78:0/2807047920 conn(0x7f14800fb9d0 msgr2=0x7f14800fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:05.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 shutdown_connections 2023-10-10T17:41:05.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:05.217+0000 7f1487516700 1 -- 172.21.15.78:0/2807047920 wait complete. 2023-10-10T17:41:05.593 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:41:05.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-10-10T17:41:05.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-10T17:41:05.636 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-10T17:41:05.637 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-10-10T17:41:05.638 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph mgr dump --format=json 2023-10-10T17:41:05.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:05 smithi078 ceph-mon[99494]: pgmap v273: 129 pgs: 129 active+clean; 583 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2023-10-10T17:41:05.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:05 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2807047920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-10T17:41:05.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:05 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2807047920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-10T17:41:05.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:05 smithi190 ceph-mon[101220]: pgmap v273: 129 pgs: 129 active+clean; 583 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2023-10-10T17:41:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:05 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2807047920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-10T17:41:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:05 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2807047920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-10T17:41:05.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:05 smithi157 ceph-mon[100896]: pgmap v273: 129 pgs: 129 active+clean; 583 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2023-10-10T17:41:05.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:05 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2807047920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-10T17:41:05.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:05 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2807047920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-10T17:41:06.312 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:06.859 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:06 smithi078 ceph-mon[99494]: pgmap v274: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-10-10T17:41:06.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:06 smithi190 ceph-mon[101220]: pgmap v274: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-10-10T17:41:06.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:06 smithi157 ceph-mon[100896]: pgmap v274: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-10-10T17:41:07.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1365624566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ff980 msgr2=0x7f5c940ffdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:07.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1365624566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ff980 0x7f5c940ffdf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5c800099c0 tx=0x7f5c80009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1365624566 shutdown_connections 2023-10-10T17:41:07.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1365624566 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c94100330 0x7f5c9410c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1365624566 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c941041c0 0x7f5c94104590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1365624566 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c940ff980 0x7f5c940ffdf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1365624566 >> 172.21.15.78:0/1365624566 conn(0x7f5c940758a0 msgr2=0x7f5c94075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:07.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1365624566 shutdown_connections 2023-10-10T17:41:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.536+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1365624566 wait complete. 2023-10-10T17:41:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.537+0000 7f5c9aaea700 1 Processor -- start 2023-10-10T17:41:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.537+0000 7f5c9aaea700 1 -- start start 2023-10-10T17:41:07.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.537+0000 7f5c9aaea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c940ff980 0x7f5c94195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.537+0000 7f5c9aaea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c94100330 0x7f5c941957a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.537+0000 7f5c9aaea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c941041c0 0x7f5c94199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.537+0000 7f5c9aaea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9419a220 con 0x7f5c941041c0 2023-10-10T17:41:07.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.537+0000 7f5c9aaea700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9419a390 con 0x7f5c94100330 2023-10-10T17:41:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c9aaea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9419a550 con 0x7f5c940ff980 2023-10-10T17:41:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c941041c0 0x7f5c94199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:07.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c941041c0 0x7f5c94199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42676/0 (socket says 172.21.15.78:42676) 2023-10-10T17:41:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 -- 172.21.15.78:0/1982270113 learned_addr learned my addr 172.21.15.78:0/1982270113 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c98886700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c940ff980 0x7f5c94195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c93fff700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c94100330 0x7f5c941957a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 -- 172.21.15.78:0/1982270113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c940ff980 msgr2=0x7f5c94195260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:07.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c940ff980 0x7f5c94195260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 -- 172.21.15.78:0/1982270113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c94100330 msgr2=0x7f5c941957a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:07.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c94100330 0x7f5c941957a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 -- 172.21.15.78:0/1982270113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c800095f0 con 0x7f5c941041c0 2023-10-10T17:41:07.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c93fff700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c94100330 0x7f5c941957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c99087700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c941041c0 0x7f5c94199c10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c007ee0 tx=0x7f5c8c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c91ffb700 1 -- 172.21.15.78:0/1982270113 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c017070 con 0x7f5c941041c0 2023-10-10T17:41:07.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c91ffb700 1 -- 172.21.15.78:0/1982270113 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c8c00f040 con 0x7f5c941041c0 2023-10-10T17:41:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c9410f2a0 con 0x7f5c941041c0 2023-10-10T17:41:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.538+0000 7f5c91ffb700 1 -- 172.21.15.78:0/1982270113 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c005740 con 0x7f5c941041c0 2023-10-10T17:41:07.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.539+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c9410f110 con 0x7f5c941041c0 2023-10-10T17:41:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.539+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c9410f430 con 0x7f5c941041c0 2023-10-10T17:41:07.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.540+0000 7f5c91ffb700 1 -- 172.21.15.78:0/1982270113 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5c8c00dc60 con 0x7f5c941041c0 2023-10-10T17:41:07.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.541+0000 7f5c91ffb700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5c840a1150 0x7f5c840a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:07.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.541+0000 7f5c98886700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5c840a1150 0x7f5c840a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:07.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.541+0000 7f5c91ffb700 1 -- 172.21.15.78:0/1982270113 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5c8c0cc340 con 0x7f5c941041c0 2023-10-10T17:41:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.541+0000 7f5c98886700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5c840a1150 0x7f5c840a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009a10 tx=0x7f5c88009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:07.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.545+0000 7f5c91ffb700 1 -- 172.21.15.78:0/1982270113 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5c8c0958a0 con 0x7f5c941041c0 2023-10-10T17:41:07.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.785+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5c94066ea0 con 0x7f5c941041c0 2023-10-10T17:41:07.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.789+0000 7f5c91ffb700 1 -- 172.21.15.78:0/1982270113 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+260033 (secure 0 0 0) 0x7f5c8c095020 con 0x7f5c941041c0 2023-10-10T17:41:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:07.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5c840a1150 msgr2=0x7f5c840a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:07.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5c840a1150 0x7f5c840a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5c88009a10 tx=0x7f5c88009190 comp rx=0 tx=0).stop 2023-10-10T17:41:07.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c941041c0 msgr2=0x7f5c94199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:07.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c941041c0 0x7f5c94199c10 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c007ee0 tx=0x7f5c8c00d3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 shutdown_connections 2023-10-10T17:41:07.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5c840a1150 0x7f5c840a3600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c940ff980 0x7f5c94195260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c941041c0 0x7f5c94199c10 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 --2- 172.21.15.78:0/1982270113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c94100330 0x7f5c941957a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:07.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 >> 172.21.15.78:0/1982270113 conn(0x7f5c940758a0 msgr2=0x7f5c940fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:07.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 shutdown_connections 2023-10-10T17:41:07.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:07.792+0000 7f5c9aaea700 1 -- 172.21.15.78:0/1982270113 wait complete. 2023-10-10T17:41:08.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:07 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1982270113' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-10T17:41:08.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:07 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1982270113' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-10T17:41:08.176 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":16,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":321866266},{"type":"v1","addr":"172.21.15.78:6801","nonce":321866266}]},"active_addr":"172.21.15.78:6801/321866266","active_change":"2023-10-10T17:34:36.143648+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.78:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":540877351}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1054117069}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":1411676003}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":951456648}]}]} 2023-10-10T17:41:08.182 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-10-10T17:41:08.183 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-10-10T17:41:08.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd dump --format=json 2023-10-10T17:41:08.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:07 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1982270113' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-10T17:41:08.827 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:09.617 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:09 smithi078 ceph-mon[99494]: pgmap v275: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-10-10T17:41:09.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:09 smithi190 ceph-mon[101220]: pgmap v275: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-10-10T17:41:09.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:09 smithi157 ceph-mon[100896]: pgmap v275: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 2.2 KiB/s wr, 9 op/s 2023-10-10T17:41:10.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 -- 172.21.15.78:0/2351542949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 msgr2=0x7f60b0101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:10.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/2351542949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0101910 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009990 tx=0x7f60a4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 -- 172.21.15.78:0/2351542949 shutdown_connections 2023-10-10T17:41:10.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/2351542949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b010a890 0x7f60b010ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/2351542949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0101910 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/2351542949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60b0101e50 0x7f60b010a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 -- 172.21.15.78:0/2351542949 >> 172.21.15.78:0/2351542949 conn(0x7f60b00770a0 msgr2=0x7f60b00774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:10.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 -- 172.21.15.78:0/2351542949 shutdown_connections 2023-10-10T17:41:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 -- 172.21.15.78:0/2351542949 wait complete. 2023-10-10T17:41:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.069+0000 7f60b5e6f700 1 Processor -- start 2023-10-10T17:41:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60b5e6f700 1 -- start start 2023-10-10T17:41:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60b5e6f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:10.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60b5e6f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60b0101e50 0x7f60b01969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60b5e6f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b010a890 0x7f60b019ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60b5e6f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019b430 con 0x7f60b0101540 2023-10-10T17:41:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60b5e6f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019b5a0 con 0x7f60b0101e50 2023-10-10T17:41:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60b5e6f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b019b760 con 0x7f60b010a890 2023-10-10T17:41:10.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0196470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42684/0 (socket says 172.21.15.78:42684) 2023-10-10T17:41:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 -- 172.21.15.78:0/4056513491 learned_addr learned my addr 172.21.15.78:0/4056513491 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60affff700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b010a890 0x7f60b019ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 -- 172.21.15.78:0/4056513491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b010a890 msgr2=0x7f60b019ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:10.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b010a890 0x7f60b019ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 -- 172.21.15.78:0/4056513491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60b0101e50 msgr2=0x7f60b01969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:10.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60b0101e50 0x7f60b01969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 -- 172.21.15.78:0/4056513491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a40095f0 con 0x7f60b0101540 2023-10-10T17:41:10.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60affff700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b010a890 0x7f60b019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:10.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60af7fe700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0196470 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f60a4006010 tx=0x7f60a400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:10.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.070+0000 7f60acff9700 1 -- 172.21.15.78:0/4056513491 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a401d070 con 0x7f60b0101540 2023-10-10T17:41:10.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.071+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b0110310 con 0x7f60b0101540 2023-10-10T17:41:10.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.071+0000 7f60acff9700 1 -- 172.21.15.78:0/4056513491 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f60a400fb80 con 0x7f60b0101540 2023-10-10T17:41:10.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.071+0000 7f60acff9700 1 -- 172.21.15.78:0/4056513491 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4017650 con 0x7f60b0101540 2023-10-10T17:41:10.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.071+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b0110180 con 0x7f60b0101540 2023-10-10T17:41:10.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.071+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b01104a0 con 0x7f60b0101540 2023-10-10T17:41:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.073+0000 7f60acff9700 1 -- 172.21.15.78:0/4056513491 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f60a4022470 con 0x7f60b0101540 2023-10-10T17:41:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.074+0000 7f60acff9700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f60980a1150 0x7f60980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.074+0000 7f60aeffd700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f60980a1150 0x7f60980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.074+0000 7f60acff9700 1 -- 172.21.15.78:0/4056513491 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f60a40cdf40 con 0x7f60b0101540 2023-10-10T17:41:10.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.074+0000 7f60aeffd700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f60980a1150 0x7f60980a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f60a0005950 tx=0x7f60a00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:10.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.077+0000 7f60acff9700 1 -- 172.21.15.78:0/4056513491 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f60a4097530 con 0x7f60b0101540 2023-10-10T17:41:10.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.267+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60b0066ea0 con 0x7f60b0101540 2023-10-10T17:41:10.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.268+0000 7f60acff9700 1 -- 172.21.15.78:0/4056513491 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f60a409ab50 con 0x7f60b0101540 2023-10-10T17:41:10.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:10.269 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:41:10.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f60980a1150 msgr2=0x7f60980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:10.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f60980a1150 0x7f60980a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f60a0005950 tx=0x7f60a00058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 msgr2=0x7f60b0196470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0196470 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f60a4006010 tx=0x7f60a400f690 comp rx=0 tx=0).stop 2023-10-10T17:41:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 shutdown_connections 2023-10-10T17:41:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f60980a1150 0x7f60980a3600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f60b010a890 0x7f60b019ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f60b0101540 0x7f60b0196470 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 --2- 172.21.15.78:0/4056513491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f60b0101e50 0x7f60b01969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.270+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 >> 172.21.15.78:0/4056513491 conn(0x7f60b00770a0 msgr2=0x7f60b00fbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.271+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 shutdown_connections 2023-10-10T17:41:10.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:10.271+0000 7f60b5e6f700 1 -- 172.21.15.78:0/4056513491 wait complete. 2023-10-10T17:41:10.687 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-10-10T17:41:10.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd dump --format=json 2023-10-10T17:41:10.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:10 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4056513491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:10.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:10 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4056513491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:10.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:10 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4056513491' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:11.333 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:11 smithi078 ceph-mon[99494]: pgmap v276: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2023-10-10T17:41:11.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:11 smithi190 ceph-mon[101220]: pgmap v276: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2023-10-10T17:41:11.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:11 smithi157 ceph-mon[100896]: pgmap v276: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.2 KiB/s wr, 11 op/s 2023-10-10T17:41:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 -- 172.21.15.78:0/3904710373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ffec0 msgr2=0x7f1904103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 --2- 172.21.15.78:0/3904710373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ffec0 0x7f1904103e90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f18f80099c0 tx=0x7f18f8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 -- 172.21.15.78:0/3904710373 shutdown_connections 2023-10-10T17:41:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 --2- 172.21.15.78:0/3904710373 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19040ff520 0x7f19040ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 --2- 172.21.15.78:0/3904710373 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19041045c0 0x7f190410c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 --2- 172.21.15.78:0/3904710373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ffec0 0x7f1904103e90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 -- 172.21.15.78:0/3904710373 >> 172.21.15.78:0/3904710373 conn(0x7f1904075010 msgr2=0x7f1904075410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 -- 172.21.15.78:0/3904710373 shutdown_connections 2023-10-10T17:41:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 -- 172.21.15.78:0/3904710373 wait complete. 2023-10-10T17:41:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.611+0000 7f1909eb1700 1 Processor -- start 2023-10-10T17:41:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1909eb1700 1 -- start start 2023-10-10T17:41:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1909eb1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ff520 0x7f1904195300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1909eb1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19040ffec0 0x7f1904195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1909eb1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19041045c0 0x7f1904199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1909eb1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190419a2c0 con 0x7f19040ff520 2023-10-10T17:41:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1909eb1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190419a430 con 0x7f19040ffec0 2023-10-10T17:41:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1909eb1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190419a5f0 con 0x7f19041045c0 2023-10-10T17:41:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ff520 0x7f1904195300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ff520 0x7f1904195300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42698/0 (socket says 172.21.15.78:42698) 2023-10-10T17:41:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 -- 172.21.15.78:0/434086785 learned_addr learned my addr 172.21.15.78:0/434086785 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1902ffd700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19040ffec0 0x7f1904195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f1903fff700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19041045c0 0x7f1904199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 -- 172.21.15.78:0/434086785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19041045c0 msgr2=0x7f1904199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19041045c0 0x7f1904199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 -- 172.21.15.78:0/434086785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19040ffec0 msgr2=0x7f1904195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19040ffec0 0x7f1904195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.612+0000 7f19037fe700 1 -- 172.21.15.78:0/434086785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f80095f0 con 0x7f19040ff520 2023-10-10T17:41:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.613+0000 7f19037fe700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ff520 0x7f1904195300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d950 tx=0x7f18f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.613+0000 7f1900ff9700 1 -- 172.21.15.78:0/434086785 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f400f890 con 0x7f19040ff520 2023-10-10T17:41:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.613+0000 7f1900ff9700 1 -- 172.21.15.78:0/434086785 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18f4010460 con 0x7f19040ff520 2023-10-10T17:41:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.613+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190410f2f0 con 0x7f19040ff520 2023-10-10T17:41:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.613+0000 7f1900ff9700 1 -- 172.21.15.78:0/434086785 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f400e610 con 0x7f19040ff520 2023-10-10T17:41:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.613+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190410f160 con 0x7f19040ff520 2023-10-10T17:41:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.614+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190410f480 con 0x7f19040ff520 2023-10-10T17:41:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.615+0000 7f1900ff9700 1 -- 172.21.15.78:0/434086785 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f18f40108d0 con 0x7f19040ff520 2023-10-10T17:41:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.615+0000 7f1900ff9700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18ec0a1150 0x7f18ec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.615+0000 7f1900ff9700 1 -- 172.21.15.78:0/434086785 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f18f40cc8e0 con 0x7f19040ff520 2023-10-10T17:41:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.615+0000 7f1902ffd700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18ec0a1150 0x7f18ec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.616+0000 7f1902ffd700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18ec0a1150 0x7f18ec0a3600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f18f800b580 tx=0x7f18f8003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.620+0000 7f1900ff9700 1 -- 172.21.15.78:0/434086785 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f18f4095e40 con 0x7f19040ff520 2023-10-10T17:41:12.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.807+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1904066ea0 con 0x7f19040ff520 2023-10-10T17:41:12.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.808+0000 7f1900ff9700 1 -- 172.21.15.78:0/434086785 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f18f4016090 con 0x7f19040ff520 2023-10-10T17:41:12.809 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:12.810 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:40:46.095933+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:41:12.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18ec0a1150 msgr2=0x7f18ec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:12.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18ec0a1150 0x7f18ec0a3600 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f18f800b580 tx=0x7f18f8003680 comp rx=0 tx=0).stop 2023-10-10T17:41:12.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ff520 msgr2=0x7f1904195300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:12.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ff520 0x7f1904195300 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f18f400d950 tx=0x7f18f400dc60 comp rx=0 tx=0).stop 2023-10-10T17:41:12.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 shutdown_connections 2023-10-10T17:41:12.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18ec0a1150 0x7f18ec0a3600 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f19041045c0 0x7f1904199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f19040ff520 0x7f1904195300 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 --2- 172.21.15.78:0/434086785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f19040ffec0 0x7f1904195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:12.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 >> 172.21.15.78:0/434086785 conn(0x7f1904075010 msgr2=0x7f19040fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:12.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 shutdown_connections 2023-10-10T17:41:12.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:12.810+0000 7f1909eb1700 1 -- 172.21.15.78:0/434086785 wait complete. 2023-10-10T17:41:13.193 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-10-10T17:41:13.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-10-10T17:41:13.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-10-10T17:41:13.195 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-10-10T17:41:13.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-10-10T17:41:13.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-10-10T17:41:13.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-10-10T17:41:13.198 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-10-10T17:41:13.198 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.8 flush_pg_stats 2023-10-10T17:41:13.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.9 flush_pg_stats 2023-10-10T17:41:13.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.10 flush_pg_stats 2023-10-10T17:41:13.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph tell osd.11 flush_pg_stats 2023-10-10T17:41:13.625 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:13 smithi078 ceph-mon[99494]: pgmap v277: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2023-10-10T17:41:13.626 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:13 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/434086785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:13.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:13 smithi190 ceph-mon[101220]: pgmap v277: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2023-10-10T17:41:13.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:13 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/434086785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:13.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:13 smithi157 ceph-mon[100896]: pgmap v277: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.0 KiB/s wr, 12 op/s 2023-10-10T17:41:13.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:13 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/434086785' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:14.554 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:14.581 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:14.677 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:14.871 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:14.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:14 smithi078 ceph-mon[99494]: pgmap v278: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 11 op/s 2023-10-10T17:41:14.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:14 smithi190 ceph-mon[101220]: pgmap v278: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 11 op/s 2023-10-10T17:41:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:14 smithi157 ceph-mon[100896]: pgmap v278: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 11 op/s 2023-10-10T17:41:15.186 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:15.335 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:15.436 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:15.676 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:16.101 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:16.435 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:17.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:17 smithi078 ceph-mon[99494]: pgmap v279: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-10T17:41:17.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:17 smithi190 ceph-mon[101220]: pgmap v279: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-10T17:41:17.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:17 smithi157 ceph-mon[100896]: pgmap v279: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 7 op/s 2023-10-10T17:41:18.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 -- 172.21.15.78:0/4218468698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 msgr2=0x7f18d4103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:18.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 --2- 172.21.15.78:0/4218468698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 0x7f18d4103dd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f18c80099c0 tx=0x7f18c8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 -- 172.21.15.78:0/4218468698 shutdown_connections 2023-10-10T17:41:18.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 --2- 172.21.15.78:0/4218468698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d4104500 0x7f18d410c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 --2- 172.21.15.78:0/4218468698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18d40ff460 0x7f18d40ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 --2- 172.21.15.78:0/4218468698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 0x7f18d4103dd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 -- 172.21.15.78:0/4218468698 >> 172.21.15.78:0/4218468698 conn(0x7f18d40770a0 msgr2=0x7f18d40774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:18.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 -- 172.21.15.78:0/4218468698 shutdown_connections 2023-10-10T17:41:18.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.329+0000 7f18da5f9700 1 -- 172.21.15.78:0/4218468698 wait complete. 2023-10-10T17:41:18.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 Processor -- start 2023-10-10T17:41:18.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 -- start start 2023-10-10T17:41:18.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d40ff460 0x7f18d4197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:18.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 0x7f18d41977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:18.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18d4104500 0x7f18d41a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:18.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d4198060 con 0x7f18d40ffe00 2023-10-10T17:41:18.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41a2820 con 0x7f18d4104500 2023-10-10T17:41:18.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.330+0000 7f18da5f9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d41a2960 con 0x7f18d40ff460 2023-10-10T17:41:18.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 0x7f18d41977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:18.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 0x7f18d41977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42712/0 (socket says 172.21.15.78:42712) 2023-10-10T17:41:18.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 -- 172.21.15.78:0/1405932750 learned_addr learned my addr 172.21.15.78:0/1405932750 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:18.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d8b96700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18d4104500 0x7f18d41a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:18.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d3fff700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d40ff460 0x7f18d4197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:18.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 -- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d40ff460 msgr2=0x7f18d4197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:18.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d40ff460 0x7f18d4197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 -- 172.21.15.78:0/1405932750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18d4104500 msgr2=0x7f18d41a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:18.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18d4104500 0x7f18d41a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 -- 172.21.15.78:0/1405932750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c80095f0 con 0x7f18d40ffe00 2023-10-10T17:41:18.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d3fff700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18d40ff460 0x7f18d4197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:18.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d37fe700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 0x7f18d41977a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009990 tx=0x7f18c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:18.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c801d070 con 0x7f18d40ffe00 2023-10-10T17:41:18.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d410f250 con 0x7f18d40ffe00 2023-10-10T17:41:18.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18c80054f0 con 0x7f18d40ffe00 2023-10-10T17:41:18.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c8003ad0 con 0x7f18d40ffe00 2023-10-10T17:41:18.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.331+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d410f0c0 con 0x7f18d40ffe00 2023-10-10T17:41:18.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.332+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18d40fd6f0 con 0x7f18d40ffe00 2023-10-10T17:41:18.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.333+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f18c80047a0 con 0x7f18d40ffe00 2023-10-10T17:41:18.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.334+0000 7f18d17fa700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18bc0a1150 0x7f18bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:18.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.334+0000 7f18d3fff700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18bc0a1150 0x7f18bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:18.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.334+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f18c80ce330 con 0x7f18d40ffe00 2023-10-10T17:41:18.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.334+0000 7f18d17fa700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] conn(0x7f18bc0abcc0 0x7f18bc0ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:18.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.334+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 --> [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18bc0ae7c0 con 0x7f18bc0abcc0 2023-10-10T17:41:18.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.334+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18c80ce740 con 0x7f18d40ffe00 2023-10-10T17:41:18.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.334+0000 7f18d3fff700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18bc0a1150 0x7f18bc0a3600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18c4005950 tx=0x7f18c40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:18.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.335+0000 7f18d8b96700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] conn(0x7f18bc0abcc0 0x7f18bc0ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:18.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.335+0000 7f18d8b96700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] conn(0x7f18bc0abcc0 0x7f18bc0ae0f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:18.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.336+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== osd.11 v2:172.21.15.190:6824/2501616180 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f18bc0ae7c0 con 0x7f18bc0abcc0 2023-10-10T17:41:18.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.353+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 --> [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18d410f3e0 con 0x7f18bc0abcc0 2023-10-10T17:41:18.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18d17fa700 1 -- 172.21.15.78:0/1405932750 <== osd.11 v2:172.21.15.190:6824/2501616180 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f18d410f3e0 con 0x7f18bc0abcc0 2023-10-10T17:41:18.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] conn(0x7f18bc0abcc0 msgr2=0x7f18bc0ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:18.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] conn(0x7f18bc0abcc0 0x7f18bc0ae0f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18bc0a1150 msgr2=0x7f18bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:18.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18bc0a1150 0x7f18bc0a3600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18c4005950 tx=0x7f18c40058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:18.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 msgr2=0x7f18d41977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:18.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 --2- 172.21.15.78:0/1405932750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18d40ffe00 0x7f18d41977a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f18c8009990 tx=0x7f18c8004730 comp rx=0 tx=0).stop 2023-10-10T17:41:18.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18d8b96700 1 -- 172.21.15.78:0/1405932750 reap_dead start 2023-10-10T17:41:18.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 shutdown_connections 2023-10-10T17:41:18.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.354+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 >> 172.21.15.78:0/1405932750 conn(0x7f18d40770a0 msgr2=0x7f18d40fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:18.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.355+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 shutdown_connections 2023-10-10T17:41:18.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:18.355+0000 7f18da5f9700 1 -- 172.21.15.78:0/1405932750 wait complete. 2023-10-10T17:41:19.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:19 smithi078 ceph-mon[99494]: pgmap v280: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-10-10T17:41:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:19 smithi190 ceph-mon[101220]: pgmap v280: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-10-10T17:41:19.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:19 smithi157 ceph-mon[100896]: pgmap v280: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-10-10T17:41:20.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 -- 172.21.15.78:0/540437523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ffe10 msgr2=0x7f6a84103de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 --2- 172.21.15.78:0/540437523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ffe10 0x7f6a84103de0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6a700099c0 tx=0x7f6a70009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 -- 172.21.15.78:0/540437523 shutdown_connections 2023-10-10T17:41:20.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 --2- 172.21.15.78:0/540437523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a84104510 0x7f6a8410c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 --2- 172.21.15.78:0/540437523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a840ff470 0x7f6a840ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 --2- 172.21.15.78:0/540437523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ffe10 0x7f6a84103de0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 -- 172.21.15.78:0/540437523 >> 172.21.15.78:0/540437523 conn(0x7f6a840770d0 msgr2=0x7f6a840774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 -- 172.21.15.78:0/540437523 shutdown_connections 2023-10-10T17:41:20.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.898+0000 7f6a8abfa700 1 -- 172.21.15.78:0/540437523 wait complete. 2023-10-10T17:41:20.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 Processor -- start 2023-10-10T17:41:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 -- start start 2023-10-10T17:41:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ff470 0x7f6a84100e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a840ffe10 0x7f6a841013c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a84104510 0x7f6a84101940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a841a4970 con 0x7f6a840ff470 2023-10-10T17:41:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a841a4ae0 con 0x7f6a84104510 2023-10-10T17:41:20.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a8abfa700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a841a4ca0 con 0x7f6a840ffe10 2023-10-10T17:41:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ff470 0x7f6a84100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ff470 0x7f6a84100e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42758/0 (socket says 172.21.15.78:42758) 2023-10-10T17:41:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 -- 172.21.15.78:0/2440931844 learned_addr learned my addr 172.21.15.78:0/2440931844 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a83fff700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a840ffe10 0x7f6a841013c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 -- 172.21.15.78:0/2440931844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a840ffe10 msgr2=0x7f6a841013c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a840ffe10 0x7f6a841013c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 -- 172.21.15.78:0/2440931844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a84104510 msgr2=0x7f6a84101940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:41:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a84104510 0x7f6a84101940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.899+0000 7f6a88996700 1 -- 172.21.15.78:0/2440931844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a700095f0 con 0x7f6a840ff470 2023-10-10T17:41:20.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.900+0000 7f6a88996700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ff470 0x7f6a84100e80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800e9a0 tx=0x7f6a7800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.900+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78009790 con 0x7f6a840ff470 2023-10-10T17:41:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.900+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a7800cbd0 con 0x7f6a840ff470 2023-10-10T17:41:20.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.900+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78018680 con 0x7f6a840ff470 2023-10-10T17:41:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.900+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8410f330 con 0x7f6a840ff470 2023-10-10T17:41:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.900+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8410f1a0 con 0x7f6a840ff470 2023-10-10T17:41:20.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.901+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a840698b0 con 0x7f6a840ff470 2023-10-10T17:41:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.903+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6a7800cd40 con 0x7f6a840ff470 2023-10-10T17:41:20.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a81ffb700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6a740a11d0 0x7f6a740a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a83fff700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6a740a11d0 0x7f6a740a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6a78014070 con 0x7f6a840ff470 2023-10-10T17:41:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a81ffb700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] conn(0x7f6a740abd40 0x7f6a740ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 --> [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a740ae840 con 0x7f6a740abd40 2023-10-10T17:41:20.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a780cd4b0 con 0x7f6a840ff470 2023-10-10T17:41:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a89197700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] conn(0x7f6a740abd40 0x7f6a740ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a83fff700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6a740a11d0 0x7f6a740a3680 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009fd0 tx=0x7f6a70005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.904+0000 7f6a89197700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] conn(0x7f6a740abd40 0x7f6a740ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:20.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.905+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== osd.9 v2:172.21.15.78:6826/2313179218 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6a740ae840 con 0x7f6a740abd40 2023-10-10T17:41:20.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.921+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 --> [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6a8410f4c0 con 0x7f6a740abd40 2023-10-10T17:41:20.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.921+0000 7f6a81ffb700 1 -- 172.21.15.78:0/2440931844 <== osd.9 v2:172.21.15.78:6826/2313179218 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6a8410f4c0 con 0x7f6a740abd40 2023-10-10T17:41:20.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] conn(0x7f6a740abd40 msgr2=0x7f6a740ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] conn(0x7f6a740abd40 0x7f6a740ae170 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6a740a11d0 msgr2=0x7f6a740a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f6a740a11d0 0x7f6a740a3680 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6a70009fd0 tx=0x7f6a70005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ff470 msgr2=0x7f6a84100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 --2- 172.21.15.78:0/2440931844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a840ff470 0x7f6a84100e80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800e9a0 tx=0x7f6a7800ecb0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a89197700 1 -- 172.21.15.78:0/2440931844 reap_dead start 2023-10-10T17:41:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 shutdown_connections 2023-10-10T17:41:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 >> 172.21.15.78:0/2440931844 conn(0x7f6a840770d0 msgr2=0x7f6a84068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 shutdown_connections 2023-10-10T17:41:20.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.922+0000 7f6a8abfa700 1 -- 172.21.15.78:0/2440931844 wait complete. 2023-10-10T17:41:20.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 -- 172.21.15.78:0/2391279395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ffe00 msgr2=0x7f1c70103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 --2- 172.21.15.78:0/2391279395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ffe00 0x7f1c70103dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1c600099c0 tx=0x7f1c60009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 -- 172.21.15.78:0/2391279395 shutdown_connections 2023-10-10T17:41:20.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 --2- 172.21.15.78:0/2391279395 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c70104500 0x7f1c7010c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 --2- 172.21.15.78:0/2391279395 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c700ff460 0x7f1c700ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 --2- 172.21.15.78:0/2391279395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ffe00 0x7f1c70103dd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 -- 172.21.15.78:0/2391279395 >> 172.21.15.78:0/2391279395 conn(0x7f1c700770a0 msgr2=0x7f1c700774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:20.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 -- 172.21.15.78:0/2391279395 shutdown_connections 2023-10-10T17:41:20.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.748+0000 7f1c77571700 1 -- 172.21.15.78:0/2391279395 wait complete. 2023-10-10T17:41:20.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 Processor -- start 2023-10-10T17:41:20.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 -- start start 2023-10-10T17:41:20.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ff460 0x7f1c70196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ffe00 0x7f1c70196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c70104500 0x7f1c7019adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019b3d0 con 0x7f1c700ff460 2023-10-10T17:41:20.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019b540 con 0x7f1c70104500 2023-10-10T17:41:20.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c77571700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7019b700 con 0x7f1c700ffe00 2023-10-10T17:41:20.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ff460 0x7f1c70196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ff460 0x7f1c70196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42742/0 (socket says 172.21.15.78:42742) 2023-10-10T17:41:20.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 -- 172.21.15.78:0/2541948904 learned_addr learned my addr 172.21.15.78:0/2541948904 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:20.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 -- 172.21.15.78:0/2541948904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ffe00 msgr2=0x7f1c70196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c74b0c700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ffe00 0x7f1c70196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1c700ffe00 0x7f1c70196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 -- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c70104500 msgr2=0x7f1c7019adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c75b0e700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c70104500 0x7f1c7019adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c70104500 0x7f1c7019adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c7530d700 1 -- 172.21.15.78:0/2541948904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c600095f0 con 0x7f1c700ff460 2023-10-10T17:41:20.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.749+0000 7f1c75b0e700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1c70104500 0x7f1c7019adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:20.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.750+0000 7f1c7530d700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ff460 0x7f1c70196410 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00c920 tx=0x7f1c6c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:20.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.750+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c007b00 con 0x7f1c700ff460 2023-10-10T17:41:20.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.750+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1c6c004cf0 con 0x7f1c700ff460 2023-10-10T17:41:20.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.750+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c005610 con 0x7f1c700ff460 2023-10-10T17:41:20.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.750+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7010f230 con 0x7f1c700ff460 2023-10-10T17:41:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.750+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7010f0a0 con 0x7f1c700ff460 2023-10-10T17:41:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.750+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1c70103a60 con 0x7f1c700ff460 2023-10-10T17:41:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.752+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1c6c004750 con 0x7f1c700ff460 2023-10-10T17:41:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.752+0000 7f1c667fc700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1c5c0a1250 0x7f1c5c0a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.752+0000 7f1c74b0c700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1c5c0a1250 0x7f1c5c0a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.752+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1c6c0ccc80 con 0x7f1c700ff460 2023-10-10T17:41:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.753+0000 7f1c667fc700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] conn(0x7f1c5c0abdc0 0x7f1c5c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.753+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 --> [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1c5c0ae8c0 con 0x7f1c5c0abdc0 2023-10-10T17:41:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.753+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c6c0ccff0 con 0x7f1c700ff460 2023-10-10T17:41:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.753+0000 7f1c74b0c700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1c5c0a1250 0x7f1c5c0a3700 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1c600098e0 tx=0x7f1c60000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.753+0000 7f1c75b0e700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] conn(0x7f1c5c0abdc0 0x7f1c5c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.753+0000 7f1c75b0e700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] conn(0x7f1c5c0abdc0 0x7f1c5c0ae1f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.754+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== osd.1 v2:172.21.15.157:6800/4215285299 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1c5c0ae8c0 con 0x7f1c5c0abdc0 2023-10-10T17:41:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.771+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 --> [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1c7010f3c0 con 0x7f1c5c0abdc0 2023-10-10T17:41:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c667fc700 1 -- 172.21.15.78:0/2541948904 <== osd.1 v2:172.21.15.157:6800/4215285299 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1c7010f3c0 con 0x7f1c5c0abdc0 2023-10-10T17:41:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] conn(0x7f1c5c0abdc0 msgr2=0x7f1c5c0ae1f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] conn(0x7f1c5c0abdc0 0x7f1c5c0ae1f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1c5c0a1250 msgr2=0x7f1c5c0a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1c5c0a1250 0x7f1c5c0a3700 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1c600098e0 tx=0x7f1c60000bc0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ff460 msgr2=0x7f1c70196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 --2- 172.21.15.78:0/2541948904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1c700ff460 0x7f1c70196410 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c00c920 tx=0x7f1c6c00cce0 comp rx=0 tx=0).stop 2023-10-10T17:41:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c75b0e700 1 -- 172.21.15.78:0/2541948904 reap_dead start 2023-10-10T17:41:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 shutdown_connections 2023-10-10T17:41:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 >> 172.21.15.78:0/2541948904 conn(0x7f1c700770a0 msgr2=0x7f1c70103010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 shutdown_connections 2023-10-10T17:41:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.772+0000 7f1c77571700 1 -- 172.21.15.78:0/2541948904 wait complete. 2023-10-10T17:41:21.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 -- 172.21.15.78:0/556940048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6900ff470 msgr2=0x7fc6900ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 --2- 172.21.15.78:0/556940048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6900ff470 0x7fc6900ff840 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc68c0099c0 tx=0x7fc68c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 -- 172.21.15.78:0/556940048 shutdown_connections 2023-10-10T17:41:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 --2- 172.21.15.78:0/556940048 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc690104510 0x7fc69010c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 --2- 172.21.15.78:0/556940048 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6900ff470 0x7fc6900ff840 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 --2- 172.21.15.78:0/556940048 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6900ffe10 0x7fc690103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 -- 172.21.15.78:0/556940048 >> 172.21.15.78:0/556940048 conn(0x7fc6900770d0 msgr2=0x7fc6900774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 -- 172.21.15.78:0/556940048 shutdown_connections 2023-10-10T17:41:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.559+0000 7fc698366700 1 -- 172.21.15.78:0/556940048 wait complete. 2023-10-10T17:41:21.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 Processor -- start 2023-10-10T17:41:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 -- start start 2023-10-10T17:41:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6900ff470 0x7fc690197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6900ffe10 0x7fc690197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc690104510 0x7fc6901a21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc690197fb0 con 0x7fc690104510 2023-10-10T17:41:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6901a26f0 con 0x7fc6900ff470 2023-10-10T17:41:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc698366700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6901a28b0 con 0x7fc6900ffe10 2023-10-10T17:41:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.560+0000 7fc696903700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc690104510 0x7fc6901a21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc690104510 0x7fc6901a21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42730/0 (socket says 172.21.15.78:42730) 2023-10-10T17:41:21.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 -- 172.21.15.78:0/3938233253 learned_addr learned my addr 172.21.15.78:0/3938233253 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:21.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696102700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6900ff470 0x7fc690197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc695901700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6900ffe10 0x7fc690197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 -- 172.21.15.78:0/3938233253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6900ffe10 msgr2=0x7fc690197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6900ffe10 0x7fc690197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 -- 172.21.15.78:0/3938233253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6900ff470 msgr2=0x7fc690197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6900ff470 0x7fc690197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 -- 172.21.15.78:0/3938233253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c0095f0 con 0x7fc690104510 2023-10-10T17:41:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc695901700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6900ffe10 0x7fc690197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc696903700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc690104510 0x7fc6901a21b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc688007ee0 tx=0x7fc68800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:21.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688017070 con 0x7fc690104510 2023-10-10T17:41:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc68800f040 con 0x7fc690104510 2023-10-10T17:41:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69010f250 con 0x7fc690104510 2023-10-10T17:41:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688005740 con 0x7fc690104510 2023-10-10T17:41:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.561+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69010f0c0 con 0x7fc690104510 2023-10-10T17:41:21.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.562+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6900fe7a0 con 0x7fc690104510 2023-10-10T17:41:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.566+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc68800dc60 con 0x7fc690104510 2023-10-10T17:41:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.566+0000 7fc6877fe700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc67c0a11d0 0x7fc67c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.566+0000 7fc696102700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc67c0a11d0 0x7fc67c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.566+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc6880cc8d0 con 0x7fc690104510 2023-10-10T17:41:21.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.566+0000 7fc6877fe700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] conn(0x7fc67c0abd40 0x7fc67c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.566+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 --> [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc67c0ae840 con 0x7fc67c0abd40 2023-10-10T17:41:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.567+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6880ccce0 con 0x7fc690104510 2023-10-10T17:41:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.567+0000 7fc696102700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc67c0a11d0 0x7fc67c0a3680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009990 tx=0x7fc68c003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.567+0000 7fc695901700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] conn(0x7fc67c0abd40 0x7fc67c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.567+0000 7fc695901700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] conn(0x7fc67c0abd40 0x7fc67c0ae170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.568+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== osd.4 v2:172.21.15.157:6808/1484677990 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc67c0ae840 con 0x7fc67c0abd40 2023-10-10T17:41:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.584+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 --> [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc69010f3e0 con 0x7fc67c0abd40 2023-10-10T17:41:21.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc6877fe700 1 -- 172.21.15.78:0/3938233253 <== osd.4 v2:172.21.15.157:6808/1484677990 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc69010f3e0 con 0x7fc67c0abd40 2023-10-10T17:41:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 >> [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] conn(0x7fc67c0abd40 msgr2=0x7fc67c0ae170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] conn(0x7fc67c0abd40 0x7fc67c0ae170 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc67c0a11d0 msgr2=0x7fc67c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc67c0a11d0 0x7fc67c0a3680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009990 tx=0x7fc68c003680 comp rx=0 tx=0).stop 2023-10-10T17:41:21.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc690104510 msgr2=0x7fc6901a21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 --2- 172.21.15.78:0/3938233253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc690104510 0x7fc6901a21b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc688007ee0 tx=0x7fc68800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc696903700 1 -- 172.21.15.78:0/3938233253 reap_dead start 2023-10-10T17:41:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 shutdown_connections 2023-10-10T17:41:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 >> 172.21.15.78:0/3938233253 conn(0x7fc6900770d0 msgr2=0x7fc6900fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 shutdown_connections 2023-10-10T17:41:21.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:20.585+0000 7fc698366700 1 -- 172.21.15.78:0/3938233253 wait complete. 2023-10-10T17:41:21.672 INFO:teuthology.orchestra.run.smithi078.stdout:292057776156 2023-10-10T17:41:21.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.9 2023-10-10T17:41:21.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 -- 172.21.15.78:0/3439108114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 msgr2=0x7f78240ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 --2- 172.21.15.78:0/3439108114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 0x7f78240ff830 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f78180099c0 tx=0x7f7818009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 -- 172.21.15.78:0/3439108114 shutdown_connections 2023-10-10T17:41:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 --2- 172.21.15.78:0/3439108114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7824104500 0x7f782410c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 --2- 172.21.15.78:0/3439108114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 0x7f78240ff830 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 --2- 172.21.15.78:0/3439108114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78240ffe00 0x7f7824103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 -- 172.21.15.78:0/3439108114 >> 172.21.15.78:0/3439108114 conn(0x7f78240770a0 msgr2=0x7f78240774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:21.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 -- 172.21.15.78:0/3439108114 shutdown_connections 2023-10-10T17:41:21.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 -- 172.21.15.78:0/3439108114 wait complete. 2023-10-10T17:41:21.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.518+0000 7f7829b58700 1 Processor -- start 2023-10-10T17:41:21.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7829b58700 1 -- start start 2023-10-10T17:41:21.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7829b58700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 0x7f78241952e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7829b58700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78240ffe00 0x7f7824195820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7829b58700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7824104500 0x7f7824199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7829b58700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782419a2a0 con 0x7f78240ff460 2023-10-10T17:41:21.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7829b58700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782419a410 con 0x7f78240ffe00 2023-10-10T17:41:21.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7829b58700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f782419a5d0 con 0x7f7824104500 2023-10-10T17:41:21.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 0x7f78241952e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 0x7f78241952e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42772/0 (socket says 172.21.15.78:42772) 2023-10-10T17:41:21.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 -- 172.21.15.78:0/2595112076 learned_addr learned my addr 172.21.15.78:0/2595112076 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:21.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 -- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7824104500 msgr2=0x7f7824199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7823fff700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7824104500 0x7f7824199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7824104500 0x7f7824199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f7822ffd700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78240ffe00 0x7f7824195820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 -- 172.21.15.78:0/2595112076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78240ffe00 msgr2=0x7f7824195820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f78240ffe00 0x7f7824195820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 -- 172.21.15.78:0/2595112076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78180095f0 con 0x7f78240ff460 2023-10-10T17:41:21.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.519+0000 7f78237fe700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 0x7f78241952e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f781800b580 tx=0x7f781800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:21.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.520+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f781801d070 con 0x7f78240ff460 2023-10-10T17:41:21.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.520+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f78180054f0 con 0x7f78240ff460 2023-10-10T17:41:21.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.520+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782410f240 con 0x7f78240ff460 2023-10-10T17:41:21.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.520+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7818004bd0 con 0x7f78240ff460 2023-10-10T17:41:21.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.520+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782410f0b0 con 0x7f78240ff460 2023-10-10T17:41:21.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.521+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f78240fd760 con 0x7f78240ff460 2023-10-10T17:41:21.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.522+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f781800fc30 con 0x7f78240ff460 2023-10-10T17:41:21.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.522+0000 7f7820ff9700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f780c0a1150 0x7f780c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.522+0000 7f7822ffd700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f780c0a1150 0x7f780c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.522+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f78180ce0d0 con 0x7f78240ff460 2023-10-10T17:41:21.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.522+0000 7f7820ff9700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] conn(0x7f780c0abcc0 0x7f780c0ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:21.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.522+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 --> [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f780c0ae7c0 con 0x7f780c0abcc0 2023-10-10T17:41:21.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.522+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78180ce4e0 con 0x7f78240ff460 2023-10-10T17:41:21.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.523+0000 7f7822ffd700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f780c0a1150 0x7f780c0a3600 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7824196900 tx=0x7f7814005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:21.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.523+0000 7f7823fff700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] conn(0x7f780c0abcc0 0x7f780c0ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:21.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.523+0000 7f7823fff700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] conn(0x7f780c0abcc0 0x7f780c0ae0f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:21.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.524+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== osd.2 v2:172.21.15.190:6800/678998123 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f780c0ae7c0 con 0x7f780c0abcc0 2023-10-10T17:41:21.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.539+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 --> [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f782410f3d0 con 0x7f780c0abcc0 2023-10-10T17:41:21.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.540+0000 7f7820ff9700 1 -- 172.21.15.78:0/2595112076 <== osd.2 v2:172.21.15.190:6800/678998123 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f782410f3d0 con 0x7f780c0abcc0 2023-10-10T17:41:21.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.540+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] conn(0x7f780c0abcc0 msgr2=0x7f780c0ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.540+0000 7f7829b58700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] conn(0x7f780c0abcc0 0x7f780c0ae0f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f780c0a1150 msgr2=0x7f780c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f780c0a1150 0x7f780c0a3600 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7824196900 tx=0x7f7814005cf0 comp rx=0 tx=0).stop 2023-10-10T17:41:21.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 msgr2=0x7f78241952e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:21.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 --2- 172.21.15.78:0/2595112076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f78240ff460 0x7f78241952e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f781800b580 tx=0x7f781800f740 comp rx=0 tx=0).stop 2023-10-10T17:41:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7823fff700 1 -- 172.21.15.78:0/2595112076 reap_dead start 2023-10-10T17:41:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 shutdown_connections 2023-10-10T17:41:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 >> 172.21.15.78:0/2595112076 conn(0x7f78240770a0 msgr2=0x7f78240fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 shutdown_connections 2023-10-10T17:41:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:21.541+0000 7f7829b58700 1 -- 172.21.15.78:0/2595112076 wait complete. 2023-10-10T17:41:21.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:21 smithi078 ceph-mon[99494]: pgmap v281: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:21.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:21 smithi190 ceph-mon[101220]: pgmap v281: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:21.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:21 smithi157 ceph-mon[100896]: pgmap v281: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:22.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1679519302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 msgr2=0x7fe8e4100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:22.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 --2- 172.21.15.78:0/1679519302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 0x7fe8e4100920 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe8d40099c0 tx=0x7fe8d4009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:22.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1679519302 shutdown_connections 2023-10-10T17:41:22.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 --2- 172.21.15.78:0/1679519302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8e41055f0 0x7fe8e410d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:22.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 --2- 172.21.15.78:0/1679519302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 0x7fe8e4100920 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:22.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 --2- 172.21.15.78:0/1679519302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e4100ef0 0x7fe8e4104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:22.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1679519302 >> 172.21.15.78:0/1679519302 conn(0x7fe8e40fbdb0 msgr2=0x7fe8e40fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:22.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1679519302 shutdown_connections 2023-10-10T17:41:22.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.489+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1679519302 wait complete. 2023-10-10T17:41:22.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 Processor -- start 2023-10-10T17:41:22.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 -- start start 2023-10-10T17:41:22.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 0x7fe8e41a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:22.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8e4100ef0 0x7fe8e41a1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:22.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e41055f0 0x7fe8e410bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:22.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e410c540 con 0x7fe8e4100550 2023-10-10T17:41:22.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e410c6b0 con 0x7fe8e4100ef0 2023-10-10T17:41:22.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e8ffc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e410c870 con 0x7fe8e41055f0 2023-10-10T17:41:22.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 0x7fe8e41a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:22.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 0x7fe8e41a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42786/0 (socket says 172.21.15.78:42786) 2023-10-10T17:41:22.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.490+0000 7fe8e259c700 1 -- 172.21.15.78:0/1641609722 learned_addr learned my addr 172.21.15.78:0/1641609722 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:22.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e1d9b700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8e4100ef0 0x7fe8e41a1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:22.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e259c700 1 -- 172.21.15.78:0/1641609722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e41055f0 msgr2=0x7fe8e410bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:22.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e2d9d700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e41055f0 0x7fe8e410bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:22.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e259c700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e41055f0 0x7fe8e410bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:22.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e259c700 1 -- 172.21.15.78:0/1641609722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8e4100ef0 msgr2=0x7fe8e41a1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:22.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e259c700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe8e4100ef0 0x7fe8e41a1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:22.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e259c700 1 -- 172.21.15.78:0/1641609722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d40095f0 con 0x7fe8e4100550 2023-10-10T17:41:22.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e2d9d700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe8e41055f0 0x7fe8e410bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:22.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e259c700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 0x7fe8e41a1720 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400b580 tx=0x7fe8d400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:22.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d401d070 con 0x7fe8e4100550 2023-10-10T17:41:22.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e4110340 con 0x7fe8e4100550 2023-10-10T17:41:22.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe8d40054f0 con 0x7fe8e4100550 2023-10-10T17:41:22.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4004b30 con 0x7fe8e4100550 2023-10-10T17:41:22.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.491+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e41101b0 con 0x7fe8e4100550 2023-10-10T17:41:22.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.492+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe8e40fd2e0 con 0x7fe8e4100550 2023-10-10T17:41:22.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.493+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe8d400fb80 con 0x7fe8e4100550 2023-10-10T17:41:22.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8d37fe700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:22.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8e1d9b700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:22.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe8d40cd8c0 con 0x7fe8e4100550 2023-10-10T17:41:22.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8d37fe700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] conn(0x7fe8cc0abcc0 0x7fe8cc0ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:22.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 --> [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe8cc0ae7c0 con 0x7fe8cc0abcc0 2023-10-10T17:41:22.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe8d40d22a0 con 0x7fe8e4100550 2023-10-10T17:41:22.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8e2d9d700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] conn(0x7fe8cc0abcc0 0x7fe8cc0ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:22.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8e1d9b700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe8e410cf30 tx=0x7fe8d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:22.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.494+0000 7fe8e2d9d700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] conn(0x7fe8cc0abcc0 0x7fe8cc0ae0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:22.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.495+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== osd.6 v2:172.21.15.78:6818/2340006051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe8cc0ae7c0 con 0x7fe8cc0abcc0 2023-10-10T17:41:22.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.512+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 --> [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe8e41104d0 con 0x7fe8cc0abcc0 2023-10-10T17:41:22.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.512+0000 7fe8d37fe700 1 -- 172.21.15.78:0/1641609722 <== osd.6 v2:172.21.15.78:6818/2340006051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe8e41104d0 con 0x7fe8cc0abcc0 2023-10-10T17:41:22.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] conn(0x7fe8cc0abcc0 msgr2=0x7fe8cc0ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:22.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] conn(0x7fe8cc0abcc0 0x7fe8cc0ae0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:22.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe8cc0a1150 msgr2=0x7fe8cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:22.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe8cc0a1150 0x7fe8cc0a3600 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe8e410cf30 tx=0x7fe8d8009260 comp rx=0 tx=0).stop 2023-10-10T17:41:22.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 msgr2=0x7fe8e41a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:22.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 --2- 172.21.15.78:0/1641609722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe8e4100550 0x7fe8e41a1720 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe8d400b580 tx=0x7fe8d400f690 comp rx=0 tx=0).stop 2023-10-10T17:41:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e2d9d700 1 -- 172.21.15.78:0/1641609722 reap_dead start 2023-10-10T17:41:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 shutdown_connections 2023-10-10T17:41:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 >> 172.21.15.78:0/1641609722 conn(0x7fe8e40fbdb0 msgr2=0x7fe8e40fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 shutdown_connections 2023-10-10T17:41:22.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:22.513+0000 7fe8e8ffc700 1 -- 172.21.15.78:0/1641609722 wait complete. 2023-10-10T17:41:22.818 INFO:teuthology.orchestra.run.smithi078.stdout:146028888114 2023-10-10T17:41:22.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-10-10T17:41:23.110 INFO:teuthology.orchestra.run.smithi078.stdout:343597383699 2023-10-10T17:41:23.111 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.11 2023-10-10T17:41:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1704375347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e81011b0 msgr2=0x7fc8e8101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 --2- 172.21.15.78:0/1704375347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e81011b0 0x7fc8e8101580 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009990 tx=0x7fc8e4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1704375347 shutdown_connections 2023-10-10T17:41:23.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 --2- 172.21.15.78:0/1704375347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8e80ff2c0 0x7fc8e810efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 --2- 172.21.15.78:0/1704375347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e81011b0 0x7fc8e8101580 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 --2- 172.21.15.78:0/1704375347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8e80fe830 0x7fc8e80feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1704375347 >> 172.21.15.78:0/1704375347 conn(0x7fc8e80758a0 msgr2=0x7fc8e8075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1704375347 shutdown_connections 2023-10-10T17:41:23.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.605+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1704375347 wait complete. 2023-10-10T17:41:23.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 Processor -- start 2023-10-10T17:41:23.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 -- start start 2023-10-10T17:41:23.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8e80fe830 0x7fc8e8198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:23.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e80ff2c0 0x7fc8e8198a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:23.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8e81011b0 0x7fc8e819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:23.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e819d4e0 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e819d650 con 0x7fc8e80fe830 2023-10-10T17:41:23.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8f00c6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e819d810 con 0x7fc8e81011b0 2023-10-10T17:41:23.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8ed661700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e80ff2c0 0x7fc8e8198a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:23.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8ed661700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e80ff2c0 0x7fc8e8198a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42804/0 (socket says 172.21.15.78:42804) 2023-10-10T17:41:23.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8ed661700 1 -- 172.21.15.78:0/1883615309 learned_addr learned my addr 172.21.15.78:0/1883615309 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:23.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.606+0000 7fc8ede62700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8e80fe830 0x7fc8e8198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:23.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8ee663700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8e81011b0 0x7fc8e819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:23.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8ed661700 1 -- 172.21.15.78:0/1883615309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8e81011b0 msgr2=0x7fc8e819ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:23.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8ed661700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8e81011b0 0x7fc8e819ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8ed661700 1 -- 172.21.15.78:0/1883615309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8e80fe830 msgr2=0x7fc8e8198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8ed661700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8e80fe830 0x7fc8e8198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8ed661700 1 -- 172.21.15.78:0/1883615309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e40095f0 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8ed661700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e80ff2c0 0x7fc8e8198a60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800c9b0 tx=0x7fc8d800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8007b00 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8e8112560 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc8d8007c60 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8018660 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.607+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e81123d0 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.608+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc8e80fcd50 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.609+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc8d800f450 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.609+0000 7fc8deffd700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc8d40a1150 0x7fc8d40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:23.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8ede62700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc8d40a1150 0x7fc8d40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:23.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc8d80ccf60 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8deffd700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] conn(0x7fc8d40abcc0 0x7fc8d40ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:23.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 --> [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8d40ae7c0 con 0x7fc8d40abcc0 2023-10-10T17:41:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8d80cd370 con 0x7fc8e80ff2c0 2023-10-10T17:41:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8ee663700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] conn(0x7fc8d40abcc0 0x7fc8d40ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8ede62700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc8d40a1150 0x7fc8d40a3600 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b2e0 tx=0x7fc8e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:23.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.610+0000 7fc8ee663700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] conn(0x7fc8d40abcc0 0x7fc8d40ae0f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:23.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.611+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== osd.0 v2:172.21.15.78:6802/857914992 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc8d40ae7c0 con 0x7fc8d40abcc0 2023-10-10T17:41:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.626+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 --> [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc8e81126f0 con 0x7fc8d40abcc0 2023-10-10T17:41:23.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8deffd700 1 -- 172.21.15.78:0/1883615309 <== osd.0 v2:172.21.15.78:6802/857914992 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc8e81126f0 con 0x7fc8d40abcc0 2023-10-10T17:41:23.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] conn(0x7fc8d40abcc0 msgr2=0x7fc8d40ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] conn(0x7fc8d40abcc0 0x7fc8d40ae0f0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc8d40a1150 msgr2=0x7fc8d40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:23.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc8d40a1150 0x7fc8d40a3600 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400b2e0 tx=0x7fc8e4005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e80ff2c0 msgr2=0x7fc8e8198a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 --2- 172.21.15.78:0/1883615309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8e80ff2c0 0x7fc8e8198a60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800c9b0 tx=0x7fc8d800ccc0 comp rx=0 tx=0).stop 2023-10-10T17:41:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8ee663700 1 -- 172.21.15.78:0/1883615309 reap_dead start 2023-10-10T17:41:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 shutdown_connections 2023-10-10T17:41:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 >> 172.21.15.78:0/1883615309 conn(0x7fc8e80758a0 msgr2=0x7fc8e80fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:23.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 shutdown_connections 2023-10-10T17:41:23.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:23.627+0000 7fc8f00c6700 1 -- 172.21.15.78:0/1883615309 wait complete. 2023-10-10T17:41:23.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:23 smithi078 ceph-mon[99494]: pgmap v282: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:23.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:23 smithi190 ceph-mon[101220]: pgmap v282: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:23.923 INFO:teuthology.orchestra.run.smithi078.stdout:201863462953 2023-10-10T17:41:23.924 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-10-10T17:41:23.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:23 smithi157 ceph-mon[100896]: pgmap v282: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:24.489 INFO:teuthology.orchestra.run.smithi078.stdout:34359738435 2023-10-10T17:41:24.489 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-10-10T17:41:24.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/3068755020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180ff1e0 msgr2=0x7f9f1810d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 --2- 172.21.15.78:0/3068755020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180ff1e0 0x7f9f1810d150 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9f1000b0a0 tx=0x7f9f1000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/3068755020 shutdown_connections 2023-10-10T17:41:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 --2- 172.21.15.78:0/3068755020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180ff1e0 0x7f9f1810d150 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 --2- 172.21.15.78:0/3068755020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f18105290 0x7f9f18105660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 --2- 172.21.15.78:0/3068755020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f180fe830 0x7f9f180feca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/3068755020 >> 172.21.15.78:0/3068755020 conn(0x7f9f180758a0 msgr2=0x7f9f18075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/3068755020 shutdown_connections 2023-10-10T17:41:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.070+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/3068755020 wait complete. 2023-10-10T17:41:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 Processor -- start 2023-10-10T17:41:24.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 -- start start 2023-10-10T17:41:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180fe830 0x7f9f180ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f180ff1e0 0x7f9f180fff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f18105290 0x7f9f181004d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181a48f0 con 0x7f9f180fe830 2023-10-10T17:41:24.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181a4a60 con 0x7f9f180ff1e0 2023-10-10T17:41:24.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.071+0000 7f9f1cc2b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f181a4c20 con 0x7f9f18105290 2023-10-10T17:41:24.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180fe830 0x7f9f180ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180fe830 0x7f9f180ffa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42818/0 (socket says 172.21.15.78:42818) 2023-10-10T17:41:24.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 -- 172.21.15.78:0/2391255236 learned_addr learned my addr 172.21.15.78:0/2391255236 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:24.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f15d9b700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f180ff1e0 0x7f9f180fff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f16d9d700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f18105290 0x7f9f181004d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 -- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f18105290 msgr2=0x7f9f181004d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f18105290 0x7f9f181004d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 -- 172.21.15.78:0/2391255236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f180ff1e0 msgr2=0x7f9f180fff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9f180ff1e0 0x7f9f180fff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 -- 172.21.15.78:0/2391255236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f10009d00 con 0x7f9f180fe830 2023-10-10T17:41:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f16d9d700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9f18105290 0x7f9f181004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1659c700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180fe830 0x7f9f180ffa10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800c920 tx=0x7f9f0800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08007b00 con 0x7f9f180fe830 2023-10-10T17:41:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f18110410 con 0x7f9f180fe830 2023-10-10T17:41:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9f0800ce70 con 0x7f9f180fe830 2023-10-10T17:41:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.072+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f08005460 con 0x7f9f180fe830 2023-10-10T17:41:24.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.073+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f18110280 con 0x7f9f180fe830 2023-10-10T17:41:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.073+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9f180fd970 con 0x7f9f180fe830 2023-10-10T17:41:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.075+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9f08004750 con 0x7f9f180fe830 2023-10-10T17:41:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f077fe700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9f000a1150 0x7f9f000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9f080cccb0 con 0x7f9f180fe830 2023-10-10T17:41:24.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f15d9b700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9f000a1150 0x7f9f000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f077fe700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] conn(0x7f9f000abcc0 0x7f9f000ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 --> [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f000ae7c0 con 0x7f9f000abcc0 2023-10-10T17:41:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9f080cd0c0 con 0x7f9f180fe830 2023-10-10T17:41:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f15d9b700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9f000a1150 0x7f9f000a3600 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9f18101200 tx=0x7f9f0c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.076+0000 7f9f16d9d700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] conn(0x7f9f000abcc0 0x7f9f000ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.077+0000 7f9f16d9d700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] conn(0x7f9f000abcc0 0x7f9f000ae0f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.078+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== osd.8 v2:172.21.15.190:6816/2390054900 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9f000ae7c0 con 0x7f9f000abcc0 2023-10-10T17:41:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.093+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 --> [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9f181105a0 con 0x7f9f000abcc0 2023-10-10T17:41:24.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.094+0000 7f9f077fe700 1 -- 172.21.15.78:0/2391255236 <== osd.8 v2:172.21.15.190:6816/2390054900 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9f181105a0 con 0x7f9f000abcc0 2023-10-10T17:41:24.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] conn(0x7f9f000abcc0 msgr2=0x7f9f000ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] conn(0x7f9f000abcc0 0x7f9f000ae0f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9f000a1150 msgr2=0x7f9f000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9f000a1150 0x7f9f000a3600 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9f18101200 tx=0x7f9f0c00b410 comp rx=0 tx=0).stop 2023-10-10T17:41:24.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180fe830 msgr2=0x7f9f180ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 --2- 172.21.15.78:0/2391255236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9f180fe830 0x7f9f180ffa10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800c920 tx=0x7f9f0800cc30 comp rx=0 tx=0).stop 2023-10-10T17:41:24.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f16d9d700 1 -- 172.21.15.78:0/2391255236 reap_dead start 2023-10-10T17:41:24.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 shutdown_connections 2023-10-10T17:41:24.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 >> 172.21.15.78:0/2391255236 conn(0x7f9f180758a0 msgr2=0x7f9f180fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:24.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 shutdown_connections 2023-10-10T17:41:24.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.095+0000 7f9f1cc2b700 1 -- 172.21.15.78:0/2391255236 wait complete. 2023-10-10T17:41:24.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 -- 172.21.15.78:0/2177911518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c0ff470 msgr2=0x7f946c0ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 --2- 172.21.15.78:0/2177911518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c0ff470 0x7f946c0ff840 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f945c0099c0 tx=0x7f945c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 -- 172.21.15.78:0/2177911518 shutdown_connections 2023-10-10T17:41:24.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 --2- 172.21.15.78:0/2177911518 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f946c104510 0x7f946c10c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 --2- 172.21.15.78:0/2177911518 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c0ff470 0x7f946c0ff840 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 --2- 172.21.15.78:0/2177911518 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f946c0ffe10 0x7f946c103de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 -- 172.21.15.78:0/2177911518 >> 172.21.15.78:0/2177911518 conn(0x7f946c0770d0 msgr2=0x7f946c0774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:24.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 -- 172.21.15.78:0/2177911518 shutdown_connections 2023-10-10T17:41:24.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.516+0000 7f9472565700 1 -- 172.21.15.78:0/2177911518 wait complete. 2023-10-10T17:41:24.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.517+0000 7f9472565700 1 Processor -- start 2023-10-10T17:41:24.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.517+0000 7f9472565700 1 -- start start 2023-10-10T17:41:24.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9472565700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f946c0ff470 0x7f946c1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9472565700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f946c0ffe10 0x7f946c198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9472565700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c104510 0x7f946c19cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9472565700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c19d580 con 0x7f946c104510 2023-10-10T17:41:24.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9472565700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c19d6f0 con 0x7f946c0ff470 2023-10-10T17:41:24.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9472565700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c19d8b0 con 0x7f946c0ffe10 2023-10-10T17:41:24.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c104510 0x7f946c19cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c104510 0x7f946c19cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42854/0 (socket says 172.21.15.78:42854) 2023-10-10T17:41:24.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 -- 172.21.15.78:0/1486465420 learned_addr learned my addr 172.21.15.78:0/1486465420 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:24.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f946b7fe700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f946c0ffe10 0x7f946c198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f946bfff700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f946c0ff470 0x7f946c1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 -- 172.21.15.78:0/1486465420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f946c0ffe10 msgr2=0x7f946c198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f946c0ffe10 0x7f946c198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 -- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f946c0ff470 msgr2=0x7f946c1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f946c0ff470 0x7f946c1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 -- 172.21.15.78:0/1486465420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c0095f0 con 0x7f946c104510 2023-10-10T17:41:24.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.518+0000 7f9470b02700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c104510 0x7f946c19cf70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f946400ef30 tx=0x7f946400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.519+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9464018070 con 0x7f946c104510 2023-10-10T17:41:24.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.519+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f946400f040 con 0x7f946c104510 2023-10-10T17:41:24.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.519+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c110410 con 0x7f946c104510 2023-10-10T17:41:24.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.519+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f946400cdb0 con 0x7f946c104510 2023-10-10T17:41:24.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.519+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c110280 con 0x7f946c104510 2023-10-10T17:41:24.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.520+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f946c0fd700 con 0x7f946c104510 2023-10-10T17:41:24.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.521+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9464003a00 con 0x7f946c104510 2023-10-10T17:41:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.522+0000 7f94697fa700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94540a1150 0x7f94540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.522+0000 7f946bfff700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94540a1150 0x7f94540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.522+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f94640ccfe0 con 0x7f946c104510 2023-10-10T17:41:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.522+0000 7f94697fa700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] conn(0x7f94540abcc0 0x7f94540ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.523+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 --> [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94540ae7c0 con 0x7f94540abcc0 2023-10-10T17:41:24.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.523+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94640cd390 con 0x7f946c104510 2023-10-10T17:41:24.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.523+0000 7f946b7fe700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] conn(0x7f94540abcc0 0x7f94540ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.523+0000 7f946bfff700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94540a1150 0x7f94540a3600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f945c009990 tx=0x7f945c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.523+0000 7f946b7fe700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] conn(0x7f94540abcc0 0x7f94540ae0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.524+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== osd.10 v2:172.21.15.157:6824/936155371 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f94540ae7c0 con 0x7f94540abcc0 2023-10-10T17:41:24.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.541+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 --> [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f946c1105a0 con 0x7f94540abcc0 2023-10-10T17:41:24.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f94697fa700 1 -- 172.21.15.78:0/1486465420 <== osd.10 v2:172.21.15.157:6824/936155371 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f946c1105a0 con 0x7f94540abcc0 2023-10-10T17:41:24.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] conn(0x7f94540abcc0 msgr2=0x7f94540ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9472565700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] conn(0x7f94540abcc0 0x7f94540ae0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94540a1150 msgr2=0x7f94540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9472565700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94540a1150 0x7f94540a3600 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f945c009990 tx=0x7f945c01a040 comp rx=0 tx=0).stop 2023-10-10T17:41:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c104510 msgr2=0x7f946c19cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9472565700 1 --2- 172.21.15.78:0/1486465420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f946c104510 0x7f946c19cf70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f946400ef30 tx=0x7f946400c3a0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9470b02700 1 -- 172.21.15.78:0/1486465420 reap_dead start 2023-10-10T17:41:24.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.542+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 shutdown_connections 2023-10-10T17:41:24.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.543+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 >> 172.21.15.78:0/1486465420 conn(0x7f946c0770d0 msgr2=0x7f946c0fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:24.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.543+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 shutdown_connections 2023-10-10T17:41:24.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.543+0000 7f9472565700 1 -- 172.21.15.78:0/1486465420 wait complete. 2023-10-10T17:41:24.683 INFO:teuthology.orchestra.run.smithi078.stdout:77309411386 2023-10-10T17:41:24.684 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-10-10T17:41:24.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.871+0000 7f6962416700 1 -- 172.21.15.78:0/3397857588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f695c0ffe00 msgr2=0x7f695c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.871+0000 7f6962416700 1 --2- 172.21.15.78:0/3397857588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f695c0ffe00 0x7f695c103dd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f69440099c0 tx=0x7f6944009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.871+0000 7f6962416700 1 -- 172.21.15.78:0/3397857588 shutdown_connections 2023-10-10T17:41:24.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.871+0000 7f6962416700 1 --2- 172.21.15.78:0/3397857588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f695c0ff460 0x7f695c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.871+0000 7f6962416700 1 --2- 172.21.15.78:0/3397857588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f695c104500 0x7f695c10c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.871+0000 7f6962416700 1 --2- 172.21.15.78:0/3397857588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f695c0ffe00 0x7f695c103dd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.871+0000 7f6962416700 1 -- 172.21.15.78:0/3397857588 >> 172.21.15.78:0/3397857588 conn(0x7f695c0770a0 msgr2=0x7f695c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:24.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.872+0000 7f6962416700 1 -- 172.21.15.78:0/3397857588 shutdown_connections 2023-10-10T17:41:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.872+0000 7f6962416700 1 -- 172.21.15.78:0/3397857588 wait complete. 2023-10-10T17:41:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.872+0000 7f6962416700 1 Processor -- start 2023-10-10T17:41:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f6962416700 1 -- start start 2023-10-10T17:41:24.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f6962416700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f695c0ff460 0x7f695c10cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f6962416700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f695c0ffe00 0x7f695c10d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f6962416700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f695c104500 0x7f695c10d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f6962416700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c1a4840 con 0x7f695c104500 2023-10-10T17:41:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f6962416700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c1a49b0 con 0x7f695c0ffe00 2023-10-10T17:41:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f6962416700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695c1a4b70 con 0x7f695c0ff460 2023-10-10T17:41:24.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f695c0ff460 0x7f695c10cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f695c0ff460 0x7f695c10cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:34396/0 (socket says 172.21.15.78:34396) 2023-10-10T17:41:24.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 -- 172.21.15.78:0/4241503459 learned_addr learned my addr 172.21.15.78:0/4241503459 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:24.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f69609b3700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f695c104500 0x7f695c10d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f69537fe700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f695c0ffe00 0x7f695c10d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 -- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f695c0ffe00 msgr2=0x7f695c10d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f695c0ffe00 0x7f695c10d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 -- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f695c104500 msgr2=0x7f695c10d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f695c104500 0x7f695c10d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.873+0000 7f695bfff700 1 -- 172.21.15.78:0/4241503459 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69440095f0 con 0x7f695c0ff460 2023-10-10T17:41:24.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.874+0000 7f695bfff700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f695c0ff460 0x7f695c10cbe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f694c00ef10 tx=0x7f694c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.876+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c00ca70 con 0x7f695c0ff460 2023-10-10T17:41:24.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.876+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f694c00f040 con 0x7f695c0ff460 2023-10-10T17:41:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.876+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c014480 con 0x7f695c0ff460 2023-10-10T17:41:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.876+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f695c110340 con 0x7f695c0ff460 2023-10-10T17:41:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.877+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f695c1101b0 con 0x7f695c0ff460 2023-10-10T17:41:24.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.878+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f695c0fd700 con 0x7f695c0ff460 2023-10-10T17:41:24.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.879+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f694c007500 con 0x7f695c0ff460 2023-10-10T17:41:24.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.880+0000 7f6959ffb700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f69480a12a0 0x7f69480a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.880+0000 7f69537fe700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f69480a12a0 0x7f69480a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.881+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f694c0cd5b0 con 0x7f695c0ff460 2023-10-10T17:41:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.881+0000 7f6959ffb700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] conn(0x7f69480abe10 0x7f69480ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.881+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 --> [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69480ae910 con 0x7f69480abe10 2023-10-10T17:41:24.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.881+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f694c0cd920 con 0x7f695c0ff460 2023-10-10T17:41:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.881+0000 7f69537fe700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f69480a12a0 0x7f69480a3750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6944009fd0 tx=0x7f694401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.881+0000 7f69609b3700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] conn(0x7f69480abe10 0x7f69480ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:24.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.882+0000 7f69609b3700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] conn(0x7f69480abe10 0x7f69480ae240 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:24.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.882+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== osd.7 v2:172.21.15.157:6816/1149014920 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f69480ae910 con 0x7f69480abe10 2023-10-10T17:41:24.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.900+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 --> [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f695c1104d0 con 0x7f69480abe10 2023-10-10T17:41:24.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6959ffb700 1 -- 172.21.15.78:0/4241503459 <== osd.7 v2:172.21.15.157:6816/1149014920 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f695c1104d0 con 0x7f69480abe10 2023-10-10T17:41:24.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] conn(0x7f69480abe10 msgr2=0x7f69480ae240 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] conn(0x7f69480abe10 0x7f69480ae240 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f69480a12a0 msgr2=0x7f69480a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f69480a12a0 0x7f69480a3750 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6944009fd0 tx=0x7f694401a040 comp rx=0 tx=0).stop 2023-10-10T17:41:24.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f695c0ff460 msgr2=0x7f695c10cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:24.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 --2- 172.21.15.78:0/4241503459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f695c0ff460 0x7f695c10cbe0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f694c00ef10 tx=0x7f694c00c3a0 comp rx=0 tx=0).stop 2023-10-10T17:41:24.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f69609b3700 1 -- 172.21.15.78:0/4241503459 reap_dead start 2023-10-10T17:41:24.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 shutdown_connections 2023-10-10T17:41:24.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 >> 172.21.15.78:0/4241503459 conn(0x7f695c0770a0 msgr2=0x7f695c0fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:24.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.901+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 shutdown_connections 2023-10-10T17:41:24.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.902+0000 7f6962416700 1 -- 172.21.15.78:0/4241503459 wait complete. 2023-10-10T17:41:25.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 -- 172.21.15.78:0/3073803495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 msgr2=0x7f9db00ff830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 --2- 172.21.15.78:0/3073803495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 0x7f9db00ff830 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9da00099c0 tx=0x7f9da0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 -- 172.21.15.78:0/3073803495 shutdown_connections 2023-10-10T17:41:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 --2- 172.21.15.78:0/3073803495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db0104500 0x7f9db010c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 --2- 172.21.15.78:0/3073803495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 0x7f9db00ff830 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 --2- 172.21.15.78:0/3073803495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db00ffe00 0x7f9db0103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 -- 172.21.15.78:0/3073803495 >> 172.21.15.78:0/3073803495 conn(0x7f9db00770a0 msgr2=0x7f9db00774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 -- 172.21.15.78:0/3073803495 shutdown_connections 2023-10-10T17:41:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.181+0000 7f9db6139700 1 -- 172.21.15.78:0/3073803495 wait complete. 2023-10-10T17:41:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 Processor -- start 2023-10-10T17:41:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 -- start start 2023-10-10T17:41:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 0x7f9db0195330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db00ffe00 0x7f9db0195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db0104500 0x7f9db0199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db019a2f0 con 0x7f9db00ff460 2023-10-10T17:41:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db019a460 con 0x7f9db0104500 2023-10-10T17:41:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9db6139700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db019a620 con 0x7f9db00ffe00 2023-10-10T17:41:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9daf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 0x7f9db0195330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9daf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 0x7f9db0195330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42834/0 (socket says 172.21.15.78:42834) 2023-10-10T17:41:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.182+0000 7f9daf7fe700 1 -- 172.21.15.78:0/3490132732 learned_addr learned my addr 172.21.15.78:0/3490132732 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daeffd700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db00ffe00 0x7f9db0195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daffff700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db0104500 0x7f9db0199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daf7fe700 1 -- 172.21.15.78:0/3490132732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db00ffe00 msgr2=0x7f9db0195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daf7fe700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db00ffe00 0x7f9db0195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daf7fe700 1 -- 172.21.15.78:0/3490132732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db0104500 msgr2=0x7f9db0199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daf7fe700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db0104500 0x7f9db0199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daf7fe700 1 -- 172.21.15.78:0/3490132732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9da00095f0 con 0x7f9db00ff460 2023-10-10T17:41:25.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daeffd700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db00ffe00 0x7f9db0195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:25.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9daf7fe700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 0x7f9db0195330 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009990 tx=0x7f9da0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:25.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da001d070 con 0x7f9db00ff460 2023-10-10T17:41:25.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db010f250 con 0x7f9db00ff460 2023-10-10T17:41:25.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9da0003f90 con 0x7f9db00ff460 2023-10-10T17:41:25.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da0022ce0 con 0x7f9db00ff460 2023-10-10T17:41:25.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.183+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db010f0c0 con 0x7f9db00ff460 2023-10-10T17:41:25.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.184+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9db00fd770 con 0x7f9db00ff460 2023-10-10T17:41:25.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.186+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9da0022470 con 0x7f9db00ff460 2023-10-10T17:41:25.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.186+0000 7f9dacff9700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d980a1150 0x7f9d980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.186+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9da00ce220 con 0x7f9db00ff460 2023-10-10T17:41:25.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.186+0000 7f9daeffd700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d980a1150 0x7f9d980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.186+0000 7f9dacff9700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] conn(0x7f9d980abcc0 0x7f9d980ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.186+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 --> [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d980ae7c0 con 0x7f9d980abcc0 2023-10-10T17:41:25.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.186+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9da00ce630 con 0x7f9db00ff460 2023-10-10T17:41:25.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.187+0000 7f9daffff700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] conn(0x7f9d980abcc0 0x7f9d980ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.187+0000 7f9daeffd700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d980a1150 0x7f9d980a3600 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9db0196950 tx=0x7f9da400b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:25.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.187+0000 7f9daffff700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] conn(0x7f9d980abcc0 0x7f9d980ae0f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:25.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.188+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== osd.3 v2:172.21.15.78:6810/2152346860 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9d980ae7c0 con 0x7f9d980abcc0 2023-10-10T17:41:25.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.209+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 --> [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9db010f3e0 con 0x7f9d980abcc0 2023-10-10T17:41:25.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.209+0000 7f9dacff9700 1 -- 172.21.15.78:0/3490132732 <== osd.3 v2:172.21.15.78:6810/2152346860 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9db010f3e0 con 0x7f9d980abcc0 2023-10-10T17:41:25.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] conn(0x7f9d980abcc0 msgr2=0x7f9d980ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] conn(0x7f9d980abcc0 0x7f9d980ae0f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d980a1150 msgr2=0x7f9d980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d980a1150 0x7f9d980a3600 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9db0196950 tx=0x7f9da400b2c0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 msgr2=0x7f9db0195330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 --2- 172.21.15.78:0/3490132732 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db00ff460 0x7f9db0195330 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009990 tx=0x7f9da0009580 comp rx=0 tx=0).stop 2023-10-10T17:41:25.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9daffff700 1 -- 172.21.15.78:0/3490132732 reap_dead start 2023-10-10T17:41:25.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 shutdown_connections 2023-10-10T17:41:25.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 >> 172.21.15.78:0/3490132732 conn(0x7f9db00770a0 msgr2=0x7f9db00fcda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:25.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 shutdown_connections 2023-10-10T17:41:25.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:24.210+0000 7f9db6139700 1 -- 172.21.15.78:0/3490132732 wait complete. 2023-10-10T17:41:25.466 INFO:teuthology.orchestra.run.smithi078.stdout:55834574910 2023-10-10T17:41:25.467 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-10-10T17:41:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 -- 172.21.15.78:0/1226743747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73841013e0 msgr2=0x7f73841017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 --2- 172.21.15.78:0/1226743747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73841013e0 0x7f73841017b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7380009990 tx=0x7f7380009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 -- 172.21.15.78:0/1226743747 shutdown_connections 2023-10-10T17:41:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 --2- 172.21.15.78:0/1226743747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f738410a900 0x7f738410cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 --2- 172.21.15.78:0/1226743747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f73841013e0 0x7f73841017b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 --2- 172.21.15.78:0/1226743747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7384101cf0 0x7f738410a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 -- 172.21.15.78:0/1226743747 >> 172.21.15.78:0/1226743747 conn(0x7f73840770d0 msgr2=0x7f73840774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 -- 172.21.15.78:0/1226743747 shutdown_connections 2023-10-10T17:41:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.493+0000 7f738c7a2700 1 -- 172.21.15.78:0/1226743747 wait complete. 2023-10-10T17:41:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 Processor -- start 2023-10-10T17:41:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 -- start start 2023-10-10T17:41:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73841013e0 0x7f7384196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7384101cf0 0x7f73841969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738410a900 0x7f738419ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738419b440 con 0x7f738410a900 2023-10-10T17:41:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738419b5b0 con 0x7f7384101cf0 2023-10-10T17:41:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738c7a2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738419b770 con 0x7f73841013e0 2023-10-10T17:41:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738ad3f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738410a900 0x7f738419ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738ad3f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738410a900 0x7f738419ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42886/0 (socket says 172.21.15.78:42886) 2023-10-10T17:41:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f738ad3f700 1 -- 172.21.15.78:0/3059414121 learned_addr learned my addr 172.21.15.78:0/3059414121 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.494+0000 7f7389d3d700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7384101cf0 0x7f73841969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738a53e700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73841013e0 0x7f7384196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738ad3f700 1 -- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73841013e0 msgr2=0x7f7384196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738ad3f700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73841013e0 0x7f7384196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738ad3f700 1 -- 172.21.15.78:0/3059414121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7384101cf0 msgr2=0x7f73841969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738ad3f700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7384101cf0 0x7f73841969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738ad3f700 1 -- 172.21.15.78:0/3059414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73800095f0 con 0x7f738410a900 2023-10-10T17:41:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738a53e700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f73841013e0 0x7f7384196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f7389d3d700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7384101cf0 0x7f73841969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:41:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738ad3f700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738410a900 0x7f738419ae30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f737c009fd0 tx=0x7f737c00ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c018070 con 0x7f738410a900 2023-10-10T17:41:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7384110370 con 0x7f738410a900 2023-10-10T17:41:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f737c00f040 con 0x7f738410a900 2023-10-10T17:41:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c00cdb0 con 0x7f738410a900 2023-10-10T17:41:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.495+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73841101e0 con 0x7f738410a900 2023-10-10T17:41:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.497+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f73840fc750 con 0x7f738410a900 2023-10-10T17:41:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.497+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f737c004ad0 con 0x7f738410a900 2023-10-10T17:41:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f737b7fe700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73700a1150 0x7f73700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f737c0cc570 con 0x7f738410a900 2023-10-10T17:41:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f738a53e700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73700a1150 0x7f73700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f737b7fe700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] conn(0x7f73700abcc0 0x7f73700ae0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 --> [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73700ae7c0 con 0x7f73700abcc0 2023-10-10T17:41:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f737c0cebd0 con 0x7f738410a900 2023-10-10T17:41:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f738a53e700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73700a1150 0x7f73700a3600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f738000b2e0 tx=0x7f7380005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.498+0000 7f7389d3d700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] conn(0x7f73700abcc0 0x7f73700ae0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.499+0000 7f7389d3d700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] conn(0x7f73700abcc0 0x7f73700ae0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:25.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.500+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== osd.5 v2:172.21.15.190:6808/934325798 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f73700ae7c0 con 0x7f73700abcc0 2023-10-10T17:41:25.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.516+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 --> [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7384110500 con 0x7f73700abcc0 2023-10-10T17:41:25.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f737b7fe700 1 -- 172.21.15.78:0/3059414121 <== osd.5 v2:172.21.15.190:6808/934325798 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7384110500 con 0x7f73700abcc0 2023-10-10T17:41:25.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] conn(0x7f73700abcc0 msgr2=0x7f73700ae0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] conn(0x7f73700abcc0 0x7f73700ae0f0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73700a1150 msgr2=0x7f73700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f73700a1150 0x7f73700a3600 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f738000b2e0 tx=0x7f7380005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738410a900 msgr2=0x7f738419ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 --2- 172.21.15.78:0/3059414121 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f738410a900 0x7f738419ae30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f737c009fd0 tx=0x7f737c00ef90 comp rx=0 tx=0).stop 2023-10-10T17:41:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738ad3f700 1 -- 172.21.15.78:0/3059414121 reap_dead start 2023-10-10T17:41:25.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 shutdown_connections 2023-10-10T17:41:25.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 >> 172.21.15.78:0/3059414121 conn(0x7f73840770d0 msgr2=0x7f73840fbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:25.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 shutdown_connections 2023-10-10T17:41:25.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:25.517+0000 7f738c7a2700 1 -- 172.21.15.78:0/3059414121 wait complete. 2023-10-10T17:41:25.608 INFO:teuthology.orchestra.run.smithi078.stdout:257698037793 2023-10-10T17:41:25.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.8 2023-10-10T17:41:25.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:25 smithi078 ceph-mon[99494]: pgmap v283: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:25.879 INFO:teuthology.orchestra.run.smithi078.stdout:313532612631 2023-10-10T17:41:25.879 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.10 2023-10-10T17:41:25.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:25 smithi190 ceph-mon[101220]: pgmap v283: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:25.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:25 smithi157 ceph-mon[100896]: pgmap v283: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:26.475 INFO:teuthology.orchestra.run.smithi078.stdout:223338299429 2023-10-10T17:41:26.475 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-10-10T17:41:26.711 INFO:teuthology.orchestra.run.smithi078.stdout:167503724590 2023-10-10T17:41:26.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-10-10T17:41:26.748 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:26.795 INFO:teuthology.orchestra.run.smithi078.stdout:111669149749 2023-10-10T17:41:26.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-10-10T17:41:26.873 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:26.876 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:26.922 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:27.614 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:27.615 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:27 smithi078 ceph-mon[99494]: pgmap v284: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-10-10T17:41:27.626 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:27.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:27 smithi190 ceph-mon[101220]: pgmap v284: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-10-10T17:41:27.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:27 smithi157 ceph-mon[100896]: pgmap v284: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-10-10T17:41:28.004 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:28.005 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:28.434 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:29.811 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:29 smithi078 ceph-mon[99494]: pgmap v285: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-10-10T17:41:29.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:29 smithi190 ceph-mon[101220]: pgmap v285: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-10-10T17:41:29.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:29 smithi157 ceph-mon[100896]: pgmap v285: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-10-10T17:41:30.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 -- 172.21.15.78:0/2407372556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 msgr2=0x7f013c103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 --2- 172.21.15.78:0/2407372556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c103dd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f01300099c0 tx=0x7f0130009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 -- 172.21.15.78:0/2407372556 shutdown_connections 2023-10-10T17:41:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 --2- 172.21.15.78:0/2407372556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f013c104500 0x7f013c10c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 --2- 172.21.15.78:0/2407372556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f013c0ff460 0x7f013c0ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 --2- 172.21.15.78:0/2407372556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c103dd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 -- 172.21.15.78:0/2407372556 >> 172.21.15.78:0/2407372556 conn(0x7f013c0770a0 msgr2=0x7f013c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 -- 172.21.15.78:0/2407372556 shutdown_connections 2023-10-10T17:41:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.088+0000 7f0140bf5700 1 -- 172.21.15.78:0/2407372556 wait complete. 2023-10-10T17:41:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 Processor -- start 2023-10-10T17:41:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 -- start start 2023-10-10T17:41:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f013c0ff460 0x7f013c197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c1977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f013c104500 0x7f013c1a22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c1980b0 con 0x7f013c0ffe00 2023-10-10T17:41:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c1a2820 con 0x7f013c104500 2023-10-10T17:41:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0140bf5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013c1a2990 con 0x7f013c0ff460 2023-10-10T17:41:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c1977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c1977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42900/0 (socket says 172.21.15.78:42900) 2023-10-10T17:41:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 -- 172.21.15.78:0/614671538 learned_addr learned my addr 172.21.15.78:0/614671538 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f013a59c700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f013c0ff460 0x7f013c197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 -- 172.21.15.78:0/614671538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f013c0ff460 msgr2=0x7f013c197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f013ad9d700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f013c104500 0x7f013c1a22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f013c0ff460 0x7f013c197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 -- 172.21.15.78:0/614671538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f013c104500 msgr2=0x7f013c1a22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f013c104500 0x7f013c1a22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 -- 172.21.15.78:0/614671538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01300095f0 con 0x7f013c0ffe00 2023-10-10T17:41:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f013ad9d700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f013c104500 0x7f013c1a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f0139d9b700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c1977a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f0130005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.089+0000 7f012b7fe700 1 -- 172.21.15.78:0/614671538 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013001d070 con 0x7f013c0ffe00 2023-10-10T17:41:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.090+0000 7f012b7fe700 1 -- 172.21.15.78:0/614671538 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0130004090 con 0x7f013c0ffe00 2023-10-10T17:41:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.090+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013c10f230 con 0x7f013c0ffe00 2023-10-10T17:41:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.090+0000 7f012b7fe700 1 -- 172.21.15.78:0/614671538 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130022d20 con 0x7f013c0ffe00 2023-10-10T17:41:30.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.090+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013c10f0a0 con 0x7f013c0ffe00 2023-10-10T17:41:30.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.091+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013c10f3c0 con 0x7f013c0ffe00 2023-10-10T17:41:30.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.091+0000 7f012b7fe700 1 -- 172.21.15.78:0/614671538 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0130022470 con 0x7f013c0ffe00 2023-10-10T17:41:30.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.092+0000 7f012b7fe700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f01240a1150 0x7f01240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:30.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.092+0000 7f013a59c700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f01240a1150 0x7f01240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:30.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.092+0000 7f012b7fe700 1 -- 172.21.15.78:0/614671538 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f01300cdca0 con 0x7f013c0ffe00 2023-10-10T17:41:30.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.093+0000 7f013a59c700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f01240a1150 0x7f01240a3600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f012c005950 tx=0x7f012c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:30.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.096+0000 7f012b7fe700 1 -- 172.21.15.78:0/614671538 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f013005c730 con 0x7f013c0ffe00 2023-10-10T17:41:30.211 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.304+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f013c066ea0 con 0x7f013c0ffe00 2023-10-10T17:41:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.304+0000 7f012b7fe700 1 -- 172.21.15.78:0/614671538 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0130097290 con 0x7f013c0ffe00 2023-10-10T17:41:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f01240a1150 msgr2=0x7f01240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:30.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f01240a1150 0x7f01240a3600 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f012c005950 tx=0x7f012c0058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 msgr2=0x7f013c1977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:30.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c1977a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0130009990 tx=0x7f0130005e80 comp rx=0 tx=0).stop 2023-10-10T17:41:30.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 shutdown_connections 2023-10-10T17:41:30.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f01240a1150 0x7f01240a3600 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f013c0ff460 0x7f013c197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f013c104500 0x7f013c1a22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 --2- 172.21.15.78:0/614671538 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f013c0ffe00 0x7f013c1977a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 >> 172.21.15.78:0/614671538 conn(0x7f013c0770a0 msgr2=0x7f013c0fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:30.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 shutdown_connections 2023-10-10T17:41:30.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.306+0000 7f0140bf5700 1 -- 172.21.15.78:0/614671538 wait complete. 2023-10-10T17:41:30.315 INFO:teuthology.orchestra.run.smithi078.stdout:292057776157 2023-10-10T17:41:30.737 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:30 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/614671538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-10-10T17:41:30.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:30 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/614671538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-10-10T17:41:30.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:30 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/614671538' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-10-10T17:41:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 -- 172.21.15.78:0/4022799321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 msgr2=0x7f9514104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 --2- 172.21.15.78:0/4022799321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f9514104d70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9504009990 tx=0x7f9504009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 -- 172.21.15.78:0/4022799321 shutdown_connections 2023-10-10T17:41:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 --2- 172.21.15.78:0/4022799321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95140ff350 0x7f951410b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 --2- 172.21.15.78:0/4022799321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f9514104d70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 --2- 172.21.15.78:0/4022799321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95140fe9a0 0x7f95140fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 -- 172.21.15.78:0/4022799321 >> 172.21.15.78:0/4022799321 conn(0x7f95140758a0 msgr2=0x7f9514075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:31.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 -- 172.21.15.78:0/4022799321 shutdown_connections 2023-10-10T17:41:31.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 -- 172.21.15.78:0/4022799321 wait complete. 2023-10-10T17:41:31.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 Processor -- start 2023-10-10T17:41:31.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.153+0000 7f951a902700 1 -- start start 2023-10-10T17:41:31.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f951a902700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95140fe9a0 0x7f9514196380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f951a902700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95140ff350 0x7f95141968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f951a902700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f951419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f951a902700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951419b340 con 0x7f95141049a0 2023-10-10T17:41:31.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f951a902700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951419b4b0 con 0x7f95140fe9a0 2023-10-10T17:41:31.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f951a902700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951419b670 con 0x7f95140ff350 2023-10-10T17:41:31.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f951419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:31.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f951419ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42930/0 (socket says 172.21.15.78:42930) 2023-10-10T17:41:31.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 -- 172.21.15.78:0/1580732784 learned_addr learned my addr 172.21.15.78:0/1580732784 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:31.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9513fff700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95140fe9a0 0x7f9514196380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:31.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f95137fe700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95140ff350 0x7f95141968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:31.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 -- 172.21.15.78:0/1580732784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95140ff350 msgr2=0x7f95141968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95140ff350 0x7f95141968c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 -- 172.21.15.78:0/1580732784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95140fe9a0 msgr2=0x7f9514196380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95140fe9a0 0x7f9514196380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 -- 172.21.15.78:0/1580732784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95040095f0 con 0x7f95141049a0 2023-10-10T17:41:31.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9513fff700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95140fe9a0 0x7f9514196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:31.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.154+0000 7f9518e9f700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f951419ad30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f950c00ada0 tx=0x7f950c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:31.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.155+0000 7f95117fa700 1 -- 172.21.15.78:0/1580732784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c00f070 con 0x7f95141049a0 2023-10-10T17:41:31.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.155+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951410e9d0 con 0x7f95141049a0 2023-10-10T17:41:31.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.155+0000 7f95117fa700 1 -- 172.21.15.78:0/1580732784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f950c007b30 con 0x7f95141049a0 2023-10-10T17:41:31.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.155+0000 7f95117fa700 1 -- 172.21.15.78:0/1580732784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c005440 con 0x7f95141049a0 2023-10-10T17:41:31.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.155+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951410e840 con 0x7f95141049a0 2023-10-10T17:41:31.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.156+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951410eb60 con 0x7f95141049a0 2023-10-10T17:41:31.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.157+0000 7f95117fa700 1 -- 172.21.15.78:0/1580732784 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f950c004ad0 con 0x7f95141049a0 2023-10-10T17:41:31.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.157+0000 7f95117fa700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94fc0a1150 0x7f94fc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.158+0000 7f9513fff700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94fc0a1150 0x7f94fc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:31.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.158+0000 7f95117fa700 1 -- 172.21.15.78:0/1580732784 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f950c0cc7a0 con 0x7f95141049a0 2023-10-10T17:41:31.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.158+0000 7f9513fff700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94fc0a1150 0x7f94fc0a3600 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9504006010 tx=0x7f950400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:31.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.161+0000 7f95117fa700 1 -- 172.21.15.78:0/1580732784 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f950c095d90 con 0x7f95141049a0 2023-10-10T17:41:31.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.362+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9514066ea0 con 0x7f95141049a0 2023-10-10T17:41:31.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.362+0000 7f95117fa700 1 -- 172.21.15.78:0/1580732784 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f950c0993b0 con 0x7f95141049a0 2023-10-10T17:41:31.597 INFO:teuthology.orchestra.run.smithi078.stdout:201863462955 2023-10-10T17:41:31.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94fc0a1150 msgr2=0x7f94fc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94fc0a1150 0x7f94fc0a3600 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9504006010 tx=0x7f950400b540 comp rx=0 tx=0).stop 2023-10-10T17:41:31.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 msgr2=0x7f951419ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f951419ad30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f950c00ada0 tx=0x7f950c00c330 comp rx=0 tx=0).stop 2023-10-10T17:41:31.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 shutdown_connections 2023-10-10T17:41:31.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f94fc0a1150 0x7f94fc0a3600 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95141049a0 0x7f951419ad30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f95140fe9a0 0x7f9514196380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 --2- 172.21.15.78:0/1580732784 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f95140ff350 0x7f95141968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 >> 172.21.15.78:0/1580732784 conn(0x7f95140758a0 msgr2=0x7f95140fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:31.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 shutdown_connections 2023-10-10T17:41:31.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:31.364+0000 7f951a902700 1 -- 172.21.15.78:0/1580732784 wait complete. 2023-10-10T17:41:31.601 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776156 got 292057776157 for osd.9 2023-10-10T17:41:31.601 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:31.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 -- 172.21.15.78:0/3165415865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28081045e0 msgr2=0x7f280810c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 --2- 172.21.15.78:0/3165415865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28081045e0 0x7f280810c160 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f280000b0d0 tx=0x7f280000b3e0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 -- 172.21.15.78:0/3165415865 shutdown_connections 2023-10-10T17:41:31.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 --2- 172.21.15.78:0/3165415865 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28081045e0 0x7f280810c160 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 --2- 172.21.15.78:0/3165415865 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28080ff540 0x7f28080ff910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 --2- 172.21.15.78:0/3165415865 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28080ffee0 0x7f2808103eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 -- 172.21.15.78:0/3165415865 >> 172.21.15.78:0/3165415865 conn(0x7f2808075010 msgr2=0x7f2808075410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:31.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 -- 172.21.15.78:0/3165415865 shutdown_connections 2023-10-10T17:41:31.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 -- 172.21.15.78:0/3165415865 wait complete. 2023-10-10T17:41:31.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.504+0000 7f280e077700 1 Processor -- start 2023-10-10T17:41:31.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f280e077700 1 -- start start 2023-10-10T17:41:31.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f280e077700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28080ff540 0x7f2808197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f280e077700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28080ffee0 0x7f2808197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f280e077700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28081045e0 0x7f28081a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f280e077700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2808198180 con 0x7f28080ffee0 2023-10-10T17:41:31.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f280e077700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28081a28f0 con 0x7f28081045e0 2023-10-10T17:41:31.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f280e077700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28081a2a60 con 0x7f28080ff540 2023-10-10T17:41:31.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28080ffee0 0x7f2808197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28080ffee0 0x7f2808197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42914/0 (socket says 172.21.15.78:42914) 2023-10-10T17:41:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 -- 172.21.15.78:0/2669185289 learned_addr learned my addr 172.21.15.78:0/2669185289 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f28077fe700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28080ff540 0x7f2808197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 -- 172.21.15.78:0/2669185289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28080ff540 msgr2=0x7f2808197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28080ff540 0x7f2808197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 -- 172.21.15.78:0/2669185289 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28081045e0 msgr2=0x7f28081a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28081045e0 0x7f28081a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 -- 172.21.15.78:0/2669185289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2800009d00 con 0x7f28080ffee0 2023-10-10T17:41:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2806ffd700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28080ffee0 0x7f2808197870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00d980 tx=0x7f27fc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:31.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.505+0000 7f2804ff9700 1 -- 172.21.15.78:0/2669185289 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc009b40 con 0x7f28080ffee0 2023-10-10T17:41:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.506+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280810f340 con 0x7f28080ffee0 2023-10-10T17:41:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.506+0000 7f2804ff9700 1 -- 172.21.15.78:0/2669185289 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27fc00de60 con 0x7f28080ffee0 2023-10-10T17:41:31.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.506+0000 7f2804ff9700 1 -- 172.21.15.78:0/2669185289 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27fc00f620 con 0x7f28080ffee0 2023-10-10T17:41:31.744 INFO:teuthology.orchestra.run.smithi078.stdout:343597383701 2023-10-10T17:41:31.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.506+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f280810f1b0 con 0x7f28080ffee0 2023-10-10T17:41:31.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.506+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280810f4d0 con 0x7f28080ffee0 2023-10-10T17:41:31.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.507+0000 7f2804ff9700 1 -- 172.21.15.78:0/2669185289 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f27fc010460 con 0x7f28080ffee0 2023-10-10T17:41:31.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.508+0000 7f2804ff9700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f27f00a11d0 0x7f27f00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.508+0000 7f2804ff9700 1 -- 172.21.15.78:0/2669185289 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f27fc0cc6b0 con 0x7f28080ffee0 2023-10-10T17:41:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.508+0000 7f28077fe700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f27f00a11d0 0x7f27f00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:31.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.508+0000 7f28077fe700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f27f00a11d0 0x7f27f00a3680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f27f8005fd0 tx=0x7f27f8005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.511+0000 7f2804ff9700 1 -- 172.21.15.78:0/2669185289 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f27fc095bb0 con 0x7f28080ffee0 2023-10-10T17:41:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.709+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f28080696b0 con 0x7f28080ffee0 2023-10-10T17:41:31.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.709+0000 7f2804ff9700 1 -- 172.21.15.78:0/2669185289 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f27fc017020 con 0x7f28080ffee0 2023-10-10T17:41:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f27f00a11d0 msgr2=0x7f27f00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f27f00a11d0 0x7f27f00a3680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f27f8005fd0 tx=0x7f27f8005f60 comp rx=0 tx=0).stop 2023-10-10T17:41:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28080ffee0 msgr2=0x7f2808197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28080ffee0 0x7f2808197870 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f27fc00d980 tx=0x7f27fc00dd40 comp rx=0 tx=0).stop 2023-10-10T17:41:31.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 shutdown_connections 2023-10-10T17:41:31.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f27f00a11d0 0x7f27f00a3680 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f28080ff540 0x7f2808197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f28081045e0 0x7f28081a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 --2- 172.21.15.78:0/2669185289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28080ffee0 0x7f2808197870 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:31.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 >> 172.21.15.78:0/2669185289 conn(0x7f2808075010 msgr2=0x7f28080fce20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:31.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 shutdown_connections 2023-10-10T17:41:31.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:30.711+0000 7f280e077700 1 -- 172.21.15.78:0/2669185289 wait complete. 2023-10-10T17:41:31.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:31 smithi078 ceph-mon[99494]: pgmap v286: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:31.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:31 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2669185289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-10-10T17:41:31.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:31 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1580732784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-10T17:41:31.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:31 smithi190 ceph-mon[101220]: pgmap v286: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:31.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:31 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2669185289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-10-10T17:41:31.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:31 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1580732784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-10T17:41:31.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:31 smithi157 ceph-mon[100896]: pgmap v286: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:31 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2669185289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-10-10T17:41:31.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:31 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1580732784' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-10T17:41:32.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 -- 172.21.15.78:0/759631961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 msgr2=0x7f5dbc103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:32.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 --2- 172.21.15.78:0/759631961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc103e40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5db00099c0 tx=0x7f5db0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:32.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 -- 172.21.15.78:0/759631961 shutdown_connections 2023-10-10T17:41:32.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 --2- 172.21.15.78:0/759631961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dbc104570 0x7f5dbc10c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:32.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 --2- 172.21.15.78:0/759631961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dbc0ffba0 0x7f5dbc0fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:32.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 --2- 172.21.15.78:0/759631961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc103e40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:32.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 -- 172.21.15.78:0/759631961 >> 172.21.15.78:0/759631961 conn(0x7f5dbc0758a0 msgr2=0x7f5dbc075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:32.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 -- 172.21.15.78:0/759631961 shutdown_connections 2023-10-10T17:41:32.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.925+0000 7f5dc2577700 1 -- 172.21.15.78:0/759631961 wait complete. 2023-10-10T17:41:32.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.926+0000 7f5dc2577700 1 Processor -- start 2023-10-10T17:41:32.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.926+0000 7f5dc2577700 1 -- start start 2023-10-10T17:41:32.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.926+0000 7f5dc2577700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dbc0ffba0 0x7f5dbc197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:32.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.926+0000 7f5dc2577700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:32.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.926+0000 7f5dc2577700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dbc104570 0x7f5dbc1a2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:32.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dc2577700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc198080 con 0x7f5dbc100540 2023-10-10T17:41:32.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dc2577700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc1a27c0 con 0x7f5dbc0ffba0 2023-10-10T17:41:32.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dc2577700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dbc1a2980 con 0x7f5dbc104570 2023-10-10T17:41:32.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:32.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42940/0 (socket says 172.21.15.78:42940) 2023-10-10T17:41:32.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 -- 172.21.15.78:0/2281863311 learned_addr learned my addr 172.21.15.78:0/2281863311 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:32.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbbfff700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dbc0ffba0 0x7f5dbc197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:32.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dc0b14700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dbc104570 0x7f5dbc1a2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:32.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 -- 172.21.15.78:0/2281863311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dbc104570 msgr2=0x7f5dbc1a2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:32.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dbc104570 0x7f5dbc1a2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:32.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 -- 172.21.15.78:0/2281863311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dbc0ffba0 msgr2=0x7f5dbc197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:32.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dbc0ffba0 0x7f5dbc197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:32.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 -- 172.21.15.78:0/2281863311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db00095f0 con 0x7f5dbc100540 2023-10-10T17:41:32.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dc0b14700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dbc104570 0x7f5dbc1a2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:32.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dbb7fe700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc197740 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5db0006010 tx=0x7f5db0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:32.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5db97fa700 1 -- 172.21.15.78:0/2281863311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db001d070 con 0x7f5dbc100540 2023-10-10T17:41:32.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dbc10f330 con 0x7f5dbc100540 2023-10-10T17:41:32.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5db97fa700 1 -- 172.21.15.78:0/2281863311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5db0004010 con 0x7f5dbc100540 2023-10-10T17:41:32.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5db97fa700 1 -- 172.21.15.78:0/2281863311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0022e50 con 0x7f5dbc100540 2023-10-10T17:41:32.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.927+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dbc10f1a0 con 0x7f5dbc100540 2023-10-10T17:41:32.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.929+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dbc10f4c0 con 0x7f5dbc100540 2023-10-10T17:41:32.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.930+0000 7f5db97fa700 1 -- 172.21.15.78:0/2281863311 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5db0022470 con 0x7f5dbc100540 2023-10-10T17:41:32.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.931+0000 7f5db97fa700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5da40a1150 0x7f5da40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:32.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.931+0000 7f5dbbfff700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5da40a1150 0x7f5da40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:32.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.931+0000 7f5db97fa700 1 -- 172.21.15.78:0/2281863311 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5db00cdc90 con 0x7f5dbc100540 2023-10-10T17:41:32.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.931+0000 7f5dbbfff700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5da40a1150 0x7f5da40a3600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5dac005950 tx=0x7f5dac00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:32.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:32.935+0000 7f5db97fa700 1 -- 172.21.15.78:0/2281863311 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5db005c730 con 0x7f5dbc100540 2023-10-10T17:41:33.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.147+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f5dbc1a3390 con 0x7f5dbc100540 2023-10-10T17:41:33.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.147+0000 7f5db97fa700 1 -- 172.21.15.78:0/2281863311 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5db0027020 con 0x7f5dbc100540 2023-10-10T17:41:33.150 INFO:teuthology.orchestra.run.smithi078.stdout:257698037795 2023-10-10T17:41:33.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5da40a1150 msgr2=0x7f5da40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5da40a1150 0x7f5da40a3600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5dac005950 tx=0x7f5dac00a300 comp rx=0 tx=0).stop 2023-10-10T17:41:33.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 msgr2=0x7f5dbc197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc197740 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f5db0006010 tx=0x7f5db0005740 comp rx=0 tx=0).stop 2023-10-10T17:41:33.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 shutdown_connections 2023-10-10T17:41:33.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f5da40a1150 0x7f5da40a3600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5dbc104570 0x7f5dbc1a2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5dbc0ffba0 0x7f5dbc197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 --2- 172.21.15.78:0/2281863311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5dbc100540 0x7f5dbc197740 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 >> 172.21.15.78:0/2281863311 conn(0x7f5dbc0758a0 msgr2=0x7f5dbc0fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:33.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 shutdown_connections 2023-10-10T17:41:33.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.150+0000 7f5dc2577700 1 -- 172.21.15.78:0/2281863311 wait complete. 2023-10-10T17:41:33.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 -- 172.21.15.78:0/3438717112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 msgr2=0x7f6260103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 --2- 172.21.15.78:0/3438717112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260103dd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f62540099c0 tx=0x7f6254009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 -- 172.21.15.78:0/3438717112 shutdown_connections 2023-10-10T17:41:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 --2- 172.21.15.78:0/3438717112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6260104500 0x7f626010c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 --2- 172.21.15.78:0/3438717112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62600ff460 0x7f62600ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 --2- 172.21.15.78:0/3438717112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260103dd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 -- 172.21.15.78:0/3438717112 >> 172.21.15.78:0/3438717112 conn(0x7f62600770a0 msgr2=0x7f62600774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 -- 172.21.15.78:0/3438717112 shutdown_connections 2023-10-10T17:41:33.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.317+0000 7f6265b67700 1 -- 172.21.15.78:0/3438717112 wait complete. 2023-10-10T17:41:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 Processor -- start 2023-10-10T17:41:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 -- start start 2023-10-10T17:41:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62600ff460 0x7f62601952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6260104500 0x7f6260199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626019a2b0 con 0x7f62600ffe00 2023-10-10T17:41:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626019a420 con 0x7f62600ff460 2023-10-10T17:41:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f6265b67700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f626019a5e0 con 0x7f6260104500 2023-10-10T17:41:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f625effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f625effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42968/0 (socket says 172.21.15.78:42968) 2023-10-10T17:41:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.318+0000 7f625effd700 1 -- 172.21.15.78:0/22944447 learned_addr learned my addr 172.21.15.78:0/22944447 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625f7fe700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62600ff460 0x7f62601952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625ffff700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6260104500 0x7f6260199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625effd700 1 -- 172.21.15.78:0/22944447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6260104500 msgr2=0x7f6260199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625effd700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6260104500 0x7f6260199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625effd700 1 -- 172.21.15.78:0/22944447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62600ff460 msgr2=0x7f62601952f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625effd700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62600ff460 0x7f62601952f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625effd700 1 -- 172.21.15.78:0/22944447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62540095f0 con 0x7f62600ffe00 2023-10-10T17:41:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625ffff700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6260104500 0x7f6260199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625effd700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260195830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6254009fd0 tx=0x7f6254009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625cff9700 1 -- 172.21.15.78:0/22944447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f625401d070 con 0x7f62600ffe00 2023-10-10T17:41:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625cff9700 1 -- 172.21.15.78:0/22944447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6254005030 con 0x7f62600ffe00 2023-10-10T17:41:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f625cff9700 1 -- 172.21.15.78:0/22944447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6254004b10 con 0x7f62600ffe00 2023-10-10T17:41:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f626010f230 con 0x7f62600ffe00 2023-10-10T17:41:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.319+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f626010f0a0 con 0x7f62600ffe00 2023-10-10T17:41:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.320+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f626019ab20 con 0x7f62600ffe00 2023-10-10T17:41:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.321+0000 7f625cff9700 1 -- 172.21.15.78:0/22944447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f62540054c0 con 0x7f62600ffe00 2023-10-10T17:41:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.322+0000 7f625cff9700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62480a1150 0x7f62480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.322+0000 7f625f7fe700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62480a1150 0x7f62480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.322+0000 7f625cff9700 1 -- 172.21.15.78:0/22944447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f62540cd860 con 0x7f62600ffe00 2023-10-10T17:41:33.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.322+0000 7f625f7fe700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62480a1150 0x7f62480a3600 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6250009c60 tx=0x7f6250009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:33.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.326+0000 7f625cff9700 1 -- 172.21.15.78:0/22944447 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6254096e50 con 0x7f62600ffe00 2023-10-10T17:41:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 -- 172.21.15.78:0/352913240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 msgr2=0x7ff988103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 --2- 172.21.15.78:0/352913240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff988103dd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ff9700099c0 tx=0x7ff970009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 -- 172.21.15.78:0/352913240 shutdown_connections 2023-10-10T17:41:33.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 --2- 172.21.15.78:0/352913240 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff988104500 0x7ff98810c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 --2- 172.21.15.78:0/352913240 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9880ff460 0x7ff9880ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 --2- 172.21.15.78:0/352913240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff988103dd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 -- 172.21.15.78:0/352913240 >> 172.21.15.78:0/352913240 conn(0x7ff9880770a0 msgr2=0x7ff9880774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 -- 172.21.15.78:0/352913240 shutdown_connections 2023-10-10T17:41:33.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.185+0000 7ff98dc40700 1 -- 172.21.15.78:0/352913240 wait complete. 2023-10-10T17:41:33.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 Processor -- start 2023-10-10T17:41:33.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 -- start start 2023-10-10T17:41:33.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9880ff460 0x7ff988195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff9881957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff988104500 0x7ff988199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98819a230 con 0x7ff9880ffe00 2023-10-10T17:41:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98819a3a0 con 0x7ff9880ff460 2023-10-10T17:41:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.186+0000 7ff98dc40700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98819a560 con 0x7ff988104500 2023-10-10T17:41:33.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff9881957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff9881957b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42954/0 (socket says 172.21.15.78:42954) 2023-10-10T17:41:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 -- 172.21.15.78:0/1324235469 learned_addr learned my addr 172.21.15.78:0/1324235469 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff987fff700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff988104500 0x7ff988199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 -- 172.21.15.78:0/1324235469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff988104500 msgr2=0x7ff988199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff988104500 0x7ff988199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff9877fe700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9880ff460 0x7ff988195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 -- 172.21.15.78:0/1324235469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9880ff460 msgr2=0x7ff988195270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9880ff460 0x7ff988195270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 -- 172.21.15.78:0/1324235469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9700095f0 con 0x7ff9880ffe00 2023-10-10T17:41:33.507 INFO:teuthology.orchestra.run.smithi078.stdout:313532612632 2023-10-10T17:41:33.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff9877fe700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9880ff460 0x7ff988195270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:33.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff986ffd700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff9881957b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff970009f50 tx=0x7ff970005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:33.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff984ff9700 1 -- 172.21.15.78:0/1324235469 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97001d070 con 0x7ff9880ffe00 2023-10-10T17:41:33.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff984ff9700 1 -- 172.21.15.78:0/1324235469 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff970004010 con 0x7ff9880ffe00 2023-10-10T17:41:33.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff98810f230 con 0x7ff9880ffe00 2023-10-10T17:41:33.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff984ff9700 1 -- 172.21.15.78:0/1324235469 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff970022dd0 con 0x7ff9880ffe00 2023-10-10T17:41:33.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.187+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff98810f0a0 con 0x7ff9880ffe00 2023-10-10T17:41:33.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.188+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff98810f3c0 con 0x7ff9880ffe00 2023-10-10T17:41:33.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.189+0000 7ff984ff9700 1 -- 172.21.15.78:0/1324235469 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff970022500 con 0x7ff9880ffe00 2023-10-10T17:41:33.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.190+0000 7ff984ff9700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9740a1150 0x7ff9740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:33.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.190+0000 7ff9877fe700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9740a1150 0x7ff9740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:33.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.190+0000 7ff984ff9700 1 -- 172.21.15.78:0/1324235469 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff9700ce5c0 con 0x7ff9880ffe00 2023-10-10T17:41:33.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.191+0000 7ff9877fe700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9740a1150 0x7ff9740a3600 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff9800094f0 tx=0x7ff980009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:33.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.194+0000 7ff984ff9700 1 -- 172.21.15.78:0/1324235469 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff970097bb0 con 0x7ff9880ffe00 2023-10-10T17:41:33.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.405+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7ff9881a22f0 con 0x7ff9880ffe00 2023-10-10T17:41:33.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.405+0000 7ff984ff9700 1 -- 172.21.15.78:0/1324235469 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7ff97009b1d0 con 0x7ff9880ffe00 2023-10-10T17:41:33.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9740a1150 msgr2=0x7ff9740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9740a1150 0x7ff9740a3600 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff9800094f0 tx=0x7ff980009040 comp rx=0 tx=0).stop 2023-10-10T17:41:33.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 msgr2=0x7ff9881957b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff9881957b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7ff970009f50 tx=0x7ff970005ef0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 shutdown_connections 2023-10-10T17:41:33.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff9740a1150 0x7ff9740a3600 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff988104500 0x7ff988199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff9880ff460 0x7ff988195270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 --2- 172.21.15.78:0/1324235469 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9880ffe00 0x7ff9881957b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 >> 172.21.15.78:0/1324235469 conn(0x7ff9880770a0 msgr2=0x7ff9880fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:33.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 shutdown_connections 2023-10-10T17:41:33.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.407+0000 7ff98dc40700 1 -- 172.21.15.78:0/1324235469 wait complete. 2023-10-10T17:41:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.537+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f626010f3c0 con 0x7f62600ffe00 2023-10-10T17:41:33.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.538+0000 7f625cff9700 1 -- 172.21.15.78:0/22944447 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f625409a470 con 0x7f62600ffe00 2023-10-10T17:41:33.540 INFO:teuthology.orchestra.run.smithi078.stdout:223338299430 2023-10-10T17:41:33.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62480a1150 msgr2=0x7f62480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62480a1150 0x7f62480a3600 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6250009c60 tx=0x7f6250009260 comp rx=0 tx=0).stop 2023-10-10T17:41:33.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 msgr2=0x7f6260195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:33.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260195830 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6254009fd0 tx=0x7f6254009580 comp rx=0 tx=0).stop 2023-10-10T17:41:33.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 shutdown_connections 2023-10-10T17:41:33.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f62480a1150 0x7f62480a3600 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6260104500 0x7f6260199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f62600ff460 0x7f62601952f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 --2- 172.21.15.78:0/22944447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62600ffe00 0x7f6260195830 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:33.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 >> 172.21.15.78:0/22944447 conn(0x7f62600770a0 msgr2=0x7f62600fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:33.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 shutdown_connections 2023-10-10T17:41:33.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:33.540+0000 7f6265b67700 1 -- 172.21.15.78:0/22944447 wait complete. 2023-10-10T17:41:33.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:33 smithi078 ceph-mon[99494]: pgmap v287: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:33.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:33 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2281863311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-10-10T17:41:33.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:33 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1324235469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-10-10T17:41:33.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:33 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/22944447' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-10T17:41:33.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:33 smithi190 ceph-mon[101220]: pgmap v287: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:33.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:33 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2281863311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-10-10T17:41:33.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:33 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1324235469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-10-10T17:41:33.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:33 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/22944447' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-10T17:41:33.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:33 smithi157 ceph-mon[100896]: pgmap v287: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:33.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:33 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2281863311' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-10-10T17:41:33.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:33 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1324235469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-10-10T17:41:33.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:33 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/22944447' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-10T17:41:34.301 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:34.323 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383699 got 343597383701 for osd.11 2023-10-10T17:41:34.324 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:34.833 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:34.943 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612631 got 313532612632 for osd.10 2023-10-10T17:41:34.943 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:35.119 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299429 got 223338299430 for osd.7 2023-10-10T17:41:35.120 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:35.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 -- 172.21.15.78:0/1904559137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 msgr2=0x7f18ac10d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 --2- 172.21.15.78:0/1904559137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac10d160 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f18a400b0d0 tx=0x7f18a400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 -- 172.21.15.78:0/1904559137 shutdown_connections 2023-10-10T17:41:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 --2- 172.21.15.78:0/1904559137 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac10d160 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 --2- 172.21.15.78:0/1904559137 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac100540 0x7f18ac100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 --2- 172.21.15.78:0/1904559137 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac100ee0 0x7f18ac104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 -- 172.21.15.78:0/1904559137 >> 172.21.15.78:0/1904559137 conn(0x7f18ac0fbdc0 msgr2=0x7f18ac0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 -- 172.21.15.78:0/1904559137 shutdown_connections 2023-10-10T17:41:35.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.139+0000 7f18b1611700 1 -- 172.21.15.78:0/1904559137 wait complete. 2023-10-10T17:41:35.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 Processor -- start 2023-10-10T17:41:35.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 -- start start 2023-10-10T17:41:35.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac100540 0x7f18ac072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac100ee0 0x7f18ac073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac0738b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac1a4870 con 0x7f18ac1055e0 2023-10-10T17:41:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac1a49e0 con 0x7f18ac100540 2023-10-10T17:41:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18b1611700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18ac1a4ba0 con 0x7f18ac100ee0 2023-10-10T17:41:35.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac0738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac0738b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42982/0 (socket says 172.21.15.78:42982) 2023-10-10T17:41:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 -- 172.21.15.78:0/183746149 learned_addr learned my addr 172.21.15.78:0/183746149 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18aa7fc700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac100ee0 0x7f18ac073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 -- 172.21.15.78:0/183746149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac100ee0 msgr2=0x7f18ac073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac100ee0 0x7f18ac073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18aaffd700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac100540 0x7f18ac072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 -- 172.21.15.78:0/183746149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac100540 msgr2=0x7f18ac072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac100540 0x7f18ac072df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.140+0000 7f18ab7fe700 1 -- 172.21.15.78:0/183746149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4009d00 con 0x7f18ac1055e0 2023-10-10T17:41:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.141+0000 7f18ab7fe700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac0738b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f18a4007940 tx=0x7f18a4007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.141+0000 7f1893fff700 1 -- 172.21.15.78:0/183746149 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a400e040 con 0x7f18ac1055e0 2023-10-10T17:41:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.141+0000 7f1893fff700 1 -- 172.21.15.78:0/183746149 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18a4003e10 con 0x7f18ac1055e0 2023-10-10T17:41:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.141+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18ac110320 con 0x7f18ac1055e0 2023-10-10T17:41:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.141+0000 7f1893fff700 1 -- 172.21.15.78:0/183746149 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4005620 con 0x7f18ac1055e0 2023-10-10T17:41:35.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.141+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18ac110190 con 0x7f18ac1055e0 2023-10-10T17:41:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.142+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18ac1104b0 con 0x7f18ac1055e0 2023-10-10T17:41:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.143+0000 7f1893fff700 1 -- 172.21.15.78:0/183746149 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f18a40057a0 con 0x7f18ac1055e0 2023-10-10T17:41:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.143+0000 7f1893fff700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18940a1150 0x7f18940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.143+0000 7f1893fff700 1 -- 172.21.15.78:0/183746149 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f18a40ce2c0 con 0x7f18ac1055e0 2023-10-10T17:41:35.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.143+0000 7f18aaffd700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18940a1150 0x7f18940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.144+0000 7f18aaffd700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18940a1150 0x7f18940a3600 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f18a0005950 tx=0x7f18a0005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.148+0000 7f1893fff700 1 -- 172.21.15.78:0/183746149 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f18a40978b0 con 0x7f18ac1055e0 2023-10-10T17:41:35.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.354+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f18ac066ea0 con 0x7f18ac1055e0 2023-10-10T17:41:35.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.354+0000 7f1893fff700 1 -- 172.21.15.78:0/183746149 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f18a409aed0 con 0x7f18ac1055e0 2023-10-10T17:41:35.356 INFO:teuthology.orchestra.run.smithi078.stdout:146028888117 2023-10-10T17:41:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18940a1150 msgr2=0x7f18940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18940a1150 0x7f18940a3600 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f18a0005950 tx=0x7f18a0005cf0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 msgr2=0x7f18ac0738b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac0738b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f18a4007940 tx=0x7f18a4007a20 comp rx=0 tx=0).stop 2023-10-10T17:41:35.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 shutdown_connections 2023-10-10T17:41:35.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f18940a1150 0x7f18940a3600 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f18ac1055e0 0x7f18ac0738b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f18ac100540 0x7f18ac072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 --2- 172.21.15.78:0/183746149 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f18ac100ee0 0x7f18ac073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 >> 172.21.15.78:0/183746149 conn(0x7f18ac0fbdc0 msgr2=0x7f18ac0fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.356+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 shutdown_connections 2023-10-10T17:41:35.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.357+0000 7f18b1611700 1 -- 172.21.15.78:0/183746149 wait complete. 2023-10-10T17:41:35.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 -- 172.21.15.78:0/1802406814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 msgr2=0x7fac9c104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 --2- 172.21.15.78:0/1802406814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c104ec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fac900099c0 tx=0x7fac90009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 -- 172.21.15.78:0/1802406814 shutdown_connections 2023-10-10T17:41:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 --2- 172.21.15.78:0/1802406814 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac9c100550 0x7fac9c100920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 --2- 172.21.15.78:0/1802406814 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c1055f0 0x7fac9c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 --2- 172.21.15.78:0/1802406814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c104ec0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 -- 172.21.15.78:0/1802406814 >> 172.21.15.78:0/1802406814 conn(0x7fac9c0fbdb0 msgr2=0x7fac9c0fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 -- 172.21.15.78:0/1802406814 shutdown_connections 2023-10-10T17:41:35.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.434+0000 7faca27cb700 1 -- 172.21.15.78:0/1802406814 wait complete. 2023-10-10T17:41:35.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 Processor -- start 2023-10-10T17:41:35.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 -- start start 2023-10-10T17:41:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c100550 0x7fac9c1963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac9c1055f0 0x7fac9c19ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19b380 con 0x7fac9c100ef0 2023-10-10T17:41:35.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19b4f0 con 0x7fac9c100550 2023-10-10T17:41:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca27cb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c19b6b0 con 0x7fac9c1055f0 2023-10-10T17:41:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7fac9b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7fac9b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c196900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:42992/0 (socket says 172.21.15.78:42992) 2023-10-10T17:41:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7fac9b7fe700 1 -- 172.21.15.78:0/230499702 learned_addr learned my addr 172.21.15.78:0/230499702 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7fac9bfff700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c100550 0x7fac9c1963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7faca0d68700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac9c1055f0 0x7fac9c19ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.435+0000 7fac9b7fe700 1 -- 172.21.15.78:0/230499702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac9c1055f0 msgr2=0x7fac9c19ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac9b7fe700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac9c1055f0 0x7fac9c19ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac9b7fe700 1 -- 172.21.15.78:0/230499702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c100550 msgr2=0x7fac9c1963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac9b7fe700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c100550 0x7fac9c1963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac9b7fe700 1 -- 172.21.15.78:0/230499702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac900095f0 con 0x7fac9c100ef0 2023-10-10T17:41:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7faca0d68700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac9c1055f0 0x7fac9c19ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac9b7fe700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c196900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fac90009990 tx=0x7fac90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac997fa700 1 -- 172.21.15.78:0/230499702 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac9001d070 con 0x7fac9c100ef0 2023-10-10T17:41:35.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac997fa700 1 -- 172.21.15.78:0/230499702 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fac900054f0 con 0x7fac9c100ef0 2023-10-10T17:41:35.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c110320 con 0x7fac9c100ef0 2023-10-10T17:41:35.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7fac997fa700 1 -- 172.21.15.78:0/230499702 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90003ad0 con 0x7fac9c100ef0 2023-10-10T17:41:35.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.436+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c110190 con 0x7fac9c100ef0 2023-10-10T17:41:35.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.437+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c1104b0 con 0x7fac9c100ef0 2023-10-10T17:41:35.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.438+0000 7fac997fa700 1 -- 172.21.15.78:0/230499702 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fac900047a0 con 0x7fac9c100ef0 2023-10-10T17:41:35.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.438+0000 7fac997fa700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac840a11d0 0x7fac840a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.439+0000 7fac9bfff700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac840a11d0 0x7fac840a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.439+0000 7fac997fa700 1 -- 172.21.15.78:0/230499702 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fac900cd8d0 con 0x7fac9c100ef0 2023-10-10T17:41:35.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.439+0000 7fac9bfff700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac840a11d0 0x7fac840a3680 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fac8c007be0 tx=0x7fac8c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.443+0000 7fac997fa700 1 -- 172.21.15.78:0/230499702 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fac90096ec0 con 0x7fac9c100ef0 2023-10-10T17:41:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.554+0000 7f5376dce700 1 -- 172.21.15.78:0/81926926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5370100540 msgr2=0x7f5370100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.554+0000 7f5376dce700 1 --2- 172.21.15.78:0/81926926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5370100540 0x7f5370100910 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f53640099c0 tx=0x7f5364009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- 172.21.15.78:0/81926926 shutdown_connections 2023-10-10T17:41:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 --2- 172.21.15.78:0/81926926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53701055e0 0x7f537010d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 --2- 172.21.15.78:0/81926926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5370100540 0x7f5370100910 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 --2- 172.21.15.78:0/81926926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5370100ee0 0x7f5370104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- 172.21.15.78:0/81926926 >> 172.21.15.78:0/81926926 conn(0x7f53700fbdc0 msgr2=0x7f53700fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- 172.21.15.78:0/81926926 shutdown_connections 2023-10-10T17:41:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- 172.21.15.78:0/81926926 wait complete. 2023-10-10T17:41:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 Processor -- start 2023-10-10T17:41:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- start start 2023-10-10T17:41:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5370100540 0x7f53701963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5370100ee0 0x7f53701968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53701055e0 0x7f537019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537019b360 con 0x7f53701055e0 2023-10-10T17:41:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537019b4d0 con 0x7f5370100540 2023-10-10T17:41:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.555+0000 7f5376dce700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537019b690 con 0x7f5370100ee0 2023-10-10T17:41:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53701055e0 0x7f537019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53701055e0 0x7f537019ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43008/0 (socket says 172.21.15.78:43008) 2023-10-10T17:41:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 -- 172.21.15.78:0/3352379435 learned_addr learned my addr 172.21.15.78:0/3352379435 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f536ffff700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5370100ee0 0x7f53701968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f5374b6a700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5370100540 0x7f53701963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 -- 172.21.15.78:0/3352379435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5370100ee0 msgr2=0x7f53701968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5370100ee0 0x7f53701968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 -- 172.21.15.78:0/3352379435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5370100540 msgr2=0x7f53701963a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5370100540 0x7f53701963a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 -- 172.21.15.78:0/3352379435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53640095f0 con 0x7f53701055e0 2023-10-10T17:41:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f537536b700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53701055e0 0x7f537019ad50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f536800ada0 tx=0x7f536800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f536dffb700 1 -- 172.21.15.78:0/3352379435 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536800f070 con 0x7f53701055e0 2023-10-10T17:41:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f536dffb700 1 -- 172.21.15.78:0/3352379435 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5368007b30 con 0x7f53701055e0 2023-10-10T17:41:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f536dffb700 1 -- 172.21.15.78:0/3352379435 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5368005280 con 0x7f53701055e0 2023-10-10T17:41:35.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.556+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5370110320 con 0x7f53701055e0 2023-10-10T17:41:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.557+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5370110190 con 0x7f53701055e0 2023-10-10T17:41:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.558+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53701104b0 con 0x7f53701055e0 2023-10-10T17:41:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.559+0000 7f536dffb700 1 -- 172.21.15.78:0/3352379435 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f536800a090 con 0x7f53701055e0 2023-10-10T17:41:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.560+0000 7f536dffb700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53600a11d0 0x7f53600a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.560+0000 7f5374b6a700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53600a11d0 0x7f53600a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.560+0000 7f536dffb700 1 -- 172.21.15.78:0/3352379435 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f53680cc510 con 0x7f53701055e0 2023-10-10T17:41:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.561+0000 7f5374b6a700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53600a11d0 0x7f53600a3680 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5364000c00 tx=0x7f5364003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.564+0000 7f536dffb700 1 -- 172.21.15.78:0/3352379435 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5368095b00 con 0x7f53701055e0 2023-10-10T17:41:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 -- 172.21.15.78:0/4053508564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c10a890 msgr2=0x7faa9c10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 --2- 172.21.15.78:0/4053508564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c10a890 0x7faa9c10ccd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7faa9400b0a0 tx=0x7faa9400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 -- 172.21.15.78:0/4053508564 shutdown_connections 2023-10-10T17:41:35.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 --2- 172.21.15.78:0/4053508564 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c10a890 0x7faa9c10ccd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 --2- 172.21.15.78:0/4053508564 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa9c101540 0x7faa9c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 --2- 172.21.15.78:0/4053508564 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa9c101e50 0x7faa9c10a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 -- 172.21.15.78:0/4053508564 >> 172.21.15.78:0/4053508564 conn(0x7faa9c0770a0 msgr2=0x7faa9c0774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 -- 172.21.15.78:0/4053508564 shutdown_connections 2023-10-10T17:41:35.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.601+0000 7faaa2f16700 1 -- 172.21.15.78:0/4053508564 wait complete. 2023-10-10T17:41:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.602+0000 7faaa2f16700 1 Processor -- start 2023-10-10T17:41:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.602+0000 7faaa2f16700 1 -- start start 2023-10-10T17:41:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa2f16700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa9c101540 0x7faa9c196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa2f16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c101e50 0x7faa9c196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa2f16700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa9c10a890 0x7faa9c19aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa2f16700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c19b4d0 con 0x7faa9c101e50 2023-10-10T17:41:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa2f16700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c19b640 con 0x7faa9c101540 2023-10-10T17:41:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa2f16700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c19b800 con 0x7faa9c10a890 2023-10-10T17:41:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c101e50 0x7faa9c196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c101e50 0x7faa9c196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43020/0 (socket says 172.21.15.78:43020) 2023-10-10T17:41:35.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 -- 172.21.15.78:0/2646363886 learned_addr learned my addr 172.21.15.78:0/2646363886 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa0cb2700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa9c101540 0x7faa9c196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa14b3700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa9c10a890 0x7faa9c19aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 -- 172.21.15.78:0/2646363886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa9c10a890 msgr2=0x7faa9c19aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa9c10a890 0x7faa9c19aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 -- 172.21.15.78:0/2646363886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa9c101540 msgr2=0x7faa9c196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa9c101540 0x7faa9c196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 -- 172.21.15.78:0/2646363886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa94009d00 con 0x7faa9c101e50 2023-10-10T17:41:35.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa9bfff700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c101e50 0x7faa9c196a50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7faa8800d980 tx=0x7faa8800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa99ffb700 1 -- 172.21.15.78:0/2646363886 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8800bb60 con 0x7faa9c101e50 2023-10-10T17:41:35.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa99ffb700 1 -- 172.21.15.78:0/2646363886 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa88007640 con 0x7faa9c101e50 2023-10-10T17:41:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c110300 con 0x7faa9c101e50 2023-10-10T17:41:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.603+0000 7faa99ffb700 1 -- 172.21.15.78:0/2646363886 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa88004e90 con 0x7faa9c101e50 2023-10-10T17:41:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.604+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c110170 con 0x7faa9c101e50 2023-10-10T17:41:35.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.605+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c110490 con 0x7faa9c101e50 2023-10-10T17:41:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.606+0000 7faa99ffb700 1 -- 172.21.15.78:0/2646363886 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faa8800bcc0 con 0x7faa9c101e50 2023-10-10T17:41:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.606+0000 7faa99ffb700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa8c0a11d0 0x7faa8c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.607+0000 7faaa0cb2700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa8c0a11d0 0x7faa8c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.607+0000 7faa99ffb700 1 -- 172.21.15.78:0/2646363886 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8730+0+0 (secure 0 0 0) 0x7faa880ccdf0 con 0x7faa9c101e50 2023-10-10T17:41:35.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.607+0000 7faaa0cb2700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa8c0a11d0 0x7faa8c0a3680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faa90005950 tx=0x7faa900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:35.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.610+0000 7faa99ffb700 1 -- 172.21.15.78:0/2646363886 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faa88096350 con 0x7faa9c101e50 2023-10-10T17:41:35.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.664+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fac9c197720 con 0x7fac9c100ef0 2023-10-10T17:41:35.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.664+0000 7fac997fa700 1 -- 172.21.15.78:0/230499702 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fac9009a4e0 con 0x7fac9c100ef0 2023-10-10T17:41:35.666 INFO:teuthology.orchestra.run.smithi078.stdout:34359738437 2023-10-10T17:41:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.666+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac840a11d0 msgr2=0x7fac840a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.666+0000 7faca27cb700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac840a11d0 0x7fac840a3680 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fac8c007be0 tx=0x7fac8c0073d0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.666+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 msgr2=0x7fac9c196900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.666+0000 7faca27cb700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c196900 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fac90009990 tx=0x7fac90004730 comp rx=0 tx=0).stop 2023-10-10T17:41:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.666+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 shutdown_connections 2023-10-10T17:41:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.666+0000 7faca27cb700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fac840a11d0 0x7fac840a3680 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.666+0000 7faca27cb700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fac9c1055f0 0x7fac9c19ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.667+0000 7faca27cb700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fac9c100550 0x7fac9c1963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.667+0000 7faca27cb700 1 --2- 172.21.15.78:0/230499702 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fac9c100ef0 0x7fac9c196900 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.667+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 >> 172.21.15.78:0/230499702 conn(0x7fac9c0fbdb0 msgr2=0x7fac9c0fc830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.667+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 shutdown_connections 2023-10-10T17:41:35.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.667+0000 7faca27cb700 1 -- 172.21.15.78:0/230499702 wait complete. 2023-10-10T17:41:35.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:35 smithi078 ceph-mon[99494]: pgmap v288: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:41:35.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:35 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/183746149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-10T17:41:35.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.779+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5370066ea0 con 0x7f53701055e0 2023-10-10T17:41:35.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.780+0000 7f536dffb700 1 -- 172.21.15.78:0/3352379435 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5368099120 con 0x7f53701055e0 2023-10-10T17:41:35.782 INFO:teuthology.orchestra.run.smithi078.stdout:55834574913 2023-10-10T17:41:35.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53600a11d0 msgr2=0x7f53600a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53600a11d0 0x7f53600a3680 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5364000c00 tx=0x7f5364003680 comp rx=0 tx=0).stop 2023-10-10T17:41:35.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53701055e0 msgr2=0x7f537019ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53701055e0 0x7f537019ad50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f536800ada0 tx=0x7f536800c330 comp rx=0 tx=0).stop 2023-10-10T17:41:35.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 shutdown_connections 2023-10-10T17:41:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53600a11d0 0x7f53600a3680 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53701055e0 0x7f537019ad50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5370100540 0x7f53701963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 --2- 172.21.15.78:0/3352379435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5370100ee0 0x7f53701968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 >> 172.21.15.78:0/3352379435 conn(0x7f53700fbdc0 msgr2=0x7f53700fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 shutdown_connections 2023-10-10T17:41:35.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.782+0000 7f5376dce700 1 -- 172.21.15.78:0/3352379435 wait complete. 2023-10-10T17:41:35.829 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462953 got 201863462955 for osd.6 2023-10-10T17:41:35.829 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:35.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.828+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faa9c066ea0 con 0x7faa9c101e50 2023-10-10T17:41:35.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.828+0000 7faa99ffb700 1 -- 172.21.15.78:0/2646363886 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faa88016070 con 0x7faa9c101e50 2023-10-10T17:41:35.831 INFO:teuthology.orchestra.run.smithi078.stdout:77309411388 2023-10-10T17:41:35.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa8c0a11d0 msgr2=0x7faa8c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa8c0a11d0 0x7faa8c0a3680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faa90005950 tx=0x7faa900058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c101e50 msgr2=0x7faa9c196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:35.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c101e50 0x7faa9c196a50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7faa8800d980 tx=0x7faa8800dd40 comp rx=0 tx=0).stop 2023-10-10T17:41:35.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 shutdown_connections 2023-10-10T17:41:35.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa8c0a11d0 0x7faa8c0a3680 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa9c10a890 0x7faa9c19aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa9c101540 0x7faa9c196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 --2- 172.21.15.78:0/2646363886 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa9c101e50 0x7faa9c196a50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:35.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 >> 172.21.15.78:0/2646363886 conn(0x7faa9c0770a0 msgr2=0x7faa9c0fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:35.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 shutdown_connections 2023-10-10T17:41:35.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:35.830+0000 7faaa2f16700 1 -- 172.21.15.78:0/2646363886 wait complete. 2023-10-10T17:41:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:35 smithi190 ceph-mon[101220]: pgmap v288: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:41:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:35 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/183746149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-10T17:41:35.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:35 smithi157 ceph-mon[100896]: pgmap v288: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:41:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:35 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/183746149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-10T17:41:36.316 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411386 got 77309411388 for osd.2 2023-10-10T17:41:36.317 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:36.533 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738435 got 34359738437 for osd.0 2023-10-10T17:41:36.533 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:36.631 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037793 got 257698037795 for osd.8 2023-10-10T17:41:36.631 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:36.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/230499702' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-10T17:41:36.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3352379435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-10T17:41:36.814 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2646363886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-10T17:41:36.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2023-10-10T17:41:36.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-10-10T17:41:36.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2023-10-10T17:41:36.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2023-10-10T17:41:36.815 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-10-10T17:41:36.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2023-10-10T17:41:36.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-10-10T17:41:36.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2023-10-10T17:41:36.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]: dispatch 2023-10-10T17:41:36.816 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:36 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-10-10T17:41:36.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/230499702' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-10T17:41:36.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3352379435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-10T17:41:36.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2646363886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-10T17:41:36.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2023-10-10T17:41:36.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-10-10T17:41:36.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2023-10-10T17:41:36.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2023-10-10T17:41:36.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-10-10T17:41:36.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2023-10-10T17:41:36.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-10-10T17:41:36.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2023-10-10T17:41:36.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]: dispatch 2023-10-10T17:41:36.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:36 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-10-10T17:41:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/230499702' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-10T17:41:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3352379435' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-10T17:41:36.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2646363886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-10T17:41:36.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]: dispatch 2023-10-10T17:41:36.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-10-10T17:41:36.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2023-10-10T17:41:36.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2023-10-10T17:41:36.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2023-10-10T17:41:36.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2023-10-10T17:41:36.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]: dispatch 2023-10-10T17:41:36.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2023-10-10T17:41:36.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]: dispatch 2023-10-10T17:41:36.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:36 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]: dispatch 2023-10-10T17:41:37.155 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888114 got 146028888117 for osd.4 2023-10-10T17:41:37.156 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:37.335 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574910 got 55834574913 for osd.1 2023-10-10T17:41:37.336 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:37.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: pgmap v289: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:37.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2023-10-10T17:41:37.747 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-10-10T17:41:37.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2023-10-10T17:41:37.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2023-10-10T17:41:37.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-10-10T17:41:37.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2023-10-10T17:41:37.748 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-10-10T17:41:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2023-10-10T17:41:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]': finished 2023-10-10T17:41:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-10-10T17:41:37.749 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:37 smithi078 ceph-mon[99494]: osdmap e90: 12 total, 12 up, 12 in 2023-10-10T17:41:37.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: pgmap v289: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:37.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2023-10-10T17:41:37.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-10-10T17:41:37.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2023-10-10T17:41:37.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2023-10-10T17:41:37.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-10-10T17:41:37.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2023-10-10T17:41:37.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-10-10T17:41:37.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2023-10-10T17:41:37.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]': finished 2023-10-10T17:41:37.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-10-10T17:41:37.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:37 smithi190 ceph-mon[101220]: osdmap e90: 12 total, 12 up, 12 in 2023-10-10T17:41:37.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: pgmap v289: 129 pgs: 129 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:37.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4]}]': finished 2023-10-10T17:41:37.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-10-10T17:41:37.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2023-10-10T17:41:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2023-10-10T17:41:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2023-10-10T17:41:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2023-10-10T17:41:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [9, 8]}]': finished 2023-10-10T17:41:37.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2023-10-10T17:41:37.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [9, 10]}]': finished 2023-10-10T17:41:37.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 8]}]': finished 2023-10-10T17:41:37.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:37 smithi157 ceph-mon[100896]: osdmap e90: 12 total, 12 up, 12 in 2023-10-10T17:41:38.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 -- 172.21.15.78:0/3141624509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 msgr2=0x7fd48c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 --2- 172.21.15.78:0/3141624509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c100910 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fd4800099c0 tx=0x7fd480009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 -- 172.21.15.78:0/3141624509 shutdown_connections 2023-10-10T17:41:38.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 --2- 172.21.15.78:0/3141624509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd48c1055e0 0x7fd48c10d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 --2- 172.21.15.78:0/3141624509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c100910 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 --2- 172.21.15.78:0/3141624509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd48c100ee0 0x7fd48c104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 -- 172.21.15.78:0/3141624509 >> 172.21.15.78:0/3141624509 conn(0x7fd48c0fbdc0 msgr2=0x7fd48c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:38.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 -- 172.21.15.78:0/3141624509 shutdown_connections 2023-10-10T17:41:38.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.732+0000 7fd492e5d700 1 -- 172.21.15.78:0/3141624509 wait complete. 2023-10-10T17:41:38.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 Processor -- start 2023-10-10T17:41:38.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 -- start start 2023-10-10T17:41:38.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c1964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd48c100ee0 0x7fd48c1969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd48c1055e0 0x7fd48c19adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c19b3d0 con 0x7fd48c100540 2023-10-10T17:41:38.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c19b540 con 0x7fd48c100ee0 2023-10-10T17:41:38.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd492e5d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48c19b700 con 0x7fd48c1055e0 2023-10-10T17:41:38.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd490bf9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c1964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd490bf9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c1964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43034/0 (socket says 172.21.15.78:43034) 2023-10-10T17:41:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.733+0000 7fd490bf9700 1 -- 172.21.15.78:0/154494481 learned_addr learned my addr 172.21.15.78:0/154494481 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd48bfff700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd48c100ee0 0x7fd48c1969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd4913fa700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd48c1055e0 0x7fd48c19adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd490bf9700 1 -- 172.21.15.78:0/154494481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd48c1055e0 msgr2=0x7fd48c19adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd490bf9700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd48c1055e0 0x7fd48c19adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd490bf9700 1 -- 172.21.15.78:0/154494481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd48c100ee0 msgr2=0x7fd48c1969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd490bf9700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd48c100ee0 0x7fd48c1969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd490bf9700 1 -- 172.21.15.78:0/154494481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4800095f0 con 0x7fd48c100540 2023-10-10T17:41:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd4913fa700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd48c1055e0 0x7fd48c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd48bfff700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd48c100ee0 0x7fd48c1969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd490bf9700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c1964b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd48000b580 tx=0x7fd4800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd489ffb700 1 -- 172.21.15.78:0/154494481 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48001d070 con 0x7fd48c100540 2023-10-10T17:41:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd489ffb700 1 -- 172.21.15.78:0/154494481 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd480004110 con 0x7fd48c100540 2023-10-10T17:41:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd489ffb700 1 -- 172.21.15.78:0/154494481 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48000f460 con 0x7fd48c100540 2023-10-10T17:41:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48c1103a0 con 0x7fd48c100540 2023-10-10T17:41:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.734+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48c110210 con 0x7fd48c100540 2023-10-10T17:41:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.735+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48c19bc40 con 0x7fd48c100540 2023-10-10T17:41:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.736+0000 7fd489ffb700 1 -- 172.21.15.78:0/154494481 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd480022670 con 0x7fd48c100540 2023-10-10T17:41:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.737+0000 7fd489ffb700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd47c0a11d0 0x7fd47c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.737+0000 7fd48bfff700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd47c0a11d0 0x7fd47c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.737+0000 7fd489ffb700 1 -- 172.21.15.78:0/154494481 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd4800cdf70 con 0x7fd48c100540 2023-10-10T17:41:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.737+0000 7fd48bfff700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd47c0a11d0 0x7fd47c0a3680 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd478005fd0 tx=0x7fd478005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.742+0000 7fd489ffb700 1 -- 172.21.15.78:0/154494481 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd4800973e0 con 0x7fd48c100540 2023-10-10T17:41:38.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:38 smithi078 ceph-mon[99494]: osdmap e91: 12 total, 12 up, 12 in 2023-10-10T17:41:38.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:38 smithi078 ceph-mon[99494]: pgmap v292: 129 pgs: 3 peering, 126 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-10-10T17:41:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:38 smithi190 ceph-mon[101220]: osdmap e91: 12 total, 12 up, 12 in 2023-10-10T17:41:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:38 smithi190 ceph-mon[101220]: pgmap v292: 129 pgs: 3 peering, 126 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-10-10T17:41:38.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.943+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fd48c110530 con 0x7fd48c100540 2023-10-10T17:41:38.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.943+0000 7fd489ffb700 1 -- 172.21.15.78:0/154494481 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd480027020 con 0x7fd48c100540 2023-10-10T17:41:38.946 INFO:teuthology.orchestra.run.smithi078.stdout:167503724593 2023-10-10T17:41:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd47c0a11d0 msgr2=0x7fd47c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd47c0a11d0 0x7fd47c0a3680 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd478005fd0 tx=0x7fd478005dc0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 msgr2=0x7fd48c1964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c1964b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd48000b580 tx=0x7fd4800057f0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 shutdown_connections 2023-10-10T17:41:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd47c0a11d0 0x7fd47c0a3680 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd48c1055e0 0x7fd48c19adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd48c100540 0x7fd48c1964b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 --2- 172.21.15.78:0/154494481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd48c100ee0 0x7fd48c1969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 >> 172.21.15.78:0/154494481 conn(0x7fd48c0fbdc0 msgr2=0x7fd48c0fc860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:38.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 shutdown_connections 2023-10-10T17:41:38.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.946+0000 7fd492e5d700 1 -- 172.21.15.78:0/154494481 wait complete. 2023-10-10T17:41:38.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:38 smithi157 ceph-mon[100896]: osdmap e91: 12 total, 12 up, 12 in 2023-10-10T17:41:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:38 smithi157 ceph-mon[100896]: pgmap v292: 129 pgs: 3 peering, 126 active+clean; 584 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s 2023-10-10T17:41:38.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.981+0000 7f0343ea7700 1 -- 172.21.15.78:0/102931362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 msgr2=0x7f033c10c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.981+0000 7f0343ea7700 1 --2- 172.21.15.78:0/102931362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c10c2c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f033400b0d0 tx=0x7f033400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.981+0000 7f0343ea7700 1 -- 172.21.15.78:0/102931362 shutdown_connections 2023-10-10T17:41:38.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.981+0000 7f0343ea7700 1 --2- 172.21.15.78:0/102931362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f033c0ffbc0 0x7f033c0fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.981+0000 7f0343ea7700 1 --2- 172.21.15.78:0/102931362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c10c2c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.981+0000 7f0343ea7700 1 --2- 172.21.15.78:0/102931362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f033c100560 0x7f033c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.981+0000 7f0343ea7700 1 -- 172.21.15.78:0/102931362 >> 172.21.15.78:0/102931362 conn(0x7f033c0758a0 msgr2=0x7f033c075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:38.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 -- 172.21.15.78:0/102931362 shutdown_connections 2023-10-10T17:41:38.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 -- 172.21.15.78:0/102931362 wait complete. 2023-10-10T17:41:38.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 Processor -- start 2023-10-10T17:41:38.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 -- start start 2023-10-10T17:41:38.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f033c0ffbc0 0x7f033c195600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f033c100560 0x7f033c195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c19a5c0 con 0x7f033c104740 2023-10-10T17:41:38.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c19a730 con 0x7f033c0ffbc0 2023-10-10T17:41:38.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.982+0000 7f0343ea7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f033c19a8f0 con 0x7f033c100560 2023-10-10T17:41:38.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43052/0 (socket says 172.21.15.78:43052) 2023-10-10T17:41:38.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 -- 172.21.15.78:0/268681157 learned_addr learned my addr 172.21.15.78:0/268681157 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:38.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0341c43700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f033c0ffbc0 0x7f033c195600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0341442700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f033c100560 0x7f033c195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 -- 172.21.15.78:0/268681157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f033c100560 msgr2=0x7f033c195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f033c100560 0x7f033c195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 -- 172.21.15.78:0/268681157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f033c0ffbc0 msgr2=0x7f033c195600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:38.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f033c0ffbc0 0x7f033c195600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:38.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 -- 172.21.15.78:0/268681157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0334009d00 con 0x7f033c104740 2023-10-10T17:41:38.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0341442700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f033c100560 0x7f033c195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:38.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0342444700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c199fb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0334007870 tx=0x7f03340078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:38.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0332ffd700 1 -- 172.21.15.78:0/268681157 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033400e040 con 0x7f033c104740 2023-10-10T17:41:38.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0332ffd700 1 -- 172.21.15.78:0/268681157 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0334005070 con 0x7f033c104740 2023-10-10T17:41:38.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0332ffd700 1 -- 172.21.15.78:0/268681157 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f033401b6a0 con 0x7f033c104740 2023-10-10T17:41:38.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033c10f480 con 0x7f033c104740 2023-10-10T17:41:38.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.983+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f033c10f2f0 con 0x7f033c104740 2023-10-10T17:41:38.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.984+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f033c19ae30 con 0x7f033c104740 2023-10-10T17:41:38.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.986+0000 7f0332ffd700 1 -- 172.21.15.78:0/268681157 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f03340051e0 con 0x7f033c104740 2023-10-10T17:41:38.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.986+0000 7f0332ffd700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03280a11d0 0x7f03280a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:38.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.986+0000 7f0341c43700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03280a11d0 0x7f03280a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:38.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.986+0000 7f0332ffd700 1 -- 172.21.15.78:0/268681157 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f03340ce250 con 0x7f033c104740 2023-10-10T17:41:38.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.987+0000 7f0341c43700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03280a11d0 0x7f03280a3680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0338009c60 tx=0x7f0338009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:38.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:38.990+0000 7f0332ffd700 1 -- 172.21.15.78:0/268681157 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0334097720 con 0x7f033c104740 2023-10-10T17:41:39.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.199+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f033c066ea0 con 0x7f033c104740 2023-10-10T17:41:39.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.200+0000 7f0332ffd700 1 -- 172.21.15.78:0/268681157 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f033409ad40 con 0x7f033c104740 2023-10-10T17:41:39.201 INFO:teuthology.orchestra.run.smithi078.stdout:111669149752 2023-10-10T17:41:39.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03280a11d0 msgr2=0x7f03280a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:39.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03280a11d0 0x7f03280a3680 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0338009c60 tx=0x7f0338009260 comp rx=0 tx=0).stop 2023-10-10T17:41:39.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 msgr2=0x7f033c199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:39.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c199fb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f0334007870 tx=0x7f03340078a0 comp rx=0 tx=0).stop 2023-10-10T17:41:39.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 shutdown_connections 2023-10-10T17:41:39.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f03280a11d0 0x7f03280a3680 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:39.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f033c104740 0x7f033c199fb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:39.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f033c0ffbc0 0x7f033c195600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:39.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 --2- 172.21.15.78:0/268681157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f033c100560 0x7f033c195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:39.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 >> 172.21.15.78:0/268681157 conn(0x7f033c0758a0 msgr2=0x7f033c0fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:39.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 shutdown_connections 2023-10-10T17:41:39.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:39.202+0000 7f0343ea7700 1 -- 172.21.15.78:0/268681157 wait complete. 2023-10-10T17:41:39.292 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724590 got 167503724593 for osd.5 2023-10-10T17:41:39.293 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:39.675 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149749 got 111669149752 for osd.3 2023-10-10T17:41:39.676 DEBUG:teuthology.parallel:result is None 2023-10-10T17:41:39.676 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-10-10T17:41:39.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph pg dump --format=json 2023-10-10T17:41:39.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:39 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/154494481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-10T17:41:39.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:39 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/268681157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-10T17:41:39.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:39 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/154494481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-10T17:41:39.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:39 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/268681157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-10T17:41:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:39 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/154494481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-10T17:41:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:39 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/268681157' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-10T17:41:40.320 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:40.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:40 smithi078 ceph-mon[99494]: pgmap v293: 129 pgs: 3 peering, 126 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:41:40.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:40 smithi190 ceph-mon[101220]: pgmap v293: 129 pgs: 3 peering, 126 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:41:40.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:40 smithi157 ceph-mon[100896]: pgmap v293: 129 pgs: 3 peering, 126 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:41:41.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 -- 172.21.15.78:0/2670899500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc105650 msgr2=0x7f4cdc10d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:41.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 --2- 172.21.15.78:0/2670899500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc105650 0x7f4cdc10d1d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4cd400b0d0 tx=0x7f4cd400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 -- 172.21.15.78:0/2670899500 shutdown_connections 2023-10-10T17:41:41.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 --2- 172.21.15.78:0/2670899500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc105650 0x7f4cdc10d1d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 --2- 172.21.15.78:0/2670899500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cdc1006f0 0x7f4cdc100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 --2- 172.21.15.78:0/2670899500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cdc101090 0x7f4cdc104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 -- 172.21.15.78:0/2670899500 >> 172.21.15.78:0/2670899500 conn(0x7f4cdc078460 msgr2=0x7f4cdc078860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:41.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 -- 172.21.15.78:0/2670899500 shutdown_connections 2023-10-10T17:41:41.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 -- 172.21.15.78:0/2670899500 wait complete. 2023-10-10T17:41:41.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 Processor -- start 2023-10-10T17:41:41.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.636+0000 7f4ce3459700 1 -- start start 2023-10-10T17:41:41.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cdc1006f0 0x7f4cdc19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:41.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc101090 0x7f4cdc19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cdc105650 0x7f4cdc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc198c60 con 0x7f4cdc101090 2023-10-10T17:41:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc198dd0 con 0x7f4cdc1006f0 2023-10-10T17:41:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc198f90 con 0x7f4cdc105650 2023-10-10T17:41:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc101090 0x7f4cdc19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc101090 0x7f4cdc19eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43064/0 (socket says 172.21.15.78:43064) 2023-10-10T17:41:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 -- 172.21.15.78:0/1962742753 learned_addr learned my addr 172.21.15.78:0/1962742753 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce19f6700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cdc105650 0x7f4cdc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 -- 172.21.15.78:0/1962742753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cdc105650 msgr2=0x7f4cdc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cdc105650 0x7f4cdc198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce11f5700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cdc1006f0 0x7f4cdc19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:41.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 -- 172.21.15.78:0/1962742753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cdc1006f0 msgr2=0x7f4cdc19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:41.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cdc1006f0 0x7f4cdc19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 -- 172.21.15.78:0/1962742753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd4009d00 con 0x7f4cdc101090 2023-10-10T17:41:41.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce09f4700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc101090 0x7f4cdc19eac0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800d950 tx=0x7f4cd800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4cd27fc700 1 -- 172.21.15.78:0/1962742753 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8009b40 con 0x7f4cdc101090 2023-10-10T17:41:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4cd27fc700 1 -- 172.21.15.78:0/1962742753 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4cd800de30 con 0x7f4cdc101090 2023-10-10T17:41:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cdc1103b0 con 0x7f4cdc101090 2023-10-10T17:41:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4cd27fc700 1 -- 172.21.15.78:0/1962742753 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd8005580 con 0x7f4cdc101090 2023-10-10T17:41:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.637+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cdc110220 con 0x7f4cdc101090 2023-10-10T17:41:41.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.638+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cdc110540 con 0x7f4cdc101090 2023-10-10T17:41:41.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.640+0000 7f4cd27fc700 1 -- 172.21.15.78:0/1962742753 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4cd8004750 con 0x7f4cdc101090 2023-10-10T17:41:41.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.640+0000 7f4cd27fc700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4cc80a1150 0x7f4cc80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:41.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.640+0000 7f4ce11f5700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4cc80a1150 0x7f4cc80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:41.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.640+0000 7f4cd27fc700 1 -- 172.21.15.78:0/1962742753 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4cd80cc940 con 0x7f4cdc101090 2023-10-10T17:41:41.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.641+0000 7f4ce11f5700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4cc80a1150 0x7f4cc80a3600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc005950 tx=0x7f4ccc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:41.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.643+0000 7f4cd27fc700 1 -- 172.21.15.78:0/1962742753 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4cd8095d80 con 0x7f4cdc101090 2023-10-10T17:41:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.828+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4cdc0fe5e0 con 0x7f4cc80a1150 2023-10-10T17:41:41.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.840+0000 7f4cd27fc700 1 -- 172.21.15.78:0/1962742753 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422899 (secure 0 0 0) 0x7f4cdc0fe5e0 con 0x7f4cc80a1150 2023-10-10T17:41:41.845 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4cc80a1150 msgr2=0x7f4cc80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4cc80a1150 0x7f4cc80a3600 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4ccc005950 tx=0x7f4ccc0058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc101090 msgr2=0x7f4cdc19eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc101090 0x7f4cdc19eac0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4cd800d950 tx=0x7f4cd800dd10 comp rx=0 tx=0).stop 2023-10-10T17:41:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 shutdown_connections 2023-10-10T17:41:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4cc80a1150 0x7f4cc80a3600 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4cdc105650 0x7f4cdc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4cdc1006f0 0x7f4cdc19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 --2- 172.21.15.78:0/1962742753 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cdc101090 0x7f4cdc19eac0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 >> 172.21.15.78:0/1962742753 conn(0x7f4cdc078460 msgr2=0x7f4cdc0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 shutdown_connections 2023-10-10T17:41:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:41.844+0000 7f4ce3459700 1 -- 172.21.15.78:0/1962742753 wait complete. 2023-10-10T17:41:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-10T17:41:42.181 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2023-10-10T17:41:40.231104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597676,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":377,"num_read_kb":253,"num_write":310,"num_write_kb":2981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80448,"kb_used_data":11744,"kb_used_omap":0,"kb_used_meta":68608,"kb_avail":1124861376,"statfs":{"total":1151940427776,"available":1151858049024,"internally_reserved":0,"allocated":12025856,"data_stored":7587679,"data_compressed":286882,"data_compressed_allocated":385024,"data_compressed_original":2269184,"omap_allocated":0,"internal_metadata":70254592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002252"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277491+0000","last_change":"2023-10-10T17:40:41.604268+0000","last_active":"2023-10-10T17:40:46.277491+0000","last_peered":"2023-10-10T17:40:46.277491+0000","last_clean":"2023-10-10T17:40:46.277491+0000","last_became_active":"2023-10-10T17:40:41.603839+0000","last_became_peered":"2023-10-10T17:40:41.603839+0000","last_unstale":"2023-10-10T17:40:46.277491+0000","last_undegraded":"2023-10-10T17:40:46.277491+0000","last_fullsized":"2023-10-10T17:40:46.277491+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:06:30.271619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.599420+0000","last_change":"2023-10-10T17:40:41.603642+0000","last_active":"2023-10-10T17:41:36.599420+0000","last_peered":"2023-10-10T17:41:36.599420+0000","last_clean":"2023-10-10T17:41:36.599420+0000","last_became_active":"2023-10-10T17:40:41.603091+0000","last_became_peered":"2023-10-10T17:40:41.603091+0000","last_unstale":"2023-10-10T17:41:36.599420+0000","last_undegraded":"2023-10-10T17:41:36.599420+0000","last_fullsized":"2023-10-10T17:41:36.599420+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:38:30.995450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.598460+0000","last_change":"2023-10-10T17:40:41.610647+0000","last_active":"2023-10-10T17:41:36.598460+0000","last_peered":"2023-10-10T17:41:36.598460+0000","last_clean":"2023-10-10T17:41:36.598460+0000","last_became_active":"2023-10-10T17:40:41.610338+0000","last_became_peered":"2023-10-10T17:40:41.610338+0000","last_unstale":"2023-10-10T17:41:36.598460+0000","last_undegraded":"2023-10-10T17:41:36.598460+0000","last_fullsized":"2023-10-10T17:41:36.598460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:06:00.229803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602832+0000","last_change":"2023-10-10T17:40:41.607832+0000","last_active":"2023-10-10T17:41:37.602832+0000","last_peered":"2023-10-10T17:41:37.602832+0000","last_clean":"2023-10-10T17:41:37.602832+0000","last_became_active":"2023-10-10T17:40:41.607492+0000","last_became_peered":"2023-10-10T17:40:41.607492+0000","last_unstale":"2023-10-10T17:41:37.602832+0000","last_undegraded":"2023-10-10T17:41:37.602832+0000","last_fullsized":"2023-10-10T17:41:37.602832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:43.514129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.600112+0000","last_change":"2023-10-10T17:40:41.601975+0000","last_active":"2023-10-10T17:41:36.600112+0000","last_peered":"2023-10-10T17:41:36.600112+0000","last_clean":"2023-10-10T17:41:36.600112+0000","last_became_active":"2023-10-10T17:40:41.601668+0000","last_became_peered":"2023-10-10T17:40:41.601668+0000","last_unstale":"2023-10-10T17:41:36.600112+0000","last_undegraded":"2023-10-10T17:41:36.600112+0000","last_fullsized":"2023-10-10T17:41:36.600112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:01:25.821062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.604098+0000","last_change":"2023-10-10T17:41:37.604098+0000","last_active":"2023-10-10T17:41:37.604098+0000","last_peered":"2023-10-10T17:41:37.604098+0000","last_clean":"2023-10-10T17:41:37.604098+0000","last_became_active":"2023-10-10T17:41:37.603750+0000","last_became_peered":"2023-10-10T17:41:37.603750+0000","last_unstale":"2023-10-10T17:41:37.604098+0000","last_undegraded":"2023-10-10T17:41:37.604098+0000","last_fullsized":"2023-10-10T17:41:37.604098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:52:52.491557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602564+0000","last_change":"2023-10-10T17:40:41.594628+0000","last_active":"2023-10-10T17:41:37.602564+0000","last_peered":"2023-10-10T17:41:37.602564+0000","last_clean":"2023-10-10T17:41:37.602564+0000","last_became_active":"2023-10-10T17:40:41.594325+0000","last_became_peered":"2023-10-10T17:40:41.594325+0000","last_unstale":"2023-10-10T17:41:37.602564+0000","last_undegraded":"2023-10-10T17:41:37.602564+0000","last_fullsized":"2023-10-10T17:41:37.602564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:09:20.447064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275229+0000","last_change":"2023-10-10T17:40:41.598032+0000","last_active":"2023-10-10T17:40:46.275229+0000","last_peered":"2023-10-10T17:40:46.275229+0000","last_clean":"2023-10-10T17:40:46.275229+0000","last_became_active":"2023-10-10T17:40:41.597671+0000","last_became_peered":"2023-10-10T17:40:41.597671+0000","last_unstale":"2023-10-10T17:40:46.275229+0000","last_undegraded":"2023-10-10T17:40:46.275229+0000","last_fullsized":"2023-10-10T17:40:46.275229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:45:42.730985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598085+0000","last_change":"2023-10-10T17:40:41.599770+0000","last_active":"2023-10-10T17:41:37.598085+0000","last_peered":"2023-10-10T17:41:37.598085+0000","last_clean":"2023-10-10T17:41:37.598085+0000","last_became_active":"2023-10-10T17:40:41.599449+0000","last_became_peered":"2023-10-10T17:40:41.599449+0000","last_unstale":"2023-10-10T17:41:37.598085+0000","last_undegraded":"2023-10-10T17:41:37.598085+0000","last_fullsized":"2023-10-10T17:41:37.598085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:25:18.482096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598837+0000","last_change":"2023-10-10T17:40:41.610655+0000","last_active":"2023-10-10T17:41:37.598837+0000","last_peered":"2023-10-10T17:41:37.598837+0000","last_clean":"2023-10-10T17:41:37.598837+0000","last_became_active":"2023-10-10T17:40:41.610133+0000","last_became_peered":"2023-10-10T17:40:41.610133+0000","last_unstale":"2023-10-10T17:41:37.598837+0000","last_undegraded":"2023-10-10T17:41:37.598837+0000","last_fullsized":"2023-10-10T17:41:37.598837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:32:12.198400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602157+0000","last_change":"2023-10-10T17:40:41.601065+0000","last_active":"2023-10-10T17:41:37.602157+0000","last_peered":"2023-10-10T17:41:37.602157+0000","last_clean":"2023-10-10T17:41:37.602157+0000","last_became_active":"2023-10-10T17:40:41.600530+0000","last_became_peered":"2023-10-10T17:40:41.600530+0000","last_unstale":"2023-10-10T17:41:37.602157+0000","last_undegraded":"2023-10-10T17:41:37.602157+0000","last_fullsized":"2023-10-10T17:41:37.602157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:48:30.086020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602647+0000","last_change":"2023-10-10T17:40:41.596067+0000","last_active":"2023-10-10T17:41:37.602647+0000","last_peered":"2023-10-10T17:41:37.602647+0000","last_clean":"2023-10-10T17:41:37.602647+0000","last_became_active":"2023-10-10T17:40:41.595703+0000","last_became_peered":"2023-10-10T17:40:41.595703+0000","last_unstale":"2023-10-10T17:41:37.602647+0000","last_undegraded":"2023-10-10T17:41:37.602647+0000","last_fullsized":"2023-10-10T17:41:37.602647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:13.451772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598863+0000","last_change":"2023-10-10T17:40:41.593296+0000","last_active":"2023-10-10T17:41:37.598863+0000","last_peered":"2023-10-10T17:41:37.598863+0000","last_clean":"2023-10-10T17:41:37.598863+0000","last_became_active":"2023-10-10T17:40:41.592965+0000","last_became_peered":"2023-10-10T17:40:41.592965+0000","last_unstale":"2023-10-10T17:41:37.598863+0000","last_undegraded":"2023-10-10T17:41:37.598863+0000","last_fullsized":"2023-10-10T17:41:37.598863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:30:41.515067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.597580+0000","last_change":"2023-10-10T17:40:41.604542+0000","last_active":"2023-10-10T17:41:36.597580+0000","last_peered":"2023-10-10T17:41:36.597580+0000","last_clean":"2023-10-10T17:41:36.597580+0000","last_became_active":"2023-10-10T17:40:41.604000+0000","last_became_peered":"2023-10-10T17:40:41.604000+0000","last_unstale":"2023-10-10T17:41:36.597580+0000","last_undegraded":"2023-10-10T17:41:36.597580+0000","last_fullsized":"2023-10-10T17:41:36.597580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:43:38.639547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"peering","last_fresh":"2023-10-10T17:40:46.275141+0000","last_change":"2023-10-10T17:41:36.597743+0000","last_active":"2023-10-10T17:41:36.597743+0000","last_peered":"2023-10-10T17:40:46.275141+0000","last_clean":"2023-10-10T17:40:46.275141+0000","last_became_active":"2023-10-10T17:40:41.610304+0000","last_became_peered":"2023-10-10T17:40:41.610304+0000","last_unstale":"2023-10-10T17:40:46.275141+0000","last_undegraded":"2023-10-10T17:40:46.275141+0000","last_fullsized":"2023-10-10T17:40:46.275141+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,8],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595527+0000","last_change":"2023-10-10T17:40:41.606659+0000","last_active":"2023-10-10T17:41:37.595527+0000","last_peered":"2023-10-10T17:41:37.595527+0000","last_clean":"2023-10-10T17:41:37.595527+0000","last_became_active":"2023-10-10T17:40:41.606061+0000","last_became_peered":"2023-10-10T17:40:41.606061+0000","last_unstale":"2023-10-10T17:41:37.595527+0000","last_undegraded":"2023-10-10T17:41:37.595527+0000","last_fullsized":"2023-10-10T17:41:37.595527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:04:19.222774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276918+0000","last_change":"2023-10-10T17:40:41.607630+0000","last_active":"2023-10-10T17:40:46.276918+0000","last_peered":"2023-10-10T17:40:46.276918+0000","last_clean":"2023-10-10T17:40:46.276918+0000","last_became_active":"2023-10-10T17:40:41.607297+0000","last_became_peered":"2023-10-10T17:40:41.607297+0000","last_unstale":"2023-10-10T17:40:46.276918+0000","last_undegraded":"2023-10-10T17:40:46.276918+0000","last_fullsized":"2023-10-10T17:40:46.276918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:18:11.997842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595355+0000","last_change":"2023-10-10T17:40:41.594082+0000","last_active":"2023-10-10T17:41:37.595355+0000","last_peered":"2023-10-10T17:41:37.595355+0000","last_clean":"2023-10-10T17:41:37.595355+0000","last_became_active":"2023-10-10T17:40:41.593787+0000","last_became_peered":"2023-10-10T17:40:41.593787+0000","last_unstale":"2023-10-10T17:41:37.595355+0000","last_undegraded":"2023-10-10T17:41:37.595355+0000","last_fullsized":"2023-10-10T17:41:37.595355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:10:48.548705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596760+0000","last_change":"2023-10-10T17:40:41.608529+0000","last_active":"2023-10-10T17:41:37.596760+0000","last_peered":"2023-10-10T17:41:37.596760+0000","last_clean":"2023-10-10T17:41:37.596760+0000","last_became_active":"2023-10-10T17:40:41.608247+0000","last_became_peered":"2023-10-10T17:40:41.608247+0000","last_unstale":"2023-10-10T17:41:37.596760+0000","last_undegraded":"2023-10-10T17:41:37.596760+0000","last_fullsized":"2023-10-10T17:41:37.596760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:20:05.900726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596528+0000","last_change":"2023-10-10T17:40:45.707455+0000","last_active":"2023-10-10T17:41:37.596528+0000","last_peered":"2023-10-10T17:41:37.596528+0000","last_clean":"2023-10-10T17:41:37.596528+0000","last_became_active":"2023-10-10T17:40:45.707024+0000","last_became_peered":"2023-10-10T17:40:45.707024+0000","last_unstale":"2023-10-10T17:41:37.596528+0000","last_undegraded":"2023-10-10T17:41:37.596528+0000","last_fullsized":"2023-10-10T17:41:37.596528+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:24.408486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595807+0000","last_change":"2023-10-10T17:40:41.165983+0000","last_active":"2023-10-10T17:41:37.595807+0000","last_peered":"2023-10-10T17:41:37.595807+0000","last_clean":"2023-10-10T17:41:37.595807+0000","last_became_active":"2023-10-10T17:40:41.165487+0000","last_became_peered":"2023-10-10T17:40:41.165487+0000","last_unstale":"2023-10-10T17:41:37.595807+0000","last_undegraded":"2023-10-10T17:41:37.595807+0000","last_fullsized":"2023-10-10T17:41:37.595807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:08:45.809159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275769+0000","last_change":"2023-10-10T17:40:41.614258+0000","last_active":"2023-10-10T17:40:46.275769+0000","last_peered":"2023-10-10T17:40:46.275769+0000","last_clean":"2023-10-10T17:40:46.275769+0000","last_became_active":"2023-10-10T17:40:41.613951+0000","last_became_peered":"2023-10-10T17:40:41.613951+0000","last_unstale":"2023-10-10T17:40:46.275769+0000","last_undegraded":"2023-10-10T17:40:46.275769+0000","last_fullsized":"2023-10-10T17:40:46.275769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:40:16.468408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:45.708568+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:45.708108+0000","last_became_peered":"2023-10-10T17:40:45.708108+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:30:24.153573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276022+0000","last_change":"2023-10-10T17:40:41.167744+0000","last_active":"2023-10-10T17:40:46.276022+0000","last_peered":"2023-10-10T17:40:46.276022+0000","last_clean":"2023-10-10T17:40:46.276022+0000","last_became_active":"2023-10-10T17:40:41.167189+0000","last_became_peered":"2023-10-10T17:40:41.167189+0000","last_unstale":"2023-10-10T17:40:46.276022+0000","last_undegraded":"2023-10-10T17:40:46.276022+0000","last_fullsized":"2023-10-10T17:40:46.276022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:17:11.349794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.598957+0000","last_change":"2023-10-10T17:40:41.608514+0000","last_active":"2023-10-10T17:41:36.598957+0000","last_peered":"2023-10-10T17:41:36.598957+0000","last_clean":"2023-10-10T17:41:36.598957+0000","last_became_active":"2023-10-10T17:40:41.608208+0000","last_became_peered":"2023-10-10T17:40:41.608208+0000","last_unstale":"2023-10-10T17:41:36.598957+0000","last_undegraded":"2023-10-10T17:41:36.598957+0000","last_fullsized":"2023-10-10T17:41:36.598957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:33:50.558022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.981315+0000","last_change":"2023-10-10T17:41:37.981315+0000","last_active":"2023-10-10T17:41:37.981315+0000","last_peered":"2023-10-10T17:41:37.981315+0000","last_clean":"2023-10-10T17:41:37.981315+0000","last_became_active":"2023-10-10T17:41:37.980691+0000","last_became_peered":"2023-10-10T17:41:37.980691+0000","last_unstale":"2023-10-10T17:41:37.981315+0000","last_undegraded":"2023-10-10T17:41:37.981315+0000","last_fullsized":"2023-10-10T17:41:37.981315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:36:12.102836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.602282+0000","last_change":"2023-10-10T17:40:41.169821+0000","last_active":"2023-10-10T17:41:36.602282+0000","last_peered":"2023-10-10T17:41:36.602282+0000","last_clean":"2023-10-10T17:41:36.602282+0000","last_became_active":"2023-10-10T17:40:41.169502+0000","last_became_peered":"2023-10-10T17:40:41.169502+0000","last_unstale":"2023-10-10T17:41:36.602282+0000","last_undegraded":"2023-10-10T17:41:36.602282+0000","last_fullsized":"2023-10-10T17:41:36.602282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:43:43.240050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595719+0000","last_change":"2023-10-10T17:40:41.602720+0000","last_active":"2023-10-10T17:41:37.595719+0000","last_peered":"2023-10-10T17:41:37.595719+0000","last_clean":"2023-10-10T17:41:37.595719+0000","last_became_active":"2023-10-10T17:40:41.602187+0000","last_became_peered":"2023-10-10T17:40:41.602187+0000","last_unstale":"2023-10-10T17:41:37.595719+0000","last_undegraded":"2023-10-10T17:41:37.595719+0000","last_fullsized":"2023-10-10T17:41:37.595719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:18:35.228329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601501+0000","last_change":"2023-10-10T17:40:45.706946+0000","last_active":"2023-10-10T17:41:37.601501+0000","last_peered":"2023-10-10T17:41:37.601501+0000","last_clean":"2023-10-10T17:41:37.601501+0000","last_became_active":"2023-10-10T17:40:45.706495+0000","last_became_peered":"2023-10-10T17:40:45.706495+0000","last_unstale":"2023-10-10T17:41:37.601501+0000","last_undegraded":"2023-10-10T17:41:37.601501+0000","last_fullsized":"2023-10-10T17:41:37.601501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:11.140718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.601667+0000","last_change":"2023-10-10T17:40:41.171855+0000","last_active":"2023-10-10T17:41:36.601667+0000","last_peered":"2023-10-10T17:41:36.601667+0000","last_clean":"2023-10-10T17:41:36.601667+0000","last_became_active":"2023-10-10T17:40:41.171515+0000","last_became_peered":"2023-10-10T17:40:41.171515+0000","last_unstale":"2023-10-10T17:41:36.601667+0000","last_undegraded":"2023-10-10T17:41:36.601667+0000","last_fullsized":"2023-10-10T17:41:36.601667+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:19:22.043725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276592+0000","last_change":"2023-10-10T17:40:41.587083+0000","last_active":"2023-10-10T17:40:46.276592+0000","last_peered":"2023-10-10T17:40:46.276592+0000","last_clean":"2023-10-10T17:40:46.276592+0000","last_became_active":"2023-10-10T17:40:41.586758+0000","last_became_peered":"2023-10-10T17:40:41.586758+0000","last_unstale":"2023-10-10T17:40:46.276592+0000","last_undegraded":"2023-10-10T17:40:46.276592+0000","last_fullsized":"2023-10-10T17:40:46.276592+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:04:36.358639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596265+0000","last_change":"2023-10-10T17:40:45.710492+0000","last_active":"2023-10-10T17:41:37.596265+0000","last_peered":"2023-10-10T17:41:37.596265+0000","last_clean":"2023-10-10T17:41:37.596265+0000","last_became_active":"2023-10-10T17:40:45.710069+0000","last_became_peered":"2023-10-10T17:40:45.710069+0000","last_unstale":"2023-10-10T17:41:37.596265+0000","last_undegraded":"2023-10-10T17:41:37.596265+0000","last_fullsized":"2023-10-10T17:41:37.596265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:58:28.507189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:41:34.645176+0000","last_change":"2023-10-10T17:40:41.167449+0000","last_active":"2023-10-10T17:41:34.645176+0000","last_peered":"2023-10-10T17:41:34.645176+0000","last_clean":"2023-10-10T17:41:34.645176+0000","last_became_active":"2023-10-10T17:40:41.167038+0000","last_became_peered":"2023-10-10T17:40:41.167038+0000","last_unstale":"2023-10-10T17:41:34.645176+0000","last_undegraded":"2023-10-10T17:41:34.645176+0000","last_fullsized":"2023-10-10T17:41:34.645176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:11:53.872618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.600527+0000","last_change":"2023-10-10T17:40:41.594327+0000","last_active":"2023-10-10T17:41:36.600527+0000","last_peered":"2023-10-10T17:41:36.600527+0000","last_clean":"2023-10-10T17:41:36.600527+0000","last_became_active":"2023-10-10T17:40:41.594008+0000","last_became_peered":"2023-10-10T17:40:41.594008+0000","last_unstale":"2023-10-10T17:41:36.600527+0000","last_undegraded":"2023-10-10T17:41:36.600527+0000","last_fullsized":"2023-10-10T17:41:36.600527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:58:12.213841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.653403+0000","last_change":"2023-10-10T17:41:37.653403+0000","last_active":"2023-10-10T17:41:37.653403+0000","last_peered":"2023-10-10T17:41:37.653403+0000","last_clean":"2023-10-10T17:41:37.653403+0000","last_became_active":"2023-10-10T17:41:37.652946+0000","last_became_peered":"2023-10-10T17:41:37.652946+0000","last_unstale":"2023-10-10T17:41:37.653403+0000","last_undegraded":"2023-10-10T17:41:37.653403+0000","last_fullsized":"2023-10-10T17:41:37.653403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:16:29.417131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598235+0000","last_change":"2023-10-10T17:40:41.171926+0000","last_active":"2023-10-10T17:41:37.598235+0000","last_peered":"2023-10-10T17:41:37.598235+0000","last_clean":"2023-10-10T17:41:37.598235+0000","last_became_active":"2023-10-10T17:40:41.171627+0000","last_became_peered":"2023-10-10T17:40:41.171627+0000","last_unstale":"2023-10-10T17:41:37.598235+0000","last_undegraded":"2023-10-10T17:41:37.598235+0000","last_fullsized":"2023-10-10T17:41:37.598235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:00:51.367484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276470+0000","last_change":"2023-10-10T17:40:41.593135+0000","last_active":"2023-10-10T17:40:46.276470+0000","last_peered":"2023-10-10T17:40:46.276470+0000","last_clean":"2023-10-10T17:40:46.276470+0000","last_became_active":"2023-10-10T17:40:41.592871+0000","last_became_peered":"2023-10-10T17:40:41.592871+0000","last_unstale":"2023-10-10T17:40:46.276470+0000","last_undegraded":"2023-10-10T17:40:46.276470+0000","last_fullsized":"2023-10-10T17:40:46.276470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:42:32.503283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275747+0000","last_change":"2023-10-10T17:40:45.703557+0000","last_active":"2023-10-10T17:40:46.275747+0000","last_peered":"2023-10-10T17:40:46.275747+0000","last_clean":"2023-10-10T17:40:46.275747+0000","last_became_active":"2023-10-10T17:40:45.703085+0000","last_became_peered":"2023-10-10T17:40:45.703085+0000","last_unstale":"2023-10-10T17:40:46.275747+0000","last_undegraded":"2023-10-10T17:40:46.275747+0000","last_fullsized":"2023-10-10T17:40:46.275747+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:34:18.186700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598088+0000","last_change":"2023-10-10T17:40:41.162829+0000","last_active":"2023-10-10T17:41:37.598088+0000","last_peered":"2023-10-10T17:41:37.598088+0000","last_clean":"2023-10-10T17:41:37.598088+0000","last_became_active":"2023-10-10T17:40:41.162516+0000","last_became_peered":"2023-10-10T17:40:41.162516+0000","last_unstale":"2023-10-10T17:41:37.598088+0000","last_undegraded":"2023-10-10T17:41:37.598088+0000","last_fullsized":"2023-10-10T17:41:37.598088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:07:29.590599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.789909+0000","last_change":"2023-10-10T17:41:37.789510+0000","last_active":"2023-10-10T17:41:37.789909+0000","last_peered":"2023-10-10T17:41:37.789909+0000","last_clean":"2023-10-10T17:41:37.789909+0000","last_became_active":"2023-10-10T17:41:37.789166+0000","last_became_peered":"2023-10-10T17:41:37.789166+0000","last_unstale":"2023-10-10T17:41:37.789909+0000","last_undegraded":"2023-10-10T17:41:37.789909+0000","last_fullsized":"2023-10-10T17:41:37.789909+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:33:55.993371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.599298+0000","last_change":"2023-10-10T17:40:45.707139+0000","last_active":"2023-10-10T17:41:36.599298+0000","last_peered":"2023-10-10T17:41:36.599298+0000","last_clean":"2023-10-10T17:41:36.599298+0000","last_became_active":"2023-10-10T17:40:45.706708+0000","last_became_peered":"2023-10-10T17:40:45.706708+0000","last_unstale":"2023-10-10T17:41:36.599298+0000","last_undegraded":"2023-10-10T17:41:36.599298+0000","last_fullsized":"2023-10-10T17:41:36.599298+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:05:11.523282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276991+0000","last_change":"2023-10-10T17:40:41.170512+0000","last_active":"2023-10-10T17:40:46.276991+0000","last_peered":"2023-10-10T17:40:46.276991+0000","last_clean":"2023-10-10T17:40:46.276991+0000","last_became_active":"2023-10-10T17:40:41.170264+0000","last_became_peered":"2023-10-10T17:40:41.170264+0000","last_unstale":"2023-10-10T17:40:46.276991+0000","last_undegraded":"2023-10-10T17:40:46.276991+0000","last_fullsized":"2023-10-10T17:40:46.276991+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:29:22.102865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594921+0000","last_change":"2023-10-10T17:40:41.597647+0000","last_active":"2023-10-10T17:41:37.594921+0000","last_peered":"2023-10-10T17:41:37.594921+0000","last_clean":"2023-10-10T17:41:37.594921+0000","last_became_active":"2023-10-10T17:40:41.597245+0000","last_became_peered":"2023-10-10T17:40:41.597245+0000","last_unstale":"2023-10-10T17:41:37.594921+0000","last_undegraded":"2023-10-10T17:41:37.594921+0000","last_fullsized":"2023-10-10T17:41:37.594921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:02.702932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275791+0000","last_change":"2023-10-10T17:40:45.705089+0000","last_active":"2023-10-10T17:40:46.275791+0000","last_peered":"2023-10-10T17:40:46.275791+0000","last_clean":"2023-10-10T17:40:46.275791+0000","last_became_active":"2023-10-10T17:40:45.704475+0000","last_became_peered":"2023-10-10T17:40:45.704475+0000","last_unstale":"2023-10-10T17:40:46.275791+0000","last_undegraded":"2023-10-10T17:40:46.275791+0000","last_fullsized":"2023-10-10T17:40:46.275791+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:51:15.851183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598714+0000","last_change":"2023-10-10T17:40:41.167409+0000","last_active":"2023-10-10T17:41:37.598714+0000","last_peered":"2023-10-10T17:41:37.598714+0000","last_clean":"2023-10-10T17:41:37.598714+0000","last_became_active":"2023-10-10T17:40:41.167125+0000","last_became_peered":"2023-10-10T17:40:41.167125+0000","last_unstale":"2023-10-10T17:41:37.598714+0000","last_undegraded":"2023-10-10T17:41:37.598714+0000","last_fullsized":"2023-10-10T17:41:37.598714+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:12:36.905039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598010+0000","last_change":"2023-10-10T17:40:41.606216+0000","last_active":"2023-10-10T17:41:37.598010+0000","last_peered":"2023-10-10T17:41:37.598010+0000","last_clean":"2023-10-10T17:41:37.598010+0000","last_became_active":"2023-10-10T17:40:41.603258+0000","last_became_peered":"2023-10-10T17:40:41.603258+0000","last_unstale":"2023-10-10T17:41:37.598010+0000","last_undegraded":"2023-10-10T17:41:37.598010+0000","last_fullsized":"2023-10-10T17:41:37.598010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:30:40.923843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594917+0000","last_change":"2023-10-10T17:40:45.705623+0000","last_active":"2023-10-10T17:41:37.594917+0000","last_peered":"2023-10-10T17:41:37.594917+0000","last_clean":"2023-10-10T17:41:37.594917+0000","last_became_active":"2023-10-10T17:40:45.705242+0000","last_became_peered":"2023-10-10T17:40:45.705242+0000","last_unstale":"2023-10-10T17:41:37.594917+0000","last_undegraded":"2023-10-10T17:41:37.594917+0000","last_fullsized":"2023-10-10T17:41:37.594917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:43:18.391641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601242+0000","last_change":"2023-10-10T17:40:41.171552+0000","last_active":"2023-10-10T17:41:37.601242+0000","last_peered":"2023-10-10T17:41:37.601242+0000","last_clean":"2023-10-10T17:41:37.601242+0000","last_became_active":"2023-10-10T17:40:41.171207+0000","last_became_peered":"2023-10-10T17:40:41.171207+0000","last_unstale":"2023-10-10T17:41:37.601242+0000","last_undegraded":"2023-10-10T17:41:37.601242+0000","last_fullsized":"2023-10-10T17:41:37.601242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:35.436090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.599065+0000","last_change":"2023-10-10T17:40:41.609655+0000","last_active":"2023-10-10T17:41:36.599065+0000","last_peered":"2023-10-10T17:41:36.599065+0000","last_clean":"2023-10-10T17:41:36.599065+0000","last_became_active":"2023-10-10T17:40:41.609318+0000","last_became_peered":"2023-10-10T17:40:41.609318+0000","last_unstale":"2023-10-10T17:41:36.599065+0000","last_undegraded":"2023-10-10T17:41:36.599065+0000","last_fullsized":"2023-10-10T17:41:36.599065+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:04.935363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602379+0000","last_change":"2023-10-10T17:40:45.709513+0000","last_active":"2023-10-10T17:41:37.602379+0000","last_peered":"2023-10-10T17:41:37.602379+0000","last_clean":"2023-10-10T17:41:37.602379+0000","last_became_active":"2023-10-10T17:40:45.708963+0000","last_became_peered":"2023-10-10T17:40:45.708963+0000","last_unstale":"2023-10-10T17:41:37.602379+0000","last_undegraded":"2023-10-10T17:41:37.602379+0000","last_fullsized":"2023-10-10T17:41:37.602379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:15:02.687427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.589227+0000","last_change":"2023-10-10T17:40:41.163234+0000","last_active":"2023-10-10T17:40:59.589227+0000","last_peered":"2023-10-10T17:40:59.589227+0000","last_clean":"2023-10-10T17:40:59.589227+0000","last_became_active":"2023-10-10T17:40:41.162916+0000","last_became_peered":"2023-10-10T17:40:41.162916+0000","last_unstale":"2023-10-10T17:40:59.589227+0000","last_undegraded":"2023-10-10T17:40:59.589227+0000","last_fullsized":"2023-10-10T17:40:59.589227+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:54.672510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598352+0000","last_change":"2023-10-10T17:40:41.602852+0000","last_active":"2023-10-10T17:41:37.598352+0000","last_peered":"2023-10-10T17:41:37.598352+0000","last_clean":"2023-10-10T17:41:37.598352+0000","last_became_active":"2023-10-10T17:40:41.602530+0000","last_became_peered":"2023-10-10T17:40:41.602530+0000","last_unstale":"2023-10-10T17:41:37.598352+0000","last_undegraded":"2023-10-10T17:41:37.598352+0000","last_fullsized":"2023-10-10T17:41:37.598352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:58:41.949886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595188+0000","last_change":"2023-10-10T17:40:45.713714+0000","last_active":"2023-10-10T17:41:37.595188+0000","last_peered":"2023-10-10T17:41:37.595188+0000","last_clean":"2023-10-10T17:41:37.595188+0000","last_became_active":"2023-10-10T17:40:45.712953+0000","last_became_peered":"2023-10-10T17:40:45.712953+0000","last_unstale":"2023-10-10T17:41:37.595188+0000","last_undegraded":"2023-10-10T17:41:37.595188+0000","last_fullsized":"2023-10-10T17:41:37.595188+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:15.222779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.648673+0000","last_change":"2023-10-10T17:40:41.168328+0000","last_active":"2023-10-10T17:41:37.648673+0000","last_peered":"2023-10-10T17:41:37.648673+0000","last_clean":"2023-10-10T17:41:37.648673+0000","last_became_active":"2023-10-10T17:40:41.168044+0000","last_became_peered":"2023-10-10T17:40:41.168044+0000","last_unstale":"2023-10-10T17:41:37.648673+0000","last_undegraded":"2023-10-10T17:41:37.648673+0000","last_fullsized":"2023-10-10T17:41:37.648673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:52:24.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":76,"num_read_kb":42,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.601961+0000","last_change":"2023-10-10T17:40:41.597795+0000","last_active":"2023-10-10T17:41:36.601961+0000","last_peered":"2023-10-10T17:41:36.601961+0000","last_clean":"2023-10-10T17:41:36.601961+0000","last_became_active":"2023-10-10T17:40:41.597537+0000","last_became_peered":"2023-10-10T17:40:41.597537+0000","last_unstale":"2023-10-10T17:41:36.601961+0000","last_undegraded":"2023-10-10T17:41:36.601961+0000","last_fullsized":"2023-10-10T17:41:36.601961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:31:19.492161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277087+0000","last_change":"2023-10-10T17:40:45.709827+0000","last_active":"2023-10-10T17:40:46.277087+0000","last_peered":"2023-10-10T17:40:46.277087+0000","last_clean":"2023-10-10T17:40:46.277087+0000","last_became_active":"2023-10-10T17:40:45.709373+0000","last_became_peered":"2023-10-10T17:40:45.709373+0000","last_unstale":"2023-10-10T17:40:46.277087+0000","last_undegraded":"2023-10-10T17:40:46.277087+0000","last_fullsized":"2023-10-10T17:40:46.277087+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:56:06.379501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596527+0000","last_change":"2023-10-10T17:40:41.160804+0000","last_active":"2023-10-10T17:41:37.596527+0000","last_peered":"2023-10-10T17:41:37.596527+0000","last_clean":"2023-10-10T17:41:37.596527+0000","last_became_active":"2023-10-10T17:40:41.160493+0000","last_became_peered":"2023-10-10T17:40:41.160493+0000","last_unstale":"2023-10-10T17:41:37.596527+0000","last_undegraded":"2023-10-10T17:41:37.596527+0000","last_fullsized":"2023-10-10T17:41:37.596527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:59.561663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598334+0000","last_change":"2023-10-10T17:40:41.601357+0000","last_active":"2023-10-10T17:41:37.598334+0000","last_peered":"2023-10-10T17:41:37.598334+0000","last_clean":"2023-10-10T17:41:37.598334+0000","last_became_active":"2023-10-10T17:40:41.601057+0000","last_became_peered":"2023-10-10T17:40:41.601057+0000","last_unstale":"2023-10-10T17:41:37.598334+0000","last_undegraded":"2023-10-10T17:41:37.598334+0000","last_fullsized":"2023-10-10T17:41:37.598334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:07.286077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275966+0000","last_change":"2023-10-10T17:40:45.708067+0000","last_active":"2023-10-10T17:40:46.275966+0000","last_peered":"2023-10-10T17:40:46.275966+0000","last_clean":"2023-10-10T17:40:46.275966+0000","last_became_active":"2023-10-10T17:40:45.707686+0000","last_became_peered":"2023-10-10T17:40:45.707686+0000","last_unstale":"2023-10-10T17:40:46.275966+0000","last_undegraded":"2023-10-10T17:40:46.275966+0000","last_fullsized":"2023-10-10T17:40:46.275966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:00:14.708537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.591858+0000","last_change":"2023-10-10T17:40:41.168006+0000","last_active":"2023-10-10T17:40:59.591858+0000","last_peered":"2023-10-10T17:40:59.591858+0000","last_clean":"2023-10-10T17:40:59.591858+0000","last_became_active":"2023-10-10T17:40:41.167777+0000","last_became_peered":"2023-10-10T17:40:41.167777+0000","last_unstale":"2023-10-10T17:40:59.591858+0000","last_undegraded":"2023-10-10T17:40:59.591858+0000","last_fullsized":"2023-10-10T17:40:59.591858+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:38:44.110762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601103+0000","last_change":"2023-10-10T17:40:41.591215+0000","last_active":"2023-10-10T17:41:37.601103+0000","last_peered":"2023-10-10T17:41:37.601103+0000","last_clean":"2023-10-10T17:41:37.601103+0000","last_became_active":"2023-10-10T17:40:41.590929+0000","last_became_peered":"2023-10-10T17:40:41.590929+0000","last_unstale":"2023-10-10T17:41:37.601103+0000","last_undegraded":"2023-10-10T17:41:37.601103+0000","last_fullsized":"2023-10-10T17:41:37.601103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:11:13.144245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595485+0000","last_change":"2023-10-10T17:40:45.709038+0000","last_active":"2023-10-10T17:41:37.595485+0000","last_peered":"2023-10-10T17:41:37.595485+0000","last_clean":"2023-10-10T17:41:37.595485+0000","last_became_active":"2023-10-10T17:40:45.708683+0000","last_became_peered":"2023-10-10T17:40:45.708683+0000","last_unstale":"2023-10-10T17:41:37.595485+0000","last_undegraded":"2023-10-10T17:41:37.595485+0000","last_fullsized":"2023-10-10T17:41:37.595485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:50:01.174506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":69,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.661285+0000","last_change":"2023-10-10T17:40:41.163817+0000","last_active":"2023-10-10T17:41:37.661285+0000","last_peered":"2023-10-10T17:41:37.661285+0000","last_clean":"2023-10-10T17:41:37.661285+0000","last_became_active":"2023-10-10T17:40:41.163497+0000","last_became_peered":"2023-10-10T17:40:41.163497+0000","last_unstale":"2023-10-10T17:41:37.661285+0000","last_undegraded":"2023-10-10T17:41:37.661285+0000","last_fullsized":"2023-10-10T17:41:37.661285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:54:45.382546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.607553+0000","last_change":"2023-10-10T17:41:37.607553+0000","last_active":"2023-10-10T17:41:37.607553+0000","last_peered":"2023-10-10T17:41:37.607553+0000","last_clean":"2023-10-10T17:41:37.607553+0000","last_became_active":"2023-10-10T17:41:37.606536+0000","last_became_peered":"2023-10-10T17:41:37.606536+0000","last_unstale":"2023-10-10T17:41:37.607553+0000","last_undegraded":"2023-10-10T17:41:37.607553+0000","last_fullsized":"2023-10-10T17:41:37.607553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:58:09.182227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600059+0000","last_change":"2023-10-10T17:40:45.714900+0000","last_active":"2023-10-10T17:41:37.600059+0000","last_peered":"2023-10-10T17:41:37.600059+0000","last_clean":"2023-10-10T17:41:37.600059+0000","last_became_active":"2023-10-10T17:40:45.714525+0000","last_became_peered":"2023-10-10T17:40:45.714525+0000","last_unstale":"2023-10-10T17:41:37.600059+0000","last_undegraded":"2023-10-10T17:41:37.600059+0000","last_fullsized":"2023-10-10T17:41:37.600059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:01:36.283546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"peering","last_fresh":"2023-10-10T17:41:01.596947+0000","last_change":"2023-10-10T17:41:36.600732+0000","last_active":"2023-10-10T17:41:36.600732+0000","last_peered":"2023-10-10T17:41:01.596947+0000","last_clean":"2023-10-10T17:41:01.596947+0000","last_became_active":"2023-10-10T17:40:41.164046+0000","last_became_peered":"2023-10-10T17:40:41.164046+0000","last_unstale":"2023-10-10T17:41:01.596947+0000","last_undegraded":"2023-10-10T17:41:01.596947+0000","last_fullsized":"2023-10-10T17:41:01.596947+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,4],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277194+0000","last_change":"2023-10-10T17:40:41.586591+0000","last_active":"2023-10-10T17:40:46.277194+0000","last_peered":"2023-10-10T17:40:46.277194+0000","last_clean":"2023-10-10T17:40:46.277194+0000","last_became_active":"2023-10-10T17:40:41.586293+0000","last_became_peered":"2023-10-10T17:40:41.586293+0000","last_unstale":"2023-10-10T17:40:46.277194+0000","last_undegraded":"2023-10-10T17:40:46.277194+0000","last_fullsized":"2023-10-10T17:40:46.277194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:10.920114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277050+0000","last_change":"2023-10-10T17:40:45.708348+0000","last_active":"2023-10-10T17:40:46.277050+0000","last_peered":"2023-10-10T17:40:46.277050+0000","last_clean":"2023-10-10T17:40:46.277050+0000","last_became_active":"2023-10-10T17:40:45.707899+0000","last_became_peered":"2023-10-10T17:40:45.707899+0000","last_unstale":"2023-10-10T17:40:46.277050+0000","last_undegraded":"2023-10-10T17:40:46.277050+0000","last_fullsized":"2023-10-10T17:40:46.277050+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:00:08.005331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.602115+0000","last_change":"2023-10-10T17:40:41.168209+0000","last_active":"2023-10-10T17:41:36.602115+0000","last_peered":"2023-10-10T17:41:36.602115+0000","last_clean":"2023-10-10T17:41:36.602115+0000","last_became_active":"2023-10-10T17:40:41.167907+0000","last_became_peered":"2023-10-10T17:40:41.167907+0000","last_unstale":"2023-10-10T17:41:36.602115+0000","last_undegraded":"2023-10-10T17:41:36.602115+0000","last_fullsized":"2023-10-10T17:41:36.602115+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:14:15.506006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276368+0000","last_change":"2023-10-10T17:40:41.594848+0000","last_active":"2023-10-10T17:40:46.276368+0000","last_peered":"2023-10-10T17:40:46.276368+0000","last_clean":"2023-10-10T17:40:46.276368+0000","last_became_active":"2023-10-10T17:40:41.594562+0000","last_became_peered":"2023-10-10T17:40:41.594562+0000","last_unstale":"2023-10-10T17:40:46.276368+0000","last_undegraded":"2023-10-10T17:40:46.276368+0000","last_fullsized":"2023-10-10T17:40:46.276368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:58:54.606753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.600387+0000","last_change":"2023-10-10T17:40:45.711043+0000","last_active":"2023-10-10T17:41:36.600387+0000","last_peered":"2023-10-10T17:41:36.600387+0000","last_clean":"2023-10-10T17:41:36.600387+0000","last_became_active":"2023-10-10T17:40:45.710617+0000","last_became_peered":"2023-10-10T17:40:45.710617+0000","last_unstale":"2023-10-10T17:41:36.600387+0000","last_undegraded":"2023-10-10T17:41:36.600387+0000","last_fullsized":"2023-10-10T17:41:36.600387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:49:02.846255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595133+0000","last_change":"2023-10-10T17:40:41.163238+0000","last_active":"2023-10-10T17:41:37.595133+0000","last_peered":"2023-10-10T17:41:37.595133+0000","last_clean":"2023-10-10T17:41:37.595133+0000","last_became_active":"2023-10-10T17:40:41.162934+0000","last_became_peered":"2023-10-10T17:40:41.162934+0000","last_unstale":"2023-10-10T17:41:37.595133+0000","last_undegraded":"2023-10-10T17:41:37.595133+0000","last_fullsized":"2023-10-10T17:41:37.595133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:23:10.440706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598371+0000","last_change":"2023-10-10T17:40:41.597026+0000","last_active":"2023-10-10T17:41:37.598371+0000","last_peered":"2023-10-10T17:41:37.598371+0000","last_clean":"2023-10-10T17:41:37.598371+0000","last_became_active":"2023-10-10T17:40:41.596747+0000","last_became_peered":"2023-10-10T17:40:41.596747+0000","last_unstale":"2023-10-10T17:41:37.598371+0000","last_undegraded":"2023-10-10T17:41:37.598371+0000","last_fullsized":"2023-10-10T17:41:37.598371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:52.010575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599056+0000","last_change":"2023-10-10T17:40:45.707218+0000","last_active":"2023-10-10T17:41:37.599056+0000","last_peered":"2023-10-10T17:41:37.599056+0000","last_clean":"2023-10-10T17:41:37.599056+0000","last_became_active":"2023-10-10T17:40:45.706850+0000","last_became_peered":"2023-10-10T17:40:45.706850+0000","last_unstale":"2023-10-10T17:41:37.599056+0000","last_undegraded":"2023-10-10T17:41:37.599056+0000","last_fullsized":"2023-10-10T17:41:37.599056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:36:44.235237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:41.166713+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:41.165951+0000","last_became_peered":"2023-10-10T17:40:41.165951+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:56:00.129165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597783+0000","last_change":"2023-10-10T17:40:41.605860+0000","last_active":"2023-10-10T17:41:37.597783+0000","last_peered":"2023-10-10T17:41:37.597783+0000","last_clean":"2023-10-10T17:41:37.597783+0000","last_became_active":"2023-10-10T17:40:41.605556+0000","last_became_peered":"2023-10-10T17:40:41.605556+0000","last_unstale":"2023-10-10T17:41:37.597783+0000","last_undegraded":"2023-10-10T17:41:37.597783+0000","last_fullsized":"2023-10-10T17:41:37.597783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:07.438058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599757+0000","last_change":"2023-10-10T17:40:45.711844+0000","last_active":"2023-10-10T17:41:37.599757+0000","last_peered":"2023-10-10T17:41:37.599757+0000","last_clean":"2023-10-10T17:41:37.599757+0000","last_became_active":"2023-10-10T17:40:45.711413+0000","last_became_peered":"2023-10-10T17:40:45.711413+0000","last_unstale":"2023-10-10T17:41:37.599757+0000","last_undegraded":"2023-10-10T17:41:37.599757+0000","last_fullsized":"2023-10-10T17:41:37.599757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:13:03.476627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277401+0000","last_change":"2023-10-10T17:40:41.608625+0000","last_active":"2023-10-10T17:40:46.277401+0000","last_peered":"2023-10-10T17:40:46.277401+0000","last_clean":"2023-10-10T17:40:46.277401+0000","last_became_active":"2023-10-10T17:40:41.608332+0000","last_became_peered":"2023-10-10T17:40:41.608332+0000","last_unstale":"2023-10-10T17:40:46.277401+0000","last_undegraded":"2023-10-10T17:40:46.277401+0000","last_fullsized":"2023-10-10T17:40:46.277401+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:17:17.274726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":67,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598001+0000","last_change":"2023-10-10T17:40:41.169807+0000","last_active":"2023-10-10T17:41:37.598001+0000","last_peered":"2023-10-10T17:41:37.598001+0000","last_clean":"2023-10-10T17:41:37.598001+0000","last_became_active":"2023-10-10T17:40:41.169475+0000","last_became_peered":"2023-10-10T17:40:41.169475+0000","last_unstale":"2023-10-10T17:41:37.598001+0000","last_undegraded":"2023-10-10T17:41:37.598001+0000","last_fullsized":"2023-10-10T17:41:37.598001+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:22.887502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":37,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601878+0000","last_change":"2023-10-10T17:40:45.710544+0000","last_active":"2023-10-10T17:41:37.601878+0000","last_peered":"2023-10-10T17:41:37.601878+0000","last_clean":"2023-10-10T17:41:37.601878+0000","last_became_active":"2023-10-10T17:40:45.710084+0000","last_became_peered":"2023-10-10T17:40:45.710084+0000","last_unstale":"2023-10-10T17:41:37.601878+0000","last_undegraded":"2023-10-10T17:41:37.601878+0000","last_fullsized":"2023-10-10T17:41:37.601878+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:44:01.303666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275269+0000","last_change":"2023-10-10T17:40:41.599028+0000","last_active":"2023-10-10T17:40:46.275269+0000","last_peered":"2023-10-10T17:40:46.275269+0000","last_clean":"2023-10-10T17:40:46.275269+0000","last_became_active":"2023-10-10T17:40:41.598684+0000","last_became_peered":"2023-10-10T17:40:41.598684+0000","last_unstale":"2023-10-10T17:40:46.275269+0000","last_undegraded":"2023-10-10T17:40:46.275269+0000","last_fullsized":"2023-10-10T17:40:46.275269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:30:10.790500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602213+0000","last_change":"2023-10-10T17:40:41.165756+0000","last_active":"2023-10-10T17:41:37.602213+0000","last_peered":"2023-10-10T17:41:37.602213+0000","last_clean":"2023-10-10T17:41:37.602213+0000","last_became_active":"2023-10-10T17:40:41.165368+0000","last_became_peered":"2023-10-10T17:40:41.165368+0000","last_unstale":"2023-10-10T17:41:37.602213+0000","last_undegraded":"2023-10-10T17:41:37.602213+0000","last_fullsized":"2023-10-10T17:41:37.602213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:16.089880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600889+0000","last_change":"2023-10-10T17:39:42.508828+0000","last_active":"2023-10-10T17:41:37.600889+0000","last_peered":"2023-10-10T17:41:37.600889+0000","last_clean":"2023-10-10T17:41:37.600889+0000","last_became_active":"2023-10-10T17:39:42.508475+0000","last_became_peered":"2023-10-10T17:39:42.508475+0000","last_unstale":"2023-10-10T17:41:37.600889+0000","last_undegraded":"2023-10-10T17:41:37.600889+0000","last_fullsized":"2023-10-10T17:41:37.600889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_clean_scrub_stamp":"2023-10-10T17:36:46.553699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:08:38.008210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277319+0000","last_change":"2023-10-10T17:40:45.714604+0000","last_active":"2023-10-10T17:40:46.277319+0000","last_peered":"2023-10-10T17:40:46.277319+0000","last_clean":"2023-10-10T17:40:46.277319+0000","last_became_active":"2023-10-10T17:40:45.714185+0000","last_became_peered":"2023-10-10T17:40:45.714185+0000","last_unstale":"2023-10-10T17:40:46.277319+0000","last_undegraded":"2023-10-10T17:40:46.277319+0000","last_fullsized":"2023-10-10T17:40:46.277319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:18:36.214433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597853+0000","last_change":"2023-10-10T17:40:41.610002+0000","last_active":"2023-10-10T17:41:37.597853+0000","last_peered":"2023-10-10T17:41:37.597853+0000","last_clean":"2023-10-10T17:41:37.597853+0000","last_became_active":"2023-10-10T17:40:41.609708+0000","last_became_peered":"2023-10-10T17:40:41.609708+0000","last_unstale":"2023-10-10T17:41:37.597853+0000","last_undegraded":"2023-10-10T17:41:37.597853+0000","last_fullsized":"2023-10-10T17:41:37.597853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:34.611184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601055+0000","last_change":"2023-10-10T17:40:41.167715+0000","last_active":"2023-10-10T17:41:37.601055+0000","last_peered":"2023-10-10T17:41:37.601055+0000","last_clean":"2023-10-10T17:41:37.601055+0000","last_became_active":"2023-10-10T17:40:41.166385+0000","last_became_peered":"2023-10-10T17:40:41.166385+0000","last_unstale":"2023-10-10T17:41:37.601055+0000","last_undegraded":"2023-10-10T17:41:37.601055+0000","last_fullsized":"2023-10-10T17:41:37.601055+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:49:05.385383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276936+0000","last_change":"2023-10-10T17:40:45.707236+0000","last_active":"2023-10-10T17:40:46.276936+0000","last_peered":"2023-10-10T17:40:46.276936+0000","last_clean":"2023-10-10T17:40:46.276936+0000","last_became_active":"2023-10-10T17:40:45.706827+0000","last_became_peered":"2023-10-10T17:40:45.706827+0000","last_unstale":"2023-10-10T17:40:46.276936+0000","last_undegraded":"2023-10-10T17:40:46.276936+0000","last_fullsized":"2023-10-10T17:40:46.276936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:38:34.181039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.602021+0000","last_change":"2023-10-10T17:40:41.170523+0000","last_active":"2023-10-10T17:41:36.602021+0000","last_peered":"2023-10-10T17:41:36.602021+0000","last_clean":"2023-10-10T17:41:36.602021+0000","last_became_active":"2023-10-10T17:40:41.170243+0000","last_became_peered":"2023-10-10T17:40:41.170243+0000","last_unstale":"2023-10-10T17:41:36.602021+0000","last_undegraded":"2023-10-10T17:41:36.602021+0000","last_fullsized":"2023-10-10T17:41:36.602021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:01:16.607917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599195+0000","last_change":"2023-10-10T17:40:41.610472+0000","last_active":"2023-10-10T17:41:37.599195+0000","last_peered":"2023-10-10T17:41:37.599195+0000","last_clean":"2023-10-10T17:41:37.599195+0000","last_became_active":"2023-10-10T17:40:41.610139+0000","last_became_peered":"2023-10-10T17:40:41.610139+0000","last_unstale":"2023-10-10T17:41:37.599195+0000","last_undegraded":"2023-10-10T17:41:37.599195+0000","last_fullsized":"2023-10-10T17:41:37.599195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:22:52.414485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275493+0000","last_change":"2023-10-10T17:40:45.712179+0000","last_active":"2023-10-10T17:40:46.275493+0000","last_peered":"2023-10-10T17:40:46.275493+0000","last_clean":"2023-10-10T17:40:46.275493+0000","last_became_active":"2023-10-10T17:40:45.711708+0000","last_became_peered":"2023-10-10T17:40:45.711708+0000","last_unstale":"2023-10-10T17:40:46.275493+0000","last_undegraded":"2023-10-10T17:40:46.275493+0000","last_fullsized":"2023-10-10T17:40:46.275493+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:49:03.281192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595330+0000","last_change":"2023-10-10T17:40:41.164882+0000","last_active":"2023-10-10T17:41:37.595330+0000","last_peered":"2023-10-10T17:41:37.595330+0000","last_clean":"2023-10-10T17:41:37.595330+0000","last_became_active":"2023-10-10T17:40:41.164536+0000","last_became_peered":"2023-10-10T17:40:41.164536+0000","last_unstale":"2023-10-10T17:41:37.595330+0000","last_undegraded":"2023-10-10T17:41:37.595330+0000","last_fullsized":"2023-10-10T17:41:37.595330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:20:38.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276385+0000","last_change":"2023-10-10T17:40:41.595043+0000","last_active":"2023-10-10T17:40:46.276385+0000","last_peered":"2023-10-10T17:40:46.276385+0000","last_clean":"2023-10-10T17:40:46.276385+0000","last_became_active":"2023-10-10T17:40:41.594703+0000","last_became_peered":"2023-10-10T17:40:41.594703+0000","last_unstale":"2023-10-10T17:40:46.276385+0000","last_undegraded":"2023-10-10T17:40:46.276385+0000","last_fullsized":"2023-10-10T17:40:46.276385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:30:41.976969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596130+0000","last_change":"2023-10-10T17:40:45.705232+0000","last_active":"2023-10-10T17:41:37.596130+0000","last_peered":"2023-10-10T17:41:37.596130+0000","last_clean":"2023-10-10T17:41:37.596130+0000","last_became_active":"2023-10-10T17:40:45.704811+0000","last_became_peered":"2023-10-10T17:40:45.704811+0000","last_unstale":"2023-10-10T17:41:37.596130+0000","last_undegraded":"2023-10-10T17:41:37.596130+0000","last_fullsized":"2023-10-10T17:41:37.596130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:50.034871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602441+0000","last_change":"2023-10-10T17:40:41.165620+0000","last_active":"2023-10-10T17:41:37.602441+0000","last_peered":"2023-10-10T17:41:37.602441+0000","last_clean":"2023-10-10T17:41:37.602441+0000","last_became_active":"2023-10-10T17:40:41.165262+0000","last_became_peered":"2023-10-10T17:40:41.165262+0000","last_unstale":"2023-10-10T17:41:37.602441+0000","last_undegraded":"2023-10-10T17:41:37.602441+0000","last_fullsized":"2023-10-10T17:41:37.602441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:19.126274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.599538+0000","last_change":"2023-10-10T17:40:41.603549+0000","last_active":"2023-10-10T17:41:36.599538+0000","last_peered":"2023-10-10T17:41:36.599538+0000","last_clean":"2023-10-10T17:41:36.599538+0000","last_became_active":"2023-10-10T17:40:41.602991+0000","last_became_peered":"2023-10-10T17:40:41.602991+0000","last_unstale":"2023-10-10T17:41:36.599538+0000","last_undegraded":"2023-10-10T17:41:36.599538+0000","last_fullsized":"2023-10-10T17:41:36.599538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:56:48.836023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.606557+0000","last_change":"2023-10-10T17:41:37.606557+0000","last_active":"2023-10-10T17:41:37.606557+0000","last_peered":"2023-10-10T17:41:37.606557+0000","last_clean":"2023-10-10T17:41:37.606557+0000","last_became_active":"2023-10-10T17:41:37.606070+0000","last_became_peered":"2023-10-10T17:41:37.606070+0000","last_unstale":"2023-10-10T17:41:37.606557+0000","last_undegraded":"2023-10-10T17:41:37.606557+0000","last_fullsized":"2023-10-10T17:41:37.606557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:09:50.492126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595549+0000","last_change":"2023-10-10T17:40:41.169859+0000","last_active":"2023-10-10T17:41:37.595549+0000","last_peered":"2023-10-10T17:41:37.595549+0000","last_clean":"2023-10-10T17:41:37.595549+0000","last_became_active":"2023-10-10T17:40:41.169507+0000","last_became_peered":"2023-10-10T17:40:41.169507+0000","last_unstale":"2023-10-10T17:41:37.595549+0000","last_undegraded":"2023-10-10T17:41:37.595549+0000","last_fullsized":"2023-10-10T17:41:37.595549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:29:19.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.600092+0000","last_change":"2023-10-10T17:40:41.593929+0000","last_active":"2023-10-10T17:41:36.600092+0000","last_peered":"2023-10-10T17:41:36.600092+0000","last_clean":"2023-10-10T17:41:36.600092+0000","last_became_active":"2023-10-10T17:40:41.593565+0000","last_became_peered":"2023-10-10T17:40:41.593565+0000","last_unstale":"2023-10-10T17:41:36.600092+0000","last_undegraded":"2023-10-10T17:41:36.600092+0000","last_fullsized":"2023-10-10T17:41:36.600092+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:38:49.137496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595626+0000","last_change":"2023-10-10T17:40:45.710759+0000","last_active":"2023-10-10T17:41:37.595626+0000","last_peered":"2023-10-10T17:41:37.595626+0000","last_clean":"2023-10-10T17:41:37.595626+0000","last_became_active":"2023-10-10T17:40:45.710118+0000","last_became_peered":"2023-10-10T17:40:45.710118+0000","last_unstale":"2023-10-10T17:41:37.595626+0000","last_undegraded":"2023-10-10T17:41:37.595626+0000","last_fullsized":"2023-10-10T17:41:37.595626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:03:22.069829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:59.590803+0000","last_change":"2023-10-10T17:40:41.165988+0000","last_active":"2023-10-10T17:40:59.590803+0000","last_peered":"2023-10-10T17:40:59.590803+0000","last_clean":"2023-10-10T17:40:59.590803+0000","last_became_active":"2023-10-10T17:40:41.165407+0000","last_became_peered":"2023-10-10T17:40:41.165407+0000","last_unstale":"2023-10-10T17:40:59.590803+0000","last_undegraded":"2023-10-10T17:40:59.590803+0000","last_fullsized":"2023-10-10T17:40:59.590803+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:54.001805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275819+0000","last_change":"2023-10-10T17:40:41.609834+0000","last_active":"2023-10-10T17:40:46.275819+0000","last_peered":"2023-10-10T17:40:46.275819+0000","last_clean":"2023-10-10T17:40:46.275819+0000","last_became_active":"2023-10-10T17:40:41.609509+0000","last_became_peered":"2023-10-10T17:40:41.609509+0000","last_unstale":"2023-10-10T17:40:46.275819+0000","last_undegraded":"2023-10-10T17:40:46.275819+0000","last_fullsized":"2023-10-10T17:40:46.275819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:42:46.393277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602247+0000","last_change":"2023-10-10T17:40:45.709003+0000","last_active":"2023-10-10T17:41:37.602247+0000","last_peered":"2023-10-10T17:41:37.602247+0000","last_clean":"2023-10-10T17:41:37.602247+0000","last_became_active":"2023-10-10T17:40:45.708610+0000","last_became_peered":"2023-10-10T17:40:45.708610+0000","last_unstale":"2023-10-10T17:41:37.602247+0000","last_undegraded":"2023-10-10T17:41:37.602247+0000","last_fullsized":"2023-10-10T17:41:37.602247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:04:55.725864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596673+0000","last_change":"2023-10-10T17:40:41.168104+0000","last_active":"2023-10-10T17:41:37.596673+0000","last_peered":"2023-10-10T17:41:37.596673+0000","last_clean":"2023-10-10T17:41:37.596673+0000","last_became_active":"2023-10-10T17:40:41.167802+0000","last_became_peered":"2023-10-10T17:40:41.167802+0000","last_unstale":"2023-10-10T17:41:37.596673+0000","last_undegraded":"2023-10-10T17:41:37.596673+0000","last_fullsized":"2023-10-10T17:41:37.596673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:22:40.050459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276168+0000","last_change":"2023-10-10T17:40:41.606553+0000","last_active":"2023-10-10T17:40:46.276168+0000","last_peered":"2023-10-10T17:40:46.276168+0000","last_clean":"2023-10-10T17:40:46.276168+0000","last_became_active":"2023-10-10T17:40:41.606087+0000","last_became_peered":"2023-10-10T17:40:41.606087+0000","last_unstale":"2023-10-10T17:40:46.276168+0000","last_undegraded":"2023-10-10T17:40:46.276168+0000","last_fullsized":"2023-10-10T17:40:46.276168+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:13:31.697453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602105+0000","last_change":"2023-10-10T17:40:45.712028+0000","last_active":"2023-10-10T17:41:37.602105+0000","last_peered":"2023-10-10T17:41:37.602105+0000","last_clean":"2023-10-10T17:41:37.602105+0000","last_became_active":"2023-10-10T17:40:45.711534+0000","last_became_peered":"2023-10-10T17:40:45.711534+0000","last_unstale":"2023-10-10T17:41:37.602105+0000","last_undegraded":"2023-10-10T17:41:37.602105+0000","last_fullsized":"2023-10-10T17:41:37.602105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:51:36.426627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.601290+0000","last_change":"2023-10-10T17:40:41.164456+0000","last_active":"2023-10-10T17:41:36.601290+0000","last_peered":"2023-10-10T17:41:36.601290+0000","last_clean":"2023-10-10T17:41:36.601290+0000","last_became_active":"2023-10-10T17:40:41.164018+0000","last_became_peered":"2023-10-10T17:40:41.164018+0000","last_unstale":"2023-10-10T17:41:36.601290+0000","last_undegraded":"2023-10-10T17:41:36.601290+0000","last_fullsized":"2023-10-10T17:41:36.601290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:52:52.201237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601647+0000","last_change":"2023-10-10T17:40:41.599724+0000","last_active":"2023-10-10T17:41:37.601647+0000","last_peered":"2023-10-10T17:41:37.601647+0000","last_clean":"2023-10-10T17:41:37.601647+0000","last_became_active":"2023-10-10T17:40:41.599356+0000","last_became_peered":"2023-10-10T17:40:41.599356+0000","last_unstale":"2023-10-10T17:41:37.601647+0000","last_undegraded":"2023-10-10T17:41:37.601647+0000","last_fullsized":"2023-10-10T17:41:37.601647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:57.775792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596111+0000","last_change":"2023-10-10T17:40:45.704911+0000","last_active":"2023-10-10T17:41:37.596111+0000","last_peered":"2023-10-10T17:41:37.596111+0000","last_clean":"2023-10-10T17:41:37.596111+0000","last_became_active":"2023-10-10T17:40:45.704299+0000","last_became_peered":"2023-10-10T17:40:45.704299+0000","last_unstale":"2023-10-10T17:41:37.596111+0000","last_undegraded":"2023-10-10T17:41:37.596111+0000","last_fullsized":"2023-10-10T17:41:37.596111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:13:42.759241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597150+0000","last_change":"2023-10-10T17:40:41.166091+0000","last_active":"2023-10-10T17:41:37.597150+0000","last_peered":"2023-10-10T17:41:37.597150+0000","last_clean":"2023-10-10T17:41:37.597150+0000","last_became_active":"2023-10-10T17:40:41.165610+0000","last_became_peered":"2023-10-10T17:40:41.165610+0000","last_unstale":"2023-10-10T17:41:37.597150+0000","last_undegraded":"2023-10-10T17:41:37.597150+0000","last_fullsized":"2023-10-10T17:41:37.597150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:31:09.299708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602252+0000","last_change":"2023-10-10T17:40:41.600837+0000","last_active":"2023-10-10T17:41:37.602252+0000","last_peered":"2023-10-10T17:41:37.602252+0000","last_clean":"2023-10-10T17:41:37.602252+0000","last_became_active":"2023-10-10T17:40:41.600090+0000","last_became_peered":"2023-10-10T17:40:41.600090+0000","last_unstale":"2023-10-10T17:41:37.602252+0000","last_undegraded":"2023-10-10T17:41:37.602252+0000","last_fullsized":"2023-10-10T17:41:37.602252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:15:21.942555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"peering","last_fresh":"2023-10-10T17:40:47.578233+0000","last_change":"2023-10-10T17:41:36.598264+0000","last_active":"2023-10-10T17:41:36.598264+0000","last_peered":"2023-10-10T17:40:47.578233+0000","last_clean":"2023-10-10T17:40:47.578233+0000","last_became_active":"2023-10-10T17:40:45.705301+0000","last_became_peered":"2023-10-10T17:40:45.705301+0000","last_unstale":"2023-10-10T17:40:47.578233+0000","last_undegraded":"2023-10-10T17:40:47.578233+0000","last_fullsized":"2023-10-10T17:40:47.578233+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,8,9],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276798+0000","last_change":"2023-10-10T17:40:41.169820+0000","last_active":"2023-10-10T17:40:46.276798+0000","last_peered":"2023-10-10T17:40:46.276798+0000","last_clean":"2023-10-10T17:40:46.276798+0000","last_became_active":"2023-10-10T17:40:41.168767+0000","last_became_peered":"2023-10-10T17:40:41.168767+0000","last_unstale":"2023-10-10T17:40:46.276798+0000","last_undegraded":"2023-10-10T17:40:46.276798+0000","last_fullsized":"2023-10-10T17:40:46.276798+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:55:51.881314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603091+0000","last_change":"2023-10-10T17:40:41.597482+0000","last_active":"2023-10-10T17:41:37.603091+0000","last_peered":"2023-10-10T17:41:37.603091+0000","last_clean":"2023-10-10T17:41:37.603091+0000","last_became_active":"2023-10-10T17:40:41.597166+0000","last_became_peered":"2023-10-10T17:40:41.597166+0000","last_unstale":"2023-10-10T17:41:37.603091+0000","last_undegraded":"2023-10-10T17:41:37.603091+0000","last_fullsized":"2023-10-10T17:41:37.603091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:11.444261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276549+0000","last_change":"2023-10-10T17:40:45.707175+0000","last_active":"2023-10-10T17:40:46.276549+0000","last_peered":"2023-10-10T17:40:46.276549+0000","last_clean":"2023-10-10T17:40:46.276549+0000","last_became_active":"2023-10-10T17:40:45.706448+0000","last_became_peered":"2023-10-10T17:40:45.706448+0000","last_unstale":"2023-10-10T17:40:46.276549+0000","last_undegraded":"2023-10-10T17:40:46.276549+0000","last_fullsized":"2023-10-10T17:40:46.276549+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:59:16.405321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600136+0000","last_change":"2023-10-10T17:40:41.169806+0000","last_active":"2023-10-10T17:41:37.600136+0000","last_peered":"2023-10-10T17:41:37.600136+0000","last_clean":"2023-10-10T17:41:37.600136+0000","last_became_active":"2023-10-10T17:40:41.169496+0000","last_became_peered":"2023-10-10T17:40:41.169496+0000","last_unstale":"2023-10-10T17:41:37.600136+0000","last_undegraded":"2023-10-10T17:41:37.600136+0000","last_fullsized":"2023-10-10T17:41:37.600136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:28:16.024298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603755+0000","last_change":"2023-10-10T17:40:41.598366+0000","last_active":"2023-10-10T17:41:37.603755+0000","last_peered":"2023-10-10T17:41:37.603755+0000","last_clean":"2023-10-10T17:41:37.603755+0000","last_became_active":"2023-10-10T17:40:41.598036+0000","last_became_peered":"2023-10-10T17:40:41.598036+0000","last_unstale":"2023-10-10T17:41:37.603755+0000","last_undegraded":"2023-10-10T17:41:37.603755+0000","last_fullsized":"2023-10-10T17:41:37.603755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:59:22.632882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.275953+0000","last_change":"2023-10-10T17:40:41.609923+0000","last_active":"2023-10-10T17:40:46.275953+0000","last_peered":"2023-10-10T17:40:46.275953+0000","last_clean":"2023-10-10T17:40:46.275953+0000","last_became_active":"2023-10-10T17:40:41.609575+0000","last_became_peered":"2023-10-10T17:40:41.609575+0000","last_unstale":"2023-10-10T17:40:46.275953+0000","last_undegraded":"2023-10-10T17:40:46.275953+0000","last_fullsized":"2023-10-10T17:40:46.275953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:51:30.575597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596871+0000","last_change":"2023-10-10T17:40:41.609012+0000","last_active":"2023-10-10T17:41:37.596871+0000","last_peered":"2023-10-10T17:41:37.596871+0000","last_clean":"2023-10-10T17:41:37.596871+0000","last_became_active":"2023-10-10T17:40:41.608738+0000","last_became_peered":"2023-10-10T17:40:41.608738+0000","last_unstale":"2023-10-10T17:41:37.596871+0000","last_undegraded":"2023-10-10T17:41:37.596871+0000","last_fullsized":"2023-10-10T17:41:37.596871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:11:39.107240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596125+0000","last_change":"2023-10-10T17:40:41.608801+0000","last_active":"2023-10-10T17:41:37.596125+0000","last_peered":"2023-10-10T17:41:37.596125+0000","last_clean":"2023-10-10T17:41:37.596125+0000","last_became_active":"2023-10-10T17:40:41.608510+0000","last_became_peered":"2023-10-10T17:40:41.608510+0000","last_unstale":"2023-10-10T17:41:37.596125+0000","last_undegraded":"2023-10-10T17:41:37.596125+0000","last_fullsized":"2023-10-10T17:41:37.596125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:16:53.945359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-10T17:41:36.598109+0000","last_change":"2023-10-10T17:40:41.603516+0000","last_active":"2023-10-10T17:41:36.598109+0000","last_peered":"2023-10-10T17:41:36.598109+0000","last_clean":"2023-10-10T17:41:36.598109+0000","last_became_active":"2023-10-10T17:40:41.602948+0000","last_became_peered":"2023-10-10T17:40:41.602948+0000","last_unstale":"2023-10-10T17:41:36.598109+0000","last_undegraded":"2023-10-10T17:41:36.598109+0000","last_fullsized":"2023-10-10T17:41:36.598109+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:53:00.813483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602112+0000","last_change":"2023-10-10T17:41:37.602112+0000","last_active":"2023-10-10T17:41:37.602112+0000","last_peered":"2023-10-10T17:41:37.602112+0000","last_clean":"2023-10-10T17:41:37.602112+0000","last_became_active":"2023-10-10T17:41:37.601781+0000","last_became_peered":"2023-10-10T17:41:37.601781+0000","last_unstale":"2023-10-10T17:41:37.602112+0000","last_undegraded":"2023-10-10T17:41:37.602112+0000","last_fullsized":"2023-10-10T17:41:37.602112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:00.565768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276363+0000","last_change":"2023-10-10T17:40:41.597595+0000","last_active":"2023-10-10T17:40:46.276363+0000","last_peered":"2023-10-10T17:40:46.276363+0000","last_clean":"2023-10-10T17:40:46.276363+0000","last_became_active":"2023-10-10T17:40:41.597301+0000","last_became_peered":"2023-10-10T17:40:41.597301+0000","last_unstale":"2023-10-10T17:40:46.276363+0000","last_undegraded":"2023-10-10T17:40:46.276363+0000","last_fullsized":"2023-10-10T17:40:46.276363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:14.213403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596929+0000","last_change":"2023-10-10T17:40:41.611841+0000","last_active":"2023-10-10T17:41:37.596929+0000","last_peered":"2023-10-10T17:41:37.596929+0000","last_clean":"2023-10-10T17:41:37.596929+0000","last_became_active":"2023-10-10T17:40:41.611438+0000","last_became_peered":"2023-10-10T17:40:41.611438+0000","last_unstale":"2023-10-10T17:41:37.596929+0000","last_undegraded":"2023-10-10T17:41:37.596929+0000","last_fullsized":"2023-10-10T17:41:37.596929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:29:05.260738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276423+0000","last_change":"2023-10-10T17:40:41.607707+0000","last_active":"2023-10-10T17:40:46.276423+0000","last_peered":"2023-10-10T17:40:46.276423+0000","last_clean":"2023-10-10T17:40:46.276423+0000","last_became_active":"2023-10-10T17:40:41.607412+0000","last_became_peered":"2023-10-10T17:40:41.607412+0000","last_unstale":"2023-10-10T17:40:46.276423+0000","last_undegraded":"2023-10-10T17:40:46.276423+0000","last_fullsized":"2023-10-10T17:40:46.276423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:46:36.504686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.277088+0000","last_change":"2023-10-10T17:40:41.593674+0000","last_active":"2023-10-10T17:40:46.277088+0000","last_peered":"2023-10-10T17:40:46.277088+0000","last_clean":"2023-10-10T17:40:46.277088+0000","last_became_active":"2023-10-10T17:40:41.593339+0000","last_became_peered":"2023-10-10T17:40:41.593339+0000","last_unstale":"2023-10-10T17:40:46.277088+0000","last_undegraded":"2023-10-10T17:40:46.277088+0000","last_fullsized":"2023-10-10T17:40:46.277088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:41:56.490267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276071+0000","last_change":"2023-10-10T17:40:41.610075+0000","last_active":"2023-10-10T17:40:46.276071+0000","last_peered":"2023-10-10T17:40:46.276071+0000","last_clean":"2023-10-10T17:40:46.276071+0000","last_became_active":"2023-10-10T17:40:41.609669+0000","last_became_peered":"2023-10-10T17:40:41.609669+0000","last_unstale":"2023-10-10T17:40:46.276071+0000","last_undegraded":"2023-10-10T17:40:46.276071+0000","last_fullsized":"2023-10-10T17:40:46.276071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:28:56.002362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596551+0000","last_change":"2023-10-10T17:40:41.596780+0000","last_active":"2023-10-10T17:41:37.596551+0000","last_peered":"2023-10-10T17:41:37.596551+0000","last_clean":"2023-10-10T17:41:37.596551+0000","last_became_active":"2023-10-10T17:40:41.596477+0000","last_became_peered":"2023-10-10T17:40:41.596477+0000","last_unstale":"2023-10-10T17:41:37.596551+0000","last_undegraded":"2023-10-10T17:41:37.596551+0000","last_fullsized":"2023-10-10T17:41:37.596551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:51:14.200732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598899+0000","last_change":"2023-10-10T17:40:41.606226+0000","last_active":"2023-10-10T17:41:37.598899+0000","last_peered":"2023-10-10T17:41:37.598899+0000","last_clean":"2023-10-10T17:41:37.598899+0000","last_became_active":"2023-10-10T17:40:41.605843+0000","last_became_peered":"2023-10-10T17:40:41.605843+0000","last_unstale":"2023-10-10T17:41:37.598899+0000","last_undegraded":"2023-10-10T17:41:37.598899+0000","last_fullsized":"2023-10-10T17:41:37.598899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:44.165622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-10-10T17:40:46.276695+0000","last_change":"2023-10-10T17:40:41.596122+0000","last_active":"2023-10-10T17:40:46.276695+0000","last_peered":"2023-10-10T17:40:46.276695+0000","last_clean":"2023-10-10T17:40:46.276695+0000","last_became_active":"2023-10-10T17:40:41.595818+0000","last_became_peered":"2023-10-10T17:40:41.595818+0000","last_unstale":"2023-10-10T17:40:46.276695+0000","last_undegraded":"2023-10-10T17:40:46.276695+0000","last_fullsized":"2023-10-10T17:40:46.276695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:46:33.542672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250871,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7308,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":241,"num_read_kb":134,"num_write":129,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":21924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383703,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6068,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739084,"statfs":{"total":95995035648,"available":95988822016,"internally_reserved":0,"allocated":962560,"data_stored":473302,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":2.6739999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.70499999999999996}]},{"osd":2,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.61499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":4.1319999999999997,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.748}]},{"osd":5,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.72499999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.66800000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.83599999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.81200000000000006}]},{"osd":10,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.82499999999999996}]}]},{"osd":10,"up_from":73,"seq":313532612634,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6432,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738720,"statfs":{"total":95995035648,"available":95988449280,"internally_reserved":0,"allocated":1073152,"data_stored":1087165,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":1,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.371}]},{"osd":2,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.621},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.73699999999999999}]},{"osd":3,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.437}]},{"osd":4,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.52500000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.70599999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.59299999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.70899999999999996}]},{"osd":8,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.751},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.80800000000000005}]},{"osd":9,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.004,"5min":1.004,"15min":1.004},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.67500000000000004}]},{"osd":11,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.79500000000000004}]}]},{"osd":9,"up_from":68,"seq":292057776160,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":1003520,"data_stored":492622,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48199999999999998,"15min":0.502},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54000000000000004,"5min":1.016,"15min":1.016},"last":0.27800000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85799999999999998,"5min":1.103,"15min":1.103},"last":0.371},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.755,"5min":1.03,"15min":1.03},"last":0.496}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59899999999999998,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69199999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70199999999999996,"5min":1.151,"15min":1.151},"last":0.40699999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.54000000000000004,"15min":0.56299999999999994},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.62,"5min":0.748,"15min":0.748},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.54600000000000004,"15min":0.56899999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64300000000000002,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.29899999999999999}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73799999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.378},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68999999999999995,"15min":0.70999999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48299999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.69599999999999995,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.436,"15min":0.436},"max":{"1min":0.77000000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69899999999999995,"15min":0.72099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.71299999999999997,"5min":1.417,"15min":1.417},"last":0.39200000000000002}]},{"osd":6,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.58099999999999996}]},{"osd":7,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.76700000000000002,"15min":0.79100000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.80900000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.46600000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.753,"15min":0.77500000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77600000000000002,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.78900000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79400000000000004,"5min":1.377,"15min":1.377},"last":0.50800000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:40:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":8,"up_from":60,"seq":257698037796,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6404,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738748,"statfs":{"total":95995035648,"available":95988477952,"internally_reserved":0,"allocated":978944,"data_stored":473458,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.54700000000000004}]},{"osd":1,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.47199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76000000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.438,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92900000000000005,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39200000000000002},"min":{"1min":0.32400000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61499999999999999,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.45200000000000001,"15min":0.44400000000000001},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63700000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.65300000000000002}]},{"osd":3,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72899999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.748}]},{"osd":4,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.47799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82899999999999996,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60199999999999998,"15min":0.59399999999999997},"min":{"1min":0.52100000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80900000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.72099999999999997}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.52000000000000002},"min":{"1min":0.36099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.84799999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.5,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81499999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96999999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.879}]},{"osd":7,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69199999999999995,"15min":0.67900000000000005},"min":{"1min":0.57999999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0029999999999999,"5min":1.022,"15min":1.022},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.59999999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86799999999999999,"5min":1.101,"15min":1.101},"last":0.89300000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.76900000000000002,"15min":0.71599999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":6.3630000000000004,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.74399999999999999,"15min":0.73399999999999999},"min":{"1min":0.64100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96999999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.90200000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:40:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.91500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92600000000000005}]}]},{"osd":7,"up_from":52,"seq":223338299432,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1044480,"data_stored":1083595,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72099999999999997,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.374}]},{"osd":1,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.39700000000000002,"15min":0.38900000000000001},"min":{"1min":0.31900000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.438,"15min":0.432},"min":{"1min":0.31,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.58199999999999996,"15min":0.57399999999999995},"min":{"1min":0.42999999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.35899999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.42299999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.65800000000000003}]},{"osd":4,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.50600000000000001},"min":{"1min":0.34599999999999997,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.55700000000000005}]},{"osd":5,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.46700000000000003,"5min":0.438,"15min":0.438},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.44800000000000001,"5min":0.441,"15min":0.441},"max":{"1min":0.871,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.749}]},{"osd":6,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.66000000000000003,"15min":0.65600000000000003},"min":{"1min":0.54100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.64600000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:41:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.53300000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93100000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.67900000000000005}]},{"osd":9,"last update":"Tue Oct 10 17:41:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.54700000000000004,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.94399999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.871}]},{"osd":10,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.92300000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462957,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6400,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738752,"statfs":{"total":95995035648,"available":95988482048,"internally_reserved":0,"allocated":974848,"data_stored":482320,"data_compressed":14582,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.374},{"interface":"front","average":{"1min":0.38,"5min":0.375,"15min":0.375},"min":{"1min":0.27700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.40500000000000003}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.41299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.499}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83099999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.67900000000000005}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.442,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.46600000000000003}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57399999999999995,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.52700000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.65000000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.57799999999999996},"min":{"1min":0.46400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89500000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.63400000000000001}]},{"osd":8,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64600000000000002,"15min":0.64300000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88600000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66300000000000003,"15min":0.66600000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.85599999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.625}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76500000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.59199999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.71399999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7368,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737784,"statfs":{"total":95995035648,"available":95987490816,"internally_reserved":0,"allocated":983040,"data_stored":486024,"data_compressed":14582,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57999999999999996,"15min":0.67700000000000005},"min":{"1min":0.39900000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80400000000000005,"5min":5.859,"15min":5.859},"last":0.58599999999999997}]},{"osd":2,"last update":"Tue Oct 10 17:40:53 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.433,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79000000000000004,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.81899999999999995}]},{"osd":3,"last update":"Tue Oct 10 17:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.56100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76900000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.56200000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80600000000000005,"5min":0.875,"15min":0.875},"last":0.68400000000000005}]},{"osd":4,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.45800000000000002,"15min":0.40400000000000003},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.46999999999999997,"15min":0.41499999999999998},"min":{"1min":0.38700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.78300000000000003}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.749},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88700000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.95799999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:41:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.627,"15min":0.60899999999999999},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.88800000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.56000000000000005,"15min":0.54400000000000004},"min":{"1min":0.36099999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.85299999999999998,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.874},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.83299999999999996,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.86499999999999999}]},{"osd":8,"last update":"Tue Oct 10 17:41:01 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.45600000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.68200000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.93899999999999995}]},{"osd":9,"last update":"Tue Oct 10 17:41:19 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75700000000000001,"15min":0.747},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":1.01}]},{"osd":10,"last update":"Tue Oct 10 17:40:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":1.0009999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7384,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737768,"statfs":{"total":95995035648,"available":95987474432,"internally_reserved":0,"allocated":999424,"data_stored":483708,"data_compressed":14582,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 17:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54500000000000004,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.61799999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.56599999999999995,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.54300000000000004}]},{"osd":3,"last update":"Tue Oct 10 17:39:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27700000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63800000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.41099999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:41:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55600000000000005,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.69899999999999995}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.50800000000000001},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.45800000000000002}]},{"osd":6,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.433},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.75800000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.49399999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.626,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.59899999999999998}]},{"osd":8,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.69999999999999996,"15min":0.70699999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89700000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.70299999999999996,"15min":0.70899999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90800000000000003,"5min":1.056,"15min":1.056},"last":0.64400000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.627}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.71699999999999997}]},{"osd":11,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.66300000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7456,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737696,"statfs":{"total":95995035648,"available":95987400704,"internally_reserved":0,"allocated":1007616,"data_stored":496169,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.31,"5min":0.252,"15min":0.252},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48799999999999999,"15min":0.48099999999999998},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.56799999999999995,"15min":0.52000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.484,"5min":5.484,"15min":5.484},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54700000000000004,"15min":0.51700000000000002},"min":{"1min":0.40500000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.753,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55900000000000005,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72799999999999998,"5min":1.099,"15min":1.099},"last":0.72299999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51900000000000002,"15min":0.502},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71999999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52100000000000002,"15min":0.501},"min":{"1min":0.42199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.439}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.439,"15min":0.40699999999999997},"min":{"1min":0.33200000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47799999999999998,"15min":0.45400000000000001},"min":{"1min":0.31,"5min":0.307,"15min":0.307},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.498}]},{"osd":6,"last update":"Tue Oct 10 17:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.74199999999999999,"5min":1.129,"15min":1.129},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59299999999999997,"15min":0.58599999999999997},"min":{"1min":0.44,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83799999999999997,"5min":1.075,"15min":1.075},"last":0.629}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77900000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.75900000000000001}]},{"osd":8,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.60999999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93600000000000005,"5min":1.087,"15min":1.087},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.996,"5min":1.226,"15min":1.226},"last":0.879}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.77900000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6716,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738436,"statfs":{"total":95995035648,"available":95988158464,"internally_reserved":0,"allocated":970752,"data_stored":472673,"data_compressed":11629,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:11 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.34799999999999998},"min":{"1min":0.29499999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.47299999999999998,"5min":0.752,"15min":0.752},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.497,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.39200000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.35799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72399999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.51800000000000002},"min":{"1min":0.36699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.628}]},{"osd":2,"last update":"Tue Oct 10 17:41:11 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56100000000000005,"15min":0.53800000000000003},"min":{"1min":0.435,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56200000000000006,"15min":0.53800000000000003},"min":{"1min":0.44900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74399999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.69999999999999996}]},{"osd":4,"last update":"Tue Oct 10 17:41:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.41099999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84099999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61299999999999999,"15min":0.59299999999999997},"min":{"1min":0.46000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.746,"5min":1.214,"15min":1.214},"last":0.64700000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.48099999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72099999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:41:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.49199999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.40500000000000003}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.73899999999999999,"5min":0.88,"15min":0.88},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86699999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.56999999999999995}]},{"osd":8,"last update":"Tue Oct 10 17:39:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53700000000000003}]},{"osd":9,"last update":"Tue Oct 10 17:41:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65700000000000003,"15min":0.66500000000000004},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.497},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.67800000000000005,"15min":0.69399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.755,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.51000000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60799999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6612,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738540,"statfs":{"total":95995035648,"available":95988264960,"internally_reserved":0,"allocated":995328,"data_stored":492400,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.432,"5min":0.372,"15min":0.372},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.36599999999999999},"min":{"1min":0.31,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65200000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.40799999999999997,"15min":0.372},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.495}]},{"osd":2,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.63400000000000001,"15min":0.57499999999999996},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.80500000000000005}]},{"osd":3,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.60899999999999999,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.1289999999999996,"5min":6.1289999999999996,"15min":6.1289999999999996},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.75900000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.55800000000000005}]},{"osd":5,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.57799999999999996},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86799999999999999,"5min":1.589,"15min":1.589},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.754}]},{"osd":6,"last update":"Tue Oct 10 17:41:16 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.45500000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95099999999999996,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:40:36 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.53100000000000003,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.57699999999999996,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.83999999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.74399999999999999,"15min":0.71199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":6.1639999999999997,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.82699999999999996}]},{"osd":9,"last update":"Tue Oct 10 17:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.502,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.92000000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.76600000000000001,"15min":0.76100000000000001},"min":{"1min":0.53000000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.018,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.93899999999999995}]},{"osd":10,"last update":"Tue Oct 10 17:40:40 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.622},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.88900000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91500000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724593,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6648,"kb_used_data":1008,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738504,"statfs":{"total":95995035648,"available":95988228096,"internally_reserved":0,"allocated":1032192,"data_stored":1064243,"data_compressed":47944,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.58199999999999996,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52000000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.75900000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.434}]},{"osd":1,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73899999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.60599999999999998,"15min":0.55000000000000004},"min":{"1min":0.41199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":7.0570000000000004,"5min":7.0570000000000004,"15min":7.0570000000000004},"last":0.55500000000000005}]},{"osd":2,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.40500000000000003,"15min":0.39900000000000002},"min":{"1min":0.28699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65500000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.45300000000000001,"15min":0.42199999999999999},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.38400000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":2.9950000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57699999999999996,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.49299999999999999}]},{"osd":4,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.64100000000000001,"15min":0.58899999999999997},"min":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":7.3479999999999999,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60799999999999998,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64200000000000002}]},{"osd":6,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":2.9620000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.432,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.54500000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.62,"15min":0.61199999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.88400000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.624,"15min":0.622},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66000000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.59699999999999998,"15min":0.55100000000000005},"min":{"1min":0.38700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.7229999999999999,"5min":6.7229999999999999,"15min":6.7229999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.76500000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70899999999999996,"15min":0.71299999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90300000000000002,"5min":1.034,"15min":1.034},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.71299999999999997,"15min":0.69899999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":0.63}]},{"osd":10,"last update":"Tue Oct 10 17:40:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.69899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T17:41:42.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph pg dump --format=json 2023-10-10T17:41:42.834 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:43.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:43 smithi078 ceph-mon[99494]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:43.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:43 smithi078 ceph-mon[99494]: pgmap v294: 129 pgs: 2 peering, 127 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 13 B/s, 0 objects/s recovering 2023-10-10T17:41:43.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:43 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:41:43.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:43 smithi190 ceph-mon[101220]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:43.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:43 smithi190 ceph-mon[101220]: pgmap v294: 129 pgs: 2 peering, 127 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 13 B/s, 0 objects/s recovering 2023-10-10T17:41:43.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:43 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:41:43.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:43 smithi157 ceph-mon[100896]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:43.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:43 smithi157 ceph-mon[100896]: pgmap v294: 129 pgs: 2 peering, 127 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 13 B/s, 0 objects/s recovering 2023-10-10T17:41:43.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:43 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:41:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 -- 172.21.15.78:0/764395746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ffe00 msgr2=0x7f72c8103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 --2- 172.21.15.78:0/764395746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ffe00 0x7f72c8103dd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f72b80099c0 tx=0x7f72b8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 -- 172.21.15.78:0/764395746 shutdown_connections 2023-10-10T17:41:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 --2- 172.21.15.78:0/764395746 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c80ff460 0x7f72c80ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 --2- 172.21.15.78:0/764395746 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c8104500 0x7f72c810c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 --2- 172.21.15.78:0/764395746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ffe00 0x7f72c8103dd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 -- 172.21.15.78:0/764395746 >> 172.21.15.78:0/764395746 conn(0x7f72c80770a0 msgr2=0x7f72c80774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.044+0000 7f72cfe12700 1 -- 172.21.15.78:0/764395746 shutdown_connections 2023-10-10T17:41:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 -- 172.21.15.78:0/764395746 wait complete. 2023-10-10T17:41:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 Processor -- start 2023-10-10T17:41:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 -- start start 2023-10-10T17:41:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ff460 0x7f72c81952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c80ffe00 0x7f72c8195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c8104500 0x7f72c8199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c819a280 con 0x7f72c80ff460 2023-10-10T17:41:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c819a3f0 con 0x7f72c8104500 2023-10-10T17:41:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.045+0000 7f72cfe12700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c819a5b0 con 0x7f72c80ffe00 2023-10-10T17:41:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ff460 0x7f72c81952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ff460 0x7f72c81952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43076/0 (socket says 172.21.15.78:43076) 2023-10-10T17:41:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 -- 172.21.15.78:0/1920979197 learned_addr learned my addr 172.21.15.78:0/1920979197 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cd3ad700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c80ffe00 0x7f72c8195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72ce3af700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c8104500 0x7f72c8199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 -- 172.21.15.78:0/1920979197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c80ffe00 msgr2=0x7f72c8195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c80ffe00 0x7f72c8195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 -- 172.21.15.78:0/1920979197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c8104500 msgr2=0x7f72c8199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c8104500 0x7f72c8199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 -- 172.21.15.78:0/1920979197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b80095f0 con 0x7f72c80ff460 2023-10-10T17:41:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72ce3af700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c8104500 0x7f72c8199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cdbae700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ff460 0x7f72c81952c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f72c400e9a0 tx=0x7f72c400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72beffd700 1 -- 172.21.15.78:0/1920979197 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4009790 con 0x7f72c80ff460 2023-10-10T17:41:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72beffd700 1 -- 172.21.15.78:0/1920979197 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72c400cbd0 con 0x7f72c80ff460 2023-10-10T17:41:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c810f230 con 0x7f72c80ff460 2023-10-10T17:41:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72beffd700 1 -- 172.21.15.78:0/1920979197 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4018680 con 0x7f72c80ff460 2023-10-10T17:41:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.046+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c810f0a0 con 0x7f72c80ff460 2023-10-10T17:41:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.047+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c810f3c0 con 0x7f72c80ff460 2023-10-10T17:41:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.049+0000 7f72beffd700 1 -- 172.21.15.78:0/1920979197 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f72c400cd40 con 0x7f72c80ff460 2023-10-10T17:41:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.050+0000 7f72beffd700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72b40a1150 0x7f72b40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.050+0000 7f72cd3ad700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72b40a1150 0x7f72b40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.050+0000 7f72beffd700 1 -- 172.21.15.78:0/1920979197 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f72c4014070 con 0x7f72c80ff460 2023-10-10T17:41:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.050+0000 7f72cd3ad700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72b40a1150 0x7f72b40a3600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009990 tx=0x7f72b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.053+0000 7f72beffd700 1 -- 172.21.15.78:0/1920979197 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f72c4096590 con 0x7f72c80ff460 2023-10-10T17:41:44.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.246+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72c80fd440 con 0x7f72b40a1150 2023-10-10T17:41:44.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.259+0000 7f72beffd700 1 -- 172.21.15.78:0/1920979197 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+424000 (secure 0 0 0) 0x7f72c80fd440 con 0x7f72b40a1150 2023-10-10T17:41:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72b40a1150 msgr2=0x7f72b40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:44.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72b40a1150 0x7f72b40a3600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f72b8009990 tx=0x7f72b8005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ff460 msgr2=0x7f72c81952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:44.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ff460 0x7f72c81952c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f72c400e9a0 tx=0x7f72c400ecb0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 shutdown_connections 2023-10-10T17:41:44.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72b40a1150 0x7f72b40a3600 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c8104500 0x7f72c8199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c80ff460 0x7f72c81952c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 --2- 172.21.15.78:0/1920979197 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c80ffe00 0x7f72c8195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:44.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 >> 172.21.15.78:0/1920979197 conn(0x7f72c80770a0 msgr2=0x7f72c80fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 shutdown_connections 2023-10-10T17:41:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:44.263+0000 7f72cfe12700 1 -- 172.21.15.78:0/1920979197 wait complete. 2023-10-10T17:41:44.271 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-10T17:41:44.889 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2023-10-10T17:41:44.232138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597676,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":407,"num_read_kb":268,"num_write":310,"num_write_kb":2981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":416,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80844,"kb_used_data":11884,"kb_used_omap":0,"kb_used_meta":68864,"kb_avail":1124860980,"statfs":{"total":1151940427776,"available":1151857643520,"internally_reserved":0,"allocated":12169216,"data_stored":7692849,"data_compressed":319365,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70516736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":55,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002081"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788069+0000","last_change":"2023-10-10T17:40:41.604268+0000","last_active":"2023-10-10T17:41:37.788069+0000","last_peered":"2023-10-10T17:41:37.788069+0000","last_clean":"2023-10-10T17:41:37.788069+0000","last_became_active":"2023-10-10T17:40:41.603839+0000","last_became_peered":"2023-10-10T17:40:41.603839+0000","last_unstale":"2023-10-10T17:41:37.788069+0000","last_undegraded":"2023-10-10T17:41:37.788069+0000","last_fullsized":"2023-10-10T17:41:37.788069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:00:22.805728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599055+0000","last_change":"2023-10-10T17:40:41.603642+0000","last_active":"2023-10-10T17:41:37.599055+0000","last_peered":"2023-10-10T17:41:37.599055+0000","last_clean":"2023-10-10T17:41:37.599055+0000","last_became_active":"2023-10-10T17:40:41.603091+0000","last_became_peered":"2023-10-10T17:40:41.603091+0000","last_unstale":"2023-10-10T17:41:37.599055+0000","last_undegraded":"2023-10-10T17:41:37.599055+0000","last_fullsized":"2023-10-10T17:41:37.599055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:10:38.807138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598740+0000","last_change":"2023-10-10T17:40:41.610647+0000","last_active":"2023-10-10T17:41:37.598740+0000","last_peered":"2023-10-10T17:41:37.598740+0000","last_clean":"2023-10-10T17:41:37.598740+0000","last_became_active":"2023-10-10T17:40:41.610338+0000","last_became_peered":"2023-10-10T17:40:41.610338+0000","last_unstale":"2023-10-10T17:41:37.598740+0000","last_undegraded":"2023-10-10T17:41:37.598740+0000","last_fullsized":"2023-10-10T17:41:37.598740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:16:20.110335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602832+0000","last_change":"2023-10-10T17:40:41.607832+0000","last_active":"2023-10-10T17:41:37.602832+0000","last_peered":"2023-10-10T17:41:37.602832+0000","last_clean":"2023-10-10T17:41:37.602832+0000","last_became_active":"2023-10-10T17:40:41.607492+0000","last_became_peered":"2023-10-10T17:40:41.607492+0000","last_unstale":"2023-10-10T17:41:37.602832+0000","last_undegraded":"2023-10-10T17:41:37.602832+0000","last_fullsized":"2023-10-10T17:41:37.602832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:43.514129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600669+0000","last_change":"2023-10-10T17:40:41.601975+0000","last_active":"2023-10-10T17:41:37.600669+0000","last_peered":"2023-10-10T17:41:37.600669+0000","last_clean":"2023-10-10T17:41:37.600669+0000","last_became_active":"2023-10-10T17:40:41.601668+0000","last_became_peered":"2023-10-10T17:40:41.601668+0000","last_unstale":"2023-10-10T17:41:37.600669+0000","last_undegraded":"2023-10-10T17:41:37.600669+0000","last_fullsized":"2023-10-10T17:41:37.600669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:25.433615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.604098+0000","last_change":"2023-10-10T17:41:37.604098+0000","last_active":"2023-10-10T17:41:37.604098+0000","last_peered":"2023-10-10T17:41:37.604098+0000","last_clean":"2023-10-10T17:41:37.604098+0000","last_became_active":"2023-10-10T17:41:37.603750+0000","last_became_peered":"2023-10-10T17:41:37.603750+0000","last_unstale":"2023-10-10T17:41:37.604098+0000","last_undegraded":"2023-10-10T17:41:37.604098+0000","last_fullsized":"2023-10-10T17:41:37.604098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:52:52.491557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602564+0000","last_change":"2023-10-10T17:40:41.594628+0000","last_active":"2023-10-10T17:41:37.602564+0000","last_peered":"2023-10-10T17:41:37.602564+0000","last_clean":"2023-10-10T17:41:37.602564+0000","last_became_active":"2023-10-10T17:40:41.594325+0000","last_became_peered":"2023-10-10T17:40:41.594325+0000","last_unstale":"2023-10-10T17:41:37.602564+0000","last_undegraded":"2023-10-10T17:41:37.602564+0000","last_fullsized":"2023-10-10T17:41:37.602564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:09:20.447064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978940+0000","last_change":"2023-10-10T17:40:41.598032+0000","last_active":"2023-10-10T17:41:37.978940+0000","last_peered":"2023-10-10T17:41:37.978940+0000","last_clean":"2023-10-10T17:41:37.978940+0000","last_became_active":"2023-10-10T17:40:41.597671+0000","last_became_peered":"2023-10-10T17:40:41.597671+0000","last_unstale":"2023-10-10T17:41:37.978940+0000","last_undegraded":"2023-10-10T17:41:37.978940+0000","last_fullsized":"2023-10-10T17:41:37.978940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:56:22.583298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598085+0000","last_change":"2023-10-10T17:40:41.599770+0000","last_active":"2023-10-10T17:41:37.598085+0000","last_peered":"2023-10-10T17:41:37.598085+0000","last_clean":"2023-10-10T17:41:37.598085+0000","last_became_active":"2023-10-10T17:40:41.599449+0000","last_became_peered":"2023-10-10T17:40:41.599449+0000","last_unstale":"2023-10-10T17:41:37.598085+0000","last_undegraded":"2023-10-10T17:41:37.598085+0000","last_fullsized":"2023-10-10T17:41:37.598085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:25:18.482096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598837+0000","last_change":"2023-10-10T17:40:41.610655+0000","last_active":"2023-10-10T17:41:37.598837+0000","last_peered":"2023-10-10T17:41:37.598837+0000","last_clean":"2023-10-10T17:41:37.598837+0000","last_became_active":"2023-10-10T17:40:41.610133+0000","last_became_peered":"2023-10-10T17:40:41.610133+0000","last_unstale":"2023-10-10T17:41:37.598837+0000","last_undegraded":"2023-10-10T17:41:37.598837+0000","last_fullsized":"2023-10-10T17:41:37.598837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:32:12.198400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602157+0000","last_change":"2023-10-10T17:40:41.601065+0000","last_active":"2023-10-10T17:41:37.602157+0000","last_peered":"2023-10-10T17:41:37.602157+0000","last_clean":"2023-10-10T17:41:37.602157+0000","last_became_active":"2023-10-10T17:40:41.600530+0000","last_became_peered":"2023-10-10T17:40:41.600530+0000","last_unstale":"2023-10-10T17:41:37.602157+0000","last_undegraded":"2023-10-10T17:41:37.602157+0000","last_fullsized":"2023-10-10T17:41:37.602157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:48:30.086020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602647+0000","last_change":"2023-10-10T17:40:41.596067+0000","last_active":"2023-10-10T17:41:37.602647+0000","last_peered":"2023-10-10T17:41:37.602647+0000","last_clean":"2023-10-10T17:41:37.602647+0000","last_became_active":"2023-10-10T17:40:41.595703+0000","last_became_peered":"2023-10-10T17:40:41.595703+0000","last_unstale":"2023-10-10T17:41:37.602647+0000","last_undegraded":"2023-10-10T17:41:37.602647+0000","last_fullsized":"2023-10-10T17:41:37.602647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:13.451772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598863+0000","last_change":"2023-10-10T17:40:41.593296+0000","last_active":"2023-10-10T17:41:37.598863+0000","last_peered":"2023-10-10T17:41:37.598863+0000","last_clean":"2023-10-10T17:41:37.598863+0000","last_became_active":"2023-10-10T17:40:41.592965+0000","last_became_peered":"2023-10-10T17:40:41.592965+0000","last_unstale":"2023-10-10T17:41:37.598863+0000","last_undegraded":"2023-10-10T17:41:37.598863+0000","last_fullsized":"2023-10-10T17:41:37.598863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:30:41.515067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598115+0000","last_change":"2023-10-10T17:40:41.604542+0000","last_active":"2023-10-10T17:41:37.598115+0000","last_peered":"2023-10-10T17:41:37.598115+0000","last_clean":"2023-10-10T17:41:37.598115+0000","last_became_active":"2023-10-10T17:40:41.604000+0000","last_became_peered":"2023-10-10T17:40:41.604000+0000","last_unstale":"2023-10-10T17:41:37.598115+0000","last_undegraded":"2023-10-10T17:41:37.598115+0000","last_fullsized":"2023-10-10T17:41:37.598115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:42:32.658592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602108+0000","last_change":"2023-10-10T17:41:37.602108+0000","last_active":"2023-10-10T17:41:37.602108+0000","last_peered":"2023-10-10T17:41:37.602108+0000","last_clean":"2023-10-10T17:41:37.602108+0000","last_became_active":"2023-10-10T17:41:37.601778+0000","last_became_peered":"2023-10-10T17:41:37.601778+0000","last_unstale":"2023-10-10T17:41:37.602108+0000","last_undegraded":"2023-10-10T17:41:37.602108+0000","last_fullsized":"2023-10-10T17:41:37.602108+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:48:51.539525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595527+0000","last_change":"2023-10-10T17:40:41.606659+0000","last_active":"2023-10-10T17:41:37.595527+0000","last_peered":"2023-10-10T17:41:37.595527+0000","last_clean":"2023-10-10T17:41:37.595527+0000","last_became_active":"2023-10-10T17:40:41.606061+0000","last_became_peered":"2023-10-10T17:40:41.606061+0000","last_unstale":"2023-10-10T17:41:37.595527+0000","last_undegraded":"2023-10-10T17:41:37.595527+0000","last_fullsized":"2023-10-10T17:41:37.595527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:04:19.222774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788308+0000","last_change":"2023-10-10T17:40:41.607630+0000","last_active":"2023-10-10T17:41:37.788308+0000","last_peered":"2023-10-10T17:41:37.788308+0000","last_clean":"2023-10-10T17:41:37.788308+0000","last_became_active":"2023-10-10T17:40:41.607297+0000","last_became_peered":"2023-10-10T17:40:41.607297+0000","last_unstale":"2023-10-10T17:41:37.788308+0000","last_undegraded":"2023-10-10T17:41:37.788308+0000","last_fullsized":"2023-10-10T17:41:37.788308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:15:06.512286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595355+0000","last_change":"2023-10-10T17:40:41.594082+0000","last_active":"2023-10-10T17:41:37.595355+0000","last_peered":"2023-10-10T17:41:37.595355+0000","last_clean":"2023-10-10T17:41:37.595355+0000","last_became_active":"2023-10-10T17:40:41.593787+0000","last_became_peered":"2023-10-10T17:40:41.593787+0000","last_unstale":"2023-10-10T17:41:37.595355+0000","last_undegraded":"2023-10-10T17:41:37.595355+0000","last_fullsized":"2023-10-10T17:41:37.595355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:10:48.548705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596760+0000","last_change":"2023-10-10T17:40:41.608529+0000","last_active":"2023-10-10T17:41:37.596760+0000","last_peered":"2023-10-10T17:41:37.596760+0000","last_clean":"2023-10-10T17:41:37.596760+0000","last_became_active":"2023-10-10T17:40:41.608247+0000","last_became_peered":"2023-10-10T17:40:41.608247+0000","last_unstale":"2023-10-10T17:41:37.596760+0000","last_undegraded":"2023-10-10T17:41:37.596760+0000","last_fullsized":"2023-10-10T17:41:37.596760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:20:05.900726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596528+0000","last_change":"2023-10-10T17:40:45.707455+0000","last_active":"2023-10-10T17:41:37.596528+0000","last_peered":"2023-10-10T17:41:37.596528+0000","last_clean":"2023-10-10T17:41:37.596528+0000","last_became_active":"2023-10-10T17:40:45.707024+0000","last_became_peered":"2023-10-10T17:40:45.707024+0000","last_unstale":"2023-10-10T17:41:37.596528+0000","last_undegraded":"2023-10-10T17:41:37.596528+0000","last_fullsized":"2023-10-10T17:41:37.596528+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:24.408486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595807+0000","last_change":"2023-10-10T17:40:41.165983+0000","last_active":"2023-10-10T17:41:37.595807+0000","last_peered":"2023-10-10T17:41:37.595807+0000","last_clean":"2023-10-10T17:41:37.595807+0000","last_became_active":"2023-10-10T17:40:41.165487+0000","last_became_peered":"2023-10-10T17:40:41.165487+0000","last_unstale":"2023-10-10T17:41:37.595807+0000","last_undegraded":"2023-10-10T17:41:37.595807+0000","last_fullsized":"2023-10-10T17:41:37.595807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:08:45.809159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979220+0000","last_change":"2023-10-10T17:40:41.614258+0000","last_active":"2023-10-10T17:41:37.979220+0000","last_peered":"2023-10-10T17:41:37.979220+0000","last_clean":"2023-10-10T17:41:37.979220+0000","last_became_active":"2023-10-10T17:40:41.613951+0000","last_became_peered":"2023-10-10T17:40:41.613951+0000","last_unstale":"2023-10-10T17:41:37.979220+0000","last_undegraded":"2023-10-10T17:41:37.979220+0000","last_fullsized":"2023-10-10T17:41:37.979220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:43:34.983899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:45.708568+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:45.708108+0000","last_became_peered":"2023-10-10T17:40:45.708108+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:30:24.153573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980255+0000","last_change":"2023-10-10T17:40:41.167744+0000","last_active":"2023-10-10T17:41:37.980255+0000","last_peered":"2023-10-10T17:41:37.980255+0000","last_clean":"2023-10-10T17:41:37.980255+0000","last_became_active":"2023-10-10T17:40:41.167189+0000","last_became_peered":"2023-10-10T17:40:41.167189+0000","last_unstale":"2023-10-10T17:41:37.980255+0000","last_undegraded":"2023-10-10T17:41:37.980255+0000","last_fullsized":"2023-10-10T17:41:37.980255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:47.667198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598064+0000","last_change":"2023-10-10T17:40:41.608514+0000","last_active":"2023-10-10T17:41:37.598064+0000","last_peered":"2023-10-10T17:41:37.598064+0000","last_clean":"2023-10-10T17:41:37.598064+0000","last_became_active":"2023-10-10T17:40:41.608208+0000","last_became_peered":"2023-10-10T17:40:41.608208+0000","last_unstale":"2023-10-10T17:41:37.598064+0000","last_undegraded":"2023-10-10T17:41:37.598064+0000","last_fullsized":"2023-10-10T17:41:37.598064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:48:25.411862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.981315+0000","last_change":"2023-10-10T17:41:37.981315+0000","last_active":"2023-10-10T17:41:37.981315+0000","last_peered":"2023-10-10T17:41:37.981315+0000","last_clean":"2023-10-10T17:41:37.981315+0000","last_became_active":"2023-10-10T17:41:37.980691+0000","last_became_peered":"2023-10-10T17:41:37.980691+0000","last_unstale":"2023-10-10T17:41:37.981315+0000","last_undegraded":"2023-10-10T17:41:37.981315+0000","last_fullsized":"2023-10-10T17:41:37.981315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:36:12.102836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601523+0000","last_change":"2023-10-10T17:40:41.169821+0000","last_active":"2023-10-10T17:41:37.601523+0000","last_peered":"2023-10-10T17:41:37.601523+0000","last_clean":"2023-10-10T17:41:37.601523+0000","last_became_active":"2023-10-10T17:40:41.169502+0000","last_became_peered":"2023-10-10T17:40:41.169502+0000","last_unstale":"2023-10-10T17:41:37.601523+0000","last_undegraded":"2023-10-10T17:41:37.601523+0000","last_fullsized":"2023-10-10T17:41:37.601523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:32:01.106471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595719+0000","last_change":"2023-10-10T17:40:41.602720+0000","last_active":"2023-10-10T17:41:37.595719+0000","last_peered":"2023-10-10T17:41:37.595719+0000","last_clean":"2023-10-10T17:41:37.595719+0000","last_became_active":"2023-10-10T17:40:41.602187+0000","last_became_peered":"2023-10-10T17:40:41.602187+0000","last_unstale":"2023-10-10T17:41:37.595719+0000","last_undegraded":"2023-10-10T17:41:37.595719+0000","last_fullsized":"2023-10-10T17:41:37.595719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:18:35.228329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601501+0000","last_change":"2023-10-10T17:40:45.706946+0000","last_active":"2023-10-10T17:41:37.601501+0000","last_peered":"2023-10-10T17:41:37.601501+0000","last_clean":"2023-10-10T17:41:37.601501+0000","last_became_active":"2023-10-10T17:40:45.706495+0000","last_became_peered":"2023-10-10T17:40:45.706495+0000","last_unstale":"2023-10-10T17:41:37.601501+0000","last_undegraded":"2023-10-10T17:41:37.601501+0000","last_fullsized":"2023-10-10T17:41:37.601501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:11.140718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601201+0000","last_change":"2023-10-10T17:40:41.171855+0000","last_active":"2023-10-10T17:41:37.601201+0000","last_peered":"2023-10-10T17:41:37.601201+0000","last_clean":"2023-10-10T17:41:37.601201+0000","last_became_active":"2023-10-10T17:40:41.171515+0000","last_became_peered":"2023-10-10T17:40:41.171515+0000","last_unstale":"2023-10-10T17:41:37.601201+0000","last_undegraded":"2023-10-10T17:41:37.601201+0000","last_fullsized":"2023-10-10T17:41:37.601201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:15.556629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652087+0000","last_change":"2023-10-10T17:40:41.587083+0000","last_active":"2023-10-10T17:41:37.652087+0000","last_peered":"2023-10-10T17:41:37.652087+0000","last_clean":"2023-10-10T17:41:37.652087+0000","last_became_active":"2023-10-10T17:40:41.586758+0000","last_became_peered":"2023-10-10T17:40:41.586758+0000","last_unstale":"2023-10-10T17:41:37.652087+0000","last_undegraded":"2023-10-10T17:41:37.652087+0000","last_fullsized":"2023-10-10T17:41:37.652087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:31:30.273631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596265+0000","last_change":"2023-10-10T17:40:45.710492+0000","last_active":"2023-10-10T17:41:37.596265+0000","last_peered":"2023-10-10T17:41:37.596265+0000","last_clean":"2023-10-10T17:41:37.596265+0000","last_became_active":"2023-10-10T17:40:45.710069+0000","last_became_peered":"2023-10-10T17:40:45.710069+0000","last_unstale":"2023-10-10T17:41:37.596265+0000","last_undegraded":"2023-10-10T17:41:37.596265+0000","last_fullsized":"2023-10-10T17:41:37.596265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:58:28.507189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:39.655314+0000","last_change":"2023-10-10T17:40:41.167449+0000","last_active":"2023-10-10T17:41:39.655314+0000","last_peered":"2023-10-10T17:41:39.655314+0000","last_clean":"2023-10-10T17:41:39.655314+0000","last_became_active":"2023-10-10T17:40:41.167038+0000","last_became_peered":"2023-10-10T17:40:41.167038+0000","last_unstale":"2023-10-10T17:41:39.655314+0000","last_undegraded":"2023-10-10T17:41:39.655314+0000","last_fullsized":"2023-10-10T17:41:39.655314+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:48:38.144880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599067+0000","last_change":"2023-10-10T17:40:41.594327+0000","last_active":"2023-10-10T17:41:37.599067+0000","last_peered":"2023-10-10T17:41:37.599067+0000","last_clean":"2023-10-10T17:41:37.599067+0000","last_became_active":"2023-10-10T17:40:41.594008+0000","last_became_peered":"2023-10-10T17:40:41.594008+0000","last_unstale":"2023-10-10T17:41:37.599067+0000","last_undegraded":"2023-10-10T17:41:37.599067+0000","last_fullsized":"2023-10-10T17:41:37.599067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:56:08.300633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.653403+0000","last_change":"2023-10-10T17:41:37.653403+0000","last_active":"2023-10-10T17:41:37.653403+0000","last_peered":"2023-10-10T17:41:37.653403+0000","last_clean":"2023-10-10T17:41:37.653403+0000","last_became_active":"2023-10-10T17:41:37.652946+0000","last_became_peered":"2023-10-10T17:41:37.652946+0000","last_unstale":"2023-10-10T17:41:37.653403+0000","last_undegraded":"2023-10-10T17:41:37.653403+0000","last_fullsized":"2023-10-10T17:41:37.653403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:16:29.417131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598235+0000","last_change":"2023-10-10T17:40:41.171926+0000","last_active":"2023-10-10T17:41:37.598235+0000","last_peered":"2023-10-10T17:41:37.598235+0000","last_clean":"2023-10-10T17:41:37.598235+0000","last_became_active":"2023-10-10T17:40:41.171627+0000","last_became_peered":"2023-10-10T17:40:41.171627+0000","last_unstale":"2023-10-10T17:41:37.598235+0000","last_undegraded":"2023-10-10T17:41:37.598235+0000","last_fullsized":"2023-10-10T17:41:37.598235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:00:51.367484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651905+0000","last_change":"2023-10-10T17:40:41.593135+0000","last_active":"2023-10-10T17:41:37.651905+0000","last_peered":"2023-10-10T17:41:37.651905+0000","last_clean":"2023-10-10T17:41:37.651905+0000","last_became_active":"2023-10-10T17:40:41.592871+0000","last_became_peered":"2023-10-10T17:40:41.592871+0000","last_unstale":"2023-10-10T17:41:37.651905+0000","last_undegraded":"2023-10-10T17:41:37.651905+0000","last_fullsized":"2023-10-10T17:41:37.651905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:32.143194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788755+0000","last_change":"2023-10-10T17:40:45.703557+0000","last_active":"2023-10-10T17:41:37.788755+0000","last_peered":"2023-10-10T17:41:37.788755+0000","last_clean":"2023-10-10T17:41:37.788755+0000","last_became_active":"2023-10-10T17:40:45.703085+0000","last_became_peered":"2023-10-10T17:40:45.703085+0000","last_unstale":"2023-10-10T17:41:37.788755+0000","last_undegraded":"2023-10-10T17:41:37.788755+0000","last_fullsized":"2023-10-10T17:41:37.788755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:09:11.130227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598088+0000","last_change":"2023-10-10T17:40:41.162829+0000","last_active":"2023-10-10T17:41:37.598088+0000","last_peered":"2023-10-10T17:41:37.598088+0000","last_clean":"2023-10-10T17:41:37.598088+0000","last_became_active":"2023-10-10T17:40:41.162516+0000","last_became_peered":"2023-10-10T17:40:41.162516+0000","last_unstale":"2023-10-10T17:41:37.598088+0000","last_undegraded":"2023-10-10T17:41:37.598088+0000","last_fullsized":"2023-10-10T17:41:37.598088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:07:29.590599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.789909+0000","last_change":"2023-10-10T17:41:37.789510+0000","last_active":"2023-10-10T17:41:37.789909+0000","last_peered":"2023-10-10T17:41:37.789909+0000","last_clean":"2023-10-10T17:41:37.789909+0000","last_became_active":"2023-10-10T17:41:37.789166+0000","last_became_peered":"2023-10-10T17:41:37.789166+0000","last_unstale":"2023-10-10T17:41:37.789909+0000","last_undegraded":"2023-10-10T17:41:37.789909+0000","last_fullsized":"2023-10-10T17:41:37.789909+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:33:55.993371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598938+0000","last_change":"2023-10-10T17:40:45.707139+0000","last_active":"2023-10-10T17:41:37.598938+0000","last_peered":"2023-10-10T17:41:37.598938+0000","last_clean":"2023-10-10T17:41:37.598938+0000","last_became_active":"2023-10-10T17:40:45.706708+0000","last_became_peered":"2023-10-10T17:40:45.706708+0000","last_unstale":"2023-10-10T17:41:37.598938+0000","last_undegraded":"2023-10-10T17:41:37.598938+0000","last_fullsized":"2023-10-10T17:41:37.598938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:46:42.381782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651470+0000","last_change":"2023-10-10T17:40:41.170512+0000","last_active":"2023-10-10T17:41:37.651470+0000","last_peered":"2023-10-10T17:41:37.651470+0000","last_clean":"2023-10-10T17:41:37.651470+0000","last_became_active":"2023-10-10T17:40:41.170264+0000","last_became_peered":"2023-10-10T17:40:41.170264+0000","last_unstale":"2023-10-10T17:41:37.651470+0000","last_undegraded":"2023-10-10T17:41:37.651470+0000","last_fullsized":"2023-10-10T17:41:37.651470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:21:55.138815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594921+0000","last_change":"2023-10-10T17:40:41.597647+0000","last_active":"2023-10-10T17:41:37.594921+0000","last_peered":"2023-10-10T17:41:37.594921+0000","last_clean":"2023-10-10T17:41:37.594921+0000","last_became_active":"2023-10-10T17:40:41.597245+0000","last_became_peered":"2023-10-10T17:40:41.597245+0000","last_unstale":"2023-10-10T17:41:37.594921+0000","last_undegraded":"2023-10-10T17:41:37.594921+0000","last_fullsized":"2023-10-10T17:41:37.594921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:02.702932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979667+0000","last_change":"2023-10-10T17:40:45.705089+0000","last_active":"2023-10-10T17:41:37.979667+0000","last_peered":"2023-10-10T17:41:37.979667+0000","last_clean":"2023-10-10T17:41:37.979667+0000","last_became_active":"2023-10-10T17:40:45.704475+0000","last_became_peered":"2023-10-10T17:40:45.704475+0000","last_unstale":"2023-10-10T17:41:37.979667+0000","last_undegraded":"2023-10-10T17:41:37.979667+0000","last_fullsized":"2023-10-10T17:41:37.979667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:26.896709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":71,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:41.657549+0000","last_change":"2023-10-10T17:40:41.167409+0000","last_active":"2023-10-10T17:41:41.657549+0000","last_peered":"2023-10-10T17:41:41.657549+0000","last_clean":"2023-10-10T17:41:41.657549+0000","last_became_active":"2023-10-10T17:40:41.167125+0000","last_became_peered":"2023-10-10T17:40:41.167125+0000","last_unstale":"2023-10-10T17:41:41.657549+0000","last_undegraded":"2023-10-10T17:41:41.657549+0000","last_fullsized":"2023-10-10T17:41:41.657549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:12:36.905039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598010+0000","last_change":"2023-10-10T17:40:41.606216+0000","last_active":"2023-10-10T17:41:37.598010+0000","last_peered":"2023-10-10T17:41:37.598010+0000","last_clean":"2023-10-10T17:41:37.598010+0000","last_became_active":"2023-10-10T17:40:41.603258+0000","last_became_peered":"2023-10-10T17:40:41.603258+0000","last_unstale":"2023-10-10T17:41:37.598010+0000","last_undegraded":"2023-10-10T17:41:37.598010+0000","last_fullsized":"2023-10-10T17:41:37.598010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:30:40.923843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594917+0000","last_change":"2023-10-10T17:40:45.705623+0000","last_active":"2023-10-10T17:41:37.594917+0000","last_peered":"2023-10-10T17:41:37.594917+0000","last_clean":"2023-10-10T17:41:37.594917+0000","last_became_active":"2023-10-10T17:40:45.705242+0000","last_became_peered":"2023-10-10T17:40:45.705242+0000","last_unstale":"2023-10-10T17:41:37.594917+0000","last_undegraded":"2023-10-10T17:41:37.594917+0000","last_fullsized":"2023-10-10T17:41:37.594917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:43:18.391641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601242+0000","last_change":"2023-10-10T17:40:41.171552+0000","last_active":"2023-10-10T17:41:37.601242+0000","last_peered":"2023-10-10T17:41:37.601242+0000","last_clean":"2023-10-10T17:41:37.601242+0000","last_became_active":"2023-10-10T17:40:41.171207+0000","last_became_peered":"2023-10-10T17:40:41.171207+0000","last_unstale":"2023-10-10T17:41:37.601242+0000","last_undegraded":"2023-10-10T17:41:37.601242+0000","last_fullsized":"2023-10-10T17:41:37.601242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:35.436090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599824+0000","last_change":"2023-10-10T17:40:41.609655+0000","last_active":"2023-10-10T17:41:37.599824+0000","last_peered":"2023-10-10T17:41:37.599824+0000","last_clean":"2023-10-10T17:41:37.599824+0000","last_became_active":"2023-10-10T17:40:41.609318+0000","last_became_peered":"2023-10-10T17:40:41.609318+0000","last_unstale":"2023-10-10T17:41:37.599824+0000","last_undegraded":"2023-10-10T17:41:37.599824+0000","last_fullsized":"2023-10-10T17:41:37.599824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:06:22.359139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602379+0000","last_change":"2023-10-10T17:40:45.709513+0000","last_active":"2023-10-10T17:41:37.602379+0000","last_peered":"2023-10-10T17:41:37.602379+0000","last_clean":"2023-10-10T17:41:37.602379+0000","last_became_active":"2023-10-10T17:40:45.708963+0000","last_became_peered":"2023-10-10T17:40:45.708963+0000","last_unstale":"2023-10-10T17:41:37.602379+0000","last_undegraded":"2023-10-10T17:41:37.602379+0000","last_fullsized":"2023-10-10T17:41:37.602379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:15:02.687427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652007+0000","last_change":"2023-10-10T17:40:41.163234+0000","last_active":"2023-10-10T17:41:37.652007+0000","last_peered":"2023-10-10T17:41:37.652007+0000","last_clean":"2023-10-10T17:41:37.652007+0000","last_became_active":"2023-10-10T17:40:41.162916+0000","last_became_peered":"2023-10-10T17:40:41.162916+0000","last_unstale":"2023-10-10T17:41:37.652007+0000","last_undegraded":"2023-10-10T17:41:37.652007+0000","last_fullsized":"2023-10-10T17:41:37.652007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:09:06.358768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598352+0000","last_change":"2023-10-10T17:40:41.602852+0000","last_active":"2023-10-10T17:41:37.598352+0000","last_peered":"2023-10-10T17:41:37.598352+0000","last_clean":"2023-10-10T17:41:37.598352+0000","last_became_active":"2023-10-10T17:40:41.602530+0000","last_became_peered":"2023-10-10T17:40:41.602530+0000","last_unstale":"2023-10-10T17:41:37.598352+0000","last_undegraded":"2023-10-10T17:41:37.598352+0000","last_fullsized":"2023-10-10T17:41:37.598352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:58:41.949886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595188+0000","last_change":"2023-10-10T17:40:45.713714+0000","last_active":"2023-10-10T17:41:37.595188+0000","last_peered":"2023-10-10T17:41:37.595188+0000","last_clean":"2023-10-10T17:41:37.595188+0000","last_became_active":"2023-10-10T17:40:45.712953+0000","last_became_peered":"2023-10-10T17:40:45.712953+0000","last_unstale":"2023-10-10T17:41:37.595188+0000","last_undegraded":"2023-10-10T17:41:37.595188+0000","last_fullsized":"2023-10-10T17:41:37.595188+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:15.222779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:42.659039+0000","last_change":"2023-10-10T17:40:41.168328+0000","last_active":"2023-10-10T17:41:42.659039+0000","last_peered":"2023-10-10T17:41:42.659039+0000","last_clean":"2023-10-10T17:41:42.659039+0000","last_became_active":"2023-10-10T17:40:41.168044+0000","last_became_peered":"2023-10-10T17:40:41.168044+0000","last_unstale":"2023-10-10T17:41:42.659039+0000","last_undegraded":"2023-10-10T17:41:42.659039+0000","last_fullsized":"2023-10-10T17:41:42.659039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:52:24.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":86,"num_read_kb":47,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599535+0000","last_change":"2023-10-10T17:40:41.597795+0000","last_active":"2023-10-10T17:41:37.599535+0000","last_peered":"2023-10-10T17:41:37.599535+0000","last_clean":"2023-10-10T17:41:37.599535+0000","last_became_active":"2023-10-10T17:40:41.597537+0000","last_became_peered":"2023-10-10T17:40:41.597537+0000","last_unstale":"2023-10-10T17:41:37.599535+0000","last_undegraded":"2023-10-10T17:41:37.599535+0000","last_fullsized":"2023-10-10T17:41:37.599535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:16:34.371564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652449+0000","last_change":"2023-10-10T17:40:45.709827+0000","last_active":"2023-10-10T17:41:37.652449+0000","last_peered":"2023-10-10T17:41:37.652449+0000","last_clean":"2023-10-10T17:41:37.652449+0000","last_became_active":"2023-10-10T17:40:45.709373+0000","last_became_peered":"2023-10-10T17:40:45.709373+0000","last_unstale":"2023-10-10T17:41:37.652449+0000","last_undegraded":"2023-10-10T17:41:37.652449+0000","last_fullsized":"2023-10-10T17:41:37.652449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:35:42.826536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596527+0000","last_change":"2023-10-10T17:40:41.160804+0000","last_active":"2023-10-10T17:41:37.596527+0000","last_peered":"2023-10-10T17:41:37.596527+0000","last_clean":"2023-10-10T17:41:37.596527+0000","last_became_active":"2023-10-10T17:40:41.160493+0000","last_became_peered":"2023-10-10T17:40:41.160493+0000","last_unstale":"2023-10-10T17:41:37.596527+0000","last_undegraded":"2023-10-10T17:41:37.596527+0000","last_fullsized":"2023-10-10T17:41:37.596527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:59.561663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598334+0000","last_change":"2023-10-10T17:40:41.601357+0000","last_active":"2023-10-10T17:41:37.598334+0000","last_peered":"2023-10-10T17:41:37.598334+0000","last_clean":"2023-10-10T17:41:37.598334+0000","last_became_active":"2023-10-10T17:40:41.601057+0000","last_became_peered":"2023-10-10T17:40:41.601057+0000","last_unstale":"2023-10-10T17:41:37.598334+0000","last_undegraded":"2023-10-10T17:41:37.598334+0000","last_fullsized":"2023-10-10T17:41:37.598334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:07.286077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788393+0000","last_change":"2023-10-10T17:40:45.708067+0000","last_active":"2023-10-10T17:41:37.788393+0000","last_peered":"2023-10-10T17:41:37.788393+0000","last_clean":"2023-10-10T17:41:37.788393+0000","last_became_active":"2023-10-10T17:40:45.707686+0000","last_became_peered":"2023-10-10T17:40:45.707686+0000","last_unstale":"2023-10-10T17:41:37.788393+0000","last_undegraded":"2023-10-10T17:41:37.788393+0000","last_fullsized":"2023-10-10T17:41:37.788393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:08.956407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651829+0000","last_change":"2023-10-10T17:40:41.168006+0000","last_active":"2023-10-10T17:41:37.651829+0000","last_peered":"2023-10-10T17:41:37.651829+0000","last_clean":"2023-10-10T17:41:37.651829+0000","last_became_active":"2023-10-10T17:40:41.167777+0000","last_became_peered":"2023-10-10T17:40:41.167777+0000","last_unstale":"2023-10-10T17:41:37.651829+0000","last_undegraded":"2023-10-10T17:41:37.651829+0000","last_fullsized":"2023-10-10T17:41:37.651829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:30:32.283329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601103+0000","last_change":"2023-10-10T17:40:41.591215+0000","last_active":"2023-10-10T17:41:37.601103+0000","last_peered":"2023-10-10T17:41:37.601103+0000","last_clean":"2023-10-10T17:41:37.601103+0000","last_became_active":"2023-10-10T17:40:41.590929+0000","last_became_peered":"2023-10-10T17:40:41.590929+0000","last_unstale":"2023-10-10T17:41:37.601103+0000","last_undegraded":"2023-10-10T17:41:37.601103+0000","last_fullsized":"2023-10-10T17:41:37.601103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:11:13.144245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595485+0000","last_change":"2023-10-10T17:40:45.709038+0000","last_active":"2023-10-10T17:41:37.595485+0000","last_peered":"2023-10-10T17:41:37.595485+0000","last_clean":"2023-10-10T17:41:37.595485+0000","last_became_active":"2023-10-10T17:40:45.708683+0000","last_became_peered":"2023-10-10T17:40:45.708683+0000","last_unstale":"2023-10-10T17:41:37.595485+0000","last_undegraded":"2023-10-10T17:41:37.595485+0000","last_fullsized":"2023-10-10T17:41:37.595485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:50:01.174506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":74,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:42.668069+0000","last_change":"2023-10-10T17:40:41.163817+0000","last_active":"2023-10-10T17:41:42.668069+0000","last_peered":"2023-10-10T17:41:42.668069+0000","last_clean":"2023-10-10T17:41:42.668069+0000","last_became_active":"2023-10-10T17:40:41.163497+0000","last_became_peered":"2023-10-10T17:40:41.163497+0000","last_unstale":"2023-10-10T17:41:42.668069+0000","last_undegraded":"2023-10-10T17:41:42.668069+0000","last_fullsized":"2023-10-10T17:41:42.668069+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:54:45.382546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.607553+0000","last_change":"2023-10-10T17:41:37.607553+0000","last_active":"2023-10-10T17:41:37.607553+0000","last_peered":"2023-10-10T17:41:37.607553+0000","last_clean":"2023-10-10T17:41:37.607553+0000","last_became_active":"2023-10-10T17:41:37.606536+0000","last_became_peered":"2023-10-10T17:41:37.606536+0000","last_unstale":"2023-10-10T17:41:37.607553+0000","last_undegraded":"2023-10-10T17:41:37.607553+0000","last_fullsized":"2023-10-10T17:41:37.607553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:58:09.182227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600059+0000","last_change":"2023-10-10T17:40:45.714900+0000","last_active":"2023-10-10T17:41:37.600059+0000","last_peered":"2023-10-10T17:41:37.600059+0000","last_clean":"2023-10-10T17:41:37.600059+0000","last_became_active":"2023-10-10T17:40:45.714525+0000","last_became_peered":"2023-10-10T17:40:45.714525+0000","last_unstale":"2023-10-10T17:41:37.600059+0000","last_undegraded":"2023-10-10T17:41:37.600059+0000","last_fullsized":"2023-10-10T17:41:37.600059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:01:36.283546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:38.795173+0000","last_change":"2023-10-10T17:41:38.795173+0000","last_active":"2023-10-10T17:41:38.795173+0000","last_peered":"2023-10-10T17:41:38.795173+0000","last_clean":"2023-10-10T17:41:38.795173+0000","last_became_active":"2023-10-10T17:41:37.603917+0000","last_became_peered":"2023-10-10T17:41:37.603917+0000","last_unstale":"2023-10-10T17:41:38.795173+0000","last_undegraded":"2023-10-10T17:41:38.795173+0000","last_fullsized":"2023-10-10T17:41:38.795173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:32:38.418130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651410+0000","last_change":"2023-10-10T17:40:41.586591+0000","last_active":"2023-10-10T17:41:37.651410+0000","last_peered":"2023-10-10T17:41:37.651410+0000","last_clean":"2023-10-10T17:41:37.651410+0000","last_became_active":"2023-10-10T17:40:41.586293+0000","last_became_peered":"2023-10-10T17:40:41.586293+0000","last_unstale":"2023-10-10T17:41:37.651410+0000","last_undegraded":"2023-10-10T17:41:37.651410+0000","last_fullsized":"2023-10-10T17:41:37.651410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:39:26.663874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651375+0000","last_change":"2023-10-10T17:40:45.708348+0000","last_active":"2023-10-10T17:41:37.651375+0000","last_peered":"2023-10-10T17:41:37.651375+0000","last_clean":"2023-10-10T17:41:37.651375+0000","last_became_active":"2023-10-10T17:40:45.707899+0000","last_became_peered":"2023-10-10T17:40:45.707899+0000","last_unstale":"2023-10-10T17:41:37.651375+0000","last_undegraded":"2023-10-10T17:41:37.651375+0000","last_fullsized":"2023-10-10T17:41:37.651375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:29:39.345572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601729+0000","last_change":"2023-10-10T17:40:41.168209+0000","last_active":"2023-10-10T17:41:37.601729+0000","last_peered":"2023-10-10T17:41:37.601729+0000","last_clean":"2023-10-10T17:41:37.601729+0000","last_became_active":"2023-10-10T17:40:41.167907+0000","last_became_peered":"2023-10-10T17:40:41.167907+0000","last_unstale":"2023-10-10T17:41:37.601729+0000","last_undegraded":"2023-10-10T17:41:37.601729+0000","last_fullsized":"2023-10-10T17:41:37.601729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:35:19.344734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.594848+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.594562+0000","last_became_peered":"2023-10-10T17:40:41.594562+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:15:21.805794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600519+0000","last_change":"2023-10-10T17:40:45.711043+0000","last_active":"2023-10-10T17:41:37.600519+0000","last_peered":"2023-10-10T17:41:37.600519+0000","last_clean":"2023-10-10T17:41:37.600519+0000","last_became_active":"2023-10-10T17:40:45.710617+0000","last_became_peered":"2023-10-10T17:40:45.710617+0000","last_unstale":"2023-10-10T17:41:37.600519+0000","last_undegraded":"2023-10-10T17:41:37.600519+0000","last_fullsized":"2023-10-10T17:41:37.600519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:00:48.152401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595133+0000","last_change":"2023-10-10T17:40:41.163238+0000","last_active":"2023-10-10T17:41:37.595133+0000","last_peered":"2023-10-10T17:41:37.595133+0000","last_clean":"2023-10-10T17:41:37.595133+0000","last_became_active":"2023-10-10T17:40:41.162934+0000","last_became_peered":"2023-10-10T17:40:41.162934+0000","last_unstale":"2023-10-10T17:41:37.595133+0000","last_undegraded":"2023-10-10T17:41:37.595133+0000","last_fullsized":"2023-10-10T17:41:37.595133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:23:10.440706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598371+0000","last_change":"2023-10-10T17:40:41.597026+0000","last_active":"2023-10-10T17:41:37.598371+0000","last_peered":"2023-10-10T17:41:37.598371+0000","last_clean":"2023-10-10T17:41:37.598371+0000","last_became_active":"2023-10-10T17:40:41.596747+0000","last_became_peered":"2023-10-10T17:40:41.596747+0000","last_unstale":"2023-10-10T17:41:37.598371+0000","last_undegraded":"2023-10-10T17:41:37.598371+0000","last_fullsized":"2023-10-10T17:41:37.598371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:52.010575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599056+0000","last_change":"2023-10-10T17:40:45.707218+0000","last_active":"2023-10-10T17:41:37.599056+0000","last_peered":"2023-10-10T17:41:37.599056+0000","last_clean":"2023-10-10T17:41:37.599056+0000","last_became_active":"2023-10-10T17:40:45.706850+0000","last_became_peered":"2023-10-10T17:40:45.706850+0000","last_unstale":"2023-10-10T17:41:37.599056+0000","last_undegraded":"2023-10-10T17:41:37.599056+0000","last_fullsized":"2023-10-10T17:41:37.599056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:36:44.235237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:41.166713+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:41.165951+0000","last_became_peered":"2023-10-10T17:40:41.165951+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:56:00.129165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597783+0000","last_change":"2023-10-10T17:40:41.605860+0000","last_active":"2023-10-10T17:41:37.597783+0000","last_peered":"2023-10-10T17:41:37.597783+0000","last_clean":"2023-10-10T17:41:37.597783+0000","last_became_active":"2023-10-10T17:40:41.605556+0000","last_became_peered":"2023-10-10T17:40:41.605556+0000","last_unstale":"2023-10-10T17:41:37.597783+0000","last_undegraded":"2023-10-10T17:41:37.597783+0000","last_fullsized":"2023-10-10T17:41:37.597783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:07.438058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599757+0000","last_change":"2023-10-10T17:40:45.711844+0000","last_active":"2023-10-10T17:41:37.599757+0000","last_peered":"2023-10-10T17:41:37.599757+0000","last_clean":"2023-10-10T17:41:37.599757+0000","last_became_active":"2023-10-10T17:40:45.711413+0000","last_became_peered":"2023-10-10T17:40:45.711413+0000","last_unstale":"2023-10-10T17:41:37.599757+0000","last_undegraded":"2023-10-10T17:41:37.599757+0000","last_fullsized":"2023-10-10T17:41:37.599757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:13:03.476627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.650650+0000","last_change":"2023-10-10T17:40:41.608625+0000","last_active":"2023-10-10T17:41:37.650650+0000","last_peered":"2023-10-10T17:41:37.650650+0000","last_clean":"2023-10-10T17:41:37.650650+0000","last_became_active":"2023-10-10T17:40:41.608332+0000","last_became_peered":"2023-10-10T17:40:41.608332+0000","last_unstale":"2023-10-10T17:41:37.650650+0000","last_undegraded":"2023-10-10T17:41:37.650650+0000","last_fullsized":"2023-10-10T17:41:37.650650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:07:55.183440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":72,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:41.665912+0000","last_change":"2023-10-10T17:40:41.169807+0000","last_active":"2023-10-10T17:41:41.665912+0000","last_peered":"2023-10-10T17:41:41.665912+0000","last_clean":"2023-10-10T17:41:41.665912+0000","last_became_active":"2023-10-10T17:40:41.169475+0000","last_became_peered":"2023-10-10T17:40:41.169475+0000","last_unstale":"2023-10-10T17:41:41.665912+0000","last_undegraded":"2023-10-10T17:41:41.665912+0000","last_fullsized":"2023-10-10T17:41:41.665912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:22.887502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":42,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601878+0000","last_change":"2023-10-10T17:40:45.710544+0000","last_active":"2023-10-10T17:41:37.601878+0000","last_peered":"2023-10-10T17:41:37.601878+0000","last_clean":"2023-10-10T17:41:37.601878+0000","last_became_active":"2023-10-10T17:40:45.710084+0000","last_became_peered":"2023-10-10T17:40:45.710084+0000","last_unstale":"2023-10-10T17:41:37.601878+0000","last_undegraded":"2023-10-10T17:41:37.601878+0000","last_fullsized":"2023-10-10T17:41:37.601878+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:44:01.303666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979130+0000","last_change":"2023-10-10T17:40:41.599028+0000","last_active":"2023-10-10T17:41:37.979130+0000","last_peered":"2023-10-10T17:41:37.979130+0000","last_clean":"2023-10-10T17:41:37.979130+0000","last_became_active":"2023-10-10T17:40:41.598684+0000","last_became_peered":"2023-10-10T17:40:41.598684+0000","last_unstale":"2023-10-10T17:41:37.979130+0000","last_undegraded":"2023-10-10T17:41:37.979130+0000","last_fullsized":"2023-10-10T17:41:37.979130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:16:57.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602213+0000","last_change":"2023-10-10T17:40:41.165756+0000","last_active":"2023-10-10T17:41:37.602213+0000","last_peered":"2023-10-10T17:41:37.602213+0000","last_clean":"2023-10-10T17:41:37.602213+0000","last_became_active":"2023-10-10T17:40:41.165368+0000","last_became_peered":"2023-10-10T17:40:41.165368+0000","last_unstale":"2023-10-10T17:41:37.602213+0000","last_undegraded":"2023-10-10T17:41:37.602213+0000","last_fullsized":"2023-10-10T17:41:37.602213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:16.089880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600889+0000","last_change":"2023-10-10T17:39:42.508828+0000","last_active":"2023-10-10T17:41:37.600889+0000","last_peered":"2023-10-10T17:41:37.600889+0000","last_clean":"2023-10-10T17:41:37.600889+0000","last_became_active":"2023-10-10T17:39:42.508475+0000","last_became_peered":"2023-10-10T17:39:42.508475+0000","last_unstale":"2023-10-10T17:41:37.600889+0000","last_undegraded":"2023-10-10T17:41:37.600889+0000","last_fullsized":"2023-10-10T17:41:37.600889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_clean_scrub_stamp":"2023-10-10T17:36:46.553699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:08:38.008210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652108+0000","last_change":"2023-10-10T17:40:45.714604+0000","last_active":"2023-10-10T17:41:37.652108+0000","last_peered":"2023-10-10T17:41:37.652108+0000","last_clean":"2023-10-10T17:41:37.652108+0000","last_became_active":"2023-10-10T17:40:45.714185+0000","last_became_peered":"2023-10-10T17:40:45.714185+0000","last_unstale":"2023-10-10T17:41:37.652108+0000","last_undegraded":"2023-10-10T17:41:37.652108+0000","last_fullsized":"2023-10-10T17:41:37.652108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:55:26.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597853+0000","last_change":"2023-10-10T17:40:41.610002+0000","last_active":"2023-10-10T17:41:37.597853+0000","last_peered":"2023-10-10T17:41:37.597853+0000","last_clean":"2023-10-10T17:41:37.597853+0000","last_became_active":"2023-10-10T17:40:41.609708+0000","last_became_peered":"2023-10-10T17:40:41.609708+0000","last_unstale":"2023-10-10T17:41:37.597853+0000","last_undegraded":"2023-10-10T17:41:37.597853+0000","last_fullsized":"2023-10-10T17:41:37.597853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:34.611184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601055+0000","last_change":"2023-10-10T17:40:41.167715+0000","last_active":"2023-10-10T17:41:37.601055+0000","last_peered":"2023-10-10T17:41:37.601055+0000","last_clean":"2023-10-10T17:41:37.601055+0000","last_became_active":"2023-10-10T17:40:41.166385+0000","last_became_peered":"2023-10-10T17:40:41.166385+0000","last_unstale":"2023-10-10T17:41:37.601055+0000","last_undegraded":"2023-10-10T17:41:37.601055+0000","last_fullsized":"2023-10-10T17:41:37.601055+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:49:05.385383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788299+0000","last_change":"2023-10-10T17:40:45.707236+0000","last_active":"2023-10-10T17:41:37.788299+0000","last_peered":"2023-10-10T17:41:37.788299+0000","last_clean":"2023-10-10T17:41:37.788299+0000","last_became_active":"2023-10-10T17:40:45.706827+0000","last_became_peered":"2023-10-10T17:40:45.706827+0000","last_unstale":"2023-10-10T17:41:37.788299+0000","last_undegraded":"2023-10-10T17:41:37.788299+0000","last_fullsized":"2023-10-10T17:41:37.788299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:58.930376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601396+0000","last_change":"2023-10-10T17:40:41.170523+0000","last_active":"2023-10-10T17:41:37.601396+0000","last_peered":"2023-10-10T17:41:37.601396+0000","last_clean":"2023-10-10T17:41:37.601396+0000","last_became_active":"2023-10-10T17:40:41.170243+0000","last_became_peered":"2023-10-10T17:40:41.170243+0000","last_unstale":"2023-10-10T17:41:37.601396+0000","last_undegraded":"2023-10-10T17:41:37.601396+0000","last_fullsized":"2023-10-10T17:41:37.601396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:40.239129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599195+0000","last_change":"2023-10-10T17:40:41.610472+0000","last_active":"2023-10-10T17:41:37.599195+0000","last_peered":"2023-10-10T17:41:37.599195+0000","last_clean":"2023-10-10T17:41:37.599195+0000","last_became_active":"2023-10-10T17:40:41.610139+0000","last_became_peered":"2023-10-10T17:40:41.610139+0000","last_unstale":"2023-10-10T17:41:37.599195+0000","last_undegraded":"2023-10-10T17:41:37.599195+0000","last_fullsized":"2023-10-10T17:41:37.599195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:22:52.414485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979558+0000","last_change":"2023-10-10T17:40:45.712179+0000","last_active":"2023-10-10T17:41:37.979558+0000","last_peered":"2023-10-10T17:41:37.979558+0000","last_clean":"2023-10-10T17:41:37.979558+0000","last_became_active":"2023-10-10T17:40:45.711708+0000","last_became_peered":"2023-10-10T17:40:45.711708+0000","last_unstale":"2023-10-10T17:41:37.979558+0000","last_undegraded":"2023-10-10T17:41:37.979558+0000","last_fullsized":"2023-10-10T17:41:37.979558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:25:37.789886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595330+0000","last_change":"2023-10-10T17:40:41.164882+0000","last_active":"2023-10-10T17:41:37.595330+0000","last_peered":"2023-10-10T17:41:37.595330+0000","last_clean":"2023-10-10T17:41:37.595330+0000","last_became_active":"2023-10-10T17:40:41.164536+0000","last_became_peered":"2023-10-10T17:40:41.164536+0000","last_unstale":"2023-10-10T17:41:37.595330+0000","last_undegraded":"2023-10-10T17:41:37.595330+0000","last_fullsized":"2023-10-10T17:41:37.595330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:20:38.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980304+0000","last_change":"2023-10-10T17:40:41.595043+0000","last_active":"2023-10-10T17:41:37.980304+0000","last_peered":"2023-10-10T17:41:37.980304+0000","last_clean":"2023-10-10T17:41:37.980304+0000","last_became_active":"2023-10-10T17:40:41.594703+0000","last_became_peered":"2023-10-10T17:40:41.594703+0000","last_unstale":"2023-10-10T17:41:37.980304+0000","last_undegraded":"2023-10-10T17:41:37.980304+0000","last_fullsized":"2023-10-10T17:41:37.980304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:29:21.277302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596130+0000","last_change":"2023-10-10T17:40:45.705232+0000","last_active":"2023-10-10T17:41:37.596130+0000","last_peered":"2023-10-10T17:41:37.596130+0000","last_clean":"2023-10-10T17:41:37.596130+0000","last_became_active":"2023-10-10T17:40:45.704811+0000","last_became_peered":"2023-10-10T17:40:45.704811+0000","last_unstale":"2023-10-10T17:41:37.596130+0000","last_undegraded":"2023-10-10T17:41:37.596130+0000","last_fullsized":"2023-10-10T17:41:37.596130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:50.034871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602441+0000","last_change":"2023-10-10T17:40:41.165620+0000","last_active":"2023-10-10T17:41:37.602441+0000","last_peered":"2023-10-10T17:41:37.602441+0000","last_clean":"2023-10-10T17:41:37.602441+0000","last_became_active":"2023-10-10T17:40:41.165262+0000","last_became_peered":"2023-10-10T17:40:41.165262+0000","last_unstale":"2023-10-10T17:41:37.602441+0000","last_undegraded":"2023-10-10T17:41:37.602441+0000","last_fullsized":"2023-10-10T17:41:37.602441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:19.126274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599185+0000","last_change":"2023-10-10T17:40:41.603549+0000","last_active":"2023-10-10T17:41:37.599185+0000","last_peered":"2023-10-10T17:41:37.599185+0000","last_clean":"2023-10-10T17:41:37.599185+0000","last_became_active":"2023-10-10T17:40:41.602991+0000","last_became_peered":"2023-10-10T17:40:41.602991+0000","last_unstale":"2023-10-10T17:41:37.599185+0000","last_undegraded":"2023-10-10T17:41:37.599185+0000","last_fullsized":"2023-10-10T17:41:37.599185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:30.283350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.606557+0000","last_change":"2023-10-10T17:41:37.606557+0000","last_active":"2023-10-10T17:41:37.606557+0000","last_peered":"2023-10-10T17:41:37.606557+0000","last_clean":"2023-10-10T17:41:37.606557+0000","last_became_active":"2023-10-10T17:41:37.606070+0000","last_became_peered":"2023-10-10T17:41:37.606070+0000","last_unstale":"2023-10-10T17:41:37.606557+0000","last_undegraded":"2023-10-10T17:41:37.606557+0000","last_fullsized":"2023-10-10T17:41:37.606557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:09:50.492126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595549+0000","last_change":"2023-10-10T17:40:41.169859+0000","last_active":"2023-10-10T17:41:37.595549+0000","last_peered":"2023-10-10T17:41:37.595549+0000","last_clean":"2023-10-10T17:41:37.595549+0000","last_became_active":"2023-10-10T17:40:41.169507+0000","last_became_peered":"2023-10-10T17:40:41.169507+0000","last_unstale":"2023-10-10T17:41:37.595549+0000","last_undegraded":"2023-10-10T17:41:37.595549+0000","last_fullsized":"2023-10-10T17:41:37.595549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:29:19.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600382+0000","last_change":"2023-10-10T17:40:41.593929+0000","last_active":"2023-10-10T17:41:37.600382+0000","last_peered":"2023-10-10T17:41:37.600382+0000","last_clean":"2023-10-10T17:41:37.600382+0000","last_became_active":"2023-10-10T17:40:41.593565+0000","last_became_peered":"2023-10-10T17:40:41.593565+0000","last_unstale":"2023-10-10T17:41:37.600382+0000","last_undegraded":"2023-10-10T17:41:37.600382+0000","last_fullsized":"2023-10-10T17:41:37.600382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:52.950921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595626+0000","last_change":"2023-10-10T17:40:45.710759+0000","last_active":"2023-10-10T17:41:37.595626+0000","last_peered":"2023-10-10T17:41:37.595626+0000","last_clean":"2023-10-10T17:41:37.595626+0000","last_became_active":"2023-10-10T17:40:45.710118+0000","last_became_peered":"2023-10-10T17:40:45.710118+0000","last_unstale":"2023-10-10T17:41:37.595626+0000","last_undegraded":"2023-10-10T17:41:37.595626+0000","last_fullsized":"2023-10-10T17:41:37.595626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:03:22.069829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979760+0000","last_change":"2023-10-10T17:40:41.165988+0000","last_active":"2023-10-10T17:41:37.979760+0000","last_peered":"2023-10-10T17:41:37.979760+0000","last_clean":"2023-10-10T17:41:37.979760+0000","last_became_active":"2023-10-10T17:40:41.165407+0000","last_became_peered":"2023-10-10T17:40:41.165407+0000","last_unstale":"2023-10-10T17:41:37.979760+0000","last_undegraded":"2023-10-10T17:41:37.979760+0000","last_fullsized":"2023-10-10T17:41:37.979760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:16:55.568098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787946+0000","last_change":"2023-10-10T17:40:41.609834+0000","last_active":"2023-10-10T17:41:37.787946+0000","last_peered":"2023-10-10T17:41:37.787946+0000","last_clean":"2023-10-10T17:41:37.787946+0000","last_became_active":"2023-10-10T17:40:41.609509+0000","last_became_peered":"2023-10-10T17:40:41.609509+0000","last_unstale":"2023-10-10T17:41:37.787946+0000","last_undegraded":"2023-10-10T17:41:37.787946+0000","last_fullsized":"2023-10-10T17:41:37.787946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:26:49.739028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602247+0000","last_change":"2023-10-10T17:40:45.709003+0000","last_active":"2023-10-10T17:41:37.602247+0000","last_peered":"2023-10-10T17:41:37.602247+0000","last_clean":"2023-10-10T17:41:37.602247+0000","last_became_active":"2023-10-10T17:40:45.708610+0000","last_became_peered":"2023-10-10T17:40:45.708610+0000","last_unstale":"2023-10-10T17:41:37.602247+0000","last_undegraded":"2023-10-10T17:41:37.602247+0000","last_fullsized":"2023-10-10T17:41:37.602247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:04:55.725864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596673+0000","last_change":"2023-10-10T17:40:41.168104+0000","last_active":"2023-10-10T17:41:37.596673+0000","last_peered":"2023-10-10T17:41:37.596673+0000","last_clean":"2023-10-10T17:41:37.596673+0000","last_became_active":"2023-10-10T17:40:41.167802+0000","last_became_peered":"2023-10-10T17:40:41.167802+0000","last_unstale":"2023-10-10T17:41:37.596673+0000","last_undegraded":"2023-10-10T17:41:37.596673+0000","last_fullsized":"2023-10-10T17:41:37.596673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:22:40.050459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979300+0000","last_change":"2023-10-10T17:40:41.606553+0000","last_active":"2023-10-10T17:41:37.979300+0000","last_peered":"2023-10-10T17:41:37.979300+0000","last_clean":"2023-10-10T17:41:37.979300+0000","last_became_active":"2023-10-10T17:40:41.606087+0000","last_became_peered":"2023-10-10T17:40:41.606087+0000","last_unstale":"2023-10-10T17:41:37.979300+0000","last_undegraded":"2023-10-10T17:41:37.979300+0000","last_fullsized":"2023-10-10T17:41:37.979300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:44:40.056635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602105+0000","last_change":"2023-10-10T17:40:45.712028+0000","last_active":"2023-10-10T17:41:37.602105+0000","last_peered":"2023-10-10T17:41:37.602105+0000","last_clean":"2023-10-10T17:41:37.602105+0000","last_became_active":"2023-10-10T17:40:45.711534+0000","last_became_peered":"2023-10-10T17:40:45.711534+0000","last_unstale":"2023-10-10T17:41:37.602105+0000","last_undegraded":"2023-10-10T17:41:37.602105+0000","last_fullsized":"2023-10-10T17:41:37.602105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:51:36.426627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600532+0000","last_change":"2023-10-10T17:40:41.164456+0000","last_active":"2023-10-10T17:41:37.600532+0000","last_peered":"2023-10-10T17:41:37.600532+0000","last_clean":"2023-10-10T17:41:37.600532+0000","last_became_active":"2023-10-10T17:40:41.164018+0000","last_became_peered":"2023-10-10T17:40:41.164018+0000","last_unstale":"2023-10-10T17:41:37.600532+0000","last_undegraded":"2023-10-10T17:41:37.600532+0000","last_fullsized":"2023-10-10T17:41:37.600532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:54:18.517460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601647+0000","last_change":"2023-10-10T17:40:41.599724+0000","last_active":"2023-10-10T17:41:37.601647+0000","last_peered":"2023-10-10T17:41:37.601647+0000","last_clean":"2023-10-10T17:41:37.601647+0000","last_became_active":"2023-10-10T17:40:41.599356+0000","last_became_peered":"2023-10-10T17:40:41.599356+0000","last_unstale":"2023-10-10T17:41:37.601647+0000","last_undegraded":"2023-10-10T17:41:37.601647+0000","last_fullsized":"2023-10-10T17:41:37.601647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:57.775792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596111+0000","last_change":"2023-10-10T17:40:45.704911+0000","last_active":"2023-10-10T17:41:37.596111+0000","last_peered":"2023-10-10T17:41:37.596111+0000","last_clean":"2023-10-10T17:41:37.596111+0000","last_became_active":"2023-10-10T17:40:45.704299+0000","last_became_peered":"2023-10-10T17:40:45.704299+0000","last_unstale":"2023-10-10T17:41:37.596111+0000","last_undegraded":"2023-10-10T17:41:37.596111+0000","last_fullsized":"2023-10-10T17:41:37.596111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:13:42.759241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597150+0000","last_change":"2023-10-10T17:40:41.166091+0000","last_active":"2023-10-10T17:41:37.597150+0000","last_peered":"2023-10-10T17:41:37.597150+0000","last_clean":"2023-10-10T17:41:37.597150+0000","last_became_active":"2023-10-10T17:40:41.165610+0000","last_became_peered":"2023-10-10T17:40:41.165610+0000","last_unstale":"2023-10-10T17:41:37.597150+0000","last_undegraded":"2023-10-10T17:41:37.597150+0000","last_fullsized":"2023-10-10T17:41:37.597150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:31:09.299708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602252+0000","last_change":"2023-10-10T17:40:41.600837+0000","last_active":"2023-10-10T17:41:37.602252+0000","last_peered":"2023-10-10T17:41:37.602252+0000","last_clean":"2023-10-10T17:41:37.602252+0000","last_became_active":"2023-10-10T17:40:41.600090+0000","last_became_peered":"2023-10-10T17:40:41.600090+0000","last_unstale":"2023-10-10T17:41:37.602252+0000","last_undegraded":"2023-10-10T17:41:37.602252+0000","last_fullsized":"2023-10-10T17:41:37.602252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:15:21.942555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603744+0000","last_change":"2023-10-10T17:41:37.603744+0000","last_active":"2023-10-10T17:41:37.603744+0000","last_peered":"2023-10-10T17:41:37.603744+0000","last_clean":"2023-10-10T17:41:37.603744+0000","last_became_active":"2023-10-10T17:41:37.603376+0000","last_became_peered":"2023-10-10T17:41:37.603376+0000","last_unstale":"2023-10-10T17:41:37.603744+0000","last_undegraded":"2023-10-10T17:41:37.603744+0000","last_fullsized":"2023-10-10T17:41:37.603744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:59:51.305467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652254+0000","last_change":"2023-10-10T17:40:41.169820+0000","last_active":"2023-10-10T17:41:37.652254+0000","last_peered":"2023-10-10T17:41:37.652254+0000","last_clean":"2023-10-10T17:41:37.652254+0000","last_became_active":"2023-10-10T17:40:41.168767+0000","last_became_peered":"2023-10-10T17:40:41.168767+0000","last_unstale":"2023-10-10T17:41:37.652254+0000","last_undegraded":"2023-10-10T17:41:37.652254+0000","last_fullsized":"2023-10-10T17:41:37.652254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:22:55.616442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603091+0000","last_change":"2023-10-10T17:40:41.597482+0000","last_active":"2023-10-10T17:41:37.603091+0000","last_peered":"2023-10-10T17:41:37.603091+0000","last_clean":"2023-10-10T17:41:37.603091+0000","last_became_active":"2023-10-10T17:40:41.597166+0000","last_became_peered":"2023-10-10T17:40:41.597166+0000","last_unstale":"2023-10-10T17:41:37.603091+0000","last_undegraded":"2023-10-10T17:41:37.603091+0000","last_fullsized":"2023-10-10T17:41:37.603091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:11.444261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651261+0000","last_change":"2023-10-10T17:40:45.707175+0000","last_active":"2023-10-10T17:41:37.651261+0000","last_peered":"2023-10-10T17:41:37.651261+0000","last_clean":"2023-10-10T17:41:37.651261+0000","last_became_active":"2023-10-10T17:40:45.706448+0000","last_became_peered":"2023-10-10T17:40:45.706448+0000","last_unstale":"2023-10-10T17:41:37.651261+0000","last_undegraded":"2023-10-10T17:41:37.651261+0000","last_fullsized":"2023-10-10T17:41:37.651261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:33.970113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600136+0000","last_change":"2023-10-10T17:40:41.169806+0000","last_active":"2023-10-10T17:41:37.600136+0000","last_peered":"2023-10-10T17:41:37.600136+0000","last_clean":"2023-10-10T17:41:37.600136+0000","last_became_active":"2023-10-10T17:40:41.169496+0000","last_became_peered":"2023-10-10T17:40:41.169496+0000","last_unstale":"2023-10-10T17:41:37.600136+0000","last_undegraded":"2023-10-10T17:41:37.600136+0000","last_fullsized":"2023-10-10T17:41:37.600136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:28:16.024298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603755+0000","last_change":"2023-10-10T17:40:41.598366+0000","last_active":"2023-10-10T17:41:37.603755+0000","last_peered":"2023-10-10T17:41:37.603755+0000","last_clean":"2023-10-10T17:41:37.603755+0000","last_became_active":"2023-10-10T17:40:41.598036+0000","last_became_peered":"2023-10-10T17:40:41.598036+0000","last_unstale":"2023-10-10T17:41:37.603755+0000","last_undegraded":"2023-10-10T17:41:37.603755+0000","last_fullsized":"2023-10-10T17:41:37.603755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:59:22.632882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787912+0000","last_change":"2023-10-10T17:40:41.609923+0000","last_active":"2023-10-10T17:41:37.787912+0000","last_peered":"2023-10-10T17:41:37.787912+0000","last_clean":"2023-10-10T17:41:37.787912+0000","last_became_active":"2023-10-10T17:40:41.609575+0000","last_became_peered":"2023-10-10T17:40:41.609575+0000","last_unstale":"2023-10-10T17:41:37.787912+0000","last_undegraded":"2023-10-10T17:41:37.787912+0000","last_fullsized":"2023-10-10T17:41:37.787912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:14:10.560266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596871+0000","last_change":"2023-10-10T17:40:41.609012+0000","last_active":"2023-10-10T17:41:37.596871+0000","last_peered":"2023-10-10T17:41:37.596871+0000","last_clean":"2023-10-10T17:41:37.596871+0000","last_became_active":"2023-10-10T17:40:41.608738+0000","last_became_peered":"2023-10-10T17:40:41.608738+0000","last_unstale":"2023-10-10T17:41:37.596871+0000","last_undegraded":"2023-10-10T17:41:37.596871+0000","last_fullsized":"2023-10-10T17:41:37.596871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:11:39.107240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596125+0000","last_change":"2023-10-10T17:40:41.608801+0000","last_active":"2023-10-10T17:41:37.596125+0000","last_peered":"2023-10-10T17:41:37.596125+0000","last_clean":"2023-10-10T17:41:37.596125+0000","last_became_active":"2023-10-10T17:40:41.608510+0000","last_became_peered":"2023-10-10T17:40:41.608510+0000","last_unstale":"2023-10-10T17:41:37.596125+0000","last_undegraded":"2023-10-10T17:41:37.596125+0000","last_fullsized":"2023-10-10T17:41:37.596125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:16:53.945359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598418+0000","last_change":"2023-10-10T17:40:41.603516+0000","last_active":"2023-10-10T17:41:37.598418+0000","last_peered":"2023-10-10T17:41:37.598418+0000","last_clean":"2023-10-10T17:41:37.598418+0000","last_became_active":"2023-10-10T17:40:41.602948+0000","last_became_peered":"2023-10-10T17:40:41.602948+0000","last_unstale":"2023-10-10T17:41:37.598418+0000","last_undegraded":"2023-10-10T17:41:37.598418+0000","last_fullsized":"2023-10-10T17:41:37.598418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:04.694328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602112+0000","last_change":"2023-10-10T17:41:37.602112+0000","last_active":"2023-10-10T17:41:37.602112+0000","last_peered":"2023-10-10T17:41:37.602112+0000","last_clean":"2023-10-10T17:41:37.602112+0000","last_became_active":"2023-10-10T17:41:37.601781+0000","last_became_peered":"2023-10-10T17:41:37.601781+0000","last_unstale":"2023-10-10T17:41:37.602112+0000","last_undegraded":"2023-10-10T17:41:37.602112+0000","last_fullsized":"2023-10-10T17:41:37.602112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:00.565768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.597595+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.597301+0000","last_became_peered":"2023-10-10T17:40:41.597301+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:51:11.601251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596929+0000","last_change":"2023-10-10T17:40:41.611841+0000","last_active":"2023-10-10T17:41:37.596929+0000","last_peered":"2023-10-10T17:41:37.596929+0000","last_clean":"2023-10-10T17:41:37.596929+0000","last_became_active":"2023-10-10T17:40:41.611438+0000","last_became_peered":"2023-10-10T17:40:41.611438+0000","last_unstale":"2023-10-10T17:41:37.596929+0000","last_undegraded":"2023-10-10T17:41:37.596929+0000","last_fullsized":"2023-10-10T17:41:37.596929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:29:05.260738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979112+0000","last_change":"2023-10-10T17:40:41.607707+0000","last_active":"2023-10-10T17:41:37.979112+0000","last_peered":"2023-10-10T17:41:37.979112+0000","last_clean":"2023-10-10T17:41:37.979112+0000","last_became_active":"2023-10-10T17:40:41.607412+0000","last_became_peered":"2023-10-10T17:40:41.607412+0000","last_unstale":"2023-10-10T17:41:37.979112+0000","last_undegraded":"2023-10-10T17:41:37.979112+0000","last_fullsized":"2023-10-10T17:41:37.979112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:09:49.015272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651033+0000","last_change":"2023-10-10T17:40:41.593674+0000","last_active":"2023-10-10T17:41:37.651033+0000","last_peered":"2023-10-10T17:41:37.651033+0000","last_clean":"2023-10-10T17:41:37.651033+0000","last_became_active":"2023-10-10T17:40:41.593339+0000","last_became_peered":"2023-10-10T17:40:41.593339+0000","last_unstale":"2023-10-10T17:41:37.651033+0000","last_undegraded":"2023-10-10T17:41:37.651033+0000","last_fullsized":"2023-10-10T17:41:37.651033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:39:37.705884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978977+0000","last_change":"2023-10-10T17:40:41.610075+0000","last_active":"2023-10-10T17:41:37.978977+0000","last_peered":"2023-10-10T17:41:37.978977+0000","last_clean":"2023-10-10T17:41:37.978977+0000","last_became_active":"2023-10-10T17:40:41.609669+0000","last_became_peered":"2023-10-10T17:40:41.609669+0000","last_unstale":"2023-10-10T17:41:37.978977+0000","last_undegraded":"2023-10-10T17:41:37.978977+0000","last_fullsized":"2023-10-10T17:41:37.978977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:42.281285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596551+0000","last_change":"2023-10-10T17:40:41.596780+0000","last_active":"2023-10-10T17:41:37.596551+0000","last_peered":"2023-10-10T17:41:37.596551+0000","last_clean":"2023-10-10T17:41:37.596551+0000","last_became_active":"2023-10-10T17:40:41.596477+0000","last_became_peered":"2023-10-10T17:40:41.596477+0000","last_unstale":"2023-10-10T17:41:37.596551+0000","last_undegraded":"2023-10-10T17:41:37.596551+0000","last_fullsized":"2023-10-10T17:41:37.596551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:51:14.200732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598899+0000","last_change":"2023-10-10T17:40:41.606226+0000","last_active":"2023-10-10T17:41:37.598899+0000","last_peered":"2023-10-10T17:41:37.598899+0000","last_clean":"2023-10-10T17:41:37.598899+0000","last_became_active":"2023-10-10T17:40:41.605843+0000","last_became_peered":"2023-10-10T17:40:41.605843+0000","last_unstale":"2023-10-10T17:41:37.598899+0000","last_undegraded":"2023-10-10T17:41:37.598899+0000","last_fullsized":"2023-10-10T17:41:37.598899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:44.165622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788525+0000","last_change":"2023-10-10T17:40:41.596122+0000","last_active":"2023-10-10T17:41:37.788525+0000","last_peered":"2023-10-10T17:41:37.788525+0000","last_clean":"2023-10-10T17:41:37.788525+0000","last_became_active":"2023-10-10T17:40:41.595818+0000","last_became_peered":"2023-10-10T17:40:41.595818+0000","last_unstale":"2023-10-10T17:41:37.788525+0000","last_undegraded":"2023-10-10T17:41:37.788525+0000","last_fullsized":"2023-10-10T17:41:37.788525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:24:32.658064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250871,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7308,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":271,"num_read_kb":149,"num_write":129,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":21924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383704,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":987136,"data_stored":492386,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":4.2000000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":2.6739999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.86099999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.621}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":4.1319999999999997,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.97499999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.71999999999999997}]},{"osd":6,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.99399999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":1.01},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.3620000000000001,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.0309999999999999}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":9,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":1.05}]},{"osd":10,"last update":"Tue Oct 10 17:41:03 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":5.0919999999999996,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":1.117},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.5510000000000002,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":1.0720000000000001}]}]},{"osd":10,"up_from":73,"seq":313532612635,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6432,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738720,"statfs":{"total":95995035648,"available":95988449280,"internally_reserved":0,"allocated":1073152,"data_stored":1087165,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.32700000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.441}]},{"osd":2,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.47399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78200000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.51500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.80200000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76700000000000002,"5min":1.071,"15min":1.071},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.47299999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.50900000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.65400000000000003},"min":{"1min":0.52900000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.81599999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.67000000000000004}]},{"osd":6,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.56699999999999995,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90100000000000002,"5min":1.05,"15min":1.05},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.56799999999999995}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.70899999999999996}]},{"osd":8,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70399999999999996,"15min":0.69999999999999996},"min":{"1min":0.60299999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.85499999999999998,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91900000000000004,"5min":1.157,"15min":1.157},"last":0.68300000000000005}]},{"osd":9,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":1.2,"15min":1.2},"last":0.626}]},{"osd":11,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.69899999999999995}]}]},{"osd":9,"up_from":68,"seq":292057776161,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":1003520,"data_stored":492622,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48199999999999998,"15min":0.502},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54000000000000004,"5min":1.016,"15min":1.016},"last":0.39900000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85799999999999998,"5min":1.103,"15min":1.103},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.755,"5min":1.03,"15min":1.03},"last":0.48799999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59899999999999998,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69199999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70199999999999996,"5min":1.151,"15min":1.151},"last":0.45300000000000001}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.54000000000000004,"15min":0.56299999999999994},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.62,"5min":0.748,"15min":0.748},"last":0.438},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.54600000000000004,"15min":0.56899999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64300000000000002,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.378}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73799999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68999999999999995,"15min":0.70999999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58399999999999996}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.69599999999999995,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.436,"15min":0.436},"max":{"1min":0.77000000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.63},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69899999999999995,"15min":0.72099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.71299999999999997,"5min":1.417,"15min":1.417},"last":0.53900000000000003}]},{"osd":6,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.67100000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.76700000000000002,"15min":0.79100000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.80900000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60399999999999998}]},{"osd":8,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.753,"15min":0.77500000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77600000000000002,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.78900000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79400000000000004,"5min":1.377,"15min":1.377},"last":0.72799999999999998}]},{"osd":10,"last update":"Tue Oct 10 17:40:40 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.749}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":8,"up_from":60,"seq":257698037797,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6492,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738660,"statfs":{"total":95995035648,"available":95988387840,"internally_reserved":0,"allocated":1003520,"data_stored":492542,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.52100000000000002,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73799999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.55900000000000005,"15min":0.55600000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.68600000000000005}]},{"osd":1,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56100000000000005,"15min":0.55400000000000005},"min":{"1min":0.47199999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.76000000000000001,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59999999999999998,"15min":0.59099999999999997},"min":{"1min":0.438,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92900000000000005,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.84899999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.40300000000000002,"15min":0.39200000000000002},"min":{"1min":0.32400000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.61499999999999999,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.45200000000000001,"15min":0.44400000000000001},"min":{"1min":0.33700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.63700000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.79000000000000004}]},{"osd":3,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56499999999999995,"15min":0.56000000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72899999999999998,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.59599999999999997}]},{"osd":4,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.47799999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82899999999999996,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.60199999999999998,"15min":0.59399999999999997},"min":{"1min":0.52100000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.80900000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.83699999999999997}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.51500000000000001,"15min":0.501},"min":{"1min":0.33800000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.499},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.53100000000000003,"15min":0.52000000000000002},"min":{"1min":0.36099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.79100000000000004,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65200000000000002,"15min":0.64700000000000002},"min":{"1min":0.5,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.81499999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.69699999999999995,"15min":0.68999999999999995},"min":{"1min":0.47999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.96999999999999997,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.752}]},{"osd":7,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.69199999999999995,"15min":0.67900000000000005},"min":{"1min":0.57999999999999996,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0029999999999999,"5min":1.022,"15min":1.022},"last":0.872},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.67400000000000004,"15min":0.66500000000000004},"min":{"1min":0.59999999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.86799999999999999,"5min":1.101,"15min":1.101},"last":0.82499999999999996}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":0.76900000000000002,"15min":0.71599999999999997},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":6.3630000000000004,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.74399999999999999,"15min":0.73399999999999999},"min":{"1min":0.64100000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.96999999999999997,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.81299999999999994}]},{"osd":10,"last update":"Tue Oct 10 17:40:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.86099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":7,"up_from":52,"seq":223338299433,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1044480,"data_stored":1083595,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72099999999999997,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.374}]},{"osd":1,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40899999999999997,"15min":0.39300000000000002},"min":{"1min":0.28699999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72099999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41199999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.628,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.78000000000000003,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55500000000000005}]},{"osd":3,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.59799999999999998,"15min":0.58899999999999997},"min":{"1min":0.48399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.755,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.67600000000000005}]},{"osd":4,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.312,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48099999999999998,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.38900000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.442,"5min":0.441,"15min":0.441},"max":{"1min":0.85499999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.48299999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60299999999999998}]},{"osd":8,"last update":"Tue Oct 10 17:41:01 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.68700000000000006,"15min":0.68400000000000005},"min":{"1min":0.53300000000000003,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.93100000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.67000000000000004,"15min":0.66800000000000004},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.63100000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:41:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71099999999999997,"15min":0.70999999999999996},"min":{"1min":0.54700000000000004,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.625},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.73599999999999999,"15min":0.73799999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.94399999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.61399999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.751,"15min":0.745},"min":{"1min":0.55600000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92300000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.59099999999999997}]},{"osd":11,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.65800000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462958,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":987136,"data_stored":491894,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.375,"15min":0.375},"min":{"1min":0.27700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.317}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.41299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.503},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.42699999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83099999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.439}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.442,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.53000000000000003}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57399999999999995,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.54900000000000004}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.495},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.54100000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.57799999999999996},"min":{"1min":0.46400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89500000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.40699999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64600000000000002,"15min":0.64300000000000002},"min":{"1min":0.48499999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.88600000000000001,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.66300000000000003,"15min":0.66600000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.85599999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.56000000000000005}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76500000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.45700000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:41:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.73199999999999998,"15min":0.74299999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.73899999999999999,"15min":0.753},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.875,"5min":1.042,"15min":1.042},"last":0.58199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":995328,"data_stored":495598,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57999999999999996,"15min":0.67700000000000005},"min":{"1min":0.39900000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80400000000000005,"5min":5.859,"15min":5.859},"last":0.68300000000000005}]},{"osd":2,"last update":"Tue Oct 10 17:40:53 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1,"5min":1,"15min":1},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.433,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79000000000000004,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.72999999999999998}]},{"osd":3,"last update":"Tue Oct 10 17:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.56100000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76900000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.58799999999999997,"15min":0.56200000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.80600000000000005,"5min":0.875,"15min":0.875},"last":0.59099999999999997}]},{"osd":4,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.45800000000000002,"15min":0.40400000000000003},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.46999999999999997,"15min":0.41499999999999998},"min":{"1min":0.38700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.38400000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56799999999999995,"15min":0.54800000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.40000000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88700000000000001,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.65300000000000002}]},{"osd":6,"last update":"Tue Oct 10 17:41:16 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.59799999999999998,"15min":0.59099999999999997},"min":{"1min":0.42399999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.622},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.627,"15min":0.60899999999999999},"min":{"1min":0.443,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.63600000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.59999999999999998,"15min":0.55700000000000005},"min":{"1min":0.42199999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.495,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.48899999999999999}]},{"osd":8,"last update":"Tue Oct 10 17:41:01 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.65600000000000003,"15min":0.64600000000000002},"min":{"1min":0.45600000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.68200000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.46999999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:41:19 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75700000000000001,"15min":0.747},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.72099999999999997,"15min":0.70899999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.71199999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:40:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.77100000000000002}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7396,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737756,"statfs":{"total":95995035648,"available":95987462144,"internally_reserved":0,"allocated":1011712,"data_stored":493282,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 17:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54500000000000004,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.56399999999999995,"15min":0.53300000000000003},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77700000000000002,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.56599999999999995,"15min":0.52800000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.48599999999999999}]},{"osd":3,"last update":"Tue Oct 10 17:39:07 2023","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35599999999999998},"min":{"1min":0.25900000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40899999999999997,"15min":0.40500000000000003},"min":{"1min":0.27700000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.63800000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.32000000000000001}]},{"osd":4,"last update":"Tue Oct 10 17:41:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55600000000000005,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.58099999999999996}]},{"osd":5,"last update":"Tue Oct 10 17:40:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.50800000000000001},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.433},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55100000000000005,"15min":0.52500000000000002},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.38500000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.75800000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.36899999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:40:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.626,"15min":0.62},"min":{"1min":0.41299999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.443}]},{"osd":8,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.69999999999999996,"15min":0.70699999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.89700000000000002,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.70299999999999996,"15min":0.70899999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90800000000000003,"5min":1.056,"15min":1.056},"last":0.69699999999999995}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.39900000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.72499999999999998}]},{"osd":11,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.54100000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7456,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737696,"statfs":{"total":95995035648,"available":95987400704,"internally_reserved":0,"allocated":1007616,"data_stored":496169,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.31,"5min":0.252,"15min":0.252},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48799999999999999,"15min":0.48099999999999998},"min":{"1min":0.33100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.68300000000000005}]},{"osd":1,"last update":"Tue Oct 10 17:40:49 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.56799999999999995,"15min":0.52000000000000002},"min":{"1min":0.34799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.484,"5min":5.484,"15min":5.484},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.54700000000000004,"15min":0.51700000000000002},"min":{"1min":0.40500000000000003,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.753,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55900000000000005,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72799999999999998,"5min":1.099,"15min":1.099},"last":0.72299999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51900000000000002,"15min":0.502},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.71999999999999997,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52100000000000002,"15min":0.501},"min":{"1min":0.42199999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.439}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.439,"15min":0.40699999999999997},"min":{"1min":0.33200000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47799999999999998,"15min":0.45400000000000001},"min":{"1min":0.31,"5min":0.307,"15min":0.307},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.498}]},{"osd":6,"last update":"Tue Oct 10 17:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.59399999999999997,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.74199999999999999,"5min":1.129,"15min":1.129},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59299999999999997,"15min":0.58599999999999997},"min":{"1min":0.44,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83799999999999997,"5min":1.075,"15min":1.075},"last":0.629}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.435,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.77900000000000003,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.81599999999999995,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.75900000000000001}]},{"osd":8,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.60999999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93600000000000005,"5min":1.087,"15min":1.087},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.996,"5min":1.226,"15min":1.226},"last":0.879}]},{"osd":10,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.77900000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6740,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738412,"statfs":{"total":95995035648,"available":95988133888,"internally_reserved":0,"allocated":995328,"data_stored":491757,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:11 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.35599999999999998,"15min":0.34799999999999998},"min":{"1min":0.29499999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.47299999999999998,"5min":0.752,"15min":0.752},"last":0.309},{"interface":"front","average":{"1min":0.371,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.497,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001}]},{"osd":1,"last update":"Tue Oct 10 17:41:11 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56299999999999994,"15min":0.53900000000000003},"min":{"1min":0.35799999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72399999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.53800000000000003,"15min":0.51800000000000002},"min":{"1min":0.36699999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Oct 10 17:41:11 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.56100000000000005,"15min":0.53800000000000003},"min":{"1min":0.435,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.2210000000000001,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56200000000000006,"15min":0.53800000000000003},"min":{"1min":0.44900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74399999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.55100000000000005}]},{"osd":4,"last update":"Tue Oct 10 17:41:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.41099999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84099999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61299999999999999,"15min":0.59299999999999997},"min":{"1min":0.46000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.746,"5min":1.214,"15min":1.214},"last":0.41599999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:41:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.59099999999999997,"15min":0.59899999999999998},"min":{"1min":0.48099999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.72099999999999997,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.61599999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:41:17 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52100000000000002,"15min":0.51500000000000001},"min":{"1min":0.32700000000000001,"5min":0.308,"15min":0.308},"max":{"1min":1.3149999999999999,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.433},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.46600000000000003,"15min":0.47999999999999998},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.49199999999999999,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.36699999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.56100000000000005}]},{"osd":8,"last update":"Tue Oct 10 17:39:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53700000000000003}]},{"osd":9,"last update":"Tue Oct 10 17:41:19 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.65700000000000003,"15min":0.66500000000000004},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.504},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.67800000000000005,"15min":0.69399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.755,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.51500000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.58999999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]}]},{"osd":4,"up_from":34,"seq":146028888119,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1003520,"data_stored":492512,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.432,"5min":0.372,"15min":0.372},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.36599999999999999},"min":{"1min":0.31,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65200000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.40799999999999997,"15min":0.372},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.33100000000000002}]},{"osd":2,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.63400000000000001,"15min":0.57499999999999996},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53000000000000003}]},{"osd":3,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.60899999999999999,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.1289999999999996,"5min":6.1289999999999996,"15min":6.1289999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.75900000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.35199999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.59999999999999998,"15min":0.57799999999999996},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86799999999999999,"5min":1.589,"15min":1.589},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.37,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.40100000000000002}]},{"osd":6,"last update":"Tue Oct 10 17:41:16 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.65300000000000002,"15min":0.63900000000000001},"min":{"1min":0.45500000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95099999999999996,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.499}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.57299999999999995,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.61799999999999999,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.59299999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:40:58 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.65100000000000002,"15min":0.64800000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.74399999999999999,"15min":0.71199999999999997},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":6.1639999999999997,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.60699999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:41:21 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.71499999999999997,"15min":0.70899999999999996},"min":{"1min":0.502,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.92000000000000004,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.627},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.76600000000000001,"15min":0.76100000000000001},"min":{"1min":0.53000000000000003,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.018,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.61699999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68200000000000005,"15min":0.67400000000000004},"min":{"1min":0.52300000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.80400000000000005,"15min":0.79600000000000004},"min":{"1min":0.56100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.038,"5min":1.169,"15min":1.169},"last":0.56999999999999995}]},{"osd":11,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.64400000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6736,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738416,"statfs":{"total":95995035648,"available":95988137984,"internally_reserved":0,"allocated":1056768,"data_stored":1083327,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.58199999999999996,"15min":0.52400000000000002},"min":{"1min":0.32100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":7.1230000000000002,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52000000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.75900000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53600000000000003,"15min":0.52900000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73899999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.60599999999999998,"15min":0.55000000000000004},"min":{"1min":0.41199999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":7.0570000000000004,"5min":7.0570000000000004,"15min":7.0570000000000004},"last":0.497}]},{"osd":2,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.40500000000000003,"15min":0.39900000000000002},"min":{"1min":0.28699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65500000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.436},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.45300000000000001,"15min":0.42199999999999999},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.39900000000000002}]},{"osd":3,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.40699999999999997,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57699999999999996,"15min":0.55900000000000005},"min":{"1min":0.42199999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4209999999999998,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.64100000000000001,"15min":0.58899999999999997},"min":{"1min":0.41499999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":7.3479999999999999,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.60799999999999998,"15min":0.60199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.77700000000000002,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.64900000000000002}]},{"osd":6,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.432,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63900000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.504,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.624},"min":{"1min":0.54300000000000004,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66300000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:41:02 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.59699999999999998,"15min":0.55100000000000005},"min":{"1min":0.38700000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.7229999999999999,"5min":6.7229999999999999,"15min":6.7229999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56000000000000005,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.72399999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70899999999999996,"15min":0.71299999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90300000000000002,"5min":1.034,"15min":1.034},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.71299999999999997,"15min":0.69899999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":0.68400000000000005}]},{"osd":10,"last update":"Tue Oct 10 17:40:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.71299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T17:41:44.900 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-10-10T17:41:44.901 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-10T17:41:44.901 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-10-10T17:41:44.902 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph health --format=json 2023-10-10T17:41:45.561 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:45.853 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:45 smithi078 ceph-mon[99494]: pgmap v295: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 13 B/s, 0 objects/s recovering 2023-10-10T17:41:45.853 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:45 smithi078 ceph-mon[99494]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:45.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:45 smithi190 ceph-mon[101220]: pgmap v295: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 13 B/s, 0 objects/s recovering 2023-10-10T17:41:45.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:45 smithi190 ceph-mon[101220]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:45.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:45 smithi157 ceph-mon[100896]: pgmap v295: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 13 B/s, 0 objects/s recovering 2023-10-10T17:41:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:45 smithi157 ceph-mon[100896]: from='client.15087 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:41:46.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:46 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:46 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.664 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:46 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:46 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:46 smithi078 ceph-mon[99494]: pgmap v296: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 11 B/s, 0 objects/s recovering 2023-10-10T17:41:46.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:46 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.666 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:46 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.821+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/606329380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a00ffef0 msgr2=0x7fe1a010a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.821+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/606329380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a00ffef0 0x7fe1a010a960 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe188009990 tx=0x7fe188009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.822+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/606329380 shutdown_connections 2023-10-10T17:41:46.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.822+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/606329380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1a010aea0 0x7fe1a010d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.822+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/606329380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1a00ff5e0 0x7fe1a00ff9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.822+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/606329380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a00ffef0 0x7fe1a010a960 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.822+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/606329380 >> 172.21.15.78:0/606329380 conn(0x7fe1a0074ba0 msgr2=0x7fe1a0074fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.822+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/606329380 shutdown_connections 2023-10-10T17:41:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.822+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/606329380 wait complete. 2023-10-10T17:41:46.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 Processor -- start 2023-10-10T17:41:46.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 -- start start 2023-10-10T17:41:46.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1a00ff5e0 0x7fe1a01963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:46.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1a00ffef0 0x7fe1a0196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:46.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a010aea0 0x7fe1a019ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:46.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a019b390 con 0x7fe1a010aea0 2023-10-10T17:41:46.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a019b500 con 0x7fe1a00ffef0 2023-10-10T17:41:46.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a6e4c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a019b6c0 con 0x7fe1a00ff5e0 2023-10-10T17:41:46.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a010aea0 0x7fe1a019ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:46.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a010aea0 0x7fe1a019ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43094/0 (socket says 172.21.15.78:43094) 2023-10-10T17:41:46.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 -- 172.21.15.78:0/4053543091 learned_addr learned my addr 172.21.15.78:0/4053543091 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:46.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a4be8700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1a00ff5e0 0x7fe1a01963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:46.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe19ffff700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1a00ffef0 0x7fe1a0196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:46.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 -- 172.21.15.78:0/4053543091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1a00ff5e0 msgr2=0x7fe1a01963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:46.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1a00ff5e0 0x7fe1a01963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:46.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 -- 172.21.15.78:0/4053543091 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1a00ffef0 msgr2=0x7fe1a0196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:46.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1a00ffef0 0x7fe1a0196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe1a53e9700 1 -- 172.21.15.78:0/4053543091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1880095f0 con 0x7fe1a010aea0 2023-10-10T17:41:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.823+0000 7fe19ffff700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1a00ffef0 0x7fe1a0196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:46.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.824+0000 7fe1a53e9700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a010aea0 0x7fe1a019ad80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe19000d720 tx=0x7fe19000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:46.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.824+0000 7fe19dffb700 1 -- 172.21.15.78:0/4053543091 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190004a30 con 0x7fe1a010aea0 2023-10-10T17:41:46.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.824+0000 7fe19dffb700 1 -- 172.21.15.78:0/4053543091 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1900054e0 con 0x7fe1a010aea0 2023-10-10T17:41:46.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.824+0000 7fe19dffb700 1 -- 172.21.15.78:0/4053543091 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe19000fbf0 con 0x7fe1a010aea0 2023-10-10T17:41:46.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.824+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a0110450 con 0x7fe1a010aea0 2023-10-10T17:41:46.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.824+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a01102c0 con 0x7fe1a010aea0 2023-10-10T17:41:46.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.825+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a01105e0 con 0x7fe1a010aea0 2023-10-10T17:41:46.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.826+0000 7fe19dffb700 1 -- 172.21.15.78:0/4053543091 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe190005650 con 0x7fe1a010aea0 2023-10-10T17:41:46.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.826+0000 7fe19dffb700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe18c0a11d0 0x7fe18c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.826+0000 7fe1a4be8700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe18c0a11d0 0x7fe18c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.826+0000 7fe19dffb700 1 -- 172.21.15.78:0/4053543091 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe1900cc300 con 0x7fe1a010aea0 2023-10-10T17:41:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.827+0000 7fe1a4be8700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe18c0a11d0 0x7fe18c0a3680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0073160 tx=0x7fe198009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:46.831+0000 7fe19dffb700 1 -- 172.21.15.78:0/4053543091 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe1900957e0 con 0x7fe1a010aea0 2023-10-10T17:41:46.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:46 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:46 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:46 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:46 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:46 smithi190 ceph-mon[101220]: pgmap v296: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 11 B/s, 0 objects/s recovering 2023-10-10T17:41:46.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:46 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:46 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:46 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:46 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:46 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:46 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:46 smithi157 ceph-mon[100896]: pgmap v296: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 11 B/s, 0 objects/s recovering 2023-10-10T17:41:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:46 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:46 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:47.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.071+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe1a0197730 con 0x7fe1a010aea0 2023-10-10T17:41:47.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.071+0000 7fe19dffb700 1 -- 172.21.15.78:0/4053543091 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe190098e00 con 0x7fe1a010aea0 2023-10-10T17:41:47.072 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:47.072 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-10T17:41:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe18c0a11d0 msgr2=0x7fe18c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe18c0a11d0 0x7fe18c0a3680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0073160 tx=0x7fe198009040 comp rx=0 tx=0).stop 2023-10-10T17:41:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a010aea0 msgr2=0x7fe1a019ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a010aea0 0x7fe1a019ad80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe19000d720 tx=0x7fe19000dae0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 shutdown_connections 2023-10-10T17:41:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe18c0a11d0 0x7fe18c0a3680 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1a00ff5e0 0x7fe1a01963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1a010aea0 0x7fe1a019ad80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 --2- 172.21.15.78:0/4053543091 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1a00ffef0 0x7fe1a0196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 >> 172.21.15.78:0/4053543091 conn(0x7fe1a0074ba0 msgr2=0x7fe1a0102900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 shutdown_connections 2023-10-10T17:41:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.073+0000 7fe1a6e4c700 1 -- 172.21.15.78:0/4053543091 wait complete. 2023-10-10T17:41:47.462 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-10-10T17:41:47.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:41:47.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 -- 172.21.15.78:0/4075160306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 msgr2=0x7fad3c107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 --2- 172.21.15.78:0/4075160306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c107680 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fad3400b0a0 tx=0x7fad3400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 -- 172.21.15.78:0/4075160306 shutdown_connections 2023-10-10T17:41:47.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 --2- 172.21.15.78:0/4075160306 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c100730 0x7fad3c100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 --2- 172.21.15.78:0/4075160306 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c107680 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 --2- 172.21.15.78:0/4075160306 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad3c0ff420 0x7fad3c0ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 -- 172.21.15.78:0/4075160306 >> 172.21.15.78:0/4075160306 conn(0x7fad3c0f9ee0 msgr2=0x7fad3c0fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:47.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 -- 172.21.15.78:0/4075160306 shutdown_connections 2023-10-10T17:41:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.587+0000 7fad42133700 1 -- 172.21.15.78:0/4075160306 wait complete. 2023-10-10T17:41:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 Processor -- start 2023-10-10T17:41:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 -- start start 2023-10-10T17:41:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c0ff420 0x7fad3c0fee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad3c100730 0x7fad3c19d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c19daf0 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c19dc60 con 0x7fad3c100730 2023-10-10T17:41:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad42133700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3c19de20 con 0x7fad3c0ff420 2023-10-10T17:41:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad3affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad3affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c19ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43104/0 (socket says 172.21.15.78:43104) 2023-10-10T17:41:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad3affd700 1 -- 172.21.15.78:0/1095172666 learned_addr learned my addr 172.21.15.78:0/1095172666 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.588+0000 7fad3b7fe700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c0ff420 0x7fad3c0fee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad3bfff700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad3c100730 0x7fad3c19d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad3affd700 1 -- 172.21.15.78:0/1095172666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c0ff420 msgr2=0x7fad3c0fee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad3affd700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c0ff420 0x7fad3c0fee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad3affd700 1 -- 172.21.15.78:0/1095172666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad3c100730 msgr2=0x7fad3c19d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad3affd700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad3c100730 0x7fad3c19d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad3affd700 1 -- 172.21.15.78:0/1095172666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad34009d00 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad3affd700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c19ad10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00c9b0 tx=0x7fad2c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad38ff9700 1 -- 172.21.15.78:0/1095172666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c007b00 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad38ff9700 1 -- 172.21.15.78:0/1095172666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad2c007c60 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c10ab20 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad38ff9700 1 -- 172.21.15.78:0/1095172666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c018660 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.589+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad3c10a990 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.590+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3c19e570 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.591+0000 7fad38ff9700 1 -- 172.21.15.78:0/1095172666 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fad2c00f450 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.591+0000 7fad38ff9700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fad240a1150 0x7fad240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.592+0000 7fad3b7fe700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fad240a1150 0x7fad240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.592+0000 7fad38ff9700 1 -- 172.21.15.78:0/1095172666 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fad2c0ccc70 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.592+0000 7fad3b7fe700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fad240a1150 0x7fad240a3600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad3c100450 tx=0x7fad3000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:47.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.596+0000 7fad38ff9700 1 -- 172.21.15.78:0/1095172666 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fad2c096050 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.786+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad3c106070 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.788+0000 7fad38ff9700 1 -- 172.21.15.78:0/1095172666 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7fad2c099670 con 0x7fad3c0ffdd0 2023-10-10T17:41:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:47.789 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:41:47.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fad240a1150 msgr2=0x7fad240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fad240a1150 0x7fad240a3600 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fad3c100450 tx=0x7fad3000b410 comp rx=0 tx=0).stop 2023-10-10T17:41:47.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 msgr2=0x7fad3c19ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c19ad10 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fad2c00c9b0 tx=0x7fad2c00ccc0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 shutdown_connections 2023-10-10T17:41:47.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fad240a1150 0x7fad240a3600 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fad3c0ff420 0x7fad3c0fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fad3c100730 0x7fad3c19d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 --2- 172.21.15.78:0/1095172666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fad3c0ffdd0 0x7fad3c19ad10 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 >> 172.21.15.78:0/1095172666 conn(0x7fad3c0f9ee0 msgr2=0x7fad3c0fc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:47.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 shutdown_connections 2023-10-10T17:41:47.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.790+0000 7fad42133700 1 -- 172.21.15.78:0/1095172666 wait complete. 2023-10-10T17:41:47.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4053543091' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-10T17:41:47.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:41:47.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:41:47.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:41:47.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:47.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:41:47.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:41:47.804 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:47 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:41:47.805 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:41:47.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:41:47.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 -- 172.21.15.78:0/1979134801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 msgr2=0x7f63bc0ff5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1979134801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0ff5d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009990 tx=0x7f63ac009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 -- 172.21.15.78:0/1979134801 shutdown_connections 2023-10-10T17:41:47.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1979134801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f63bc100d40 0x7f63bc107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1979134801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0ff5d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1979134801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc1003b0 0x7f63bc100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 -- 172.21.15.78:0/1979134801 >> 172.21.15.78:0/1979134801 conn(0x7f63bc0fa7a0 msgr2=0x7f63bc0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:47.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 -- 172.21.15.78:0/1979134801 shutdown_connections 2023-10-10T17:41:47.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.930+0000 7f63c24fb700 1 -- 172.21.15.78:0/1979134801 wait complete. 2023-10-10T17:41:47.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 Processor -- start 2023-10-10T17:41:47.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 -- start start 2023-10-10T17:41:47.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0699e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f63bc1003b0 0x7f63bc19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100d40 0x7f63bc19d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc19db80 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc19dcf0 con 0x7f63bc1003b0 2023-10-10T17:41:47.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63c24fb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc19deb0 con 0x7f63bc100d40 2023-10-10T17:41:47.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0699e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0699e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43116/0 (socket says 172.21.15.78:43116) 2023-10-10T17:41:47.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63bbfff700 1 -- 172.21.15.78:0/1428827015 learned_addr learned my addr 172.21.15.78:0/1428827015 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:47.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.931+0000 7f63bb7fe700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f63bc1003b0 0x7f63bc19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63c0a98700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100d40 0x7f63bc19d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63bbfff700 1 -- 172.21.15.78:0/1428827015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100d40 msgr2=0x7f63bc19d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63bbfff700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100d40 0x7f63bc19d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63bbfff700 1 -- 172.21.15.78:0/1428827015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f63bc1003b0 msgr2=0x7f63bc19ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:47.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63bbfff700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f63bc1003b0 0x7f63bc19ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:47.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63bbfff700 1 -- 172.21.15.78:0/1428827015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac0095f0 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63c0a98700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100d40 0x7f63bc19d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:47.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63bbfff700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0699e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f63ac000c00 tx=0x7f63ac0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63b97fa700 1 -- 172.21.15.78:0/1428827015 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac01d070 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63b97fa700 1 -- 172.21.15.78:0/1428827015 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f63ac022470 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc10ab20 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63b97fa700 1 -- 172.21.15.78:0/1428827015 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac017650 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.932+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc10a990 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.933+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc19e4b0 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.934+0000 7f63b97fa700 1 -- 172.21.15.78:0/1428827015 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f63ac0225e0 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.935+0000 7f63b97fa700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63a40a11d0 0x7f63a40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:47.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.935+0000 7f63bb7fe700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63a40a11d0 0x7f63a40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:47.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.935+0000 7f63b97fa700 1 -- 172.21.15.78:0/1428827015 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f63ac0ce1f0 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.935+0000 7f63bb7fe700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63a40a11d0 0x7f63a40a3680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009c60 tx=0x7f63b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:47.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:47.939+0000 7f63b97fa700 1 -- 172.21.15.78:0/1428827015 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f63ac0976c0 con 0x7f63bc0ff1c0 2023-10-10T17:41:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4053543091' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-10T17:41:47.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:41:47.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:41:47.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:41:47.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:47.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:41:47.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:41:47.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:47 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:41:48.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.127+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63bc19eb90 con 0x7f63bc0ff1c0 2023-10-10T17:41:48.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.127+0000 7f63b97fa700 1 -- 172.21.15.78:0/1428827015 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63ac09ace0 con 0x7f63bc0ff1c0 2023-10-10T17:41:48.128 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:41:48.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63a40a11d0 msgr2=0x7f63a40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63a40a11d0 0x7f63a40a3680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009c60 tx=0x7f63b0009260 comp rx=0 tx=0).stop 2023-10-10T17:41:48.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 msgr2=0x7f63bc0699e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0699e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f63ac000c00 tx=0x7f63ac0047a0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 shutdown_connections 2023-10-10T17:41:48.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63a40a11d0 0x7f63a40a3680 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f63bc100d40 0x7f63bc19d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f63bc0ff1c0 0x7f63bc0699e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 --2- 172.21.15.78:0/1428827015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f63bc1003b0 0x7f63bc19ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 >> 172.21.15.78:0/1428827015 conn(0x7f63bc0fa7a0 msgr2=0x7f63bc0fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:48.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 shutdown_connections 2023-10-10T17:41:48.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.130+0000 7f63c24fb700 1 -- 172.21.15.78:0/1428827015 wait complete. 2023-10-10T17:41:48.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:41:48.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4053543091' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-10T17:41:48.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:41:48.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:41:48.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:41:48.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: from='mgr.14152 172.21.15.78:0/2765705281' entity='mgr.x' 2023-10-10T17:41:48.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:41:48.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:41:48.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:47 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:41:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 -- 172.21.15.78:0/2284612233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 msgr2=0x7f620c100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 --2- 172.21.15.78:0/2284612233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c100820 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6200009990 tx=0x7f6200009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 -- 172.21.15.78:0/2284612233 shutdown_connections 2023-10-10T17:41:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 --2- 172.21.15.78:0/2284612233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f620c0ff1e0 0x7f620c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 --2- 172.21.15.78:0/2284612233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c100d60 0x7f620c107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 --2- 172.21.15.78:0/2284612233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c100820 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 -- 172.21.15.78:0/2284612233 >> 172.21.15.78:0/2284612233 conn(0x7f620c0fa800 msgr2=0x7f620c0fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 -- 172.21.15.78:0/2284612233 shutdown_connections 2023-10-10T17:41:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.267+0000 7f6211e7e700 1 -- 172.21.15.78:0/2284612233 wait complete. 2023-10-10T17:41:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 Processor -- start 2023-10-10T17:41:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 -- start start 2023-10-10T17:41:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0ff1e0 0x7f620c194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c1954f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f620c100d60 0x7f620c19c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c19cb30 con 0x7f620c1003d0 2023-10-10T17:41:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c19cc70 con 0x7f620c0ff1e0 2023-10-10T17:41:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f6211e7e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620c19ce00 con 0x7f620c100d60 2023-10-10T17:41:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f620affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c1954f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f620affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c1954f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43132/0 (socket says 172.21.15.78:43132) 2023-10-10T17:41:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f620affd700 1 -- 172.21.15.78:0/1778556304 learned_addr learned my addr 172.21.15.78:0/1778556304 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f620b7fe700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0ff1e0 0x7f620c194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f620affd700 1 -- 172.21.15.78:0/1778556304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f620c100d60 msgr2=0x7f620c19c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f620bfff700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f620c100d60 0x7f620c19c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.268+0000 7f620affd700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f620c100d60 0x7f620c19c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f620affd700 1 -- 172.21.15.78:0/1778556304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0ff1e0 msgr2=0x7f620c194fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f620affd700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0ff1e0 0x7f620c194fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f620affd700 1 -- 172.21.15.78:0/1778556304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62000095f0 con 0x7f620c1003d0 2023-10-10T17:41:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f620b7fe700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0ff1e0 0x7f620c194fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f620bfff700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f620c100d60 0x7f620c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f620affd700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c1954f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6200005910 tx=0x7f6200009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f6208ff9700 1 -- 172.21.15.78:0/1778556304 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620001d070 con 0x7f620c1003d0 2023-10-10T17:41:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620c10ac10 con 0x7f620c1003d0 2023-10-10T17:41:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f6208ff9700 1 -- 172.21.15.78:0/1778556304 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f62000041d0 con 0x7f620c1003d0 2023-10-10T17:41:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f6208ff9700 1 -- 172.21.15.78:0/1778556304 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f620000f460 con 0x7f620c1003d0 2023-10-10T17:41:48.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.269+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620c10aa80 con 0x7f620c1003d0 2023-10-10T17:41:48.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.270+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620c106720 con 0x7f620c1003d0 2023-10-10T17:41:48.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.271+0000 7f6208ff9700 1 -- 172.21.15.78:0/1778556304 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f62000226e0 con 0x7f620c1003d0 2023-10-10T17:41:48.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.271+0000 7f6208ff9700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f61f40a11d0 0x7f61f40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.271+0000 7f620b7fe700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f61f40a11d0 0x7f61f40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.271+0000 7f6208ff9700 1 -- 172.21.15.78:0/1778556304 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62000ce260 con 0x7f620c1003d0 2023-10-10T17:41:48.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.272+0000 7f620b7fe700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f61f40a11d0 0x7f61f40a3680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f620c100230 tx=0x7f61fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:48.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.275+0000 7f6208ff9700 1 -- 172.21.15.78:0/1778556304 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f62000976d0 con 0x7f620c1003d0 2023-10-10T17:41:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.462+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f620c061700 con 0x7f620c1003d0 2023-10-10T17:41:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.463+0000 7f6208ff9700 1 -- 172.21.15.78:0/1778556304 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f620009acf0 con 0x7f620c1003d0 2023-10-10T17:41:48.464 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:41:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f61f40a11d0 msgr2=0x7f61f40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f61f40a11d0 0x7f61f40a3680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f620c100230 tx=0x7f61fc005f90 comp rx=0 tx=0).stop 2023-10-10T17:41:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 msgr2=0x7f620c1954f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c1954f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6200005910 tx=0x7f6200009f90 comp rx=0 tx=0).stop 2023-10-10T17:41:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 shutdown_connections 2023-10-10T17:41:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f61f40a11d0 0x7f61f40a3680 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f620c100d60 0x7f620c19c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f620c0ff1e0 0x7f620c194fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 --2- 172.21.15.78:0/1778556304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f620c1003d0 0x7f620c1954f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 >> 172.21.15.78:0/1778556304 conn(0x7f620c0fa800 msgr2=0x7f620c103860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 shutdown_connections 2023-10-10T17:41:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.465+0000 7f6211e7e700 1 -- 172.21.15.78:0/1778556304 wait complete. 2023-10-10T17:41:48.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:41:48.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 -- 172.21.15.78:0/4244258937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1740fe930 msgr2=0x7fe1740fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 --2- 172.21.15.78:0/4244258937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1740fe930 0x7fe1740fed40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe168009990 tx=0x7fe168009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 -- 172.21.15.78:0/4244258937 shutdown_connections 2023-10-10T17:41:48.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 --2- 172.21.15.78:0/4244258937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1741004b0 0x7fe174106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 --2- 172.21.15.78:0/4244258937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1740fe930 0x7fe1740fed40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 --2- 172.21.15.78:0/4244258937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1740ffb20 0x7fe1740fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 -- 172.21.15.78:0/4244258937 >> 172.21.15.78:0/4244258937 conn(0x7fe1740f9f30 msgr2=0x7fe1740fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:48.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 -- 172.21.15.78:0/4244258937 shutdown_connections 2023-10-10T17:41:48.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.606+0000 7fe178eb5700 1 -- 172.21.15.78:0/4244258937 wait complete. 2023-10-10T17:41:48.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 Processor -- start 2023-10-10T17:41:48.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 -- start start 2023-10-10T17:41:48.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1740fe930 0x7fe174194f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1740ffb20 0x7fe174195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1741004b0 0x7fe17419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17419ca90 con 0x7fe1741004b0 2023-10-10T17:41:48.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17419cbd0 con 0x7fe1740fe930 2023-10-10T17:41:48.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe178eb5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17419cd60 con 0x7fe1740ffb20 2023-10-10T17:41:48.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1741004b0 0x7fe17419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1741004b0 0x7fe17419c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43148/0 (socket says 172.21.15.78:43148) 2023-10-10T17:41:48.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 -- 172.21.15.78:0/3277471626 learned_addr learned my addr 172.21.15.78:0/3277471626 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:48.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe17259c700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1740fe930 0x7fe174194f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe171d9b700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1740ffb20 0x7fe174195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 -- 172.21.15.78:0/3277471626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1740ffb20 msgr2=0x7fe174195450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1740ffb20 0x7fe174195450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 -- 172.21.15.78:0/3277471626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1740fe930 msgr2=0x7fe174194f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1740fe930 0x7fe174194f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 -- 172.21.15.78:0/3277471626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1680095f0 con 0x7fe1741004b0 2023-10-10T17:41:48.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe17259c700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1740fe930 0x7fe174194f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:48.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.607+0000 7fe172d9d700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1741004b0 0x7fe17419c4c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe16c007ee0 tx=0x7fe16c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:48.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.608+0000 7fe1637fe700 1 -- 172.21.15.78:0/3277471626 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c017070 con 0x7fe1741004b0 2023-10-10T17:41:48.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.608+0000 7fe1637fe700 1 -- 172.21.15.78:0/3277471626 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe16c00f040 con 0x7fe1741004b0 2023-10-10T17:41:48.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.608+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17410a290 con 0x7fe1741004b0 2023-10-10T17:41:48.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.608+0000 7fe1637fe700 1 -- 172.21.15.78:0/3277471626 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c005740 con 0x7fe1741004b0 2023-10-10T17:41:48.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.608+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17410a100 con 0x7fe1741004b0 2023-10-10T17:41:48.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.609+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe174105500 con 0x7fe1741004b0 2023-10-10T17:41:48.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.610+0000 7fe1637fe700 1 -- 172.21.15.78:0/3277471626 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe16c0058a0 con 0x7fe1741004b0 2023-10-10T17:41:48.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.610+0000 7fe1637fe700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe15c0a1150 0x7fe15c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.610+0000 7fe1637fe700 1 -- 172.21.15.78:0/3277471626 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe16c0ccab0 con 0x7fe1741004b0 2023-10-10T17:41:48.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.610+0000 7fe17259c700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe15c0a1150 0x7fe15c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.611+0000 7fe17259c700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe15c0a1150 0x7fe15c0a3600 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe168006010 tx=0x7fe16800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:48.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.613+0000 7fe1637fe700 1 -- 172.21.15.78:0/3277471626 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe16c095ef0 con 0x7fe1741004b0 2023-10-10T17:41:48.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.806+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe174061700 con 0x7fe1741004b0 2023-10-10T17:41:48.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.806+0000 7fe1637fe700 1 -- 172.21.15.78:0/3277471626 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe16c099510 con 0x7fe1741004b0 2023-10-10T17:41:48.807 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:41:48.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe15c0a1150 msgr2=0x7fe15c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe15c0a1150 0x7fe15c0a3600 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe168006010 tx=0x7fe16800b540 comp rx=0 tx=0).stop 2023-10-10T17:41:48.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1741004b0 msgr2=0x7fe17419c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1741004b0 0x7fe17419c4c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe16c007ee0 tx=0x7fe16c00d3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 shutdown_connections 2023-10-10T17:41:48.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe15c0a1150 0x7fe15c0a3600 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1741004b0 0x7fe17419c4c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1740fe930 0x7fe174194f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 --2- 172.21.15.78:0/3277471626 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1740ffb20 0x7fe174195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.808+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 >> 172.21.15.78:0/3277471626 conn(0x7fe1740f9f30 msgr2=0x7fe1740fb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:48.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.809+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 shutdown_connections 2023-10-10T17:41:48.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.809+0000 7fe178eb5700 1 -- 172.21.15.78:0/3277471626 wait complete. 2023-10-10T17:41:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:48 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1095172666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:48 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1428827015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:41:48.820 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:48 smithi078 ceph-mon[99494]: pgmap v297: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 0 objects/s recovering 2023-10-10T17:41:48.821 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:48 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1778556304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:41:48.822 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:41:48.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 -- 172.21.15.78:0/758398010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550100d40 msgr2=0x7fe550103120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 --2- 172.21.15.78:0/758398010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550100d40 0x7fe550103120 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe540009990 tx=0x7fe540009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 -- 172.21.15.78:0/758398010 shutdown_connections 2023-10-10T17:41:48.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 --2- 172.21.15.78:0/758398010 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe550105f80 0x7fe5501083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 --2- 172.21.15.78:0/758398010 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550100d40 0x7fe550103120 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 --2- 172.21.15.78:0/758398010 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550103660 0x7fe550105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 -- 172.21.15.78:0/758398010 >> 172.21.15.78:0/758398010 conn(0x7fe5500fa7a0 msgr2=0x7fe5500fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:48.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 -- 172.21.15.78:0/758398010 shutdown_connections 2023-10-10T17:41:48.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.946+0000 7fe557cf5700 1 -- 172.21.15.78:0/758398010 wait complete. 2023-10-10T17:41:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 Processor -- start 2023-10-10T17:41:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 -- start start 2023-10-10T17:41:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550100d40 0x7fe550196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550103660 0x7fe5501965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe550105f80 0x7fe55019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55019dbf0 con 0x7fe550103660 2023-10-10T17:41:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55019dd30 con 0x7fe550105f80 2023-10-10T17:41:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe557cf5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55019dec0 con 0x7fe550100d40 2023-10-10T17:41:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550103660 0x7fe5501965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550103660 0x7fe5501965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43160/0 (socket says 172.21.15.78:43160) 2023-10-10T17:41:48.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 -- 172.21.15.78:0/2452574098 learned_addr learned my addr 172.21.15.78:0/2452574098 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:48.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555a91700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550100d40 0x7fe550196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe556292700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe550105f80 0x7fe55019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 -- 172.21.15.78:0/2452574098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550100d40 msgr2=0x7fe550196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550100d40 0x7fe550196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 -- 172.21.15.78:0/2452574098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe550105f80 msgr2=0x7fe55019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:48.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe550105f80 0x7fe55019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:48.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 -- 172.21.15.78:0/2452574098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400095f0 con 0x7fe550103660 2023-10-10T17:41:48.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.947+0000 7fe555290700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550103660 0x7fe5501965b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00d950 tx=0x7fe54c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.948+0000 7fe546ffd700 1 -- 172.21.15.78:0/2452574098 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c009b40 con 0x7fe550103660 2023-10-10T17:41:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.948+0000 7fe546ffd700 1 -- 172.21.15.78:0/2452574098 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe54c009ca0 con 0x7fe550103660 2023-10-10T17:41:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.948+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55010bc70 con 0x7fe550103660 2023-10-10T17:41:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.948+0000 7fe546ffd700 1 -- 172.21.15.78:0/2452574098 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54c00f620 con 0x7fe550103660 2023-10-10T17:41:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.948+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55010bae0 con 0x7fe550103660 2023-10-10T17:41:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.948+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5500fc310 con 0x7fe550103660 2023-10-10T17:41:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.950+0000 7fe546ffd700 1 -- 172.21.15.78:0/2452574098 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe54c010460 con 0x7fe550103660 2023-10-10T17:41:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.950+0000 7fe546ffd700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe53c0a1150 0x7fe53c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.950+0000 7fe555a91700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe53c0a1150 0x7fe53c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.950+0000 7fe546ffd700 1 -- 172.21.15.78:0/2452574098 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe54c0ccfa0 con 0x7fe550103660 2023-10-10T17:41:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.951+0000 7fe555a91700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe53c0a1150 0x7fe53c0a3600 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe54000b5c0 tx=0x7fe540005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:48.954+0000 7fe546ffd700 1 -- 172.21.15.78:0/2452574098 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe54c0963e0 con 0x7fe550103660 2023-10-10T17:41:48.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:48 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1095172666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:48 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1428827015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:41:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:48 smithi157 ceph-mon[100896]: pgmap v297: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 0 objects/s recovering 2023-10-10T17:41:48.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:48 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1778556304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:41:49.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.140+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe550061700 con 0x7fe550103660 2023-10-10T17:41:49.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.141+0000 7fe546ffd700 1 -- 172.21.15.78:0/2452574098 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe54c016020 con 0x7fe550103660 2023-10-10T17:41:49.142 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:41:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe53c0a1150 msgr2=0x7fe53c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe53c0a1150 0x7fe53c0a3600 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe54000b5c0 tx=0x7fe540005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550103660 msgr2=0x7fe5501965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550103660 0x7fe5501965b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe54c00d950 tx=0x7fe54c00dd10 comp rx=0 tx=0).stop 2023-10-10T17:41:49.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 shutdown_connections 2023-10-10T17:41:49.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe53c0a1150 0x7fe53c0a3600 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe550100d40 0x7fe550196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe550105f80 0x7fe55019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 --2- 172.21.15.78:0/2452574098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe550103660 0x7fe5501965b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 >> 172.21.15.78:0/2452574098 conn(0x7fe5500fa7a0 msgr2=0x7fe5501019a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:49.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 shutdown_connections 2023-10-10T17:41:49.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.143+0000 7fe557cf5700 1 -- 172.21.15.78:0/2452574098 wait complete. 2023-10-10T17:41:49.155 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:41:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:48 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1095172666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:48 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1428827015' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:41:49.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:48 smithi190 ceph-mon[101220]: pgmap v297: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 10 B/s, 0 objects/s recovering 2023-10-10T17:41:49.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:48 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1778556304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:41:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 -- 172.21.15.78:0/413440682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d4100d90 msgr2=0x7f49d4107650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 --2- 172.21.15.78:0/413440682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d4100d90 0x7f49d4107650 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00b0a0 tx=0x7f49cc00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 -- 172.21.15.78:0/413440682 shutdown_connections 2023-10-10T17:41:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 --2- 172.21.15.78:0/413440682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d40ff210 0x7f49d40ff620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 --2- 172.21.15.78:0/413440682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d4100d90 0x7f49d4107650 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 --2- 172.21.15.78:0/413440682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4100400 0x7f49d4100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 -- 172.21.15.78:0/413440682 >> 172.21.15.78:0/413440682 conn(0x7f49d40fa810 msgr2=0x7f49d40fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 -- 172.21.15.78:0/413440682 shutdown_connections 2023-10-10T17:41:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 -- 172.21.15.78:0/413440682 wait complete. 2023-10-10T17:41:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 Processor -- start 2023-10-10T17:41:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.277+0000 7f49d9fea700 1 -- start start 2023-10-10T17:41:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d40ff210 0x7f49d4196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d4100400 0x7f49d4196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4100d90 0x7f49d419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d419dbd0 con 0x7f49d40ff210 2023-10-10T17:41:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d419dd10 con 0x7f49d4100d90 2023-10-10T17:41:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d419dea0 con 0x7f49d4100400 2023-10-10T17:41:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d40ff210 0x7f49d4196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d40ff210 0x7f49d4196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43172/0 (socket says 172.21.15.78:43172) 2023-10-10T17:41:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 -- 172.21.15.78:0/1693814536 learned_addr learned my addr 172.21.15.78:0/1693814536 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d2ffd700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d4100400 0x7f49d4196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d3fff700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4100d90 0x7f49d419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 -- 172.21.15.78:0/1693814536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d4100400 msgr2=0x7f49d4196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d4100400 0x7f49d4196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 -- 172.21.15.78:0/1693814536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4100d90 msgr2=0x7f49d419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4100d90 0x7f49d419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 -- 172.21.15.78:0/1693814536 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49cc009d00 con 0x7f49d40ff210 2023-10-10T17:41:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d37fe700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d40ff210 0x7f49d4196050 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c990 tx=0x7f49c800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d0ff9700 1 -- 172.21.15.78:0/1693814536 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8007b00 con 0x7f49d40ff210 2023-10-10T17:41:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d0ff9700 1 -- 172.21.15.78:0/1693814536 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49c8007c60 con 0x7f49d40ff210 2023-10-10T17:41:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d0ff9700 1 -- 172.21.15.78:0/1693814536 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8018660 con 0x7f49d40ff210 2023-10-10T17:41:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d410ab80 con 0x7f49d40ff210 2023-10-10T17:41:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.278+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d410a9f0 con 0x7f49d40ff210 2023-10-10T17:41:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.279+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d419e5f0 con 0x7f49d40ff210 2023-10-10T17:41:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.280+0000 7f49d0ff9700 1 -- 172.21.15.78:0/1693814536 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f49c800f450 con 0x7f49d40ff210 2023-10-10T17:41:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.281+0000 7f49d0ff9700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49bc0a11d0 0x7f49bc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.281+0000 7f49d2ffd700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49bc0a11d0 0x7f49bc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.281+0000 7f49d0ff9700 1 -- 172.21.15.78:0/1693814536 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f49c80ccb10 con 0x7f49d40ff210 2023-10-10T17:41:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.281+0000 7f49d2ffd700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49bc0a11d0 0x7f49bc0a3680 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f49c4005950 tx=0x7f49c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.285+0000 7f49d0ff9700 1 -- 172.21.15.78:0/1693814536 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f49c8095f50 con 0x7f49d40ff210 2023-10-10T17:41:49.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.519+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49d419eba0 con 0x7f49d40ff210 2023-10-10T17:41:49.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.520+0000 7f49d0ff9700 1 -- 172.21.15.78:0/1693814536 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7f49c8099570 con 0x7f49d40ff210 2023-10-10T17:41:49.521 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:49.521 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:41:49.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49bc0a11d0 msgr2=0x7f49bc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49bc0a11d0 0x7f49bc0a3680 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f49c4005950 tx=0x7f49c400b410 comp rx=0 tx=0).stop 2023-10-10T17:41:49.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d40ff210 msgr2=0x7f49d4196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d40ff210 0x7f49d4196050 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c990 tx=0x7f49c800cca0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 shutdown_connections 2023-10-10T17:41:49.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f49bc0a11d0 0x7f49bc0a3680 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d4100d90 0x7f49d419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d40ff210 0x7f49d4196050 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 --2- 172.21.15.78:0/1693814536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d4100400 0x7f49d4196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 >> 172.21.15.78:0/1693814536 conn(0x7f49d40fa810 msgr2=0x7f49d4069050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:49.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 shutdown_connections 2023-10-10T17:41:49.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.522+0000 7f49d9fea700 1 -- 172.21.15.78:0/1693814536 wait complete. 2023-10-10T17:41:49.525 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:41:49.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:41:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- 172.21.15.78:0/4169404062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 msgr2=0x7fb8ec100870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 --2- 172.21.15.78:0/4169404062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec100870 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc009990 tx=0x7fb8dc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- 172.21.15.78:0/4169404062 shutdown_connections 2023-10-10T17:41:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 --2- 172.21.15.78:0/4169404062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8ec0ff170 0x7fb8ec0ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 --2- 172.21.15.78:0/4169404062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8ec100db0 0x7fb8ec107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 --2- 172.21.15.78:0/4169404062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec100870 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- 172.21.15.78:0/4169404062 >> 172.21.15.78:0/4169404062 conn(0x7fb8ec0fa750 msgr2=0x7fb8ec0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- 172.21.15.78:0/4169404062 shutdown_connections 2023-10-10T17:41:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- 172.21.15.78:0/4169404062 wait complete. 2023-10-10T17:41:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 Processor -- start 2023-10-10T17:41:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- start start 2023-10-10T17:41:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8ec0ff170 0x7fb8ec194f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec195440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8ec100db0 0x7fb8ec19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec19ca80 con 0x7fb8ec100400 2023-10-10T17:41:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec19cbc0 con 0x7fb8ec100db0 2023-10-10T17:41:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.656+0000 7fb8f4099700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ec19cd50 con 0x7fb8ec0ff170 2023-10-10T17:41:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec195440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec195440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43188/0 (socket says 172.21.15.78:43188) 2023-10-10T17:41:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 -- 172.21.15.78:0/2828518905 learned_addr learned my addr 172.21.15.78:0/2828518905 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1e35700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8ec0ff170 0x7fb8ec194f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f2636700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8ec100db0 0x7fb8ec19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 -- 172.21.15.78:0/2828518905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8ec0ff170 msgr2=0x7fb8ec194f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8ec0ff170 0x7fb8ec194f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 -- 172.21.15.78:0/2828518905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8ec100db0 msgr2=0x7fb8ec19c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8ec100db0 0x7fb8ec19c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 -- 172.21.15.78:0/2828518905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc0095f0 con 0x7fb8ec100400 2023-10-10T17:41:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f1634700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec195440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc000c00 tx=0x7fb8dc00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8e2ffd700 1 -- 172.21.15.78:0/2828518905 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc01d070 con 0x7fb8ec100400 2023-10-10T17:41:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8e2ffd700 1 -- 172.21.15.78:0/2828518905 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8dc00fb50 con 0x7fb8ec100400 2023-10-10T17:41:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ec10ab70 con 0x7fb8ec100400 2023-10-10T17:41:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8e2ffd700 1 -- 172.21.15.78:0/2828518905 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc017650 con 0x7fb8ec100400 2023-10-10T17:41:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.657+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ec10a9e0 con 0x7fb8ec100400 2023-10-10T17:41:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.658+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ec1067f0 con 0x7fb8ec100400 2023-10-10T17:41:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.659+0000 7fb8e2ffd700 1 -- 172.21.15.78:0/2828518905 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb8dc00fcc0 con 0x7fb8ec100400 2023-10-10T17:41:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.660+0000 7fb8e2ffd700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb8d80a1150 0x7fb8d80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.660+0000 7fb8f1e35700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb8d80a1150 0x7fb8d80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.660+0000 7fb8e2ffd700 1 -- 172.21.15.78:0/2828518905 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb8dc0ce290 con 0x7fb8ec100400 2023-10-10T17:41:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.660+0000 7fb8f1e35700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb8d80a1150 0x7fb8d80a3600 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8005950 tx=0x7fb8e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.663+0000 7fb8e2ffd700 1 -- 172.21.15.78:0/2828518905 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb8dc097760 con 0x7fb8ec100400 2023-10-10T17:41:49.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.860+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8ec02cfe0 con 0x7fb8ec100400 2023-10-10T17:41:49.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.861+0000 7fb8e2ffd700 1 -- 172.21.15.78:0/2828518905 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7fb8dc09ad80 con 0x7fb8ec100400 2023-10-10T17:41:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:49.863 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:41:49.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb8d80a1150 msgr2=0x7fb8d80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb8d80a1150 0x7fb8d80a3600 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb8e8005950 tx=0x7fb8e800b410 comp rx=0 tx=0).stop 2023-10-10T17:41:49.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 msgr2=0x7fb8ec195440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:49.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec195440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc000c00 tx=0x7fb8dc00f710 comp rx=0 tx=0).stop 2023-10-10T17:41:49.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 shutdown_connections 2023-10-10T17:41:49.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb8d80a1150 0x7fb8d80a3600 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8ec0ff170 0x7fb8ec194f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb8ec100db0 0x7fb8ec19c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 --2- 172.21.15.78:0/2828518905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8ec100400 0x7fb8ec195440 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:49.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 >> 172.21.15.78:0/2828518905 conn(0x7fb8ec0fa750 msgr2=0x7fb8ec103930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:49.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 shutdown_connections 2023-10-10T17:41:49.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:49.863+0000 7fb8f4099700 1 -- 172.21.15.78:0/2828518905 wait complete. 2023-10-10T17:41:49.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:49 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3277471626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:41:49.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:49 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2452574098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:41:49.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:49 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1693814536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:41:49.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:41:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:49 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3277471626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:41:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:49 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2452574098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:41:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:49 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1693814536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:50.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 -- 172.21.15.78:0/1439102961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d0101f90 msgr2=0x7fc1d0108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 --2- 172.21.15.78:0/1439102961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d0101f90 0x7fc1d0108850 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc1c800b0a0 tx=0x7fc1c800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 -- 172.21.15.78:0/1439102961 shutdown_connections 2023-10-10T17:41:50.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 --2- 172.21.15.78:0/1439102961 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d00fe590 0x7fc1d00fe9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 --2- 172.21.15.78:0/1439102961 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d0101f90 0x7fc1d0108850 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 --2- 172.21.15.78:0/1439102961 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc1d0101600 0x7fc1d0101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 -- 172.21.15.78:0/1439102961 >> 172.21.15.78:0/1439102961 conn(0x7fc1d00f9f30 msgr2=0x7fc1d00fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:50.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 -- 172.21.15.78:0/1439102961 shutdown_connections 2023-10-10T17:41:50.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.002+0000 7fc1d5422700 1 -- 172.21.15.78:0/1439102961 wait complete. 2023-10-10T17:41:50.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 Processor -- start 2023-10-10T17:41:50.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 -- start start 2023-10-10T17:41:50.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00fe590 0x7fc1d0196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d0101600 0x7fc1d01965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc1d0101f90 0x7fc1d019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d019dbf0 con 0x7fc1d00fe590 2023-10-10T17:41:50.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d019dd30 con 0x7fc1d0101f90 2023-10-10T17:41:50.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1d5422700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d019dec0 con 0x7fc1d0101600 2023-10-10T17:41:50.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00fe590 0x7fc1d0196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00fe590 0x7fc1d0196070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43200/0 (socket says 172.21.15.78:43200) 2023-10-10T17:41:50.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ceffd700 1 -- 172.21.15.78:0/2114411133 learned_addr learned my addr 172.21.15.78:0/2114411133 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:50.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1cf7fe700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc1d0101f90 0x7fc1d019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ceffd700 1 -- 172.21.15.78:0/2114411133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d0101600 msgr2=0x7fc1d01965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ce7fc700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d0101600 0x7fc1d01965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ceffd700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d0101600 0x7fc1d01965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ceffd700 1 -- 172.21.15.78:0/2114411133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc1d0101f90 msgr2=0x7fc1d019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.003+0000 7fc1ceffd700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc1d0101f90 0x7fc1d019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1ceffd700 1 -- 172.21.15.78:0/2114411133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8009d00 con 0x7fc1d00fe590 2023-10-10T17:41:50.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1cf7fe700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc1d0101f90 0x7fc1d019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:50.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1ce7fc700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d0101600 0x7fc1d01965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:50.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1ceffd700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00fe590 0x7fc1d0196070 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c990 tx=0x7fc1c400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:50.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1b7fff700 1 -- 172.21.15.78:0/2114411133 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4007b00 con 0x7fc1d00fe590 2023-10-10T17:41:50.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d010bc30 con 0x7fc1d00fe590 2023-10-10T17:41:50.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1b7fff700 1 -- 172.21.15.78:0/2114411133 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc1c4007c60 con 0x7fc1d00fe590 2023-10-10T17:41:50.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1b7fff700 1 -- 172.21.15.78:0/2114411133 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c4018660 con 0x7fc1d00fe590 2023-10-10T17:41:50.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.004+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d010baa0 con 0x7fc1d00fe590 2023-10-10T17:41:50.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.005+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d019e610 con 0x7fc1d00fe590 2023-10-10T17:41:50.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.006+0000 7fc1b7fff700 1 -- 172.21.15.78:0/2114411133 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc1c400f450 con 0x7fc1d00fe590 2023-10-10T17:41:50.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.006+0000 7fc1b7fff700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc1b80a1150 0x7fc1b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.006+0000 7fc1ce7fc700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc1b80a1150 0x7fc1b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.006+0000 7fc1b7fff700 1 -- 172.21.15.78:0/2114411133 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc1c40ccc40 con 0x7fc1d00fe590 2023-10-10T17:41:50.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.007+0000 7fc1ce7fc700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc1b80a1150 0x7fc1b80a3600 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0005fd0 tx=0x7fc1c0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:50.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.011+0000 7fc1b7fff700 1 -- 172.21.15.78:0/2114411133 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc1c4096080 con 0x7fc1d00fe590 2023-10-10T17:41:50.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:49 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3277471626' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:41:50.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:49 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2452574098' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:41:50.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:49 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1693814536' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.199+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1d0107220 con 0x7fc1d00fe590 2023-10-10T17:41:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.200+0000 7fc1b7fff700 1 -- 172.21.15.78:0/2114411133 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1c40996a0 con 0x7fc1d00fe590 2023-10-10T17:41:50.201 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:41:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc1b80a1150 msgr2=0x7fc1b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc1b80a1150 0x7fc1b80a3600 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0005fd0 tx=0x7fc1c0005ee0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00fe590 msgr2=0x7fc1d0196070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00fe590 0x7fc1d0196070 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc1c400c990 tx=0x7fc1c400cd50 comp rx=0 tx=0).stop 2023-10-10T17:41:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 shutdown_connections 2023-10-10T17:41:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc1b80a1150 0x7fc1b80a3600 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc1d0101f90 0x7fc1d019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1d00fe590 0x7fc1d0196070 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 --2- 172.21.15.78:0/2114411133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc1d0101600 0x7fc1d01965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 >> 172.21.15.78:0/2114411133 conn(0x7fc1d00f9f30 msgr2=0x7fc1d00fc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:50.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 shutdown_connections 2023-10-10T17:41:50.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.202+0000 7fc1d5422700 1 -- 172.21.15.78:0/2114411133 wait complete. 2023-10-10T17:41:50.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:41:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 -- 172.21.15.78:0/3357573737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c1004a0 msgr2=0x7f3a6c100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 --2- 172.21.15.78:0/3357573737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c1004a0 0x7f3a6c100910 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3a60009990 tx=0x7f3a60009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 -- 172.21.15.78:0/3357573737 shutdown_connections 2023-10-10T17:41:50.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 --2- 172.21.15.78:0/3357573737 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a6c100e50 0x7f3a6c1076f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 --2- 172.21.15.78:0/3357573737 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a6c0ff210 0x7f3a6c0ff620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 --2- 172.21.15.78:0/3357573737 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c1004a0 0x7f3a6c100910 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 -- 172.21.15.78:0/3357573737 >> 172.21.15.78:0/3357573737 conn(0x7f3a6c0fa830 msgr2=0x7f3a6c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 -- 172.21.15.78:0/3357573737 shutdown_connections 2023-10-10T17:41:50.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.339+0000 7f3a738f9700 1 -- 172.21.15.78:0/3357573737 wait complete. 2023-10-10T17:41:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 Processor -- start 2023-10-10T17:41:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 -- start start 2023-10-10T17:41:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c0ff210 0x7f3a6c103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a6c1004a0 0x7f3a6c101880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a6c100e50 0x7f3a6c101dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c102560 con 0x7f3a6c0ff210 2023-10-10T17:41:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c1026d0 con 0x7f3a6c1004a0 2023-10-10T17:41:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a738f9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6c1a00f0 con 0x7f3a6c100e50 2023-10-10T17:41:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c0ff210 0x7f3a6c103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c0ff210 0x7f3a6c103240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43214/0 (socket says 172.21.15.78:43214) 2023-10-10T17:41:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 -- 172.21.15.78:0/2159051982 learned_addr learned my addr 172.21.15.78:0/2159051982 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a70e94700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a6c1004a0 0x7f3a6c101880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71e96700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a6c100e50 0x7f3a6c101dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 -- 172.21.15.78:0/2159051982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a6c100e50 msgr2=0x7f3a6c101dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a6c100e50 0x7f3a6c101dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 -- 172.21.15.78:0/2159051982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a6c1004a0 msgr2=0x7f3a6c101880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a6c1004a0 0x7f3a6c101880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71695700 1 -- 172.21.15.78:0/2159051982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600095f0 con 0x7f3a6c0ff210 2023-10-10T17:41:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.340+0000 7f3a71e96700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a6c100e50 0x7f3a6c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:50.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.341+0000 7f3a71695700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c0ff210 0x7f3a6c103240 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800d950 tx=0x7f3a6800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.341+0000 7f3a5e7fc700 1 -- 172.21.15.78:0/2159051982 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a68009940 con 0x7f3a6c0ff210 2023-10-10T17:41:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.341+0000 7f3a5e7fc700 1 -- 172.21.15.78:0/2159051982 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3a68010460 con 0x7f3a6c0ff210 2023-10-10T17:41:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.341+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6c10ac10 con 0x7f3a6c0ff210 2023-10-10T17:41:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.341+0000 7f3a5e7fc700 1 -- 172.21.15.78:0/2159051982 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800f620 con 0x7f3a6c0ff210 2023-10-10T17:41:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.341+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a6c10aa80 con 0x7f3a6c0ff210 2023-10-10T17:41:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.342+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a6c1a07e0 con 0x7f3a6c0ff210 2023-10-10T17:41:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.343+0000 7f3a5e7fc700 1 -- 172.21.15.78:0/2159051982 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3a680108d0 con 0x7f3a6c0ff210 2023-10-10T17:41:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.343+0000 7f3a5e7fc700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3a580a11d0 0x7f3a580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.344+0000 7f3a5e7fc700 1 -- 172.21.15.78:0/2159051982 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3a680ccb10 con 0x7f3a6c0ff210 2023-10-10T17:41:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.344+0000 7f3a70e94700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3a580a11d0 0x7f3a580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.344+0000 7f3a70e94700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3a580a11d0 0x7f3a580a3680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000b5c0 tx=0x7f3a60005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.347+0000 7f3a5e7fc700 1 -- 172.21.15.78:0/2159051982 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3a68095ef0 con 0x7f3a6c0ff210 2023-10-10T17:41:50.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.534+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3a6c1a0e80 con 0x7f3a6c0ff210 2023-10-10T17:41:50.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.535+0000 7f3a5e7fc700 1 -- 172.21.15.78:0/2159051982 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3a68016020 con 0x7f3a6c0ff210 2023-10-10T17:41:50.536 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:41:50.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3a580a11d0 msgr2=0x7f3a580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3a580a11d0 0x7f3a580a3680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3a6000b5c0 tx=0x7f3a60005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c0ff210 msgr2=0x7f3a6c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c0ff210 0x7f3a6c103240 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800d950 tx=0x7f3a6800dc60 comp rx=0 tx=0).stop 2023-10-10T17:41:50.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 shutdown_connections 2023-10-10T17:41:50.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3a580a11d0 0x7f3a580a3680 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3a6c100e50 0x7f3a6c101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3a6c0ff210 0x7f3a6c103240 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 --2- 172.21.15.78:0/2159051982 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3a6c1004a0 0x7f3a6c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 >> 172.21.15.78:0/2159051982 conn(0x7f3a6c0fa830 msgr2=0x7f3a6c0691c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:50.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 shutdown_connections 2023-10-10T17:41:50.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.537+0000 7f3a738f9700 1 -- 172.21.15.78:0/2159051982 wait complete. 2023-10-10T17:41:50.550 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:41:50.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 -- 172.21.15.78:0/3133076759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101600 msgr2=0x7f26a8101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 --2- 172.21.15.78:0/3133076759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101600 0x7f26a8101a50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2698009990 tx=0x7f2698009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 -- 172.21.15.78:0/3133076759 shutdown_connections 2023-10-10T17:41:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 --2- 172.21.15.78:0/3133076759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a8101f90 0x7f26a8108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 --2- 172.21.15.78:0/3133076759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26a80fe590 0x7f26a80fe9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 --2- 172.21.15.78:0/3133076759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101600 0x7f26a8101a50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 -- 172.21.15.78:0/3133076759 >> 172.21.15.78:0/3133076759 conn(0x7f26a80f9f30 msgr2=0x7f26a80fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 -- 172.21.15.78:0/3133076759 shutdown_connections 2023-10-10T17:41:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 -- 172.21.15.78:0/3133076759 wait complete. 2023-10-10T17:41:50.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.672+0000 7f26adbca700 1 Processor -- start 2023-10-10T17:41:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 -- start start 2023-10-10T17:41:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a80fe590 0x7f26a8196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26a8101600 0x7f26a8196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101f90 0x7f26a819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a819dbb0 con 0x7f26a8101f90 2023-10-10T17:41:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a819dcf0 con 0x7f26a8101600 2023-10-10T17:41:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a819de80 con 0x7f26a80fe590 2023-10-10T17:41:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101f90 0x7f26a819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101f90 0x7f26a819d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43232/0 (socket says 172.21.15.78:43232) 2023-10-10T17:41:50.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 -- 172.21.15.78:0/2837521932 learned_addr learned my addr 172.21.15.78:0/2837521932 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:50.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a77fe700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a80fe590 0x7f26a8196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a6ffd700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26a8101600 0x7f26a8196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 -- 172.21.15.78:0/2837521932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a80fe590 msgr2=0x7f26a8196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a80fe590 0x7f26a8196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 -- 172.21.15.78:0/2837521932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26a8101600 msgr2=0x7f26a8196570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26a8101600 0x7f26a8196570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 -- 172.21.15.78:0/2837521932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26980095f0 con 0x7f26a8101f90 2023-10-10T17:41:50.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a7fff700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101f90 0x7f26a819d5e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f26a0007ee0 tx=0x7f26a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:50.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a4ff9700 1 -- 172.21.15.78:0/2837521932 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0017070 con 0x7f26a8101f90 2023-10-10T17:41:50.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a810bc50 con 0x7f26a8101f90 2023-10-10T17:41:50.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a4ff9700 1 -- 172.21.15.78:0/2837521932 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f26a000f040 con 0x7f26a8101f90 2023-10-10T17:41:50.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26a4ff9700 1 -- 172.21.15.78:0/2837521932 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0005580 con 0x7f26a8101f90 2023-10-10T17:41:50.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.673+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a810bac0 con 0x7f26a8101f90 2023-10-10T17:41:50.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.674+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a8107390 con 0x7f26a8101f90 2023-10-10T17:41:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.675+0000 7f26a4ff9700 1 -- 172.21.15.78:0/2837521932 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f26a000db20 con 0x7f26a8101f90 2023-10-10T17:41:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.676+0000 7f26a4ff9700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f26900a1150 0x7f26900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.676+0000 7f26a4ff9700 1 -- 172.21.15.78:0/2837521932 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f26a00cc2b0 con 0x7f26a8101f90 2023-10-10T17:41:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.676+0000 7f26a77fe700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f26900a1150 0x7f26900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.677+0000 7f26a77fe700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f26900a1150 0x7f26900a3600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f26a8067b10 tx=0x7f269c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:50.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.680+0000 7f26a4ff9700 1 -- 172.21.15.78:0/2837521932 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f26a0095790 con 0x7f26a8101f90 2023-10-10T17:41:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.867+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f26a8061700 con 0x7f26a8101f90 2023-10-10T17:41:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.867+0000 7f26a4ff9700 1 -- 172.21.15.78:0/2837521932 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f26a0098db0 con 0x7f26a8101f90 2023-10-10T17:41:50.868 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:41:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f26900a1150 msgr2=0x7f26900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f26900a1150 0x7f26900a3600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f26a8067b10 tx=0x7f269c009260 comp rx=0 tx=0).stop 2023-10-10T17:41:50.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101f90 msgr2=0x7f26a819d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:50.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101f90 0x7f26a819d5e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f26a0007ee0 tx=0x7f26a000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 shutdown_connections 2023-10-10T17:41:50.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f26900a1150 0x7f26900a3600 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f26a80fe590 0x7f26a8196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26a8101f90 0x7f26a819d5e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 --2- 172.21.15.78:0/2837521932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f26a8101600 0x7f26a8196570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:50.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.869+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 >> 172.21.15.78:0/2837521932 conn(0x7f26a80f9f30 msgr2=0x7f26a80fc210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:50.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.870+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 shutdown_connections 2023-10-10T17:41:50.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:50.870+0000 7f26adbca700 1 -- 172.21.15.78:0/2837521932 wait complete. 2023-10-10T17:41:50.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:50 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2828518905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:50.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:50 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2114411133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:41:50.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:50 smithi078 ceph-mon[99494]: pgmap v298: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2023-10-10T17:41:50.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:50 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2159051982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:41:50.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:41:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:50 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2828518905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:50 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2114411133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:41:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:50 smithi157 ceph-mon[100896]: pgmap v298: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2023-10-10T17:41:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:50 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2159051982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:41:51.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 -- 172.21.15.78:0/4056836435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 msgr2=0x7f17f40ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 --2- 172.21.15.78:0/4056836435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f40ff600 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f17e4009990 tx=0x7f17e4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 -- 172.21.15.78:0/4056836435 shutdown_connections 2023-10-10T17:41:51.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 --2- 172.21.15.78:0/4056836435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f4100d70 0x7f17f4107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 --2- 172.21.15.78:0/4056836435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f40ff600 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 --2- 172.21.15.78:0/4056836435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f41003e0 0x7f17f4100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 -- 172.21.15.78:0/4056836435 >> 172.21.15.78:0/4056836435 conn(0x7f17f40fa7d0 msgr2=0x7f17f40fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:51.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 -- 172.21.15.78:0/4056836435 shutdown_connections 2023-10-10T17:41:51.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 -- 172.21.15.78:0/4056836435 wait complete. 2023-10-10T17:41:51.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 Processor -- start 2023-10-10T17:41:51.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.007+0000 7f17fbb40700 1 -- start start 2023-10-10T17:41:51.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f41950d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f41003e0 0x7f17f4195610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f4100d70 0x7f17f419c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f419cc50 con 0x7f17f40ff1f0 2023-10-10T17:41:51.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f419cd90 con 0x7f17f4100d70 2023-10-10T17:41:51.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f419cf20 con 0x7f17f41003e0 2023-10-10T17:41:51.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f41950d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f41950d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43242/0 (socket says 172.21.15.78:43242) 2023-10-10T17:41:51.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 -- 172.21.15.78:0/2730499093 learned_addr learned my addr 172.21.15.78:0/2730499093 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fa0dd700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f4100d70 0x7f17f419c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f90db700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f41003e0 0x7f17f4195610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 -- 172.21.15.78:0/2730499093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f41003e0 msgr2=0x7f17f4195610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f41003e0 0x7f17f4195610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 -- 172.21.15.78:0/2730499093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f4100d70 msgr2=0x7f17f419c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f4100d70 0x7f17f419c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 -- 172.21.15.78:0/2730499093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40095f0 con 0x7f17f40ff1f0 2023-10-10T17:41:51.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17f98dc700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f41950d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f17e4004730 tx=0x7f17e4009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:51.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17eaffd700 1 -- 172.21.15.78:0/2730499093 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e401d070 con 0x7f17f40ff1f0 2023-10-10T17:41:51.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17eaffd700 1 -- 172.21.15.78:0/2730499093 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17e4004cd0 con 0x7f17f40ff1f0 2023-10-10T17:41:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f410ac30 con 0x7f17f40ff1f0 2023-10-10T17:41:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17eaffd700 1 -- 172.21.15.78:0/2730499093 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e4004230 con 0x7f17f40ff1f0 2023-10-10T17:41:51.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.008+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f410aaa0 con 0x7f17f40ff1f0 2023-10-10T17:41:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.009+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f41067d0 con 0x7f17f40ff1f0 2023-10-10T17:41:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.010+0000 7f17eaffd700 1 -- 172.21.15.78:0/2730499093 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f17e4003b90 con 0x7f17f40ff1f0 2023-10-10T17:41:51.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.011+0000 7f17eaffd700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17e00a1150 0x7f17e00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.011+0000 7f17eaffd700 1 -- 172.21.15.78:0/2730499093 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f17e40cdfa0 con 0x7f17f40ff1f0 2023-10-10T17:41:51.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.011+0000 7f17f90db700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17e00a1150 0x7f17e00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.011+0000 7f17f90db700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17e00a1150 0x7f17e00a3600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f17f0005950 tx=0x7f17f00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:51.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.014+0000 7f17eaffd700 1 -- 172.21.15.78:0/2730499093 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f17e4097470 con 0x7f17f40ff1f0 2023-10-10T17:41:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:50 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2828518905' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:50 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2114411133' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:41:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:50 smithi190 ceph-mon[101220]: pgmap v298: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 9 B/s, 0 objects/s recovering 2023-10-10T17:41:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:50 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2159051982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:41:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.203+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f17f4061700 con 0x7f17f40ff1f0 2023-10-10T17:41:51.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.204+0000 7f17eaffd700 1 -- 172.21.15.78:0/2730499093 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f17e409aa90 con 0x7f17f40ff1f0 2023-10-10T17:41:51.205 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:41:51.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17e00a1150 msgr2=0x7f17e00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17e00a1150 0x7f17e00a3600 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f17f0005950 tx=0x7f17f00058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 msgr2=0x7f17f41950d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f41950d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f17e4004730 tx=0x7f17e4009510 comp rx=0 tx=0).stop 2023-10-10T17:41:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 shutdown_connections 2023-10-10T17:41:51.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f17e00a1150 0x7f17e00a3600 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17f4100d70 0x7f17f419c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17f40ff1f0 0x7f17f41950d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 --2- 172.21.15.78:0/2730499093 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17f41003e0 0x7f17f4195610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 >> 172.21.15.78:0/2730499093 conn(0x7f17f40fa7d0 msgr2=0x7f17f4103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:51.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 shutdown_connections 2023-10-10T17:41:51.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.206+0000 7f17fbb40700 1 -- 172.21.15.78:0/2730499093 wait complete. 2023-10-10T17:41:51.219 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:41:51.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- 172.21.15.78:0/3731864351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c0ffb30 msgr2=0x7f774c107700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 --2- 172.21.15.78:0/3731864351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c0ffb30 0x7f774c107700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f774400b0a0 tx=0x7f774400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- 172.21.15.78:0/3731864351 shutdown_connections 2023-10-10T17:41:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 --2- 172.21.15.78:0/3731864351 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f774c100760 0x7f774c100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 --2- 172.21.15.78:0/3731864351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c0ffb30 0x7f774c107700 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 --2- 172.21.15.78:0/3731864351 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f774c1010b0 0x7f774c0ff4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- 172.21.15.78:0/3731864351 >> 172.21.15.78:0/3731864351 conn(0x7f774c0f9ee0 msgr2=0x7f774c0fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- 172.21.15.78:0/3731864351 shutdown_connections 2023-10-10T17:41:51.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- 172.21.15.78:0/3731864351 wait complete. 2023-10-10T17:41:51.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 Processor -- start 2023-10-10T17:41:51.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- start start 2023-10-10T17:41:51.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f774c0ffb30 0x7f774c1982b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c100760 0x7f774c1987f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f774c1010b0 0x7f774c19f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c19fe30 con 0x7f774c100760 2023-10-10T17:41:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c19ff70 con 0x7f774c0ffb30 2023-10-10T17:41:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f7753382700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c1a0100 con 0x7f774c1010b0 2023-10-10T17:41:51.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.340+0000 7f775091d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c100760 0x7f774c1987f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c100760 0x7f774c1987f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43258/0 (socket says 172.21.15.78:43258) 2023-10-10T17:41:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 -- 172.21.15.78:0/3043177415 learned_addr learned my addr 172.21.15.78:0/3043177415 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775191f700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f774c1010b0 0x7f774c19f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775111e700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f774c0ffb30 0x7f774c1982b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 -- 172.21.15.78:0/3043177415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f774c1010b0 msgr2=0x7f774c19f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f774c1010b0 0x7f774c19f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 -- 172.21.15.78:0/3043177415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f774c0ffb30 msgr2=0x7f774c1982b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f774c0ffb30 0x7f774c1982b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 -- 172.21.15.78:0/3043177415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7744009d00 con 0x7f774c100760 2023-10-10T17:41:51.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f775091d700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c100760 0x7f774c1987f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f773c00e970 tx=0x7f773c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:51.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f77427fc700 1 -- 172.21.15.78:0/3043177415 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c009790 con 0x7f774c100760 2023-10-10T17:41:51.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f77427fc700 1 -- 172.21.15.78:0/3043177415 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f773c004cf0 con 0x7f774c100760 2023-10-10T17:41:51.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774c10ab30 con 0x7f774c100760 2023-10-10T17:41:51.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f77427fc700 1 -- 172.21.15.78:0/3043177415 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c005680 con 0x7f774c100760 2023-10-10T17:41:51.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.341+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774c10a9a0 con 0x7f774c100760 2023-10-10T17:41:51.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.342+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f774c1a08a0 con 0x7f774c100760 2023-10-10T17:41:51.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.343+0000 7f77427fc700 1 -- 172.21.15.78:0/3043177415 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f773c004750 con 0x7f774c100760 2023-10-10T17:41:51.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.343+0000 7f77427fc700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77380a11d0 0x7f77380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.344+0000 7f775111e700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77380a11d0 0x7f77380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.344+0000 7f77427fc700 1 -- 172.21.15.78:0/3043177415 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f773c014070 con 0x7f774c100760 2023-10-10T17:41:51.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.344+0000 7f775111e700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77380a11d0 0x7f77380a3680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7748005950 tx=0x7f77480058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:51.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.348+0000 7f77427fc700 1 -- 172.21.15.78:0/3043177415 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f773c0962c0 con 0x7f774c100760 2023-10-10T17:41:51.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.581+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f774c1a0e50 con 0x7f774c100760 2023-10-10T17:41:51.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.582+0000 7f77427fc700 1 -- 172.21.15.78:0/3043177415 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7f773c0998e0 con 0x7f774c100760 2023-10-10T17:41:51.583 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:51.583 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:41:51.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77380a11d0 msgr2=0x7f77380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77380a11d0 0x7f77380a3680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7748005950 tx=0x7f77480058e0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c100760 msgr2=0x7f774c1987f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c100760 0x7f774c1987f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f773c00e970 tx=0x7f773c00ed30 comp rx=0 tx=0).stop 2023-10-10T17:41:51.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 shutdown_connections 2023-10-10T17:41:51.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f77380a11d0 0x7f77380a3680 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f774c1010b0 0x7f774c19f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f774c0ffb30 0x7f774c1982b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 --2- 172.21.15.78:0/3043177415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f774c100760 0x7f774c1987f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 >> 172.21.15.78:0/3043177415 conn(0x7f774c0f9ee0 msgr2=0x7f774c0fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:51.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 shutdown_connections 2023-10-10T17:41:51.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.584+0000 7f7753382700 1 -- 172.21.15.78:0/3043177415 wait complete. 2023-10-10T17:41:51.588 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:41:51.597 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:41:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 -- 172.21.15.78:0/2247417232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff588100730 msgr2=0x7ff588100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 --2- 172.21.15.78:0/2247417232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff588100730 0x7ff588100b40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff578009990 tx=0x7ff578009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 -- 172.21.15.78:0/2247417232 shutdown_connections 2023-10-10T17:41:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 --2- 172.21.15.78:0/2247417232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5880ffdd0 0x7ff588107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 --2- 172.21.15.78:0/2247417232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff588100730 0x7ff588100b40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 --2- 172.21.15.78:0/2247417232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5880ff420 0x7ff5880ff890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 -- 172.21.15.78:0/2247417232 >> 172.21.15.78:0/2247417232 conn(0x7ff5880f9ee0 msgr2=0x7ff5880fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:51.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 -- 172.21.15.78:0/2247417232 shutdown_connections 2023-10-10T17:41:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.719+0000 7ff58f814700 1 -- 172.21.15.78:0/2247417232 wait complete. 2023-10-10T17:41:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 Processor -- start 2023-10-10T17:41:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 -- start start 2023-10-10T17:41:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5880ff420 0x7ff5880feea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5880ffdd0 0x7ff58819acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff588100730 0x7ff58819d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58819dae0 con 0x7ff5880ffdd0 2023-10-10T17:41:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58819dc50 con 0x7ff5880ff420 2023-10-10T17:41:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58f814700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58819de10 con 0x7ff588100730 2023-10-10T17:41:51.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5880ffdd0 0x7ff58819acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5880ffdd0 0x7ff58819acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43272/0 (socket says 172.21.15.78:43272) 2023-10-10T17:41:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 -- 172.21.15.78:0/864990279 learned_addr learned my addr 172.21.15.78:0/864990279 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58ddb1700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff588100730 0x7ff58819d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58d5b0700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5880ff420 0x7ff5880feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 -- 172.21.15.78:0/864990279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff588100730 msgr2=0x7ff58819d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff588100730 0x7ff58819d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 -- 172.21.15.78:0/864990279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5880ff420 msgr2=0x7ff5880feea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5880ff420 0x7ff5880feea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.720+0000 7ff58cdaf700 1 -- 172.21.15.78:0/864990279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5780095f0 con 0x7ff5880ffdd0 2023-10-10T17:41:51.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.721+0000 7ff58cdaf700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5880ffdd0 0x7ff58819acc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff58400d950 tx=0x7ff58400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:51.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.721+0000 7ff57e7fc700 1 -- 172.21.15.78:0/864990279 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584009940 con 0x7ff5880ffdd0 2023-10-10T17:41:51.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.721+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58810ab10 con 0x7ff5880ffdd0 2023-10-10T17:41:51.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.721+0000 7ff57e7fc700 1 -- 172.21.15.78:0/864990279 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff584010460 con 0x7ff5880ffdd0 2023-10-10T17:41:51.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.721+0000 7ff57e7fc700 1 -- 172.21.15.78:0/864990279 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58400f620 con 0x7ff5880ffdd0 2023-10-10T17:41:51.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.721+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58810a980 con 0x7ff5880ffdd0 2023-10-10T17:41:51.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.722+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5881061c0 con 0x7ff5880ffdd0 2023-10-10T17:41:51.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.723+0000 7ff57e7fc700 1 -- 172.21.15.78:0/864990279 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff5840108d0 con 0x7ff5880ffdd0 2023-10-10T17:41:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.723+0000 7ff57e7fc700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5740a1150 0x7ff5740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.724+0000 7ff58d5b0700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5740a1150 0x7ff5740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.724+0000 7ff57e7fc700 1 -- 172.21.15.78:0/864990279 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff5840ccab0 con 0x7ff5880ffdd0 2023-10-10T17:41:51.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.724+0000 7ff58d5b0700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5740a1150 0x7ff5740a3600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff578006010 tx=0x7ff57800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:51.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.727+0000 7ff57e7fc700 1 -- 172.21.15.78:0/864990279 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff584095ef0 con 0x7ff5880ffdd0 2023-10-10T17:41:51.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:51 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2837521932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:41:51.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:51 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2730499093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:41:51.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:51 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3043177415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:51.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.915+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff58802cfe0 con 0x7ff5880ffdd0 2023-10-10T17:41:51.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.917+0000 7ff57e7fc700 1 -- 172.21.15.78:0/864990279 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7ff584016090 con 0x7ff5880ffdd0 2023-10-10T17:41:51.918 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:51.918 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:41:51.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5740a1150 msgr2=0x7ff5740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5740a1150 0x7ff5740a3600 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff578006010 tx=0x7ff57800b260 comp rx=0 tx=0).stop 2023-10-10T17:41:51.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5880ffdd0 msgr2=0x7ff58819acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:51.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5880ffdd0 0x7ff58819acc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7ff58400d950 tx=0x7ff58400dc60 comp rx=0 tx=0).stop 2023-10-10T17:41:51.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 shutdown_connections 2023-10-10T17:41:51.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5740a1150 0x7ff5740a3600 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff588100730 0x7ff58819d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5880ff420 0x7ff5880feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 --2- 172.21.15.78:0/864990279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5880ffdd0 0x7ff58819acc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 >> 172.21.15.78:0/864990279 conn(0x7ff5880f9ee0 msgr2=0x7ff5880fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 shutdown_connections 2023-10-10T17:41:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:51.919+0000 7ff58f814700 1 -- 172.21.15.78:0/864990279 wait complete. 2023-10-10T17:41:51.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:41:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:51 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2837521932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:41:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:51 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2730499093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:41:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:51 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3043177415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:52.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 -- 172.21.15.78:0/859799471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc0ff1f0 msgr2=0x7faedc0ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 --2- 172.21.15.78:0/859799471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc0ff1f0 0x7faedc0ff600 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faecc009990 tx=0x7faecc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 -- 172.21.15.78:0/859799471 shutdown_connections 2023-10-10T17:41:52.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 --2- 172.21.15.78:0/859799471 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faedc100d70 0x7faedc107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 --2- 172.21.15.78:0/859799471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc0ff1f0 0x7faedc0ff600 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 --2- 172.21.15.78:0/859799471 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc1003e0 0x7faedc100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 -- 172.21.15.78:0/859799471 >> 172.21.15.78:0/859799471 conn(0x7faedc0fa7f0 msgr2=0x7faedc0fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.052+0000 7faee4377700 1 -- 172.21.15.78:0/859799471 shutdown_connections 2023-10-10T17:41:52.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 -- 172.21.15.78:0/859799471 wait complete. 2023-10-10T17:41:52.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 Processor -- start 2023-10-10T17:41:52.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 -- start start 2023-10-10T17:41:52.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc0ff1f0 0x7faedc196080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faedc1003e0 0x7faedc1965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc100d70 0x7faedc19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc19dc00 con 0x7faedc100d70 2023-10-10T17:41:52.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc19dd40 con 0x7faedc0ff1f0 2023-10-10T17:41:52.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee4377700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc19ded0 con 0x7faedc1003e0 2023-10-10T17:41:52.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee2914700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc100d70 0x7faedc19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee2914700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc100d70 0x7faedc19d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43288/0 (socket says 172.21.15.78:43288) 2023-10-10T17:41:52.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee2914700 1 -- 172.21.15.78:0/1711251302 learned_addr learned my addr 172.21.15.78:0/1711251302 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.053+0000 7faee2113700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc0ff1f0 0x7faedc196080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee2914700 1 -- 172.21.15.78:0/1711251302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faedc1003e0 msgr2=0x7faedc1965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee2914700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faedc1003e0 0x7faedc1965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee2914700 1 -- 172.21.15.78:0/1711251302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc0ff1f0 msgr2=0x7faedc196080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee2914700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc0ff1f0 0x7faedc196080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee2914700 1 -- 172.21.15.78:0/1711251302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc0095f0 con 0x7faedc100d70 2023-10-10T17:41:52.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee2914700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc100d70 0x7faedc19d630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faed4007ee0 tx=0x7faed400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faed37fe700 1 -- 172.21.15.78:0/1711251302 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4017070 con 0x7faedc100d70 2023-10-10T17:41:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faed37fe700 1 -- 172.21.15.78:0/1711251302 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faed400f040 con 0x7faedc100d70 2023-10-10T17:41:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc10bd20 con 0x7faedc100d70 2023-10-10T17:41:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faed37fe700 1 -- 172.21.15.78:0/1711251302 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4005740 con 0x7faedc100d70 2023-10-10T17:41:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.054+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc10bb90 con 0x7faedc100d70 2023-10-10T17:41:52.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.055+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc1067d0 con 0x7faedc100d70 2023-10-10T17:41:52.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.056+0000 7faed37fe700 1 -- 172.21.15.78:0/1711251302 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faed400dc60 con 0x7faedc100d70 2023-10-10T17:41:52.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.056+0000 7faed37fe700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faec80a1150 0x7faec80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.057+0000 7faee2113700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faec80a1150 0x7faec80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.057+0000 7faed37fe700 1 -- 172.21.15.78:0/1711251302 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faed40cca10 con 0x7faedc100d70 2023-10-10T17:41:52.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.057+0000 7faee2113700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faec80a1150 0x7faec80a3600 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faecc00b5c0 tx=0x7faecc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.060+0000 7faed37fe700 1 -- 172.21.15.78:0/1711251302 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faed4095e50 con 0x7faedc100d70 2023-10-10T17:41:52.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:51 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2837521932' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:41:52.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:51 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2730499093' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:41:52.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:51 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3043177415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:52.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.292+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faedc061700 con 0x7faedc100d70 2023-10-10T17:41:52.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.292+0000 7faed37fe700 1 -- 172.21.15.78:0/1711251302 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7faed4099470 con 0x7faedc100d70 2023-10-10T17:41:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:41:52.294 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:41:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faec80a1150 msgr2=0x7faec80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faec80a1150 0x7faec80a3600 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faecc00b5c0 tx=0x7faecc005fb0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc100d70 msgr2=0x7faedc19d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc100d70 0x7faedc19d630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faed4007ee0 tx=0x7faed400d3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 shutdown_connections 2023-10-10T17:41:52.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faec80a1150 0x7faec80a3600 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faedc100d70 0x7faedc19d630 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faedc0ff1f0 0x7faedc196080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.294+0000 7faee4377700 1 --2- 172.21.15.78:0/1711251302 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faedc1003e0 0x7faedc1965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.295+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 >> 172.21.15.78:0/1711251302 conn(0x7faedc0fa7f0 msgr2=0x7faedc103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.295+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 shutdown_connections 2023-10-10T17:41:52.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.295+0000 7faee4377700 1 -- 172.21.15.78:0/1711251302 wait complete. 2023-10-10T17:41:52.298 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:41:52.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-10-10T17:40:41.574125+0000', 'modified': '2023-10-10T17:41:01.590420+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24500, 'mds_1': 24490, 'mds_2': 24472}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.157:6835/2501992479', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 2501992479}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 2501992479}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6835/1269112994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 1269112994}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 1269112994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.157:6837/1666360072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6836', 'nonce': 1666360072}, {'type': 'v1', 'addr': '172.21.15.157:6837', 'nonce': 1666360072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.190:6837/661255149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 661255149}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 661255149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24490': {'gid': 24490, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.190:6839/486459111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 486459111}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 486459111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.157:6839/2554315829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 2554315829}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 2554315829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-10T17:41:52.308 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-10T17:41:52.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-10T17:41:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 -- 172.21.15.78:0/431051730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4180ff110 msgr2=0x7ff4180ff520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 --2- 172.21.15.78:0/431051730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4180ff110 0x7ff4180ff520 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff40c009990 tx=0x7ff40c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 -- 172.21.15.78:0/431051730 shutdown_connections 2023-10-10T17:41:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 --2- 172.21.15.78:0/431051730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418100c90 0x7ff418107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 --2- 172.21.15.78:0/431051730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4180ff110 0x7ff4180ff520 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 --2- 172.21.15.78:0/431051730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff418100300 0x7ff418100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 -- 172.21.15.78:0/431051730 >> 172.21.15.78:0/431051730 conn(0x7ff4180fa6f0 msgr2=0x7ff4180fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 -- 172.21.15.78:0/431051730 shutdown_connections 2023-10-10T17:41:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.430+0000 7ff41e17e700 1 -- 172.21.15.78:0/431051730 wait complete. 2023-10-10T17:41:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 Processor -- start 2023-10-10T17:41:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 -- start start 2023-10-10T17:41:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4180ff110 0x7ff418068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff418100300 0x7ff41819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418100c90 0x7ff41819d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41819db40 con 0x7ff418100300 2023-10-10T17:41:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41819dcb0 con 0x7ff4180ff110 2023-10-10T17:41:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff41e17e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41819de70 con 0x7ff418100c90 2023-10-10T17:41:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff418100300 0x7ff41819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff418100300 0x7ff41819ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43300/0 (socket says 172.21.15.78:43300) 2023-10-10T17:41:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 -- 172.21.15.78:0/3331498459 learned_addr learned my addr 172.21.15.78:0/3331498459 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff417fff700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418100c90 0x7ff41819d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff4177fe700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4180ff110 0x7ff418068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 -- 172.21.15.78:0/3331498459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418100c90 msgr2=0x7ff41819d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418100c90 0x7ff41819d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 -- 172.21.15.78:0/3331498459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4180ff110 msgr2=0x7ff418068960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4180ff110 0x7ff418068960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff416ffd700 1 -- 172.21.15.78:0/3331498459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c0095f0 con 0x7ff418100300 2023-10-10T17:41:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.431+0000 7ff4177fe700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4180ff110 0x7ff418068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.432+0000 7ff416ffd700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff418100300 0x7ff41819ad20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff40800d950 tx=0x7ff40800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.432+0000 7ff414ff9700 1 -- 172.21.15.78:0/3331498459 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff408009940 con 0x7ff418100300 2023-10-10T17:41:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.432+0000 7ff414ff9700 1 -- 172.21.15.78:0/3331498459 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff408010460 con 0x7ff418100300 2023-10-10T17:41:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.432+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41810aaf0 con 0x7ff418100300 2023-10-10T17:41:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.432+0000 7ff414ff9700 1 -- 172.21.15.78:0/3331498459 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800f620 con 0x7ff418100300 2023-10-10T17:41:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.432+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41810a960 con 0x7ff418100300 2023-10-10T17:41:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.434+0000 7ff414ff9700 1 -- 172.21.15.78:0/3331498459 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff4080108d0 con 0x7ff418100300 2023-10-10T17:41:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.435+0000 7ff414ff9700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff4000a1150 0x7ff4000a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.435+0000 7ff414ff9700 1 -- 172.21.15.78:0/3331498459 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff4080ccaf0 con 0x7ff418100300 2023-10-10T17:41:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.435+0000 7ff4177fe700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff4000a1150 0x7ff4000a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.435+0000 7ff4177fe700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff4000a1150 0x7ff4000a3600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff40c006010 tx=0x7ff40c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 Processor -- start 2023-10-10T17:41:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 -- start start 2023-10-10T17:41:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a3efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a3fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a40530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa3944010 con 0x561fa3a400c0 2023-10-10T17:41:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa3996850 con 0x561fa3a3f500 2023-10-10T17:41:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa398fce0 con 0x561fa3a3ebb0 2023-10-10T17:41:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a40530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a40530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43308/0 (socket says 172.21.15.78:43308) 2023-10-10T17:41:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 -- 172.21.15.78:0/3042569590 learned_addr learned my addr 172.21.15.78:0/3042569590 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff4177fe700 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a3efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 -- 172.21.15.78:0/3042569590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 msgr2=0x561fa3a3efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a3efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff416ffd700 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a3fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 -- 172.21.15.78:0/3042569590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 msgr2=0x561fa3a3fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a3fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff417fff700 1 -- 172.21.15.78:0/3042569590 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fa38115e0 con 0x561fa3a400c0 2023-10-10T17:41:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.436+0000 7ff4177fe700 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a3efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff417fff700 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a40530 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff41000a430 tx=0x7ff410009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c5152050c7b059d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff3fdffb700 1 -- 172.21.15.78:0/3042569590 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410010040 con 0x561fa3a400c0 2023-10-10T17:41:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff3fdffb700 1 -- 172.21.15.78:0/3042569590 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff410013a50 con 0x561fa3a400c0 2023-10-10T17:41:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff3fdffb700 1 -- 172.21.15.78:0/3042569590 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410014b20 con 0x561fa3a400c0 2023-10-10T17:41:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 -- 172.21.15.78:0/3042569590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 msgr2=0x561fa3a40530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a40530 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff41000a430 tx=0x7ff410009650 comp rx=0 tx=0).stop 2023-10-10T17:41:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 -- 172.21.15.78:0/3042569590 shutdown_connections 2023-10-10T17:41:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a3efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a40530 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/3042569590 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a3fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 -- 172.21.15.78:0/3042569590 >> 172.21.15.78:0/3042569590 conn(0x561fa3a34310 msgr2=0x561fa3a39620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 -- 172.21.15.78:0/3042569590 shutdown_connections 2023-10-10T17:41:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.437+0000 7ff42a30ab80 1 -- 172.21.15.78:0/3042569590 wait complete. 2023-10-10T17:41:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 -- start start 2023-10-10T17:41:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 Processor -- start 2023-10-10T17:41:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3ebb0 0x561fa3a48560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3f500 0x561fa3a48aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a400c0 0x561fa3a4fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa3931890 con 0x561fa3a3ebb0 2023-10-10T17:41:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa3931530 con 0x561fa3a400c0 2023-10-10T17:41:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa38138d0 con 0x561fa3a3f500 2023-10-10T17:41:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff42a30ab80 10 client.0 _ll_register_callbacks cb 0x561fa394f730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:41:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff4177fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3ebb0 0x561fa3a48560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff4177fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3ebb0 0x561fa3a48560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43310/0 (socket says 172.21.15.78:43310) 2023-10-10T17:41:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff4177fe700 1 -- 172.21.15.78:0/564949254 learned_addr learned my addr 172.21.15.78:0/564949254 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.438+0000 7ff416ffd700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3f500 0x561fa3a48aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff417fff700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a400c0 0x561fa3a4fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff4177fe700 1 -- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3f500 msgr2=0x561fa3a48aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff4177fe700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3f500 0x561fa3a48aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff4177fe700 1 -- 172.21.15.78:0/564949254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a400c0 msgr2=0x561fa3a4fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff4177fe700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a400c0 0x561fa3a4fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff4177fe700 1 -- 172.21.15.78:0/564949254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff410009c60 con 0x561fa3a3ebb0 2023-10-10T17:41:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff417fff700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a400c0 0x561fa3a4fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff4177fe700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3ebb0 0x561fa3a48560 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00bf60 tx=0x7ff40c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f100eb23229f89d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff3f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:41:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/564949254 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c0041e0 con 0x561fa3a3ebb0 2023-10-10T17:41:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/564949254 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff40c005050 con 0x561fa3a3ebb0 2023-10-10T17:41:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561fa3a31ad0 con 0x561fa3a3ebb0 2023-10-10T17:41:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/564949254 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff40c005250 con 0x561fa3a3ebb0 2023-10-10T17:41:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 10 client.15189 fetch_fsmap learned FSMap version 24 2023-10-10T17:41:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561fa3956140 con 0x561fa3a3ebb0 2023-10-10T17:41:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/564949254 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561fa3a31ad0 con 0x561fa3a3ebb0 2023-10-10T17:41:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/564949254 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff40c00f460 con 0x561fa3a3ebb0 2023-10-10T17:41:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 10 client.15189 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:41:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 10 client.15189 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-10T17:41:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 20 client.15189 populate_metadata read hostname 'smithi078' 2023-10-10T17:41:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3a53830 0x561fa3a52c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 4 client.15189 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:41:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.439+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561fa3952050 con 0x561fa3a53830 2023-10-10T17:41:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.440+0000 7ff416ffd700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3a53830 0x561fa3a52c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.440+0000 7ff3f6ffd700 10 client.15189 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:41:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.440+0000 7ff416ffd700 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3a53830 0x561fa3a52c30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ddac9479e3e0cd23 server_cookie=ecf5462601d1cd1 in_seq=0 out_seq=0 2023-10-10T17:41:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.441+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/564949254 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561fa3952050 con 0x561fa3a53830 2023-10-10T17:41:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.441+0000 7ff3f6ffd700 10 client.15189 handle_command_reply: tid=0 2023-10-10T17:41:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.441+0000 7ff42a30ab80 1 client.15189 shutdown 2023-10-10T17:41:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.441+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3ebb0 msgr2=0x561fa3a48560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.441+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3ebb0 0x561fa3a48560 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00bf60 tx=0x7ff40c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 shutdown_connections 2023-10-10T17:41:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a400c0 0x561fa3a4fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3ebb0 0x561fa3a48560 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3a53830 0x561fa3a52c30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/564949254 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3f500 0x561fa3a48aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff3f6ffd700 0 client.15189 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:41:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 >> 172.21.15.78:0/564949254 conn(0x561fa3a34310 msgr2=0x561fa3a38d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 shutdown_connections 2023-10-10T17:41:52.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 1 -- 172.21.15.78:0/564949254 wait complete. 2023-10-10T17:41:52.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.442+0000 7ff42a30ab80 20 client.15189 trim_cache size 0 max 16384 2023-10-10T17:41:52.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 Processor -- start 2023-10-10T17:41:52.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 -- start start 2023-10-10T17:41:52.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a46400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a4faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a52c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa389bed0 con 0x561fa3a400c0 2023-10-10T17:41:52.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa392be10 con 0x561fa3a3f500 2023-10-10T17:41:52.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa394dfa0 con 0x561fa3a3ebb0 2023-10-10T17:41:52.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff417fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a52c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff417fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a52c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43322/0 (socket says 172.21.15.78:43322) 2023-10-10T17:41:52.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.459+0000 7ff417fff700 1 -- 172.21.15.78:0/2726540760 learned_addr learned my addr 172.21.15.78:0/2726540760 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff4177fe700 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a46400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff416ffd700 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a4faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff417fff700 1 -- 172.21.15.78:0/2726540760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 msgr2=0x561fa3a46400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff417fff700 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a46400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff417fff700 1 -- 172.21.15.78:0/2726540760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 msgr2=0x561fa3a4faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff417fff700 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a4faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff417fff700 1 -- 172.21.15.78:0/2726540760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561fa3955e90 con 0x561fa3a400c0 2023-10-10T17:41:52.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff4177fe700 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a46400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:52.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff417fff700 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a52c30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff410018000 tx=0x7ff410009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ba35837f3918b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff3fe7fc700 1 -- 172.21.15.78:0/2726540760 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410010070 con 0x561fa3a400c0 2023-10-10T17:41:52.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff3fe7fc700 1 -- 172.21.15.78:0/2726540760 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff410013dc0 con 0x561fa3a400c0 2023-10-10T17:41:52.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.460+0000 7ff3fe7fc700 1 -- 172.21.15.78:0/2726540760 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff410026400 con 0x561fa3a400c0 2023-10-10T17:41:52.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- 172.21.15.78:0/2726540760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 msgr2=0x561fa3a52c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a52c30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7ff410018000 tx=0x7ff410009f30 comp rx=0 tx=0).stop 2023-10-10T17:41:52.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- 172.21.15.78:0/2726540760 shutdown_connections 2023-10-10T17:41:52.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a3ebb0 0x561fa3a46400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a400c0 0x561fa3a52c30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/2726540760 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3f500 0x561fa3a4faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- 172.21.15.78:0/2726540760 >> 172.21.15.78:0/2726540760 conn(0x561fa3a34310 msgr2=0x561fa3a5c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- 172.21.15.78:0/2726540760 shutdown_connections 2023-10-10T17:41:52.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- 172.21.15.78:0/2726540760 wait complete. 2023-10-10T17:41:52.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- start start 2023-10-10T17:41:52.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 Processor -- start 2023-10-10T17:41:52.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3ebb0 0x561fa3a4a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3f500 0x561fa3a4ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a400c0 0x561fa3aa76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa3a4b2d0 con 0x561fa3a3f500 2023-10-10T17:41:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa3a4b440 con 0x561fa3a3ebb0 2023-10-10T17:41:52.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561fa3aa7c30 con 0x561fa3a400c0 2023-10-10T17:41:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff42a30ab80 10 client.0 _ll_register_callbacks cb 0x7ff40c00bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:41:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.461+0000 7ff416ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3f500 0x561fa3a4ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3f500 0x561fa3a4ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43326/0 (socket says 172.21.15.78:43326) 2023-10-10T17:41:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 -- 172.21.15.78:0/270652135 learned_addr learned my addr 172.21.15.78:0/270652135 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff4177fe700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3ebb0 0x561fa3a4a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff417fff700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a400c0 0x561fa3aa76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 -- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a400c0 msgr2=0x561fa3aa76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a400c0 0x561fa3aa76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 -- 172.21.15.78:0/270652135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3ebb0 msgr2=0x561fa3a4a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3ebb0 0x561fa3a4a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 -- 172.21.15.78:0/270652135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561fa38115e0 con 0x561fa3a3f500 2023-10-10T17:41:52.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff4177fe700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3ebb0 0x561fa3a4a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:41:52.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff416ffd700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3f500 0x561fa3a4ad90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff408099890 tx=0x7ff40800b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88b68d16ef2efab server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff3f6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:41:52.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/270652135 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4080b2260 con 0x561fa3a3f500 2023-10-10T17:41:52.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/270652135 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff408099df0 con 0x561fa3a3f500 2023-10-10T17:41:52.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff410013a50 con 0x561fa3a3f500 2023-10-10T17:41:52.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/270652135 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff408099fb0 con 0x561fa3a3f500 2023-10-10T17:41:52.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff42a30ab80 10 client.15195 fetch_fsmap learned FSMap version 24 2023-10-10T17:41:52.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561fa37a6c10 con 0x561fa3a3f500 2023-10-10T17:41:52.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.462+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/270652135 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff410013a50 con 0x561fa3a3f500 2023-10-10T17:41:52.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/270652135 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff408098af0 con 0x561fa3a3f500 2023-10-10T17:41:52.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff42a30ab80 10 client.15195 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:41:52.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff42a30ab80 10 client.15195 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-10T17:41:52.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff42a30ab80 20 client.15195 populate_metadata read hostname 'smithi078' 2023-10-10T17:41:52.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3aaca00 0x561fa3aa9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff42a30ab80 4 client.15195 mds_command: new command op to 24472 tid=0[{"prefix": "status"}] 2023-10-10T17:41:52.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561fa395e610 con 0x561fa3aaca00 2023-10-10T17:41:52.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff4177fe700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3aaca00 0x561fa3aa9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff3f6ffd700 10 client.15195 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:41:52.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.463+0000 7ff4177fe700 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3aaca00 0x561fa3aa9330 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e40f16e0a5153a76 server_cookie=d866cb98a7e993b1 in_seq=0 out_seq=0 2023-10-10T17:41:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff3f6ffd700 1 -- 172.21.15.78:0/270652135 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x561fa395e610 con 0x561fa3aaca00 2023-10-10T17:41:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff3f6ffd700 10 client.15195 handle_command_reply: tid=0 2023-10-10T17:41:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 client.15195 shutdown 2023-10-10T17:41:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3f500 msgr2=0x561fa3a4ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3f500 0x561fa3a4ad90 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff408099890 tx=0x7ff40800b5d0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 shutdown_connections 2023-10-10T17:41:52.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561fa3aaca00 0x561fa3aa9330 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561fa3a400c0 0x561fa3aa76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff3f6ffd700 0 client.15195 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:41:52.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561fa3a3ebb0 0x561fa3a4a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 --2- 172.21.15.78:0/270652135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561fa3a3f500 0x561fa3a4ad90 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 >> 172.21.15.78:0/270652135 conn(0x561fa3a34310 msgr2=0x561fa3a52c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 shutdown_connections 2023-10-10T17:41:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 1 -- 172.21.15.78:0/270652135 wait complete. 2023-10-10T17:41:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff42a30ab80 20 client.15195 trim_cache size 0 max 16384 2023-10-10T17:41:52.489 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:41:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:41:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2023-10-10T17:41:52.489 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24472, 2023-10-10T17:41:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-10T17:41:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-10T17:41:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-10T17:41:52.490 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 51.877170567, 2023-10-10T17:41:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2023-10-10T17:41:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2023-10-10T17:41:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2023-10-10T17:41:52.491 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 87.628501220000004 2023-10-10T17:41:52.491 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:41:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff4000a1150 msgr2=0x7ff4000a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff41e17e700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff4000a1150 0x7ff4000a3600 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff40c006010 tx=0x7ff40c00b540 comp rx=0 tx=0).stop 2023-10-10T17:41:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff418100300 msgr2=0x7ff41819ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.464+0000 7ff41e17e700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff418100300 0x7ff41819ad20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff40800d950 tx=0x7ff40800dc60 comp rx=0 tx=0).stop 2023-10-10T17:41:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 shutdown_connections 2023-10-10T17:41:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff4000a1150 0x7ff4000a3600 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff418100c90 0x7ff41819d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff4180ff110 0x7ff418068960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 --2- 172.21.15.78:0/3331498459 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff418100300 0x7ff41819ad20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 >> 172.21.15.78:0/3331498459 conn(0x7ff4180fa6f0 msgr2=0x7ff4181037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 shutdown_connections 2023-10-10T17:41:52.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.465+0000 7ff41e17e700 1 -- 172.21.15.78:0/3331498459 wait complete. 2023-10-10T17:41:52.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-10-10T17:41:52.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.617+0000 7f53aaf39700 1 -- 172.21.15.78:0/17851692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a40ff150 msgr2=0x7f53a40ff560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.617+0000 7f53aaf39700 1 --2- 172.21.15.78:0/17851692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a40ff150 0x7f53a40ff560 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f5398009990 tx=0x7f5398009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.617+0000 7f53aaf39700 1 -- 172.21.15.78:0/17851692 shutdown_connections 2023-10-10T17:41:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.617+0000 7f53aaf39700 1 --2- 172.21.15.78:0/17851692 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53a4100d90 0x7f53a4107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.617+0000 7f53aaf39700 1 --2- 172.21.15.78:0/17851692 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a40ff150 0x7f53a40ff560 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.617+0000 7f53aaf39700 1 --2- 172.21.15.78:0/17851692 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53a41003e0 0x7f53a4100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.617+0000 7f53aaf39700 1 -- 172.21.15.78:0/17851692 >> 172.21.15.78:0/17851692 conn(0x7f53a40fa750 msgr2=0x7f53a40fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 -- 172.21.15.78:0/17851692 shutdown_connections 2023-10-10T17:41:52.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 -- 172.21.15.78:0/17851692 wait complete. 2023-10-10T17:41:52.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 Processor -- start 2023-10-10T17:41:52.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 -- start start 2023-10-10T17:41:52.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53a40ff150 0x7f53a4195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53a41003e0 0x7f53a4196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a4100d90 0x7f53a419d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a419da80 con 0x7f53a4100d90 2023-10-10T17:41:52.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a419dbc0 con 0x7f53a41003e0 2023-10-10T17:41:52.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53aaf39700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a419dd50 con 0x7f53a40ff150 2023-10-10T17:41:52.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53a94d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a4100d90 0x7f53a419d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53a94d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a4100d90 0x7f53a419d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43342/0 (socket says 172.21.15.78:43342) 2023-10-10T17:41:52.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.618+0000 7f53a94d6700 1 -- 172.21.15.78:0/3618378099 learned_addr learned my addr 172.21.15.78:0/3618378099 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a94d6700 1 -- 172.21.15.78:0/3618378099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53a40ff150 msgr2=0x7f53a4195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53a40ff150 0x7f53a4195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a94d6700 1 -- 172.21.15.78:0/3618378099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53a41003e0 msgr2=0x7f53a4196440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a3fff700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53a41003e0 0x7f53a4196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53a41003e0 0x7f53a4196440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a94d6700 1 -- 172.21.15.78:0/3618378099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53980095f0 con 0x7f53a4100d90 2023-10-10T17:41:52.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a3fff700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53a41003e0 0x7f53a4196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a4100d90 0x7f53a419d4b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f539c007d70 tx=0x7f539c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a1ffb700 1 -- 172.21.15.78:0/3618378099 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c00ad40 con 0x7f53a4100d90 2023-10-10T17:41:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a1ffb700 1 -- 172.21.15.78:0/3618378099 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f539c0128e0 con 0x7f53a4100d90 2023-10-10T17:41:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a410bc10 con 0x7f53a4100d90 2023-10-10T17:41:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53a1ffb700 1 -- 172.21.15.78:0/3618378099 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c011780 con 0x7f53a4100d90 2023-10-10T17:41:52.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.619+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a410ba80 con 0x7f53a4100d90 2023-10-10T17:41:52.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.621+0000 7f53a1ffb700 1 -- 172.21.15.78:0/3618378099 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f539c012a50 con 0x7f53a4100d90 2023-10-10T17:41:52.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.622+0000 7f53a1ffb700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53940a1150 0x7f53940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.622+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53940a1150 0x7f53940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.622+0000 7f53a1ffb700 1 -- 172.21.15.78:0/3618378099 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f539c0ccfe0 con 0x7f53a4100d90 2023-10-10T17:41:52.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.622+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53940a1150 0x7f53940a3600 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f539800b680 tx=0x7f5398009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 Processor -- start 2023-10-10T17:41:52.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 -- start start 2023-10-10T17:41:52.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c75fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c76500 0x55e7a0c76ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c77530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0b7b010 con 0x55e7a0c770c0 2023-10-10T17:41:52.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0bcd850 con 0x55e7a0c76500 2023-10-10T17:41:52.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0bc6ce0 con 0x55e7a0c75bb0 2023-10-10T17:41:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c77530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c77530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43350/0 (socket says 172.21.15.78:43350) 2023-10-10T17:41:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 -- 172.21.15.78:0/1256664220 learned_addr learned my addr 172.21.15.78:0/1256664220 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c75fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a3fff700 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c76500 0x55e7a0c76ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 -- 172.21.15.78:0/1256664220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 msgr2=0x55e7a0c75fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c75fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 -- 172.21.15.78:0/1256664220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c76500 msgr2=0x55e7a0c76ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c76500 0x55e7a0c76ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 -- 172.21.15.78:0/1256664220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7a0a485e0 con 0x55e7a0c770c0 2023-10-10T17:41:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.623+0000 7f53a94d6700 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c77530 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f539c096af0 tx=0x7f539c007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa972ccaaba14e35 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f538effd700 1 -- 172.21.15.78:0/1256664220 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c09e0b0 con 0x55e7a0c770c0 2023-10-10T17:41:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f538effd700 1 -- 172.21.15.78:0/1256664220 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f539c0a24d0 con 0x55e7a0c770c0 2023-10-10T17:41:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f538effd700 1 -- 172.21.15.78:0/1256664220 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c0a22a0 con 0x55e7a0c770c0 2023-10-10T17:41:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 -- 172.21.15.78:0/1256664220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 msgr2=0x55e7a0c77530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c77530 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f539c096af0 tx=0x7f539c007f70 comp rx=0 tx=0).stop 2023-10-10T17:41:52.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 -- 172.21.15.78:0/1256664220 shutdown_connections 2023-10-10T17:41:52.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c75fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c77530 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/1256664220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c76500 0x55e7a0c76ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 -- 172.21.15.78:0/1256664220 >> 172.21.15.78:0/1256664220 conn(0x55e7a0c6b310 msgr2=0x55e7a0c70620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 -- 172.21.15.78:0/1256664220 shutdown_connections 2023-10-10T17:41:52.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.624+0000 7f53b70c5b80 1 -- 172.21.15.78:0/1256664220 wait complete. 2023-10-10T17:41:52.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 -- start start 2023-10-10T17:41:52.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 Processor -- start 2023-10-10T17:41:52.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c7fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c86b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0b68890 con 0x55e7a0c770c0 2023-10-10T17:41:52.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0b68530 con 0x55e7a0c75bb0 2023-10-10T17:41:52.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0a4a8d0 con 0x55e7a0c76500 2023-10-10T17:41:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53b70c5b80 10 client.0 _ll_register_callbacks cb 0x55e7a0b86730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:41:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53a94d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c86b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53a94d6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c86b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43356/0 (socket says 172.21.15.78:43356) 2023-10-10T17:41:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53a94d6700 1 -- 172.21.15.78:0/3290828812 learned_addr learned my addr 172.21.15.78:0/3290828812 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.625+0000 7f53a3fff700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c7fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a94d6700 1 -- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 msgr2=0x55e7a0c7fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c7fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a94d6700 1 -- 172.21.15.78:0/3290828812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c75bb0 msgr2=0x55e7a0c7f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a94d6700 1 -- 172.21.15.78:0/3290828812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a410ba80 con 0x55e7a0c770c0 2023-10-10T17:41:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a3fff700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c7fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c86b70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f539c099fc0 tx=0x7f539c09a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1909441afb7a32 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f5383fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:41:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f5383fff700 1 -- 172.21.15.78:0/3290828812 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c0a2240 con 0x55e7a0c770c0 2023-10-10T17:41:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f5383fff700 1 -- 172.21.15.78:0/3290828812 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f539c0b8180 con 0x55e7a0c770c0 2023-10-10T17:41:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7a0c68ad0 con 0x55e7a0c770c0 2023-10-10T17:41:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f5383fff700 1 -- 172.21.15.78:0/3290828812 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f539c0b8340 con 0x55e7a0c770c0 2023-10-10T17:41:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 10 client.15207 fetch_fsmap learned FSMap version 24 2023-10-10T17:41:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e7a0b8d140 con 0x55e7a0c770c0 2023-10-10T17:41:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f5383fff700 1 -- 172.21.15.78:0/3290828812 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e7a0c68ad0 con 0x55e7a0c770c0 2023-10-10T17:41:52.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f5383fff700 1 -- 172.21.15.78:0/3290828812 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f539c004c90 con 0x55e7a0c770c0 2023-10-10T17:41:52.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 10 client.15207 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:41:52.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 10 client.15207 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-10T17:41:52.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 20 client.15207 populate_metadata read hostname 'smithi078' 2023-10-10T17:41:52.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0c8a450 0x55e7a0c89aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 4 client.15207 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:41:52.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.626+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7a0b89050 con 0x55e7a0c8a450 2023-10-10T17:41:52.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.627+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0c8a450 0x55e7a0c89aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.627+0000 7f5383fff700 10 client.15207 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:41:52.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.627+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0c8a450 0x55e7a0c89aa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bda65e03f43608e1 server_cookie=10102f276ccf2a0c in_seq=0 out_seq=0 2023-10-10T17:41:52.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f5383fff700 1 -- 172.21.15.78:0/3290828812 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e7a0b89050 con 0x55e7a0c8a450 2023-10-10T17:41:52.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f5383fff700 10 client.15207 handle_command_reply: tid=0 2023-10-10T17:41:52.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 client.15207 shutdown 2023-10-10T17:41:52.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 msgr2=0x55e7a0c86b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c86b70 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f539c099fc0 tx=0x7f539c09a1f0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 shutdown_connections 2023-10-10T17:41:52.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c770c0 0x55e7a0c86b70 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0c8a450 0x55e7a0c89aa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3290828812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c7fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f5383fff700 0 client.15207 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:41:52.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 >> 172.21.15.78:0/3290828812 conn(0x55e7a0c6b310 msgr2=0x55e7a0c6fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 shutdown_connections 2023-10-10T17:41:52.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3290828812 wait complete. 2023-10-10T17:41:52.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.628+0000 7f53b70c5b80 20 client.15207 trim_cache size 0 max 16384 2023-10-10T17:41:52.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 Processor -- start 2023-10-10T17:41:52.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 -- start start 2023-10-10T17:41:52.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c86b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c89aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0c76cf0 con 0x55e7a0c75bb0 2023-10-10T17:41:52.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0b89080 con 0x55e7a0c770c0 2023-10-10T17:41:52.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0ad2ed0 con 0x55e7a0c76500 2023-10-10T17:41:52.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53a8cd5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53a8cd5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43360/0 (socket says 172.21.15.78:43360) 2023-10-10T17:41:52.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53a8cd5700 1 -- 172.21.15.78:0/3629834221 learned_addr learned my addr 172.21.15.78:0/3629834221 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.646+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c89aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53a8cd5700 1 -- 172.21.15.78:0/3629834221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 msgr2=0x55e7a0c86b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c86b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53a8cd5700 1 -- 172.21.15.78:0/3629834221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 msgr2=0x55e7a0c89aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c89aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53a8cd5700 1 -- 172.21.15.78:0/3629834221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7a0b8ce90 con 0x55e7a0c75bb0 2023-10-10T17:41:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53a94d6700 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c89aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7d5b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5398009990 tx=0x7f5398018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c2d2c5682c5383 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f538f7fe700 1 -- 172.21.15.78:0/3629834221 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539802b040 con 0x55e7a0c75bb0 2023-10-10T17:41:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f538f7fe700 1 -- 172.21.15.78:0/3629834221 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f53980319c0 con 0x55e7a0c75bb0 2023-10-10T17:41:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f538f7fe700 1 -- 172.21.15.78:0/3629834221 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539801d070 con 0x55e7a0c75bb0 2023-10-10T17:41:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3629834221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c75bb0 msgr2=0x55e7a0c7d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7d5b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5398009990 tx=0x7f5398018000 comp rx=0 tx=0).stop 2023-10-10T17:41:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.647+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3629834221 shutdown_connections 2023-10-10T17:41:52.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c89aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c7d5b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/3629834221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c76500 0x55e7a0c86b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3629834221 >> 172.21.15.78:0/3629834221 conn(0x55e7a0c6b310 msgr2=0x55e7a0c931a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3629834221 shutdown_connections 2023-10-10T17:41:52.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 -- 172.21.15.78:0/3629834221 wait complete. 2023-10-10T17:41:52.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 -- start start 2023-10-10T17:41:52.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 Processor -- start 2023-10-10T17:41:52.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c76500 0x55e7a0c868f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c818f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0c81e80 con 0x55e7a0c76500 2023-10-10T17:41:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0c81ff0 con 0x55e7a0c770c0 2023-10-10T17:41:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a0cde320 con 0x55e7a0c75bb0 2023-10-10T17:41:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53b70c5b80 10 client.0 _ll_register_callbacks cb 0x7f5398005be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:41:52.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53a3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c76500 0x55e7a0c868f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53a3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c76500 0x55e7a0c868f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43368/0 (socket says 172.21.15.78:43368) 2023-10-10T17:41:52.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.648+0000 7f53a3fff700 1 -- 172.21.15.78:0/2960750855 learned_addr learned my addr 172.21.15.78:0/2960750855 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a94d6700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c818f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a3fff700 1 -- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 msgr2=0x55e7a0c813b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a3fff700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c813b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a3fff700 1 -- 172.21.15.78:0/2960750855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 msgr2=0x55e7a0c818f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a3fff700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c818f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a3fff700 1 -- 172.21.15.78:0/2960750855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a4068fd0 con 0x55e7a0c76500 2023-10-10T17:41:52.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:52.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53a3fff700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c76500 0x55e7a0c868f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55e7a0c71460 tx=0x7f5390013a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92dbceb0ee50b459 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f5383fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:41:52.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f5383fff700 1 -- 172.21.15.78:0/2960750855 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390011870 con 0x55e7a0c76500 2023-10-10T17:41:52.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f5383fff700 1 -- 172.21.15.78:0/2960750855 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f53900119d0 con 0x55e7a0c76500 2023-10-10T17:41:52.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f539c0a24d0 con 0x55e7a0c76500 2023-10-10T17:41:52.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f5383fff700 1 -- 172.21.15.78:0/2960750855 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5390011b90 con 0x55e7a0c76500 2023-10-10T17:41:52.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53b70c5b80 10 client.15213 fetch_fsmap learned FSMap version 24 2023-10-10T17:41:52.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e7a09ddc10 con 0x55e7a0c76500 2023-10-10T17:41:52.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f5383fff700 1 -- 172.21.15.78:0/2960750855 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f539c0a24d0 con 0x55e7a0c76500 2023-10-10T17:41:52.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.649+0000 7f5383fff700 1 -- 172.21.15.78:0/2960750855 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f539001b3e0 con 0x55e7a0c76500 2023-10-10T17:41:52.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53b70c5b80 10 client.15213 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:41:52.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53b70c5b80 10 client.15213 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-10T17:41:52.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53b70c5b80 20 client.15213 populate_metadata read hostname 'smithi078' 2023-10-10T17:41:52.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0ce3170 0x55e7a0cdfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53b70c5b80 4 client.15213 mds_command: new command op to 24490 tid=0[{"prefix": "status"}] 2023-10-10T17:41:52.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e7a0b95610 con 0x55e7a0ce3170 2023-10-10T17:41:52.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0ce3170 0x55e7a0cdfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f5383fff700 10 client.15213 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:41:52.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53a8cd5700 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0ce3170 0x55e7a0cdfad0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=603a2b2014e38233 server_cookie=6924609381f1193b in_seq=0 out_seq=0 2023-10-10T17:41:52.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f5383fff700 1 -- 172.21.15.78:0/2960750855 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55e7a0b95610 con 0x55e7a0ce3170 2023-10-10T17:41:52.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f5383fff700 10 client.15213 handle_command_reply: tid=0 2023-10-10T17:41:52.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.650+0000 7f53b70c5b80 1 client.15213 shutdown 2023-10-10T17:41:52.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c76500 msgr2=0x55e7a0c868f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c76500 0x55e7a0c868f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x55e7a0c71460 tx=0x7f5390013a10 comp rx=0 tx=0).stop 2023-10-10T17:41:52.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 shutdown_connections 2023-10-10T17:41:52.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55e7a0ce3170 0x55e7a0cdfad0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e7a0c75bb0 0x55e7a0c813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e7a0c770c0 0x55e7a0c818f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f5383fff700 0 client.15213 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:41:52.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 --2- 172.21.15.78:0/2960750855 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e7a0c76500 0x55e7a0c868f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 >> 172.21.15.78:0/2960750855 conn(0x55e7a0c6b310 msgr2=0x55e7a0c89aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 shutdown_connections 2023-10-10T17:41:52.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 1 -- 172.21.15.78:0/2960750855 wait complete. 2023-10-10T17:41:52.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53b70c5b80 20 client.15213 trim_cache size 0 max 16384 2023-10-10T17:41:52.674 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:41:52.674 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:41:52.674 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2023-10-10T17:41:52.674 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24490, 2023-10-10T17:41:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-10T17:41:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-10T17:41:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-10T17:41:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 53.071351128000003, 2023-10-10T17:41:52.675 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2023-10-10T17:41:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2023-10-10T17:41:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2023-10-10T17:41:52.676 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 83.565345973000007 2023-10-10T17:41:52.676 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:41:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53940a1150 msgr2=0x7f53940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53940a1150 0x7f53940a3600 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f539800b680 tx=0x7f5398009510 comp rx=0 tx=0).stop 2023-10-10T17:41:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a4100d90 msgr2=0x7f53a419d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a4100d90 0x7f53a419d4b0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f539c007d70 tx=0x7f539c00a340 comp rx=0 tx=0).stop 2023-10-10T17:41:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 shutdown_connections 2023-10-10T17:41:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f53940a1150 0x7f53940a3600 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f53a40ff150 0x7f53a4195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53a4100d90 0x7f53a419d4b0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 --2- 172.21.15.78:0/3618378099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f53a41003e0 0x7f53a4196440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 >> 172.21.15.78:0/3618378099 conn(0x7f53a40fa750 msgr2=0x7f53a40fc920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 shutdown_connections 2023-10-10T17:41:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.651+0000 7f53aaf39700 1 -- 172.21.15.78:0/3618378099 wait complete. 2023-10-10T17:41:52.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-10T17:41:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 -- 172.21.15.78:0/1862682526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00ff110 msgr2=0x7fc9f00ff520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/1862682526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00ff110 0x7fc9f00ff520 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009990 tx=0x7fc9e4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 -- 172.21.15.78:0/1862682526 shutdown_connections 2023-10-10T17:41:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/1862682526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9f0100c90 0x7fc9f0107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/1862682526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f00ff110 0x7fc9f00ff520 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/1862682526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0100300 0x7fc9f0100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 -- 172.21.15.78:0/1862682526 >> 172.21.15.78:0/1862682526 conn(0x7fc9f00fa6f0 msgr2=0x7fc9f00fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 -- 172.21.15.78:0/1862682526 shutdown_connections 2023-10-10T17:41:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 -- 172.21.15.78:0/1862682526 wait complete. 2023-10-10T17:41:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 Processor -- start 2023-10-10T17:41:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.803+0000 7fc9f6b96700 1 -- start start 2023-10-10T17:41:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f6b96700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9f00ff110 0x7fc9f01030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f6b96700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0100300 0x7fc9f01016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f6b96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f0100c90 0x7fc9f0101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f6b96700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f01023c0 con 0x7fc9f0100c90 2023-10-10T17:41:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f6b96700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f0102530 con 0x7fc9f0100300 2023-10-10T17:41:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f6b96700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9f019fef0 con 0x7fc9f00ff110 2023-10-10T17:41:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f0100c90 0x7fc9f0101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f0100c90 0x7fc9f0101c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43384/0 (socket says 172.21.15.78:43384) 2023-10-10T17:41:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 -- 172.21.15.78:0/3493556142 learned_addr learned my addr 172.21.15.78:0/3493556142 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f4932700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9f00ff110 0x7fc9f01030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9effff700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0100300 0x7fc9f01016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 -- 172.21.15.78:0/3493556142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9f00ff110 msgr2=0x7fc9f01030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9f00ff110 0x7fc9f01030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 -- 172.21.15.78:0/3493556142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0100300 msgr2=0x7fc9f01016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0100300 0x7fc9f01016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 -- 172.21.15.78:0/3493556142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e40095f0 con 0x7fc9f0100c90 2023-10-10T17:41:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f5133700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f0100c90 0x7fc9f0101c20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007d70 tx=0x7fc9e800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9edffb700 1 -- 172.21.15.78:0/3493556142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e800ad40 con 0x7fc9f0100c90 2023-10-10T17:41:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9edffb700 1 -- 172.21.15.78:0/3493556142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9e80128e0 con 0x7fc9f0100c90 2023-10-10T17:41:52.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9edffb700 1 -- 172.21.15.78:0/3493556142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8011780 con 0x7fc9f0100c90 2023-10-10T17:41:52.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.804+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f010aa70 con 0x7fc9f0100c90 2023-10-10T17:41:52.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.805+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f010a8e0 con 0x7fc9f0100c90 2023-10-10T17:41:52.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.806+0000 7fc9edffb700 1 -- 172.21.15.78:0/3493556142 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc9e8012a50 con 0x7fc9f0100c90 2023-10-10T17:41:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.807+0000 7fc9edffb700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9e00a11d0 0x7fc9e00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.807+0000 7fc9f4932700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9e00a11d0 0x7fc9e00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.807+0000 7fc9edffb700 1 -- 172.21.15.78:0/3493556142 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9e80ccf00 con 0x7fc9f0100c90 2023-10-10T17:41:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.808+0000 7fc9f4932700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9e00a11d0 0x7fc9e00a3680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4006010 tx=0x7fc9e400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.808+0000 7fca02d22b80 1 Processor -- start 2023-10-10T17:41:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.808+0000 7fca02d22b80 1 -- start start 2023-10-10T17:41:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.808+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f00fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f01500 0x55b5c4f01ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f020c0 0x55b5c4f02530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fca02d22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4e06010 con 0x55b5c4f020c0 2023-10-10T17:41:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fca02d22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4e58850 con 0x55b5c4f01500 2023-10-10T17:41:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fca02d22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4e51ce0 con 0x55b5c4f00bb0 2023-10-10T17:41:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f020c0 0x55b5c4f02530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f020c0 0x55b5c4f02530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43392/0 (socket says 172.21.15.78:43392) 2023-10-10T17:41:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 -- 172.21.15.78:0/2897742573 learned_addr learned my addr 172.21.15.78:0/2897742573 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9effff700 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f01500 0x55b5c4f01ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f4932700 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f00fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 -- 172.21.15.78:0/2897742573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 msgr2=0x55b5c4f00fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f00fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 -- 172.21.15.78:0/2897742573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f01500 msgr2=0x55b5c4f01ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f01500 0x55b5c4f01ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 -- 172.21.15.78:0/2897742573 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5c4cd35e0 con 0x55b5c4f020c0 2023-10-10T17:41:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9effff700 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f01500 0x55b5c4f01ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:41:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9f5133700 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f020c0 0x55b5c4f02530 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8099f90 tx=0x7fc9e8007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb0927f7401d07ae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9daffd700 1 -- 172.21.15.78:0/2897742573 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e809a000 con 0x55b5c4f020c0 2023-10-10T17:41:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9daffd700 1 -- 172.21.15.78:0/2897742573 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9e809e030 con 0x55b5c4f020c0 2023-10-10T17:41:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.809+0000 7fc9daffd700 1 -- 172.21.15.78:0/2897742573 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80a2220 con 0x55b5c4f020c0 2023-10-10T17:41:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 -- 172.21.15.78:0/2897742573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f020c0 msgr2=0x55b5c4f02530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f020c0 0x55b5c4f02530 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8099f90 tx=0x7fc9e8007f70 comp rx=0 tx=0).stop 2023-10-10T17:41:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 -- 172.21.15.78:0/2897742573 shutdown_connections 2023-10-10T17:41:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f00fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f020c0 0x55b5c4f02530 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2897742573 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f01500 0x55b5c4f01ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 -- 172.21.15.78:0/2897742573 >> 172.21.15.78:0/2897742573 conn(0x55b5c4ef6310 msgr2=0x55b5c4efb620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 -- 172.21.15.78:0/2897742573 shutdown_connections 2023-10-10T17:41:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 -- 172.21.15.78:0/2897742573 wait complete. 2023-10-10T17:41:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 -- start start 2023-10-10T17:41:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.810+0000 7fca02d22b80 1 Processor -- start 2023-10-10T17:41:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f11ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fca02d22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4df3890 con 0x55b5c4f01500 2023-10-10T17:41:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fca02d22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4df3530 con 0x55b5c4f020c0 2023-10-10T17:41:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fca02d22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4cd58d0 con 0x55b5c4f00bb0 2023-10-10T17:41:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fca02d22b80 10 client.0 _ll_register_callbacks cb 0x55b5c4e11730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:41:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43396/0 (socket says 172.21.15.78:43396) 2023-10-10T17:41:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 -- 172.21.15.78:0/1611771065 learned_addr learned my addr 172.21.15.78:0/1611771065 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9f5133700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f11ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9f4932700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 -- 172.21.15.78:0/1611771065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 msgr2=0x55b5c4f0a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 -- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 msgr2=0x55b5c4f11ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f11ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9effff700 1 -- 172.21.15.78:0/1611771065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f010a8e0 con 0x55b5c4f01500 2023-10-10T17:41:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.811+0000 7fc9f4932700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9effff700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0aaa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc010840 tx=0x7fc9dc010b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c090929724d0faa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9cffff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:41:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9cffff700 1 -- 172.21.15.78:0/1611771065 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc00e9e0 con 0x55b5c4f01500 2023-10-10T17:41:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9cffff700 1 -- 172.21.15.78:0/1611771065 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9dc017070 con 0x55b5c4f01500 2023-10-10T17:41:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b5c4ef3ad0 con 0x55b5c4f01500 2023-10-10T17:41:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9cffff700 1 -- 172.21.15.78:0/1611771065 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9dc00eb40 con 0x55b5c4f01500 2023-10-10T17:41:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 10 client.15225 fetch_fsmap learned FSMap version 24 2023-10-10T17:41:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b5c4e18140 con 0x55b5c4f01500 2023-10-10T17:41:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9cffff700 1 -- 172.21.15.78:0/1611771065 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b5c4ef3ad0 con 0x55b5c4f01500 2023-10-10T17:41:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9cffff700 1 -- 172.21.15.78:0/1611771065 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc9dc011440 con 0x55b5c4f01500 2023-10-10T17:41:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:41:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 10 client.15225 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T17:41:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 20 client.15225 populate_metadata read hostname 'smithi078' 2023-10-10T17:41:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f157b0 0x55b5c4f14bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 4 client.15225 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:41:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5c4e14050 con 0x55b5c4f157b0 2023-10-10T17:41:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.812+0000 7fc9f4932700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f157b0 0x55b5c4f14bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.813+0000 7fc9cffff700 10 client.15225 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:41:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.813+0000 7fc9f4932700 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f157b0 0x55b5c4f14bb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d77c041ec01119c server_cookie=c7e8bcfde1fff8f8 in_seq=0 out_seq=0 2023-10-10T17:41:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fc9cffff700 1 -- 172.21.15.78:0/1611771065 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b5c4e14050 con 0x55b5c4f157b0 2023-10-10T17:41:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fc9cffff700 10 client.15225 handle_command_reply: tid=0 2023-10-10T17:41:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 client.15225 shutdown 2023-10-10T17:41:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 msgr2=0x55b5c4f0aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0aaa0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc010840 tx=0x7fc9dc010b50 comp rx=0 tx=0).stop 2023-10-10T17:41:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 shutdown_connections 2023-10-10T17:41:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f157b0 0x55b5c4f14bb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f11ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fc9cffff700 0 client.15225 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:41:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 --2- 172.21.15.78:0/1611771065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0aaa0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 >> 172.21.15.78:0/1611771065 conn(0x55b5c4ef6310 msgr2=0x55b5c4efad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 shutdown_connections 2023-10-10T17:41:52.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 1 -- 172.21.15.78:0/1611771065 wait complete. 2023-10-10T17:41:52.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.814+0000 7fca02d22b80 20 client.15225 trim_cache size 0 max 16384 2023-10-10T17:41:52.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 Processor -- start 2023-10-10T17:41:52.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 -- start start 2023-10-10T17:41:52.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f04570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f11aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f09240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4f01cf0 con 0x55b5c4f01500 2023-10-10T17:41:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4e14050 con 0x55b5c4f020c0 2023-10-10T17:41:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fca02d22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4d5ded0 con 0x55b5c4f00bb0 2023-10-10T17:41:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f11aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f11aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43404/0 (socket says 172.21.15.78:43404) 2023-10-10T17:41:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 -- 172.21.15.78:0/2878325834 learned_addr learned my addr 172.21.15.78:0/2878325834 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9f4932700 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f04570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9f5133700 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f09240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 -- 172.21.15.78:0/2878325834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 msgr2=0x55b5c4f04570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f04570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 -- 172.21.15.78:0/2878325834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 msgr2=0x55b5c4f09240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f09240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 -- 172.21.15.78:0/2878325834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5c4e17e90 con 0x55b5c4f01500 2023-10-10T17:41:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9effff700 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f11aa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0117a0 tx=0x7fc9dc00e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=379d51db145d76a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.832+0000 7fc9db7fe700 1 -- 172.21.15.78:0/2878325834 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc011a90 con 0x55b5c4f01500 2023-10-10T17:41:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fc9db7fe700 1 -- 172.21.15.78:0/2878325834 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9dc015070 con 0x55b5c4f01500 2023-10-10T17:41:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fc9db7fe700 1 -- 172.21.15.78:0/2878325834 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc0052a0 con 0x55b5c4f01500 2023-10-10T17:41:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 -- 172.21.15.78:0/2878325834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 msgr2=0x55b5c4f11aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f11aa0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc0117a0 tx=0x7fc9dc00e350 comp rx=0 tx=0).stop 2023-10-10T17:41:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 -- 172.21.15.78:0/2878325834 shutdown_connections 2023-10-10T17:41:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f04570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f020c0 0x55b5c4f09240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 --2- 172.21.15.78:0/2878325834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f11aa0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 -- 172.21.15.78:0/2878325834 >> 172.21.15.78:0/2878325834 conn(0x55b5c4ef6310 msgr2=0x55b5c4f1e500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 -- 172.21.15.78:0/2878325834 shutdown_connections 2023-10-10T17:41:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 -- 172.21.15.78:0/2878325834 wait complete. 2023-10-10T17:41:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.833+0000 7fca02d22b80 1 -- start start 2023-10-10T17:41:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 1 Processor -- start 2023-10-10T17:41:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f020c0 0x55b5c4f69630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4f11c20 con 0x55b5c4f01500 2023-10-10T17:41:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4f11d90 con 0x55b5c4f00bb0 2023-10-10T17:41:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c4f69b70 con 0x55b5c4f020c0 2023-10-10T17:41:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fca02d22b80 10 client.0 _ll_register_callbacks cb 0x7fc9dc00ccb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:41:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43410/0 (socket says 172.21.15.78:43410) 2023-10-10T17:41:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 -- 172.21.15.78:0/3488771015 learned_addr learned my addr 172.21.15.78:0/3488771015 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9f4932700 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9f5133700 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f020c0 0x55b5c4f69630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 -- 172.21.15.78:0/3488771015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f020c0 msgr2=0x55b5c4f69630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f020c0 0x55b5c4f69630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 -- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f00bb0 msgr2=0x55b5c4f0c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:41:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 -- 172.21.15.78:0/3488771015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5c4e18140 con 0x55b5c4f01500 2023-10-10T17:41:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9effff700 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0cbf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc010760 tx=0x7fc9dc00c5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88494af6e6c37d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.834+0000 7fc9cffff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:41:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fc9cffff700 1 -- 172.21.15.78:0/3488771015 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9dc00e9f0 con 0x55b5c4f01500 2023-10-10T17:41:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fc9cffff700 1 -- 172.21.15.78:0/3488771015 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9dc017070 con 0x55b5c4f01500 2023-10-10T17:41:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9e809e030 con 0x55b5c4f01500 2023-10-10T17:41:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fc9cffff700 1 -- 172.21.15.78:0/3488771015 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9dc00eb50 con 0x55b5c4f01500 2023-10-10T17:41:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 10 client.15231 fetch_fsmap learned FSMap version 24 2023-10-10T17:41:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b5c4c68c10 con 0x55b5c4f01500 2023-10-10T17:41:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fc9cffff700 1 -- 172.21.15.78:0/3488771015 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9e809e030 con 0x55b5c4f01500 2023-10-10T17:41:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fc9cffff700 1 -- 172.21.15.78:0/3488771015 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc9dc003d10 con 0x55b5c4f01500 2023-10-10T17:41:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 10 client.15231 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:41:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 10 client.15231 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T17:41:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 20 client.15231 populate_metadata read hostname 'smithi078' 2023-10-10T17:41:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f6e3b0 0x55b5c4f6ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 4 client.15231 mds_command: new command op to 24500 tid=0[{"prefix": "status"}] 2023-10-10T17:41:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5c4e20610 con 0x55b5c4f6e3b0 2023-10-10T17:41:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.835+0000 7fc9f4932700 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f6e3b0 0x55b5c4f6ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fc9f4932700 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f6e3b0 0x55b5c4f6ad20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b9806f745bfceb server_cookie=87616a9dfbba0215 in_seq=0 out_seq=0 2023-10-10T17:41:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fc9cffff700 10 client.15231 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:41:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fc9cffff700 1 -- 172.21.15.78:0/3488771015 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55b5c4e20610 con 0x55b5c4f6e3b0 2023-10-10T17:41:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fc9cffff700 10 client.15231 handle_command_reply: tid=0 2023-10-10T17:41:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 client.15231 shutdown 2023-10-10T17:41:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 msgr2=0x55b5c4f0cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0cbf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc010760 tx=0x7fc9dc00c5b0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 shutdown_connections 2023-10-10T17:41:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5c4f020c0 0x55b5c4f69630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5c4f6e3b0 0x55b5c4f6ad20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5c4f00bb0 0x55b5c4f0c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 --2- 172.21.15.78:0/3488771015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5c4f01500 0x55b5c4f0cbf0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fc9cffff700 0 client.15231 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:41:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 >> 172.21.15.78:0/3488771015 conn(0x55b5c4ef6310 msgr2=0x55b5c4f04570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 shutdown_connections 2023-10-10T17:41:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.836+0000 7fca02d22b80 1 -- 172.21.15.78:0/3488771015 wait complete. 2023-10-10T17:41:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fca02d22b80 20 client.15231 trim_cache size 0 max 16384 2023-10-10T17:41:52.859 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:41:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:41:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-10T17:41:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24500, 2023-10-10T17:41:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-10T17:41:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-10T17:41:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-10T17:41:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 71.259934377999997, 2023-10-10T17:41:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2023-10-10T17:41:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2023-10-10T17:41:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2023-10-10T17:41:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 76.908465706000001 2023-10-10T17:41:52.861 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:41:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9e00a11d0 msgr2=0x7fc9e00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9e00a11d0 0x7fc9e00a3680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4006010 tx=0x7fc9e400b540 comp rx=0 tx=0).stop 2023-10-10T17:41:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f0100c90 msgr2=0x7fc9f0101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f0100c90 0x7fc9f0101c20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8007d70 tx=0x7fc9e800a340 comp rx=0 tx=0).stop 2023-10-10T17:41:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 shutdown_connections 2023-10-10T17:41:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9e00a11d0 0x7fc9e00a3680 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9f00ff110 0x7fc9f01030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9f0100c90 0x7fc9f0101c20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 --2- 172.21.15.78:0/3493556142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9f0100300 0x7fc9f01016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 >> 172.21.15.78:0/3493556142 conn(0x7fc9f00fa6f0 msgr2=0x7fc9f0067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 shutdown_connections 2023-10-10T17:41:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:52.837+0000 7fc9f6b96700 1 -- 172.21.15.78:0/3493556142 wait complete. 2023-10-10T17:41:52.865 INFO:tasks.cephadm:Setup complete, yielding 2023-10-10T17:41:52.865 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-10-10T17:41:52.877 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:41:52.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch status' 2023-10-10T17:41:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:52 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/864990279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:52 smithi157 ceph-mon[100896]: pgmap v299: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2023-10-10T17:41:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:52 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1711251302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:53.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:52 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/864990279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:52 smithi078 ceph-mon[99494]: pgmap v299: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2023-10-10T17:41:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:52 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1711251302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:52 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/864990279' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:41:53.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:52 smithi190 ceph-mon[101220]: pgmap v299: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s; 9 B/s, 0 objects/s recovering 2023-10-10T17:41:53.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:52 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1711251302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:41:53.522 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:54.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.829+0000 7f529b8fc700 1 -- 172.21.15.78:0/1026058167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 msgr2=0x7f5294107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:54.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.829+0000 7f529b8fc700 1 --2- 172.21.15.78:0/1026058167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f5294107630 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f528c00b0a0 tx=0x7f528c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:41:54.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- 172.21.15.78:0/1026058167 shutdown_connections 2023-10-10T17:41:54.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 --2- 172.21.15.78:0/1026058167 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f5294107630 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:54.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 --2- 172.21.15.78:0/1026058167 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52940ff1e0 0x7f52940ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:54.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 --2- 172.21.15.78:0/1026058167 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52941003e0 0x7f5294100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:54.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- 172.21.15.78:0/1026058167 >> 172.21.15.78:0/1026058167 conn(0x7f52940fa7f0 msgr2=0x7f52940fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:54.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- 172.21.15.78:0/1026058167 shutdown_connections 2023-10-10T17:41:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- 172.21.15.78:0/1026058167 wait complete. 2023-10-10T17:41:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 Processor -- start 2023-10-10T17:41:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- start start 2023-10-10T17:41:54.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52940ff1e0 0x7f5294195090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:54.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52941003e0 0x7f52941955d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:54.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f529419c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:54.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529419cc20 con 0x7f5294100d70 2023-10-10T17:41:54.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529419cd60 con 0x7f52940ff1e0 2023-10-10T17:41:54.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.830+0000 7f529b8fc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5294068e40 con 0x7f52941003e0 2023-10-10T17:41:54.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f529419c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:54.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f529419c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43424/0 (socket says 172.21.15.78:43424) 2023-10-10T17:41:54.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 -- 172.21.15.78:0/3291663745 learned_addr learned my addr 172.21.15.78:0/3291663745 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:54.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299698700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52940ff1e0 0x7f5294195090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:54.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5298e97700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52941003e0 0x7f52941955d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:54.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 -- 172.21.15.78:0/3291663745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52941003e0 msgr2=0x7f52941955d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:54.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52941003e0 0x7f52941955d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:54.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 -- 172.21.15.78:0/3291663745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52940ff1e0 msgr2=0x7f5294195090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:54.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52940ff1e0 0x7f5294195090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:54.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 -- 172.21.15.78:0/3291663745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c009d00 con 0x7f5294100d70 2023-10-10T17:41:54.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f5299e99700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f529419c650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f528c007ab0 tx=0x7f528c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:54.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f528a7fc700 1 -- 172.21.15.78:0/3291663745 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c00e040 con 0x7f5294100d70 2023-10-10T17:41:54.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f528a7fc700 1 -- 172.21.15.78:0/3291663745 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f528c00cec0 con 0x7f5294100d70 2023-10-10T17:41:54.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529410abd0 con 0x7f5294100d70 2023-10-10T17:41:54.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f528a7fc700 1 -- 172.21.15.78:0/3291663745 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f528c01bb30 con 0x7f5294100d70 2023-10-10T17:41:54.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.831+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529410aa40 con 0x7f5294100d70 2023-10-10T17:41:54.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.832+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5294066ea0 con 0x7f5294100d70 2023-10-10T17:41:54.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.833+0000 7f528a7fc700 1 -- 172.21.15.78:0/3291663745 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f528c01bc90 con 0x7f5294100d70 2023-10-10T17:41:54.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.834+0000 7f528a7fc700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f52800a11d0 0x7f52800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:54.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.834+0000 7f5299698700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f52800a11d0 0x7f52800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:54.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.834+0000 7f528a7fc700 1 -- 172.21.15.78:0/3291663745 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f528c0ced70 con 0x7f5294100d70 2023-10-10T17:41:54.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.835+0000 7f5299698700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f52800a11d0 0x7f52800a3680 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5294100240 tx=0x7f5284009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:54.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:54.837+0000 7f528a7fc700 1 -- 172.21.15.78:0/3291663745 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f528c098240 con 0x7f5294100d70 2023-10-10T17:41:55.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.038+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f52940694b0 con 0x7f52800a11d0 2023-10-10T17:41:55.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.039+0000 7f528a7fc700 1 -- 172.21.15.78:0/3291663745 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f52940694b0 con 0x7f52800a11d0 2023-10-10T17:41:55.040 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: cephadm 2023-10-10T17:41:55.041 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2023-10-10T17:41:55.041 INFO:teuthology.orchestra.run.smithi078.stdout:Paused: No 2023-10-10T17:41:55.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f52800a11d0 msgr2=0x7f52800a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:55.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f52800a11d0 0x7f52800a3680 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5294100240 tx=0x7f5284009190 comp rx=0 tx=0).stop 2023-10-10T17:41:55.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 msgr2=0x7f529419c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:55.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f529419c650 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f528c007ab0 tx=0x7f528c004d00 comp rx=0 tx=0).stop 2023-10-10T17:41:55.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 shutdown_connections 2023-10-10T17:41:55.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f52800a11d0 0x7f52800a3680 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:55.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5294100d70 0x7f529419c650 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:55.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f52940ff1e0 0x7f5294195090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:55.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 --2- 172.21.15.78:0/3291663745 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f52941003e0 0x7f52941955d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:55.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 >> 172.21.15.78:0/3291663745 conn(0x7f52940fa7f0 msgr2=0x7f5294103880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:55.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 shutdown_connections 2023-10-10T17:41:55.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:55.041+0000 7f529b8fc700 1 -- 172.21.15.78:0/3291663745 wait complete. 2023-10-10T17:41:55.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-10-10T17:41:55.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:55 smithi078 ceph-mon[99494]: pgmap v300: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:55.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:55 smithi190 ceph-mon[101220]: pgmap v300: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:55.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:55 smithi157 ceph-mon[100896]: pgmap v300: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:41:56.111 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:56.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:56 smithi078 ceph-mon[99494]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:41:56.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:56 smithi190 ceph-mon[101220]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:41:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:56 smithi157 ceph-mon[100896]: from='client.15237 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:41:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 -- 172.21.15.78:0/3579233212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ff430 msgr2=0x7f41c00ff8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:57.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 --2- 172.21.15.78:0/3579233212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ff430 0x7f41c00ff8a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f41bc009990 tx=0x7f41bc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 -- 172.21.15.78:0/3579233212 shutdown_connections 2023-10-10T17:41:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 --2- 172.21.15.78:0/3579233212 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41c0100860 0x7f41c0100c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 --2- 172.21.15.78:0/3579233212 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41c00ffde0 0x7f41c0107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 --2- 172.21.15.78:0/3579233212 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ff430 0x7f41c00ff8a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 -- 172.21.15.78:0/3579233212 >> 172.21.15.78:0/3579233212 conn(0x7f41c00f9f70 msgr2=0x7f41c00fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 -- 172.21.15.78:0/3579233212 shutdown_connections 2023-10-10T17:41:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.459+0000 7f41c86e2700 1 -- 172.21.15.78:0/3579233212 wait complete. 2023-10-10T17:41:57.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 Processor -- start 2023-10-10T17:41:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 -- start start 2023-10-10T17:41:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41c00ff430 0x7f41c00fee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ffde0 0x7f41c019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41c0100860 0x7f41c019d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:57.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c019db90 con 0x7f41c00ffde0 2023-10-10T17:41:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c019dd00 con 0x7f41c00ff430 2023-10-10T17:41:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c86e2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c019dec0 con 0x7f41c0100860 2023-10-10T17:41:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ffde0 0x7f41c019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:57.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ffde0 0x7f41c019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43434/0 (socket says 172.21.15.78:43434) 2023-10-10T17:41:57.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 -- 172.21.15.78:0/209610945 learned_addr learned my addr 172.21.15.78:0/209610945 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:41:57.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c6c7f700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41c0100860 0x7f41c019d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:57.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c647e700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41c00ff430 0x7f41c00fee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:57.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 -- 172.21.15.78:0/209610945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41c0100860 msgr2=0x7f41c019d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:57.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41c0100860 0x7f41c019d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 -- 172.21.15.78:0/209610945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41c00ff430 msgr2=0x7f41c00fee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:57.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41c00ff430 0x7f41c00fee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 -- 172.21.15.78:0/209610945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41bc0095f0 con 0x7f41c00ffde0 2023-10-10T17:41:57.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c6c7f700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41c0100860 0x7f41c019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:57.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c647e700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41c00ff430 0x7f41c00fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:41:57.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.460+0000 7f41c5c7d700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ffde0 0x7f41c019adb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f41bc000c00 tx=0x7f41bc00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:57.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.461+0000 7f41b77fe700 1 -- 172.21.15.78:0/209610945 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc01d070 con 0x7f41c00ffde0 2023-10-10T17:41:57.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.461+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c010abf0 con 0x7f41c00ffde0 2023-10-10T17:41:57.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.461+0000 7f41b77fe700 1 -- 172.21.15.78:0/209610945 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f41bc00fc00 con 0x7f41c00ffde0 2023-10-10T17:41:57.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.461+0000 7f41b77fe700 1 -- 172.21.15.78:0/209610945 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41bc017650 con 0x7f41c00ffde0 2023-10-10T17:41:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.461+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c010aa60 con 0x7f41c00ffde0 2023-10-10T17:41:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.462+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c01067e0 con 0x7f41c00ffde0 2023-10-10T17:41:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.462+0000 7f41b77fe700 1 -- 172.21.15.78:0/209610945 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f41bc022470 con 0x7f41c00ffde0 2023-10-10T17:41:57.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.463+0000 7f41b77fe700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f41ac0a1150 0x7f41ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:41:57.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.463+0000 7f41c647e700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f41ac0a1150 0x7f41ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:41:57.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.463+0000 7f41b77fe700 1 -- 172.21.15.78:0/209610945 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f41bc0cdf50 con 0x7f41c00ffde0 2023-10-10T17:41:57.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.463+0000 7f41c647e700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f41ac0a1150 0x7f41ac0a3600 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f41b0005fd0 tx=0x7f41b0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:41:57.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.467+0000 7f41b77fe700 1 -- 172.21.15.78:0/209610945 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f41bc0973c0 con 0x7f41c00ffde0 2023-10-10T17:41:57.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.664+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c0061c30 con 0x7f41ac0a1150 2023-10-10T17:41:57.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.677+0000 7f41b77fe700 1 -- 172.21.15.78:0/209610945 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f41c0061c30 con 0x7f41ac0a1150 2023-10-10T17:41:57.678 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-10-10T17:41:57.678 INFO:teuthology.orchestra.run.smithi078.stdout:mds.a smithi078 running (84s) 11s ago 83s 14.5M - 18.0.0-3908-g05a43c8f 2d174a053211 1e06a33ed0cd 2023-10-10T17:41:57.679 INFO:teuthology.orchestra.run.smithi078.stdout:mds.b smithi157 running (82s) 12s ago 81s 21.5M - 18.0.0-3908-g05a43c8f 2d174a053211 8b1e2f0706d1 2023-10-10T17:41:57.679 INFO:teuthology.orchestra.run.smithi078.stdout:mds.c smithi190 running (93s) 12s ago 92s 18.2M - 18.0.0-3908-g05a43c8f 2d174a053211 760d849e7fb8 2023-10-10T17:41:57.679 INFO:teuthology.orchestra.run.smithi078.stdout:mds.d smithi078 running (99s) 11s ago 99s 14.8M - 18.0.0-3908-g05a43c8f 2d174a053211 9e0dda29a3b6 2023-10-10T17:41:57.679 INFO:teuthology.orchestra.run.smithi078.stdout:mds.e smithi157 running (97s) 12s ago 97s 17.3M - 18.0.0-3908-g05a43c8f 2d174a053211 f3cf017e3dd8 2023-10-10T17:41:57.679 INFO:teuthology.orchestra.run.smithi078.stdout:mds.f smithi190 running (95s) 12s ago 94s 18.5M - 18.0.0-3908-g05a43c8f 2d174a053211 94978f176263 2023-10-10T17:41:57.680 INFO:teuthology.orchestra.run.smithi078.stdout:mds.g smithi078 running (86s) 11s ago 86s 17.2M - 18.0.0-3908-g05a43c8f 2d174a053211 26867ac92b4a 2023-10-10T17:41:57.680 INFO:teuthology.orchestra.run.smithi078.stdout:mds.h smithi157 running (91s) 12s ago 90s 17.5M - 18.0.0-3908-g05a43c8f 2d174a053211 e1aa27419eb0 2023-10-10T17:41:57.680 INFO:teuthology.orchestra.run.smithi078.stdout:mds.i smithi190 running (88s) 12s ago 88s 19.5M - 18.0.0-3908-g05a43c8f 2d174a053211 c770cd9e493e 2023-10-10T17:41:57.680 INFO:teuthology.orchestra.run.smithi078.stdout:mds.j smithi078 running (101s) 11s ago 101s 15.9M - 18.0.0-3908-g05a43c8f 2d174a053211 d26cd2cd8f32 2023-10-10T17:41:57.681 INFO:teuthology.orchestra.run.smithi078.stdout:mds.k smithi157 running (104s) 12s ago 103s 14.9M - 18.0.0-3908-g05a43c8f 2d174a053211 644474bea764 2023-10-10T17:41:57.681 INFO:teuthology.orchestra.run.smithi078.stdout:mds.l smithi190 running (106s) 12s ago 105s 14.7M - 18.0.0-3908-g05a43c8f 2d174a053211 65090a020cc4 2023-10-10T17:41:57.681 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi078 *:9283,8765 running (8m) 11s ago 8m 516M - 18.0.0-3908-g05a43c8f 2d174a053211 49476fedb0b9 2023-10-10T17:41:57.681 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi157 *:8443,8765 running (6m) 12s ago 6m 447M - 18.0.0-3908-g05a43c8f 2d174a053211 71f05455bfb5 2023-10-10T17:41:57.681 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.z smithi190 *:8443,8765 running (6m) 12s ago 6m 446M - 18.0.0-3908-g05a43c8f 2d174a053211 26af3b784537 2023-10-10T17:41:57.682 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (8m) 11s ago 8m 93.0M 2048M 18.0.0-3908-g05a43c8f 2d174a053211 d45cef894d1a 2023-10-10T17:41:57.682 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi157 running (6m) 12s ago 6m 57.3M 2048M 18.0.0-3908-g05a43c8f 2d174a053211 7e7ac842ef13 2023-10-10T17:41:57.682 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi190 running (6m) 12s ago 6m 59.3M 2048M 18.0.0-3908-g05a43c8f 2d174a053211 8cb7d5cc38e1 2023-10-10T17:41:57.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6m) 11s ago 6m 71.6M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 b097f2f2b766 2023-10-10T17:41:57.683 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi157 running (5m) 12s ago 5m 75.5M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 211412a88d0b 2023-10-10T17:41:57.683 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi190 running (5m) 12s ago 5m 72.6M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 6235858c34e8 2023-10-10T17:41:57.683 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (4m) 11s ago 4m 68.1M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 d9c0297ec4f3 2023-10-10T17:41:57.683 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi157 running (4m) 12s ago 4m 64.2M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 80ce72aeab2d 2023-10-10T17:41:57.683 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi190 running (4m) 12s ago 4m 68.7M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 97744a722103 2023-10-10T17:41:57.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi078 running (3m) 11s ago 3m 61.8M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 60a5387e7621 2023-10-10T17:41:57.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi157 running (3m) 12s ago 3m 64.2M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 5ca0d4e637bc 2023-10-10T17:41:57.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi190 running (3m) 12s ago 3m 62.8M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 20c9476d1bbb 2023-10-10T17:41:57.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (2m) 11s ago 2m 59.5M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 6098c08f4940 2023-10-10T17:41:57.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi157 running (2m) 12s ago 2m 61.2M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 fef448e89959 2023-10-10T17:41:57.685 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi190 running (2m) 12s ago 2m 54.3M 4292M 18.0.0-3908-g05a43c8f 2d174a053211 f5eb0b0e0b4b 2023-10-10T17:41:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f41ac0a1150 msgr2=0x7f41ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f41ac0a1150 0x7f41ac0a3600 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f41b0005fd0 tx=0x7f41b0005ee0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ffde0 msgr2=0x7f41c019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:41:57.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ffde0 0x7f41c019adb0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f41bc000c00 tx=0x7f41bc00f710 comp rx=0 tx=0).stop 2023-10-10T17:41:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 shutdown_connections 2023-10-10T17:41:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f41ac0a1150 0x7f41ac0a3600 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f41c0100860 0x7f41c019d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f41c00ff430 0x7f41c00fee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 --2- 172.21.15.78:0/209610945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f41c00ffde0 0x7f41c019adb0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:41:57.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.679+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 >> 172.21.15.78:0/209610945 conn(0x7f41c00f9f70 msgr2=0x7f41c0103920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:41:57.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.680+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 shutdown_connections 2023-10-10T17:41:57.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:41:57.680+0000 7f41c86e2700 1 -- 172.21.15.78:0/209610945 wait complete. 2023-10-10T17:41:57.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:57 smithi078 ceph-mon[99494]: pgmap v301: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:57.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:57 smithi190 ceph-mon[101220]: pgmap v301: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:57.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:57 smithi157 ceph-mon[100896]: pgmap v301: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:41:58.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-10-10T17:41:58.701 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:41:59.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:59 smithi078 ceph-mon[99494]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:41:59.898 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:41:59 smithi078 ceph-mon[99494]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:41:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:59 smithi190 ceph-mon[101220]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:41:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:41:59 smithi190 ceph-mon[101220]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:41:59.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:59 smithi157 ceph-mon[100896]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:41:59.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:41:59 smithi157 ceph-mon[100896]: pgmap v302: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:42:00.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 -- 172.21.15.78:0/205366455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff0100d70 msgr2=0x7f3ff0107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:00.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/205366455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff0100d70 0x7f3ff0107630 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f3fe800b0a0 tx=0x7f3fe800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 -- 172.21.15.78:0/205366455 shutdown_connections 2023-10-10T17:42:00.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/205366455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff00ff1e0 0x7f3ff00ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/205366455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff0100d70 0x7f3ff0107630 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/205366455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ff01003e0 0x7f3ff0100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 -- 172.21.15.78:0/205366455 >> 172.21.15.78:0/205366455 conn(0x7f3ff00fa7f0 msgr2=0x7f3ff00fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:00.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 -- 172.21.15.78:0/205366455 shutdown_connections 2023-10-10T17:42:00.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.113+0000 7f3ff6d89700 1 -- 172.21.15.78:0/205366455 wait complete. 2023-10-10T17:42:00.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 Processor -- start 2023-10-10T17:42:00.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 -- start start 2023-10-10T17:42:00.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff00ff1e0 0x7f3ff0194f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:00.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff01003e0 0x7f3ff0195480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:00.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ff0100d70 0x7f3ff019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:00.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff019cad0 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff019cc10 con 0x7f3ff0100d70 2023-10-10T17:42:00.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff6d89700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ff019cda0 con 0x7f3ff01003e0 2023-10-10T17:42:00.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff00ff1e0 0x7f3ff0194f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:00.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff00ff1e0 0x7f3ff0194f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43444/0 (socket says 172.21.15.78:43444) 2023-10-10T17:42:00.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 -- 172.21.15.78:0/3479568154 learned_addr learned my addr 172.21.15.78:0/3479568154 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:00.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3feffff700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff01003e0 0x7f3ff0195480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:00.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 -- 172.21.15.78:0/3479568154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff01003e0 msgr2=0x7f3ff0195480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:00.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff01003e0 0x7f3ff0195480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff5326700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ff0100d70 0x7f3ff019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:00.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 -- 172.21.15.78:0/3479568154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ff0100d70 msgr2=0x7f3ff019c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:00.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ff0100d70 0x7f3ff019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3ff4b25700 1 -- 172.21.15.78:0/3479568154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8009d00 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.114+0000 7f3feffff700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff01003e0 0x7f3ff0195480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:00.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.115+0000 7f3ff4b25700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff00ff1e0 0x7f3ff0194f40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4007b80 tx=0x7f3fe4007f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:00.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.115+0000 7f3fedffb700 1 -- 172.21.15.78:0/3479568154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4011070 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.115+0000 7f3fedffb700 1 -- 172.21.15.78:0/3479568154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3fe400c2e0 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.115+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ff010ab30 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.115+0000 7f3fedffb700 1 -- 172.21.15.78:0/3479568154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe400e600 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.115+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ff010a9a0 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.116+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ff0066ea0 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.117+0000 7f3fedffb700 1 -- 172.21.15.78:0/3479568154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3fe4003f20 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.117+0000 7f3fedffb700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3fe00a1150 0x7f3fe00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:00.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.117+0000 7f3feffff700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3fe00a1150 0x7f3fe00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:00.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.117+0000 7f3fedffb700 1 -- 172.21.15.78:0/3479568154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3fe40ce020 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.118+0000 7f3feffff700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3fe00a1150 0x7f3fe00a3600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc005fd0 tx=0x7f3fdc005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:00.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.122+0000 7f3fedffb700 1 -- 172.21.15.78:0/3479568154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3fe40974f0 con 0x7f3ff00ff1e0 2023-10-10T17:42:00.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.317+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ff00692a0 con 0x7f3fe00a1150 2023-10-10T17:42:00.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.322+0000 7f3fedffb700 1 -- 172.21.15.78:0/3479568154 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7f3ff00692a0 con 0x7f3fe00a1150 2023-10-10T17:42:00.323 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-10-10T17:42:00.323 INFO:teuthology.orchestra.run.smithi078.stdout:mds.all 12/12 14s ago 110s smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi157=b;smithi157=e;smithi157=h;smithi157=k;smithi190=c;smithi190=f;smithi190=i;smithi190=l;count:12 2023-10-10T17:42:00.323 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 3/3 14s ago 6m smithi078=x;smithi157=y;smithi190=z;count:3 2023-10-10T17:42:00.324 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 14s ago 6m smithi078:172.21.15.78=a;smithi157:172.21.15.157=b;smithi190:172.21.15.190=c;count:3 2023-10-10T17:42:00.324 INFO:teuthology.orchestra.run.smithi078.stdout:osd 12 14s ago - 2023-10-10T17:42:00.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.324+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3fe00a1150 msgr2=0x7f3fe00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:00.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.324+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3fe00a1150 0x7f3fe00a3600 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc005fd0 tx=0x7f3fdc005e30 comp rx=0 tx=0).stop 2023-10-10T17:42:00.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.324+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff00ff1e0 msgr2=0x7f3ff0194f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.324+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff00ff1e0 0x7f3ff0194f40 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4007b80 tx=0x7f3fe4007f40 comp rx=0 tx=0).stop 2023-10-10T17:42:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 shutdown_connections 2023-10-10T17:42:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f3fe00a1150 0x7f3fe00a3600 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3ff0100d70 0x7f3ff019c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3ff00ff1e0 0x7f3ff0194f40 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 --2- 172.21.15.78:0/3479568154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3ff01003e0 0x7f3ff0195480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:00.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 >> 172.21.15.78:0/3479568154 conn(0x7f3ff00fa7f0 msgr2=0x7f3ff0103800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:00.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 shutdown_connections 2023-10-10T17:42:00.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:00.325+0000 7f3ff6d89700 1 -- 172.21.15.78:0/3479568154 wait complete. 2023-10-10T17:42:00.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-10-10T17:42:01.597 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:42:01.856 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:01 smithi078 ceph-mon[99494]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:01.856 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:01 smithi078 ceph-mon[99494]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:01.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:01 smithi190 ceph-mon[101220]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:01.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:01 smithi190 ceph-mon[101220]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:01 smithi157 ceph-mon[100896]: pgmap v303: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:01.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:01 smithi157 ceph-mon[100896]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:02.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 -- 172.21.15.78:0/234494682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 msgr2=0x7fa5080ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:02.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/234494682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5080ff8f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8009990 tx=0x7fa4f8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:02.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 -- 172.21.15.78:0/234494682 shutdown_connections 2023-10-10T17:42:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/234494682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ffe30 0x7fa5081076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/234494682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5081008b0 0x7fa508100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/234494682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5080ff8f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:02.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 -- 172.21.15.78:0/234494682 >> 172.21.15.78:0/234494682 conn(0x7fa5080fa000 msgr2=0x7fa5080fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:02.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 -- 172.21.15.78:0/234494682 shutdown_connections 2023-10-10T17:42:02.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.888+0000 7fa50d5c6700 1 -- 172.21.15.78:0/234494682 wait complete. 2023-10-10T17:42:02.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 Processor -- start 2023-10-10T17:42:02.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 -- start start 2023-10-10T17:42:02.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5081960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:02.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ffe30 0x7fa508196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:02.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5081008b0 0x7fa50819d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:02.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50819dc60 con 0x7fa5080ff480 2023-10-10T17:42:02.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50819dda0 con 0x7fa5081008b0 2023-10-10T17:42:02.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa50d5c6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa50819df30 con 0x7fa5080ffe30 2023-10-10T17:42:02.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa506ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5081960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:02.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa506ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5081960d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43458/0 (socket says 172.21.15.78:43458) 2023-10-10T17:42:02.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa506ffd700 1 -- 172.21.15.78:0/2506451320 learned_addr learned my addr 172.21.15.78:0/2506451320 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:02.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa5077fe700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5081008b0 0x7fa50819d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:02.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.889+0000 7fa5067fc700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ffe30 0x7fa508196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:02.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa506ffd700 1 -- 172.21.15.78:0/2506451320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ffe30 msgr2=0x7fa508196610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:02.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa506ffd700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ffe30 0x7fa508196610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:02.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa506ffd700 1 -- 172.21.15.78:0/2506451320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5081008b0 msgr2=0x7fa50819d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:02.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa506ffd700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5081008b0 0x7fa50819d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:02.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa506ffd700 1 -- 172.21.15.78:0/2506451320 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80095f0 con 0x7fa5080ff480 2023-10-10T17:42:02.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa5067fc700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ffe30 0x7fa508196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:02.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa506ffd700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5081960d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00b560 tx=0x7fa4fc00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:02.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa4effff700 1 -- 172.21.15.78:0/2506451320 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc011830 con 0x7fa5080ff480 2023-10-10T17:42:02.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa4effff700 1 -- 172.21.15.78:0/2506451320 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4fc011e50 con 0x7fa5080ff480 2023-10-10T17:42:02.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa50810ac80 con 0x7fa5080ff480 2023-10-10T17:42:02.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa4effff700 1 -- 172.21.15.78:0/2506451320 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc00f2e0 con 0x7fa5080ff480 2023-10-10T17:42:02.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.890+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa50810aaf0 con 0x7fa5080ff480 2023-10-10T17:42:02.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.891+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5081072d0 con 0x7fa5080ff480 2023-10-10T17:42:02.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.892+0000 7fa4effff700 1 -- 172.21.15.78:0/2506451320 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa4fc011990 con 0x7fa5080ff480 2023-10-10T17:42:02.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.893+0000 7fa4effff700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4f00a1150 0x7fa4f00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:02.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.893+0000 7fa5067fc700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4f00a1150 0x7fa4f00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:02.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.893+0000 7fa4effff700 1 -- 172.21.15.78:0/2506451320 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4fc0ccd30 con 0x7fa5080ff480 2023-10-10T17:42:02.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.893+0000 7fa5067fc700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4f00a1150 0x7fa4f00a3600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800b5c0 tx=0x7fa4f8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:02.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:02.896+0000 7fa4effff700 1 -- 172.21.15.78:0/2506451320 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa4fc096110 con 0x7fa5080ff480 2023-10-10T17:42:03.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.089+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5080619f0 con 0x7fa4f00a1150 2023-10-10T17:42:03.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.091+0000 7fa4effff700 1 -- 172.21.15.78:0/2506451320 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fa5080619f0 con 0x7fa4f00a1150 2023-10-10T17:42:03.091 INFO:teuthology.orchestra.run.smithi078.stdout:HOST ADDR LABELS STATUS 2023-10-10T17:42:03.092 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 172.21.15.78 2023-10-10T17:42:03.092 INFO:teuthology.orchestra.run.smithi078.stdout:smithi157 172.21.15.157 2023-10-10T17:42:03.092 INFO:teuthology.orchestra.run.smithi078.stdout:smithi190 172.21.15.190 2023-10-10T17:42:03.092 INFO:teuthology.orchestra.run.smithi078.stdout:3 hosts in cluster 2023-10-10T17:42:03.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4f00a1150 msgr2=0x7fa4f00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:03.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4f00a1150 0x7fa4f00a3600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa4f800b5c0 tx=0x7fa4f8005fb0 comp rx=0 tx=0).stop 2023-10-10T17:42:03.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 msgr2=0x7fa5081960d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:03.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5081960d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc00b560 tx=0x7fa4fc00b920 comp rx=0 tx=0).stop 2023-10-10T17:42:03.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 shutdown_connections 2023-10-10T17:42:03.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa4f00a1150 0x7fa4f00a3600 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:03.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5081008b0 0x7fa50819d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:03.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5080ff480 0x7fa5081960d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:03.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 --2- 172.21.15.78:0/2506451320 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5080ffe30 0x7fa508196610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:03.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 >> 172.21.15.78:0/2506451320 conn(0x7fa5080fa000 msgr2=0x7fa5080fc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:03.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 shutdown_connections 2023-10-10T17:42:03.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:03.093+0000 7fa50d5c6700 1 -- 172.21.15.78:0/2506451320 wait complete. 2023-10-10T17:42:03.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-10-10T17:42:03.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:03 smithi078 ceph-mon[99494]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:03.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:03 smithi190 ceph-mon[101220]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:03.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:03 smithi157 ceph-mon[100896]: pgmap v304: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:04.203 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:42:04.743 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:04 smithi078 ceph-mon[99494]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:04.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:04 smithi190 ceph-mon[101220]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:04.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:04 smithi157 ceph-mon[100896]: from='client.15255 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.504+0000 7f9db8bb9700 1 -- 172.21.15.78:0/1214562128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db4100d70 msgr2=0x7f9db4107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:05.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.504+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/1214562128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db4100d70 0x7f9db4107630 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9dac00b0a0 tx=0x7f9dac00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.504+0000 7f9db8bb9700 1 -- 172.21.15.78:0/1214562128 shutdown_connections 2023-10-10T17:42:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.504+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/1214562128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db40ff1e0 0x7f9db40ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.504+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/1214562128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db4100d70 0x7f9db4107630 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.504+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/1214562128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db41003e0 0x7f9db4100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.504+0000 7f9db8bb9700 1 -- 172.21.15.78:0/1214562128 >> 172.21.15.78:0/1214562128 conn(0x7f9db40fa7f0 msgr2=0x7f9db40fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.505+0000 7f9db8bb9700 1 -- 172.21.15.78:0/1214562128 shutdown_connections 2023-10-10T17:42:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.505+0000 7f9db8bb9700 1 -- 172.21.15.78:0/1214562128 wait complete. 2023-10-10T17:42:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.505+0000 7f9db8bb9700 1 Processor -- start 2023-10-10T17:42:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.505+0000 7f9db8bb9700 1 -- start start 2023-10-10T17:42:05.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db40ff1e0 0x7f9db4196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:05.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db41003e0 0x7f9db41966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:05.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db4100d70 0x7f9db419d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:05.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419dd00 con 0x7f9db41003e0 2023-10-10T17:42:05.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419de40 con 0x7f9db40ff1e0 2023-10-10T17:42:05.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9db419dfd0 con 0x7f9db4100d70 2023-10-10T17:42:05.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db41003e0 0x7f9db41966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:05.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db41003e0 0x7f9db41966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43472/0 (socket says 172.21.15.78:43472) 2023-10-10T17:42:05.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 -- 172.21.15.78:0/3268008295 learned_addr learned my addr 172.21.15.78:0/3268008295 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:05.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db2d9d700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db4100d70 0x7f9db419d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:05.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db259c700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db40ff1e0 0x7f9db4196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 -- 172.21.15.78:0/3268008295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db4100d70 msgr2=0x7f9db419d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db4100d70 0x7f9db419d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 -- 172.21.15.78:0/3268008295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db40ff1e0 msgr2=0x7f9db4196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db40ff1e0 0x7f9db4196170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 -- 172.21.15.78:0/3268008295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dac009d00 con 0x7f9db41003e0 2023-10-10T17:42:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db259c700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db40ff1e0 0x7f9db4196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db1d9b700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db41003e0 0x7f9db41966b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9da400d980 tx=0x7f9da400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9da37fe700 1 -- 172.21.15.78:0/3268008295 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4009b40 con 0x7f9db41003e0 2023-10-10T17:42:05.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9da37fe700 1 -- 172.21.15.78:0/3268008295 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9da400b6e0 con 0x7f9db41003e0 2023-10-10T17:42:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9db410abd0 con 0x7f9db41003e0 2023-10-10T17:42:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9da37fe700 1 -- 172.21.15.78:0/3268008295 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da400f620 con 0x7f9db41003e0 2023-10-10T17:42:05.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.506+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9db410aa40 con 0x7f9db41003e0 2023-10-10T17:42:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.507+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9db4066ea0 con 0x7f9db41003e0 2023-10-10T17:42:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.509+0000 7f9da37fe700 1 -- 172.21.15.78:0/3268008295 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9da4010460 con 0x7f9db41003e0 2023-10-10T17:42:05.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.509+0000 7f9da37fe700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d9c0a1150 0x7f9d9c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:05.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.509+0000 7f9db259c700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d9c0a1150 0x7f9d9c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:05.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.509+0000 7f9da37fe700 1 -- 172.21.15.78:0/3268008295 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9da40ccec0 con 0x7f9db41003e0 2023-10-10T17:42:05.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.510+0000 7f9db259c700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d9c0a1150 0x7f9d9c0a3600 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9db4100240 tx=0x7f9da8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:05.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.513+0000 7f9da37fe700 1 -- 172.21.15.78:0/3268008295 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9da4096300 con 0x7f9db41003e0 2023-10-10T17:42:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.710+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9db40fb6f0 con 0x7f9d9c0a1150 2023-10-10T17:42:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.711+0000 7f9da37fe700 1 -- 172.21.15.78:0/3268008295 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f9db40fb6f0 con 0x7f9d9c0a1150 2023-10-10T17:42:05.712 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-10-10T17:42:05.712 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QR400BGN 372G No 2m ago LVM detected, locked 2023-10-10T17:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout:smithi157 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002V480DGN 447G No 2m ago LVM detected, locked 2023-10-10T17:42:05.713 INFO:teuthology.orchestra.run.smithi078.stdout:smithi190 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT63210012400BGN 372G No 2m ago LVM detected, locked 2023-10-10T17:42:05.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d9c0a1150 msgr2=0x7f9d9c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d9c0a1150 0x7f9d9c0a3600 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9db4100240 tx=0x7f9da8008040 comp rx=0 tx=0).stop 2023-10-10T17:42:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db41003e0 msgr2=0x7f9db41966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db41003e0 0x7f9db41966b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9da400d980 tx=0x7f9da400dc90 comp rx=0 tx=0).stop 2023-10-10T17:42:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 shutdown_connections 2023-10-10T17:42:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f9d9c0a1150 0x7f9d9c0a3600 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9db4100d70 0x7f9db419d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9db40ff1e0 0x7f9db4196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 --2- 172.21.15.78:0/3268008295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9db41003e0 0x7f9db41966b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 >> 172.21.15.78:0/3268008295 conn(0x7f9db40fa7f0 msgr2=0x7f9db40fafe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 shutdown_connections 2023-10-10T17:42:05.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:05.714+0000 7f9db8bb9700 1 -- 172.21.15.78:0/3268008295 wait complete. 2023-10-10T17:42:05.860 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:05 smithi078 ceph-mon[99494]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:05.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:05 smithi190 ceph-mon[101220]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:05.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:05 smithi157 ceph-mon[100896]: pgmap v305: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:06.163 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-10-10T17:42:06.174 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:42:06.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- bash -c 'ceph fs dump' 2023-10-10T17:42:06.817 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:42:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:07 smithi078 ceph-mon[99494]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:07.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:07 smithi078 ceph-mon[99494]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:07.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:07 smithi190 ceph-mon[101220]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:07.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:07 smithi190 ceph-mon[101220]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:07.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:07 smithi157 ceph-mon[100896]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:07.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:07 smithi157 ceph-mon[100896]: pgmap v306: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:08.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 -- 172.21.15.78:0/474188531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 msgr2=0x7f9974100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:08.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 --2- 172.21.15.78:0/474188531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974100930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9968009990 tx=0x7f9968009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 -- 172.21.15.78:0/474188531 shutdown_connections 2023-10-10T17:42:08.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 --2- 172.21.15.78:0/474188531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99741020b0 0x7f9974108970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 --2- 172.21.15.78:0/474188531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974100930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 --2- 172.21.15.78:0/474188531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9974101720 0x7f9974101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 -- 172.21.15.78:0/474188531 >> 172.21.15.78:0/474188531 conn(0x7f99740fbb50 msgr2=0x7f99740fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:08.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 -- 172.21.15.78:0/474188531 shutdown_connections 2023-10-10T17:42:08.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 -- 172.21.15.78:0/474188531 wait complete. 2023-10-10T17:42:08.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.040+0000 7f9979e32700 1 Processor -- start 2023-10-10T17:42:08.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9979e32700 1 -- start start 2023-10-10T17:42:08.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9979e32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974074890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:08.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9979e32700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9974101720 0x7f9974072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:08.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9979e32700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99741020b0 0x7f9974073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:08.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9979e32700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974073ba0 con 0x7f9974100520 2023-10-10T17:42:08.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9979e32700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974073d10 con 0x7f9974101720 2023-10-10T17:42:08.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9979e32700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99741a02c0 con 0x7f99741020b0 2023-10-10T17:42:08.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974074890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:08.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974074890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43482/0 (socket says 172.21.15.78:43482) 2023-10-10T17:42:08.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 -- 172.21.15.78:0/2510343945 learned_addr learned my addr 172.21.15.78:0/2510343945 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:08.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9972ffd700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9974101720 0x7f9974072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:08.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9973fff700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99741020b0 0x7f9974073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:08.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 -- 172.21.15.78:0/2510343945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99741020b0 msgr2=0x7f9974073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:08.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99741020b0 0x7f9974073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 -- 172.21.15.78:0/2510343945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9974101720 msgr2=0x7f9974072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:08.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9974101720 0x7f9974072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 -- 172.21.15.78:0/2510343945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99680095f0 con 0x7f9974100520 2023-10-10T17:42:08.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f99737fe700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974074890 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f99680098e0 tx=0x7f996800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:08.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9970ff9700 1 -- 172.21.15.78:0/2510343945 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996801d070 con 0x7f9974100520 2023-10-10T17:42:08.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.041+0000 7f9970ff9700 1 -- 172.21.15.78:0/2510343945 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9968022470 con 0x7f9974100520 2023-10-10T17:42:08.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.042+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997410be70 con 0x7f9974100520 2023-10-10T17:42:08.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.042+0000 7f9970ff9700 1 -- 172.21.15.78:0/2510343945 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9968017650 con 0x7f9974100520 2023-10-10T17:42:08.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.042+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997410bce0 con 0x7f9974100520 2023-10-10T17:42:08.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.042+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9974107b60 con 0x7f9974100520 2023-10-10T17:42:08.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.043+0000 7f9970ff9700 1 -- 172.21.15.78:0/2510343945 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f99680225e0 con 0x7f9974100520 2023-10-10T17:42:08.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.044+0000 7f9970ff9700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f995c0a1150 0x7f995c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:08.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.044+0000 7f9972ffd700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f995c0a1150 0x7f995c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:08.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.044+0000 7f9970ff9700 1 -- 172.21.15.78:0/2510343945 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f99680ce0b0 con 0x7f9974100520 2023-10-10T17:42:08.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.045+0000 7f9972ffd700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f995c0a1150 0x7f995c0a3600 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9964005950 tx=0x7f996400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:08.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.048+0000 7f9970ff9700 1 -- 172.21.15.78:0/2510343945 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9968097580 con 0x7f9974100520 2023-10-10T17:42:08.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.278+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f997402cfe0 con 0x7f9974100520 2023-10-10T17:42:08.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.278+0000 7f9970ff9700 1 -- 172.21.15.78:0/2510343945 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2900 (secure 0 0 0) 0x7f996809aba0 con 0x7f9974100520 2023-10-10T17:42:08.279 INFO:teuthology.orchestra.run.smithi078.stdout:e24 2023-10-10T17:42:08.279 INFO:teuthology.orchestra.run.smithi078.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-10-10T17:42:08.280 INFO:teuthology.orchestra.run.smithi078.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-10T17:42:08.280 INFO:teuthology.orchestra.run.smithi078.stdout:legacy client fscid: 1 2023-10-10T17:42:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:08.280 INFO:teuthology.orchestra.run.smithi078.stdout:Filesystem 'cephfs' (1) 2023-10-10T17:42:08.280 INFO:teuthology.orchestra.run.smithi078.stdout:fs_name cephfs 2023-10-10T17:42:08.281 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 24 2023-10-10T17:42:08.281 INFO:teuthology.orchestra.run.smithi078.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-10-10T17:42:08.281 INFO:teuthology.orchestra.run.smithi078.stdout:created 2023-10-10T17:40:41.574125+0000 2023-10-10T17:42:08.281 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2023-10-10T17:41:01.590420+0000 2023-10-10T17:42:08.281 INFO:teuthology.orchestra.run.smithi078.stdout:tableserver 0 2023-10-10T17:42:08.282 INFO:teuthology.orchestra.run.smithi078.stdout:root 0 2023-10-10T17:42:08.282 INFO:teuthology.orchestra.run.smithi078.stdout:session_timeout 300 2023-10-10T17:42:08.282 INFO:teuthology.orchestra.run.smithi078.stdout:session_autoclose 300 2023-10-10T17:42:08.282 INFO:teuthology.orchestra.run.smithi078.stdout:max_file_size 1099511627776 2023-10-10T17:42:08.282 INFO:teuthology.orchestra.run.smithi078.stdout:max_xattr_size 65536 2023-10-10T17:42:08.283 INFO:teuthology.orchestra.run.smithi078.stdout:required_client_features {} 2023-10-10T17:42:08.283 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure 0 2023-10-10T17:42:08.283 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure_osd_epoch 0 2023-10-10T17:42:08.283 INFO:teuthology.orchestra.run.smithi078.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-10-10T17:42:08.283 INFO:teuthology.orchestra.run.smithi078.stdout:max_mds 3 2023-10-10T17:42:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:in 0,1,2 2023-10-10T17:42:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:up {0=24500,1=24490,2=24472} 2023-10-10T17:42:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:failed 2023-10-10T17:42:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:damaged 2023-10-10T17:42:08.284 INFO:teuthology.orchestra.run.smithi078.stdout:stopped 2023-10-10T17:42:08.285 INFO:teuthology.orchestra.run.smithi078.stdout:data_pools [3,4] 2023-10-10T17:42:08.285 INFO:teuthology.orchestra.run.smithi078.stdout:metadata_pool 2 2023-10-10T17:42:08.285 INFO:teuthology.orchestra.run.smithi078.stdout:inline_data disabled 2023-10-10T17:42:08.285 INFO:teuthology.orchestra.run.smithi078.stdout:balancer 2023-10-10T17:42:08.286 INFO:teuthology.orchestra.run.smithi078.stdout:bal_rank_mask -1 2023-10-10T17:42:08.286 INFO:teuthology.orchestra.run.smithi078.stdout:standby_count_wanted 0 2023-10-10T17:42:08.286 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b{0:24500} state up:active seq 3 addr [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.286 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.h{0:24467} state up:standby-replay seq 1 addr [v2:172.21.15.157:6836/1666360072,v1:172.21.15.157:6837/1666360072] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.286 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.i{1:24490} state up:active seq 9 addr [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.287 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.f{1:24460} state up:standby-replay seq 1 addr [v2:172.21.15.190:6834/1269112994,v1:172.21.15.190:6835/1269112994] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.287 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c{2:24472} state up:active seq 10 addr [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.287 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.e{2:24437} state up:standby-replay seq 1 addr [v2:172.21.15.157:6834/2501992479,v1:172.21.15.157:6835/2501992479] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:08.288 INFO:teuthology.orchestra.run.smithi078.stdout:Standby daemons: 2023-10-10T17:42:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:08.288 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.j{-1:14559} state up:standby seq 1 addr [v2:172.21.15.78:6834/1904698406,v1:172.21.15.78:6835/1904698406] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.288 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d{-1:14565} state up:standby seq 1 addr [v2:172.21.15.78:6836/4004555102,v1:172.21.15.78:6837/4004555102] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.288 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.g{-1:14625} state up:standby seq 1 addr [v2:172.21.15.78:6838/943709499,v1:172.21.15.78:6839/943709499] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.289 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a{-1:14637} state up:standby seq 1 addr [v2:172.21.15.78:6840/504515550,v1:172.21.15.78:6841/504515550] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.289 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.l{-1:24418} state up:standby seq 1 addr [v2:172.21.15.190:6832/1708598976,v1:172.21.15.190:6833/1708598976] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.289 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.k{-1:24419} state up:standby seq 1 addr [v2:172.21.15.157:6832/1712500424,v1:172.21.15.157:6833/1712500424] compat {c=[1],r=[1],i=[7ff]}] 2023-10-10T17:42:08.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f995c0a1150 msgr2=0x7f995c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:08.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f995c0a1150 0x7f995c0a3600 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9964005950 tx=0x7f996400b410 comp rx=0 tx=0).stop 2023-10-10T17:42:08.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 msgr2=0x7f9974074890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974074890 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f99680098e0 tx=0x7f996800f690 comp rx=0 tx=0).stop 2023-10-10T17:42:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 shutdown_connections 2023-10-10T17:42:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f995c0a1150 0x7f995c0a3600 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f99741020b0 0x7f9974073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9974100520 0x7f9974074890 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 --2- 172.21.15.78:0/2510343945 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9974101720 0x7f9974072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 >> 172.21.15.78:0/2510343945 conn(0x7f99740fbb50 msgr2=0x7f9974104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.280+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 shutdown_connections 2023-10-10T17:42:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:08.281+0000 7f9979e32700 1 -- 172.21.15.78:0/2510343945 wait complete. 2023-10-10T17:42:08.292 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:42:08.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- bash -c 'ceph osd dump' 2023-10-10T17:42:08.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:08 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2510343945' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-10-10T17:42:08.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:08 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2510343945' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-10-10T17:42:08.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:08 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2510343945' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-10-10T17:42:09.319 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:42:09.850 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:09 smithi078 ceph-mon[99494]: pgmap v307: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:42:09.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:09 smithi190 ceph-mon[101220]: pgmap v307: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:42:09.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:09 smithi157 ceph-mon[100896]: pgmap v307: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:42:10.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 -- 172.21.15.78:0/2860193911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ffe30 msgr2=0x7f36cc1076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:10.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 --2- 172.21.15.78:0/2860193911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ffe30 0x7f36cc1076e0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f36c400b0d0 tx=0x7f36c400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 -- 172.21.15.78:0/2860193911 shutdown_connections 2023-10-10T17:42:10.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 --2- 172.21.15.78:0/2860193911 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36cc100920 0x7f36cc100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 --2- 172.21.15.78:0/2860193911 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ffe30 0x7f36cc1076e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 --2- 172.21.15.78:0/2860193911 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc0ff480 0x7f36cc0ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 -- 172.21.15.78:0/2860193911 >> 172.21.15.78:0/2860193911 conn(0x7f36cc0f9fe0 msgr2=0x7f36cc0fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:10.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 -- 172.21.15.78:0/2860193911 shutdown_connections 2023-10-10T17:42:10.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 -- 172.21.15.78:0/2860193911 wait complete. 2023-10-10T17:42:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 Processor -- start 2023-10-10T17:42:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.554+0000 7f36d4721700 1 -- start start 2023-10-10T17:42:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d4721700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ff480 0x7f36cc0feed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d4721700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36cc0ffe30 0x7f36cc19af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:10.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d4721700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc100920 0x7f36cc19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d4721700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc19dd30 con 0x7f36cc0ff480 2023-10-10T17:42:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d4721700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc19dea0 con 0x7f36cc100920 2023-10-10T17:42:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d4721700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36cc19e060 con 0x7f36cc0ffe30 2023-10-10T17:42:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ff480 0x7f36cc0feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:10.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ff480 0x7f36cc0feed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43494/0 (socket says 172.21.15.78:43494) 2023-10-10T17:42:10.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 -- 172.21.15.78:0/493866225 learned_addr learned my addr 172.21.15.78:0/493866225 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:10.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d2cbe700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc100920 0x7f36cc19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:10.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d1cbc700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36cc0ffe30 0x7f36cc19af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:10.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 -- 172.21.15.78:0/493866225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36cc0ffe30 msgr2=0x7f36cc19af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:10.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36cc0ffe30 0x7f36cc19af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 -- 172.21.15.78:0/493866225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc100920 msgr2=0x7f36cc19d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:10.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc100920 0x7f36cc19d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 -- 172.21.15.78:0/493866225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4009d00 con 0x7f36cc0ff480 2023-10-10T17:42:10.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d2cbe700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc100920 0x7f36cc19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:10.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d24bd700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ff480 0x7f36cc0feed0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c970 tx=0x7f36c800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:10.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36c37fe700 1 -- 172.21.15.78:0/493866225 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8004220 con 0x7f36cc0ff480 2023-10-10T17:42:10.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36cc10ad10 con 0x7f36cc0ff480 2023-10-10T17:42:10.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36c37fe700 1 -- 172.21.15.78:0/493866225 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f36c8007d60 con 0x7f36cc0ff480 2023-10-10T17:42:10.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.555+0000 7f36c37fe700 1 -- 172.21.15.78:0/493866225 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8003d00 con 0x7f36cc0ff480 2023-10-10T17:42:10.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.556+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36cc10ab80 con 0x7f36cc0ff480 2023-10-10T17:42:10.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.556+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36cc066ea0 con 0x7f36cc0ff480 2023-10-10T17:42:10.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.558+0000 7f36c37fe700 1 -- 172.21.15.78:0/493866225 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f36c8004380 con 0x7f36cc0ff480 2023-10-10T17:42:10.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.558+0000 7f36c37fe700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36b80a1150 0x7f36b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:10.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.559+0000 7f36d1cbc700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36b80a1150 0x7f36b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:10.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.559+0000 7f36c37fe700 1 -- 172.21.15.78:0/493866225 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f36c80ccb30 con 0x7f36cc0ff480 2023-10-10T17:42:10.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.559+0000 7f36d1cbc700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36b80a1150 0x7f36b80a3600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0ff450 tx=0x7f36bc00b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:10.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.562+0000 7f36c37fe700 1 -- 172.21.15.78:0/493866225 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f36c8095f70 con 0x7f36cc0ff480 2023-10-10T17:42:10.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:10 smithi078 ceph-mon[99494]: pgmap v308: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:10.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.753+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f36cc195970 con 0x7f36cc0ff480 2023-10-10T17:42:10.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.754+0000 7f36c37fe700 1 -- 172.21.15.78:0/493866225 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v91) v1 ==== 56+0+5596 (secure 0 0 0) 0x7f36c8099590 con 0x7f36cc0ff480 2023-10-10T17:42:10.755 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 91 2023-10-10T17:42:10.755 INFO:teuthology.orchestra.run.smithi078.stdout:fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 2023-10-10T17:42:10.756 INFO:teuthology.orchestra.run.smithi078.stdout:created 2023-10-10T17:33:42.140151+0000 2023-10-10T17:42:10.756 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2023-10-10T17:41:37.591297+0000 2023-10-10T17:42:10.756 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-10-10T17:42:10.756 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 27 2023-10-10T17:42:10.756 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2023-10-10T17:42:10.757 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2023-10-10T17:42:10.757 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2023-10-10T17:42:10.757 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2023-10-10T17:42:10.757 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client luminous 2023-10-10T17:42:10.757 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release reef 2023-10-10T17:42:10.758 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2023-10-10T17:42:10.758 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-10-10T17:42:10.758 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2023-10-10T17:42:10.758 INFO:teuthology.orchestra.run.smithi078.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-10-10T17:42:10.758 INFO:teuthology.orchestra.run.smithi078.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-10-10T17:42:10.759 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 12 2023-10-10T17:42:10.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/857914992,v1:172.21.15.78:6803/857914992] [v2:172.21.15.78:6804/857914992,v1:172.21.15.78:6805/857914992] exists,up d74c4b96-ee1f-4f21-9424-d693d38f0335 2023-10-10T17:42:10.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6800/4215285299,v1:172.21.15.157:6801/4215285299] [v2:172.21.15.157:6802/4215285299,v1:172.21.15.157:6803/4215285299] exists,up c01e2eb5-d98b-4e22-9339-06aaa290b7ae 2023-10-10T17:42:10.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6800/678998123,v1:172.21.15.190:6801/678998123] [v2:172.21.15.190:6802/678998123,v1:172.21.15.190:6803/678998123] exists,up 4c48ed5d-d817-45b6-9c2c-1c371df35f95 2023-10-10T17:42:10.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 26 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6810/2152346860,v1:172.21.15.78:6811/2152346860] [v2:172.21.15.78:6812/2152346860,v1:172.21.15.78:6813/2152346860] exists,up 0b2e185b-92ed-4d3e-8d87-e8f34fc640e2 2023-10-10T17:42:10.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 34 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6808/1484677990,v1:172.21.15.157:6809/1484677990] [v2:172.21.15.157:6810/1484677990,v1:172.21.15.157:6811/1484677990] exists,up 11514abf-4ffd-437b-9b35-e92d90c3a963 2023-10-10T17:42:10.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6808/934325798,v1:172.21.15.190:6809/934325798] [v2:172.21.15.190:6810/934325798,v1:172.21.15.190:6811/934325798] exists,up e1a3f83a-3fe5-470e-a55b-76a851412013 2023-10-10T17:42:10.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 up in weight 1 up_from 47 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6818/2340006051,v1:172.21.15.78:6819/2340006051] [v2:172.21.15.78:6820/2340006051,v1:172.21.15.78:6821/2340006051] exists,up 149dd00c-dd70-4d62-91db-f9a6830d918f 2023-10-10T17:42:10.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 up in weight 1 up_from 52 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6816/1149014920,v1:172.21.15.157:6817/1149014920] [v2:172.21.15.157:6818/1149014920,v1:172.21.15.157:6819/1149014920] exists,up 2c72dbb4-e18a-4627-9871-2730cc29fcd9 2023-10-10T17:42:10.761 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 up in weight 1 up_from 60 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6816/2390054900,v1:172.21.15.190:6817/2390054900] [v2:172.21.15.190:6818/2390054900,v1:172.21.15.190:6819/2390054900] exists,up 82521098-f51f-4c32-ad81-5d1c12c67a24 2023-10-10T17:42:10.761 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 up in weight 1 up_from 68 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6826/2313179218,v1:172.21.15.78:6827/2313179218] [v2:172.21.15.78:6828/2313179218,v1:172.21.15.78:6829/2313179218] exists,up 7670b692-faa9-4a49-8d19-32418fa64d2a 2023-10-10T17:42:10.761 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 up in weight 1 up_from 73 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.157:6824/936155371,v1:172.21.15.157:6825/936155371] [v2:172.21.15.157:6826/936155371,v1:172.21.15.157:6827/936155371] exists,up 998b645c-fda2-4969-9917-da58dede0c03 2023-10-10T17:42:10.761 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 up in weight 1 up_from 80 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.190:6824/2501616180,v1:172.21.15.190:6825/2501616180] [v2:172.21.15.190:6826/2501616180,v1:172.21.15.190:6827/2501616180] exists,up d638b1e0-6b53-45f2-9ea0-719cdf4ff100 2023-10-10T17:42:10.761 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.10 [2,4] 2023-10-10T17:42:10.762 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.10 [5,9] 2023-10-10T17:42:10.762 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.18 [3,9] 2023-10-10T17:42:10.762 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.24 [4,9] 2023-10-10T17:42:10.762 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.31 [4,8] 2023-10-10T17:42:10.762 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.3a [4,6] 2023-10-10T17:42:10.763 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.b [9,8] 2023-10-10T17:42:10.763 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.e [5,8] 2023-10-10T17:42:10.763 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.1b [9,10] 2023-10-10T17:42:10.763 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.1c [9,8] 2023-10-10T17:42:10.763 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2447900312 expires 2023-10-11T17:34:36.143528+0000 2023-10-10T17:42:10.764 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/942965343 expires 2023-10-11T17:34:36.143528+0000 2023-10-10T17:42:10.764 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/1662641434 expires 2023-10-11T17:34:36.143528+0000 2023-10-10T17:42:10.764 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/3346226185 expires 2023-10-11T17:34:13.659771+0000 2023-10-10T17:42:10.764 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1720027577 expires 2023-10-11T17:34:36.143528+0000 2023-10-10T17:42:10.764 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/3346226185 expires 2023-10-11T17:34:13.659771+0000 2023-10-10T17:42:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/843244395 expires 2023-10-11T17:34:13.659771+0000 2023-10-10T17:42:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/667827327 expires 2023-10-11T17:34:13.659771+0000 2023-10-10T17:42:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2252830072 expires 2023-10-11T17:34:13.659771+0000 2023-10-10T17:42:10.765 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/1662641434 expires 2023-10-11T17:34:36.143528+0000 2023-10-10T17:42:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36b80a1150 msgr2=0x7f36b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36b80a1150 0x7f36b80a3600 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0ff450 tx=0x7f36bc00b5b0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ff480 msgr2=0x7f36cc0feed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ff480 0x7f36cc0feed0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c970 tx=0x7f36c800cd30 comp rx=0 tx=0).stop 2023-10-10T17:42:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 shutdown_connections 2023-10-10T17:42:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f36b80a1150 0x7f36b80a3600 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f36cc100920 0x7f36cc19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f36cc0ff480 0x7f36cc0feed0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 --2- 172.21.15.78:0/493866225 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f36cc0ffe30 0x7f36cc19af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 >> 172.21.15.78:0/493866225 conn(0x7f36cc0f9fe0 msgr2=0x7f36cc0fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.756+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 shutdown_connections 2023-10-10T17:42:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:10.757+0000 7f36d4721700 1 -- 172.21.15.78:0/493866225 wait complete. 2023-10-10T17:42:10.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:10 smithi190 ceph-mon[101220]: pgmap v308: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:10.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:10 smithi157 ceph-mon[100896]: pgmap v308: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:11.182 INFO:teuthology.run_tasks:Running task fs.ready... 2023-10-10T17:42:11.224 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:42:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 -- 172.21.15.78:0/4169920515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 msgr2=0x7fbc0c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/4169920515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c0fffb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400b0a0 tx=0x7fbc0400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 -- 172.21.15.78:0/4169920515 shutdown_connections 2023-10-10T17:42:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/4169920515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc0c0fe970 0x7fbc0c0fed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/4169920515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc0c1004f0 0x7fbc0c106db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/4169920515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c0fffb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 -- 172.21.15.78:0/4169920515 >> 172.21.15.78:0/4169920515 conn(0x7fbc0c0f9fb0 msgr2=0x7fbc0c0fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 -- 172.21.15.78:0/4169920515 shutdown_connections 2023-10-10T17:42:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.350+0000 7fbc11c7b700 1 -- 172.21.15.78:0/4169920515 wait complete. 2023-10-10T17:42:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 Processor -- start 2023-10-10T17:42:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 -- start start 2023-10-10T17:42:11.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc0c0fe970 0x7fbc0c10b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c19ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc0c1004f0 0x7fbc0c19d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c19db60 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c19dcd0 con 0x7fbc0c1004f0 2023-10-10T17:42:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc11c7b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc0c19de90 con 0x7fbc0c0fe970 2023-10-10T17:42:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c19ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc0affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c19ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43510/0 (socket says 172.21.15.78:43510) 2023-10-10T17:42:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc0affd700 1 -- 172.21.15.78:0/519331556 learned_addr learned my addr 172.21.15.78:0/519331556 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.351+0000 7fbc0bfff700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc0c1004f0 0x7fbc0c19d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0b7fe700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc0c0fe970 0x7fbc0c10b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0affd700 1 -- 172.21.15.78:0/519331556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc0c0fe970 msgr2=0x7fbc0c10b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0affd700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc0c0fe970 0x7fbc0c10b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0affd700 1 -- 172.21.15.78:0/519331556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc0c1004f0 msgr2=0x7fbc0c19d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0affd700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc0c1004f0 0x7fbc0c19d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0affd700 1 -- 172.21.15.78:0/519331556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc04009d00 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0b7fe700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc0c0fe970 0x7fbc0c10b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:11.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc0affd700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c19ad40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d010 tx=0x7fbc04004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:11.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc08ff9700 1 -- 172.21.15.78:0/519331556 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0400e040 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc0c10a2d0 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc08ff9700 1 -- 172.21.15.78:0/519331556 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc0401dd50 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc08ff9700 1 -- 172.21.15.78:0/519331556 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0401be90 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.352+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc0c10a140 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.353+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc0c105540 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.354+0000 7fbc08ff9700 1 -- 172.21.15.78:0/519331556 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fbc0401b550 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.355+0000 7fbc08ff9700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbbf80a1150 0x7fbbf80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.355+0000 7fbc0b7fe700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbbf80a1150 0x7fbbf80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.355+0000 7fbc08ff9700 1 -- 172.21.15.78:0/519331556 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbc040cee10 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.355+0000 7fbc0b7fe700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbbf80a1150 0x7fbbf80a3600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc005fd0 tx=0x7fbbfc005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:11.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.358+0000 7fbc08ff9700 1 -- 172.21.15.78:0/519331556 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fbc04098280 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.550+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc0c067eb0 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.552+0000 7fbc08ff9700 1 -- 172.21.15.78:0/519331556 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7fbc0407ae60 con 0x7fbc0c0ffb60 2023-10-10T17:42:11.553 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:11.553 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:42:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbbf80a1150 msgr2=0x7fbbf80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbbf80a1150 0x7fbbf80a3600 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbbfc005fd0 tx=0x7fbbfc005e40 comp rx=0 tx=0).stop 2023-10-10T17:42:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 msgr2=0x7fbc0c19ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c19ad40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbc0400d010 tx=0x7fbc04004db0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 shutdown_connections 2023-10-10T17:42:11.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fbbf80a1150 0x7fbbf80a3600 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbc0c0fe970 0x7fbc0c10b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbc0c1004f0 0x7fbc0c19d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 --2- 172.21.15.78:0/519331556 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc0c0ffb60 0x7fbc0c19ad40 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 >> 172.21.15.78:0/519331556 conn(0x7fbc0c0f9fb0 msgr2=0x7fbc0c0fb820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:11.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 shutdown_connections 2023-10-10T17:42:11.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.554+0000 7fbc11c7b700 1 -- 172.21.15.78:0/519331556 wait complete. 2023-10-10T17:42:11.567 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:42:11.568 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:42:11.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 -- 172.21.15.78:0/272983276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 msgr2=0x7fb7d0101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/272983276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0101910 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc009990 tx=0x7fb7cc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 -- 172.21.15.78:0/272983276 shutdown_connections 2023-10-10T17:42:11.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/272983276 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7d0101e50 0x7fb7d0108710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/272983276 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d01002d0 0x7fb7d01006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/272983276 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0101910 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 -- 172.21.15.78:0/272983276 >> 172.21.15.78:0/272983276 conn(0x7fb7d00fb8b0 msgr2=0x7fb7d00fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:11.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 -- 172.21.15.78:0/272983276 shutdown_connections 2023-10-10T17:42:11.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.693+0000 7fb7d84ad700 1 -- 172.21.15.78:0/272983276 wait complete. 2023-10-10T17:42:11.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 Processor -- start 2023-10-10T17:42:11.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 -- start start 2023-10-10T17:42:11.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d01002d0 0x7fb7d00748a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7d0101e50 0x7fb7d0073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0073bc0 con 0x7fb7d01014c0 2023-10-10T17:42:11.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d0073d30 con 0x7fb7d01002d0 2023-10-10T17:42:11.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d84ad700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d019ff70 con 0x7fb7d0101e50 2023-10-10T17:42:11.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43524/0 (socket says 172.21.15.78:43524) 2023-10-10T17:42:11.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 -- 172.21.15.78:0/1416872056 learned_addr learned my addr 172.21.15.78:0/1416872056 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:11.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d6249700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d01002d0 0x7fb7d00748a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d6a4a700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7d0101e50 0x7fb7d0073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 -- 172.21.15.78:0/1416872056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7d0101e50 msgr2=0x7fb7d0073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7d0101e50 0x7fb7d0073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 -- 172.21.15.78:0/1416872056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d01002d0 msgr2=0x7fb7d00748a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d01002d0 0x7fb7d00748a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.694+0000 7fb7d5a48700 1 -- 172.21.15.78:0/1416872056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0095f0 con 0x7fb7d01014c0 2023-10-10T17:42:11.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.695+0000 7fb7d5a48700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0072ee0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc004760 tx=0x7fb7cc004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:11.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.695+0000 7fb7c77fe700 1 -- 172.21.15.78:0/1416872056 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc01d070 con 0x7fb7d01014c0 2023-10-10T17:42:11.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.695+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d010bc40 con 0x7fb7d01014c0 2023-10-10T17:42:11.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.695+0000 7fb7c77fe700 1 -- 172.21.15.78:0/1416872056 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7cc0041d0 con 0x7fb7d01014c0 2023-10-10T17:42:11.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.695+0000 7fb7c77fe700 1 -- 172.21.15.78:0/1416872056 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0053a0 con 0x7fb7d01014c0 2023-10-10T17:42:11.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.695+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d010bab0 con 0x7fb7d01014c0 2023-10-10T17:42:11.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.696+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d0107110 con 0x7fb7d01014c0 2023-10-10T17:42:11.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.697+0000 7fb7c77fe700 1 -- 172.21.15.78:0/1416872056 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb7cc005500 con 0x7fb7d01014c0 2023-10-10T17:42:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.697+0000 7fb7c77fe700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb7bc0a1150 0x7fb7bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.698+0000 7fb7d6249700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb7bc0a1150 0x7fb7bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.698+0000 7fb7c77fe700 1 -- 172.21.15.78:0/1416872056 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb7cc0cded0 con 0x7fb7d01014c0 2023-10-10T17:42:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.698+0000 7fb7d6249700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb7bc0a1150 0x7fb7bc0a3600 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0101320 tx=0x7fb7c000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:11.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.702+0000 7fb7c77fe700 1 -- 172.21.15.78:0/1416872056 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fb7cc05b730 con 0x7fb7d01014c0 2023-10-10T17:42:11.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:11 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/493866225' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-10-10T17:42:11.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:11 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/519331556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:11.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.891+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb7d0061c40 con 0x7fb7d01014c0 2023-10-10T17:42:11.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.891+0000 7fb7c77fe700 1 -- 172.21.15.78:0/1416872056 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb7cc0973a0 con 0x7fb7d01014c0 2023-10-10T17:42:11.892 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:42:11.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb7bc0a1150 msgr2=0x7fb7bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb7bc0a1150 0x7fb7bc0a3600 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0101320 tx=0x7fb7c000a3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 msgr2=0x7fb7d0072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:11.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0072ee0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc004760 tx=0x7fb7cc004790 comp rx=0 tx=0).stop 2023-10-10T17:42:11.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 shutdown_connections 2023-10-10T17:42:11.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fb7bc0a1150 0x7fb7bc0a3600 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7d0101e50 0x7fb7d0073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7d01002d0 0x7fb7d00748a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 --2- 172.21.15.78:0/1416872056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7d01014c0 0x7fb7d0072ee0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 >> 172.21.15.78:0/1416872056 conn(0x7fb7d00fb8b0 msgr2=0x7fb7d00fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 shutdown_connections 2023-10-10T17:42:11.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:11.893+0000 7fb7d84ad700 1 -- 172.21.15.78:0/1416872056 wait complete. 2023-10-10T17:42:11.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:42:11.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:11 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/493866225' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-10-10T17:42:11.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:11 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/519331556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:11.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:11 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/493866225' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-10-10T17:42:11.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:11 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/519331556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:12.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.030+0000 7f97bab1f700 1 -- 172.21.15.78:0/4236876977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 msgr2=0x7f97b4100b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.030+0000 7f97bab1f700 1 --2- 172.21.15.78:0/4236876977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b4100b50 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009990 tx=0x7f97a8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.030+0000 7f97bab1f700 1 -- 172.21.15.78:0/4236876977 shutdown_connections 2023-10-10T17:42:12.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.030+0000 7f97bab1f700 1 --2- 172.21.15.78:0/4236876977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40ffe00 0x7f97b41076d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 --2- 172.21.15.78:0/4236876977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b4100b50 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 --2- 172.21.15.78:0/4236876977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b40ff470 0x7f97b40ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 -- 172.21.15.78:0/4236876977 >> 172.21.15.78:0/4236876977 conn(0x7f97b40f9f30 msgr2=0x7f97b40fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:12.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 -- 172.21.15.78:0/4236876977 shutdown_connections 2023-10-10T17:42:12.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 -- 172.21.15.78:0/4236876977 wait complete. 2023-10-10T17:42:12.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 Processor -- start 2023-10-10T17:42:12.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 -- start start 2023-10-10T17:42:12.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40ff470 0x7f97b40feed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b40ffe00 0x7f97b419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b419d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b419dbc0 con 0x7f97b4100740 2023-10-10T17:42:12.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b419dd30 con 0x7f97b40ffe00 2023-10-10T17:42:12.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97bab1f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b419def0 con 0x7f97b40ff470 2023-10-10T17:42:12.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97b90bc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b419d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97b90bc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b419d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43540/0 (socket says 172.21.15.78:43540) 2023-10-10T17:42:12.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.031+0000 7f97b90bc700 1 -- 172.21.15.78:0/2137057281 learned_addr learned my addr 172.21.15.78:0/2137057281 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:12.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b3fff700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b40ffe00 0x7f97b419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b88bb700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40ff470 0x7f97b40feed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b90bc700 1 -- 172.21.15.78:0/2137057281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40ff470 msgr2=0x7f97b40feed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b90bc700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40ff470 0x7f97b40feed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b90bc700 1 -- 172.21.15.78:0/2137057281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b40ffe00 msgr2=0x7f97b419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b90bc700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b40ffe00 0x7f97b419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b90bc700 1 -- 172.21.15.78:0/2137057281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80095f0 con 0x7f97b4100740 2023-10-10T17:42:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b90bc700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b419d420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00ef30 tx=0x7f97ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:12.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b1ffb700 1 -- 172.21.15.78:0/2137057281 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac018070 con 0x7f97b4100740 2023-10-10T17:42:12.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b1ffb700 1 -- 172.21.15.78:0/2137057281 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f97ac00f040 con 0x7f97b4100740 2023-10-10T17:42:12.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b410ab00 con 0x7f97b4100740 2023-10-10T17:42:12.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97b1ffb700 1 -- 172.21.15.78:0/2137057281 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00cd00 con 0x7f97b4100740 2023-10-10T17:42:12.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.032+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b410a970 con 0x7f97b4100740 2023-10-10T17:42:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.033+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b41068a0 con 0x7f97b4100740 2023-10-10T17:42:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.034+0000 7f97b1ffb700 1 -- 172.21.15.78:0/2137057281 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f97ac003a00 con 0x7f97b4100740 2023-10-10T17:42:12.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.034+0000 7f97b1ffb700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f97a40a11d0 0x7f97a40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.035+0000 7f97b88bb700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f97a40a11d0 0x7f97a40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.035+0000 7f97b1ffb700 1 -- 172.21.15.78:0/2137057281 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f97ac0cc690 con 0x7f97b4100740 2023-10-10T17:42:12.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.035+0000 7f97b88bb700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f97a40a11d0 0x7f97a40a3680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f97a800b2e0 tx=0x7f97a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:12.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.039+0000 7f97b1ffb700 1 -- 172.21.15.78:0/2137057281 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f97ac095b00 con 0x7f97b4100740 2023-10-10T17:42:12.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.227+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f97b4061b80 con 0x7f97b4100740 2023-10-10T17:42:12.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.228+0000 7f97b1ffb700 1 -- 172.21.15.78:0/2137057281 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f97ac099120 con 0x7f97b4100740 2023-10-10T17:42:12.229 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:12.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f97a40a11d0 msgr2=0x7f97a40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f97a40a11d0 0x7f97a40a3680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f97a800b2e0 tx=0x7f97a8005fb0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 msgr2=0x7f97b419d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b419d420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00ef30 tx=0x7f97ac00c3a0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 shutdown_connections 2023-10-10T17:42:12.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f97a40a11d0 0x7f97a40a3680 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f97b40ff470 0x7f97b40feed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97b4100740 0x7f97b419d420 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 --2- 172.21.15.78:0/2137057281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f97b40ffe00 0x7f97b419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 >> 172.21.15.78:0/2137057281 conn(0x7f97b40f9f30 msgr2=0x7f97b41039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 shutdown_connections 2023-10-10T17:42:12.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.230+0000 7f97bab1f700 1 -- 172.21.15.78:0/2137057281 wait complete. 2023-10-10T17:42:12.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:42:12.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/3498604994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f041003d0 msgr2=0x7f8f04100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/3498604994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f041003d0 0x7f8f04100820 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8ef0009990 tx=0x7f8ef0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/3498604994 shutdown_connections 2023-10-10T17:42:12.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/3498604994 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f040ff1e0 0x7f8f040ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/3498604994 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f04100d60 0x7f8f04107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/3498604994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f041003d0 0x7f8f04100820 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/3498604994 >> 172.21.15.78:0/3498604994 conn(0x7f8f040fa800 msgr2=0x7f8f040fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:12.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/3498604994 shutdown_connections 2023-10-10T17:42:12.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.367+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/3498604994 wait complete. 2023-10-10T17:42:12.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 Processor -- start 2023-10-10T17:42:12.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 -- start start 2023-10-10T17:42:12.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f040ff1e0 0x7f8f04103170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f041003d0 0x7f8f041017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f04100d60 0x7f8f04101cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04102490 con 0x7f8f040ff1e0 2023-10-10T17:42:12.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04102600 con 0x7f8f04100d60 2023-10-10T17:42:12.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f0ab4f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f041a0150 con 0x7f8f041003d0 2023-10-10T17:42:12.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f040ff1e0 0x7f8f04103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f040ff1e0 0x7f8f04103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43550/0 (socket says 172.21.15.78:43550) 2023-10-10T17:42:12.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 -- 172.21.15.78:0/2409748309 learned_addr learned my addr 172.21.15.78:0/2409748309 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f03fff700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f041003d0 0x7f8f041017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f090ec700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f04100d60 0x7f8f04101cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 -- 172.21.15.78:0/2409748309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f041003d0 msgr2=0x7f8f041017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f041003d0 0x7f8f041017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 -- 172.21.15.78:0/2409748309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f04100d60 msgr2=0x7f8f04101cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f04100d60 0x7f8f04101cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.368+0000 7f8f088eb700 1 -- 172.21.15.78:0/2409748309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef00095f0 con 0x7f8f040ff1e0 2023-10-10T17:42:12.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.369+0000 7f8f088eb700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f040ff1e0 0x7f8f04103170 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800e970 tx=0x7f8ef800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:12.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.369+0000 7f8f01ffb700 1 -- 172.21.15.78:0/2409748309 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8009790 con 0x7f8f040ff1e0 2023-10-10T17:42:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.369+0000 7f8f01ffb700 1 -- 172.21.15.78:0/2409748309 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ef800eec0 con 0x7f8f040ff1e0 2023-10-10T17:42:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.369+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f0410ab90 con 0x7f8f040ff1e0 2023-10-10T17:42:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.369+0000 7f8f01ffb700 1 -- 172.21.15.78:0/2409748309 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef8003cf0 con 0x7f8f040ff1e0 2023-10-10T17:42:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.369+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0410aa00 con 0x7f8f040ff1e0 2023-10-10T17:42:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.370+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f041a08a0 con 0x7f8f040ff1e0 2023-10-10T17:42:12.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.371+0000 7f8f01ffb700 1 -- 172.21.15.78:0/2409748309 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8ef800ccd0 con 0x7f8f040ff1e0 2023-10-10T17:42:12.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.371+0000 7f8f01ffb700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8ef40a1150 0x7f8ef40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.371+0000 7f8f03fff700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8ef40a1150 0x7f8ef40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.371+0000 7f8f01ffb700 1 -- 172.21.15.78:0/2409748309 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8ef8014070 con 0x7f8f040ff1e0 2023-10-10T17:42:12.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.372+0000 7f8f03fff700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8ef40a1150 0x7f8ef40a3600 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000b2e0 tx=0x7f8ef0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:12.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.376+0000 7f8f01ffb700 1 -- 172.21.15.78:0/2409748309 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ef80962b0 con 0x7f8f040ff1e0 2023-10-10T17:42:12.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.564+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8f041a0e50 con 0x7f8f040ff1e0 2023-10-10T17:42:12.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.564+0000 7f8f01ffb700 1 -- 172.21.15.78:0/2409748309 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8ef80998d0 con 0x7f8f040ff1e0 2023-10-10T17:42:12.565 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:42:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8ef40a1150 msgr2=0x7f8ef40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8ef40a1150 0x7f8ef40a3600 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000b2e0 tx=0x7f8ef0005fb0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f040ff1e0 msgr2=0x7f8f04103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f040ff1e0 0x7f8f04103170 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f8ef800e970 tx=0x7f8ef800ec80 comp rx=0 tx=0).stop 2023-10-10T17:42:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 shutdown_connections 2023-10-10T17:42:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8ef40a1150 0x7f8ef40a3600 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8f04100d60 0x7f8f04101cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8f040ff1e0 0x7f8f04103170 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 --2- 172.21.15.78:0/2409748309 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8f041003d0 0x7f8f041017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 >> 172.21.15.78:0/2409748309 conn(0x7f8f040fa800 msgr2=0x7f8f04069160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 shutdown_connections 2023-10-10T17:42:12.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.566+0000 7f8f0ab4f700 1 -- 172.21.15.78:0/2409748309 wait complete. 2023-10-10T17:42:12.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:42:12.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 -- 172.21.15.78:0/182670589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c0fe590 msgr2=0x7f642c0fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 --2- 172.21.15.78:0/182670589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c0fe590 0x7f642c0fe9a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6414009990 tx=0x7f6414009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 -- 172.21.15.78:0/182670589 shutdown_connections 2023-10-10T17:42:12.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 --2- 172.21.15.78:0/182670589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f642c101f90 0x7f642c108850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 --2- 172.21.15.78:0/182670589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c0fe590 0x7f642c0fe9a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 --2- 172.21.15.78:0/182670589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f642c101600 0x7f642c101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 -- 172.21.15.78:0/182670589 >> 172.21.15.78:0/182670589 conn(0x7f642c0f9f30 msgr2=0x7f642c0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:12.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 -- 172.21.15.78:0/182670589 shutdown_connections 2023-10-10T17:42:12.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 -- 172.21.15.78:0/182670589 wait complete. 2023-10-10T17:42:12.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 Processor -- start 2023-10-10T17:42:12.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.704+0000 7f6432780700 1 -- start start 2023-10-10T17:42:12.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6432780700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f642c0fe590 0x7f642c196030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6432780700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c101600 0x7f642c196570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6432780700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f642c101f90 0x7f642c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6432780700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c19dbb0 con 0x7f642c101600 2023-10-10T17:42:12.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6432780700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c19dcf0 con 0x7f642c0fe590 2023-10-10T17:42:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6432780700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c19de80 con 0x7f642c101f90 2023-10-10T17:42:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c101600 0x7f642c196570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c101600 0x7f642c196570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43566/0 (socket says 172.21.15.78:43566) 2023-10-10T17:42:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 -- 172.21.15.78:0/3700693214 learned_addr learned my addr 172.21.15.78:0/3700693214 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6430d1d700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f642c101f90 0x7f642c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642bfff700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f642c0fe590 0x7f642c196030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 -- 172.21.15.78:0/3700693214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f642c101f90 msgr2=0x7f642c19d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f642c101f90 0x7f642c19d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 -- 172.21.15.78:0/3700693214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f642c0fe590 msgr2=0x7f642c196030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f642c0fe590 0x7f642c196030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 -- 172.21.15.78:0/3700693214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64140095f0 con 0x7f642c101600 2023-10-10T17:42:12.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f642b7fe700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c101600 0x7f642c196570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f641c00d980 tx=0x7f641c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:12.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f64297fa700 1 -- 172.21.15.78:0/3700693214 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c004a30 con 0x7f642c101600 2023-10-10T17:42:12.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f64297fa700 1 -- 172.21.15.78:0/3700693214 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f641c0054e0 con 0x7f642c101600 2023-10-10T17:42:12.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c10bc20 con 0x7f642c101600 2023-10-10T17:42:12.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.705+0000 7f64297fa700 1 -- 172.21.15.78:0/3700693214 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c00fbc0 con 0x7f642c101600 2023-10-10T17:42:12.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.706+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c10ba90 con 0x7f642c101600 2023-10-10T17:42:12.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.707+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c107390 con 0x7f642c101600 2023-10-10T17:42:12.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.707+0000 7f64297fa700 1 -- 172.21.15.78:0/3700693214 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f641c004c20 con 0x7f642c101600 2023-10-10T17:42:12.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.708+0000 7f64297fa700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f64180a1150 0x7f64180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:12.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.708+0000 7f64297fa700 1 -- 172.21.15.78:0/3700693214 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f641c0cca70 con 0x7f642c101600 2023-10-10T17:42:12.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.708+0000 7f642bfff700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f64180a1150 0x7f64180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:12.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.709+0000 7f642bfff700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f64180a1150 0x7f64180a3600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f641400b5c0 tx=0x7f6414005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:12.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.713+0000 7f64297fa700 1 -- 172.21.15.78:0/3700693214 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f641c095eb0 con 0x7f642c101600 2023-10-10T17:42:12.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:12 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1416872056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:12 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2137057281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:12 smithi078 ceph-mon[99494]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:12.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:12 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2409748309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.902+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f642c067270 con 0x7f642c101600 2023-10-10T17:42:12.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.902+0000 7f64297fa700 1 -- 172.21.15.78:0/3700693214 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f641c0994d0 con 0x7f642c101600 2023-10-10T17:42:12.903 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:12.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f64180a1150 msgr2=0x7f64180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f64180a1150 0x7f64180a3600 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f641400b5c0 tx=0x7f6414005fd0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c101600 msgr2=0x7f642c196570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:12.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c101600 0x7f642c196570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f641c00d980 tx=0x7f641c00dc90 comp rx=0 tx=0).stop 2023-10-10T17:42:12.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 shutdown_connections 2023-10-10T17:42:12.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f64180a1150 0x7f64180a3600 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f642c101f90 0x7f642c19d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f642c0fe590 0x7f642c196030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 --2- 172.21.15.78:0/3700693214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f642c101600 0x7f642c196570 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:12.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 >> 172.21.15.78:0/3700693214 conn(0x7f642c0f9f30 msgr2=0x7f642c0fc260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:12.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 shutdown_connections 2023-10-10T17:42:12.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:12.905+0000 7f6432780700 1 -- 172.21.15.78:0/3700693214 wait complete. 2023-10-10T17:42:12.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:12 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1416872056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:12 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2137057281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:12 smithi190 ceph-mon[101220]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:12.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:12 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2409748309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:42:12.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:12 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1416872056' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:12 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2137057281' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:12.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:12 smithi157 ceph-mon[100896]: pgmap v309: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:12.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:12 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2409748309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 -- 172.21.15.78:0/335654289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 msgr2=0x7fa020107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 --2- 172.21.15.78:0/335654289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa020107600 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa01800b0a0 tx=0x7fa01800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 -- 172.21.15.78:0/335654289 shutdown_connections 2023-10-10T17:42:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 --2- 172.21.15.78:0/335654289 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0200ff1c0 0x7fa0200ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 --2- 172.21.15.78:0/335654289 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa020107600 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 --2- 172.21.15.78:0/335654289 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0201003b0 0x7fa020100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 -- 172.21.15.78:0/335654289 >> 172.21.15.78:0/335654289 conn(0x7fa0200fa7a0 msgr2=0x7fa0200fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 -- 172.21.15.78:0/335654289 shutdown_connections 2023-10-10T17:42:13.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.039+0000 7fa027e02700 1 -- 172.21.15.78:0/335654289 wait complete. 2023-10-10T17:42:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 Processor -- start 2023-10-10T17:42:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 -- start start 2023-10-10T17:42:13.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0200ff1c0 0x7fa020069990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0201003b0 0x7fa02019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa02019d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02019db50 con 0x7fa020100d40 2023-10-10T17:42:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02019dcc0 con 0x7fa0200ff1c0 2023-10-10T17:42:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02019de80 con 0x7fa0201003b0 2023-10-10T17:42:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa02019d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa02019d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43582/0 (socket says 172.21.15.78:43582) 2023-10-10T17:42:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 -- 172.21.15.78:0/3500262152 learned_addr learned my addr 172.21.15.78:0/3500262152 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:13.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa025b9e700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0200ff1c0 0x7fa020069990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02539d700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0201003b0 0x7fa02019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 -- 172.21.15.78:0/3500262152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0201003b0 msgr2=0x7fa02019ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0201003b0 0x7fa02019ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 -- 172.21.15.78:0/3500262152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0200ff1c0 msgr2=0x7fa020069990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0200ff1c0 0x7fa020069990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 -- 172.21.15.78:0/3500262152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa018009d00 con 0x7fa020100d40 2023-10-10T17:42:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02539d700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0201003b0 0x7fa02019ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa02639f700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa02019d3b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa01800b780 tx=0x7fa01800bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:13.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa016ffd700 1 -- 172.21.15.78:0/3500262152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01800e040 con 0x7fa020100d40 2023-10-10T17:42:13.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02010ab40 con 0x7fa020100d40 2023-10-10T17:42:13.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.040+0000 7fa016ffd700 1 -- 172.21.15.78:0/3500262152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa018004e60 con 0x7fa020100d40 2023-10-10T17:42:13.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.041+0000 7fa016ffd700 1 -- 172.21.15.78:0/3500262152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01801b740 con 0x7fa020100d40 2023-10-10T17:42:13.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.041+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02010a9b0 con 0x7fa020100d40 2023-10-10T17:42:13.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.041+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02019e4d0 con 0x7fa020100d40 2023-10-10T17:42:13.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.042+0000 7fa016ffd700 1 -- 172.21.15.78:0/3500262152 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa01801da40 con 0x7fa020100d40 2023-10-10T17:42:13.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.043+0000 7fa016ffd700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa00c0a1150 0x7fa00c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.043+0000 7fa025b9e700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa00c0a1150 0x7fa00c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.043+0000 7fa016ffd700 1 -- 172.21.15.78:0/3500262152 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa0180cec30 con 0x7fa020100d40 2023-10-10T17:42:13.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.043+0000 7fa025b9e700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa00c0a1150 0x7fa00c0a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa020100210 tx=0x7fa010009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:13.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.046+0000 7fa016ffd700 1 -- 172.21.15.78:0/3500262152 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa018098100 con 0x7fa020100d40 2023-10-10T17:42:13.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.275+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa020106010 con 0x7fa020100d40 2023-10-10T17:42:13.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.276+0000 7fa016ffd700 1 -- 172.21.15.78:0/3500262152 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7fa01809b720 con 0x7fa020100d40 2023-10-10T17:42:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:13.277 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:42:13.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.278+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa00c0a1150 msgr2=0x7fa00c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa00c0a1150 0x7fa00c0a3600 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fa020100210 tx=0x7fa010009260 comp rx=0 tx=0).stop 2023-10-10T17:42:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 msgr2=0x7fa02019d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa02019d3b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa01800b780 tx=0x7fa01800bd00 comp rx=0 tx=0).stop 2023-10-10T17:42:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 shutdown_connections 2023-10-10T17:42:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa00c0a1150 0x7fa00c0a3600 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa020100d40 0x7fa02019d3b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa0200ff1c0 0x7fa020069990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 --2- 172.21.15.78:0/3500262152 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa0201003b0 0x7fa02019ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 >> 172.21.15.78:0/3500262152 conn(0x7fa0200fa7a0 msgr2=0x7fa0200fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 shutdown_connections 2023-10-10T17:42:13.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.279+0000 7fa027e02700 1 -- 172.21.15.78:0/3500262152 wait complete. 2023-10-10T17:42:13.282 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:42:13.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:42:13.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/3879452221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ffe10 msgr2=0x7f4abc1076c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/3879452221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ffe10 0x7f4abc1076c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b0d0 tx=0x7f4ab400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/3879452221 shutdown_connections 2023-10-10T17:42:13.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/3879452221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc100770 0x7f4abc100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/3879452221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ffe10 0x7f4abc1076c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/3879452221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4abc0ff460 0x7f4abc0ff8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/3879452221 >> 172.21.15.78:0/3879452221 conn(0x7f4abc0f9ee0 msgr2=0x7f4abc0fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:13.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/3879452221 shutdown_connections 2023-10-10T17:42:13.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.412+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/3879452221 wait complete. 2023-10-10T17:42:13.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 Processor -- start 2023-10-10T17:42:13.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 -- start start 2023-10-10T17:42:13.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ff460 0x7f4abc0feec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4abc0ffe10 0x7f4abc19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc100770 0x7f4abc19d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc19db30 con 0x7f4abc0ff460 2023-10-10T17:42:13.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc19dca0 con 0x7f4abc0ffe10 2023-10-10T17:42:13.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac3f6c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc19de60 con 0x7f4abc100770 2023-10-10T17:42:13.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac1d08700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ff460 0x7f4abc0feec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac1d08700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ff460 0x7f4abc0feec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43592/0 (socket says 172.21.15.78:43592) 2023-10-10T17:42:13.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac1d08700 1 -- 172.21.15.78:0/262264199 learned_addr learned my addr 172.21.15.78:0/262264199 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:13.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.413+0000 7f4ac1d08700 1 -- 172.21.15.78:0/262264199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc100770 msgr2=0x7f4abc19d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac1d08700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc100770 0x7f4abc19d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac1d08700 1 -- 172.21.15.78:0/262264199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4abc0ffe10 msgr2=0x7f4abc19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac1507700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4abc0ffe10 0x7f4abc19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac1d08700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4abc0ffe10 0x7f4abc19ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac1d08700 1 -- 172.21.15.78:0/262264199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4009d00 con 0x7f4abc0ff460 2023-10-10T17:42:13.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac1d08700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ff460 0x7f4abc0feec0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800e9a0 tx=0x7f4ab800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:13.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ab2ffd700 1 -- 172.21.15.78:0/262264199 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8009790 con 0x7f4abc0ff460 2023-10-10T17:42:13.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ab2ffd700 1 -- 172.21.15.78:0/262264199 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4ab800cbd0 con 0x7f4abc0ff460 2023-10-10T17:42:13.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc10ab00 con 0x7f4abc0ff460 2023-10-10T17:42:13.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ab2ffd700 1 -- 172.21.15.78:0/262264199 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8018680 con 0x7f4abc0ff460 2023-10-10T17:42:13.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.414+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc10a970 con 0x7f4abc0ff460 2023-10-10T17:42:13.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.415+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc1061b0 con 0x7f4abc0ff460 2023-10-10T17:42:13.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.416+0000 7f4ab2ffd700 1 -- 172.21.15.78:0/262264199 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4ab800cd40 con 0x7f4abc0ff460 2023-10-10T17:42:13.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.416+0000 7f4ab2ffd700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4aa80a1150 0x7f4aa80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.416+0000 7f4ab2ffd700 1 -- 172.21.15.78:0/262264199 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4ab8014070 con 0x7f4abc0ff460 2023-10-10T17:42:13.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.416+0000 7f4ac1507700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4aa80a1150 0x7f4aa80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.417+0000 7f4ac1507700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4aa80a1150 0x7f4aa80a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4aac005950 tx=0x7f4aac005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:13.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.420+0000 7f4ab2ffd700 1 -- 172.21.15.78:0/262264199 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4ab8051730 con 0x7f4abc0ff460 2023-10-10T17:42:13.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.614+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4abc02cfe0 con 0x7f4abc0ff460 2023-10-10T17:42:13.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.616+0000 7f4ab2ffd700 1 -- 172.21.15.78:0/262264199 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7f4ab8099e70 con 0x7f4abc0ff460 2023-10-10T17:42:13.617 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:13.617 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:42:13.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4aa80a1150 msgr2=0x7f4aa80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4aa80a1150 0x7f4aa80a3600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4aac005950 tx=0x7f4aac005cf0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ff460 msgr2=0x7f4abc0feec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ff460 0x7f4abc0feec0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f4ab800e9a0 tx=0x7f4ab800ed60 comp rx=0 tx=0).stop 2023-10-10T17:42:13.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 shutdown_connections 2023-10-10T17:42:13.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f4aa80a1150 0x7f4aa80a3600 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4abc100770 0x7f4abc19d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4abc0ff460 0x7f4abc0feec0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 --2- 172.21.15.78:0/262264199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4abc0ffe10 0x7f4abc19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 >> 172.21.15.78:0/262264199 conn(0x7f4abc0f9ee0 msgr2=0x7f4abc0fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:13.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 shutdown_connections 2023-10-10T17:42:13.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.618+0000 7f4ac3f6c700 1 -- 172.21.15.78:0/262264199 wait complete. 2023-10-10T17:42:13.632 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:42:13.632 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:42:13.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.758+0000 7fe6d7785700 1 -- 172.21.15.78:0/1262821235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0105fc0 msgr2=0x7fe6d01083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.758+0000 7fe6d7785700 1 --2- 172.21.15.78:0/1262821235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0105fc0 0x7fe6d01083e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800b0a0 tx=0x7fe6c800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- 172.21.15.78:0/1262821235 shutdown_connections 2023-10-10T17:42:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 --2- 172.21.15.78:0/1262821235 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d0100d80 0x7fe6d0103160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 --2- 172.21.15.78:0/1262821235 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0105fc0 0x7fe6d01083e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 --2- 172.21.15.78:0/1262821235 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d01036a0 0x7fe6d0105a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- 172.21.15.78:0/1262821235 >> 172.21.15.78:0/1262821235 conn(0x7fe6d00fa7a0 msgr2=0x7fe6d00fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:13.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- 172.21.15.78:0/1262821235 shutdown_connections 2023-10-10T17:42:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- 172.21.15.78:0/1262821235 wait complete. 2023-10-10T17:42:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 Processor -- start 2023-10-10T17:42:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- start start 2023-10-10T17:42:13.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0100d80 0x7fe6d01006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d01036a0 0x7fe6d00fed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d0105fc0 0x7fe6d00ff270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00ffa10 con 0x7fe6d0100d80 2023-10-10T17:42:13.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00ffb80 con 0x7fe6d01036a0 2023-10-10T17:42:13.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d7785700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d01a0160 con 0x7fe6d0105fc0 2023-10-10T17:42:13.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d5521700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0100d80 0x7fe6d01006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d5521700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0100d80 0x7fe6d01006f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43606/0 (socket says 172.21.15.78:43606) 2023-10-10T17:42:13.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.759+0000 7fe6d5521700 1 -- 172.21.15.78:0/3512626392 learned_addr learned my addr 172.21.15.78:0/3512626392 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:13.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d4d20700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d01036a0 0x7fe6d00fed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5d22700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d0105fc0 0x7fe6d00ff270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5521700 1 -- 172.21.15.78:0/3512626392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d0105fc0 msgr2=0x7fe6d00ff270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5521700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d0105fc0 0x7fe6d00ff270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5521700 1 -- 172.21.15.78:0/3512626392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d01036a0 msgr2=0x7fe6d00fed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5521700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d01036a0 0x7fe6d00fed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5521700 1 -- 172.21.15.78:0/3512626392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c8009d00 con 0x7fe6d0100d80 2023-10-10T17:42:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5d22700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d0105fc0 0x7fe6d00ff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d4d20700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d01036a0 0x7fe6d00fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d5521700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0100d80 0x7fe6d01006f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00e9a0 tx=0x7fe6cc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6c67fc700 1 -- 172.21.15.78:0/3512626392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc009790 con 0x7fe6d0100d80 2023-10-10T17:42:13.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6c67fc700 1 -- 172.21.15.78:0/3512626392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6cc00cbd0 con 0x7fe6d0100d80 2023-10-10T17:42:13.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d010bc80 con 0x7fe6d0100d80 2023-10-10T17:42:13.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6c67fc700 1 -- 172.21.15.78:0/3512626392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc018680 con 0x7fe6d0100d80 2023-10-10T17:42:13.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.760+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d010baf0 con 0x7fe6d0100d80 2023-10-10T17:42:13.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.761+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d00fc310 con 0x7fe6d0100d80 2023-10-10T17:42:13.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.762+0000 7fe6c67fc700 1 -- 172.21.15.78:0/3512626392 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe6cc00cd40 con 0x7fe6d0100d80 2023-10-10T17:42:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.762+0000 7fe6c67fc700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe6bc0a1150 0x7fe6bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.763+0000 7fe6c67fc700 1 -- 172.21.15.78:0/3512626392 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe6cc014070 con 0x7fe6d0100d80 2023-10-10T17:42:13.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.763+0000 7fe6d4d20700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe6bc0a1150 0x7fe6bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.763+0000 7fe6d4d20700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe6bc0a1150 0x7fe6bc0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000bc10 tx=0x7fe6c000b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:13.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.767+0000 7fe6c67fc700 1 -- 172.21.15.78:0/3512626392 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe6cc051730 con 0x7fe6d0100d80 2023-10-10T17:42:13.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:13 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3700693214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:13.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:13 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3500262152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:13.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:13 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/262264199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:13.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:13 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3700693214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:13.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:13 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3500262152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:13.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:13 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/262264199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.960+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6d0061700 con 0x7fe6d0100d80 2023-10-10T17:42:13.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.960+0000 7fe6c67fc700 1 -- 172.21.15.78:0/3512626392 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6cc099e70 con 0x7fe6d0100d80 2023-10-10T17:42:13.961 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:42:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe6bc0a1150 msgr2=0x7fe6bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe6bc0a1150 0x7fe6bc0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000bc10 tx=0x7fe6c000b5f0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0100d80 msgr2=0x7fe6d01006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:13.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0100d80 0x7fe6d01006f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00e9a0 tx=0x7fe6cc00ed60 comp rx=0 tx=0).stop 2023-10-10T17:42:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 shutdown_connections 2023-10-10T17:42:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe6bc0a1150 0x7fe6bc0a3600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d0105fc0 0x7fe6d00ff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d0100d80 0x7fe6d01006f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 --2- 172.21.15.78:0/3512626392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d01036a0 0x7fe6d00fed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 >> 172.21.15.78:0/3512626392 conn(0x7fe6d00fa7a0 msgr2=0x7fe6d01019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 shutdown_connections 2023-10-10T17:42:13.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:13.962+0000 7fe6d7785700 1 -- 172.21.15.78:0/3512626392 wait complete. 2023-10-10T17:42:13.975 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:42:13.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:13 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3700693214' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:13.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:13 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3500262152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:13.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:13 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/262264199' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 -- 172.21.15.78:0/3936355017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0100d90 msgr2=0x7f90a0107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/3936355017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0100d90 0x7f90a0107630 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f909800b0a0 tx=0x7f909800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 -- 172.21.15.78:0/3936355017 shutdown_connections 2023-10-10T17:42:14.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/3936355017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a0100d90 0x7f90a0107630 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/3936355017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a00ff6d0 0x7f90a00ffae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/3936355017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90a01003e0 0x7f90a0100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 -- 172.21.15.78:0/3936355017 >> 172.21.15.78:0/3936355017 conn(0x7f90a00f9f30 msgr2=0x7f90a00fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 -- 172.21.15.78:0/3936355017 shutdown_connections 2023-10-10T17:42:14.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 -- 172.21.15.78:0/3936355017 wait complete. 2023-10-10T17:42:14.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.099+0000 7f90a5cd1700 1 Processor -- start 2023-10-10T17:42:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f90a5cd1700 1 -- start start 2023-10-10T17:42:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f90a5cd1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90a00ff6d0 0x7f90a0195020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f90a5cd1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a01003e0 0x7f90a0195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f90a5cd1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a0100d90 0x7f90a019c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f90a5cd1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a019cba0 con 0x7f90a01003e0 2023-10-10T17:42:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f90a5cd1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a019cce0 con 0x7f90a0100d90 2023-10-10T17:42:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f90a5cd1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a019ce70 con 0x7f90a00ff6d0 2023-10-10T17:42:14.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a01003e0 0x7f90a0195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a01003e0 0x7f90a0195560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43622/0 (socket says 172.21.15.78:43622) 2023-10-10T17:42:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 -- 172.21.15.78:0/1299406379 learned_addr learned my addr 172.21.15.78:0/1299406379 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909f7fe700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90a00ff6d0 0x7f90a0195020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909ffff700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a0100d90 0x7f90a019c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 -- 172.21.15.78:0/1299406379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90a00ff6d0 msgr2=0x7f90a0195020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90a00ff6d0 0x7f90a0195020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 -- 172.21.15.78:0/1299406379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a0100d90 msgr2=0x7f90a019c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a0100d90 0x7f90a019c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 -- 172.21.15.78:0/1299406379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9098009d00 con 0x7f90a01003e0 2023-10-10T17:42:14.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909ffff700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a0100d90 0x7f90a019c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909effd700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a01003e0 0x7f90a0195560 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f909000d980 tx=0x7f909000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.100+0000 7f909cff9700 1 -- 172.21.15.78:0/1299406379 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9090009930 con 0x7f90a01003e0 2023-10-10T17:42:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.101+0000 7f909cff9700 1 -- 172.21.15.78:0/1299406379 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9090010460 con 0x7f90a01003e0 2023-10-10T17:42:14.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.101+0000 7f909cff9700 1 -- 172.21.15.78:0/1299406379 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909000f620 con 0x7f90a01003e0 2023-10-10T17:42:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.101+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a010ab60 con 0x7f90a01003e0 2023-10-10T17:42:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.101+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a010a9d0 con 0x7f90a01003e0 2023-10-10T17:42:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.101+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a019d510 con 0x7f90a01003e0 2023-10-10T17:42:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.102+0000 7f909cff9700 1 -- 172.21.15.78:0/1299406379 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9090010860 con 0x7f90a01003e0 2023-10-10T17:42:14.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.103+0000 7f909cff9700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f90880a11d0 0x7f90880a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.103+0000 7f909f7fe700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f90880a11d0 0x7f90880a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.103+0000 7f909cff9700 1 -- 172.21.15.78:0/1299406379 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f90900cc680 con 0x7f90a01003e0 2023-10-10T17:42:14.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.103+0000 7f909f7fe700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f90880a11d0 0x7f90880a3680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9094005950 tx=0x7f909400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:14.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.107+0000 7f909cff9700 1 -- 172.21.15.78:0/1299406379 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9090095b40 con 0x7f90a01003e0 2023-10-10T17:42:14.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.296+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90a0106070 con 0x7f90a01003e0 2023-10-10T17:42:14.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.296+0000 7f909cff9700 1 -- 172.21.15.78:0/1299406379 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90900170a0 con 0x7f90a01003e0 2023-10-10T17:42:14.297 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f90880a11d0 msgr2=0x7f90880a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f90880a11d0 0x7f90880a3680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9094005950 tx=0x7f909400a300 comp rx=0 tx=0).stop 2023-10-10T17:42:14.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a01003e0 msgr2=0x7f90a0195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a01003e0 0x7f90a0195560 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f909000d980 tx=0x7f909000dc90 comp rx=0 tx=0).stop 2023-10-10T17:42:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 shutdown_connections 2023-10-10T17:42:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f90880a11d0 0x7f90880a3680 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f90a00ff6d0 0x7f90a0195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f90a0100d90 0x7f90a019c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 --2- 172.21.15.78:0/1299406379 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f90a01003e0 0x7f90a0195560 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.298+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 >> 172.21.15.78:0/1299406379 conn(0x7f90a00f9f30 msgr2=0x7f90a00fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.299+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 shutdown_connections 2023-10-10T17:42:14.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.299+0000 7f90a5cd1700 1 -- 172.21.15.78:0/1299406379 wait complete. 2023-10-10T17:42:14.311 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:42:14.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.434+0000 7fd26b911700 1 -- 172.21.15.78:0/4277371312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 msgr2=0x7fd2640ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.434+0000 7fd26b911700 1 --2- 172.21.15.78:0/4277371312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd2640ff600 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fd260009990 tx=0x7fd260009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.434+0000 7fd26b911700 1 -- 172.21.15.78:0/4277371312 shutdown_connections 2023-10-10T17:42:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.434+0000 7fd26b911700 1 --2- 172.21.15.78:0/4277371312 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd264100d70 0x7fd264107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.434+0000 7fd26b911700 1 --2- 172.21.15.78:0/4277371312 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd2640ff600 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.434+0000 7fd26b911700 1 --2- 172.21.15.78:0/4277371312 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2641003e0 0x7fd264100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.434+0000 7fd26b911700 1 -- 172.21.15.78:0/4277371312 >> 172.21.15.78:0/4277371312 conn(0x7fd2640fa810 msgr2=0x7fd2640fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:14.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 -- 172.21.15.78:0/4277371312 shutdown_connections 2023-10-10T17:42:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 -- 172.21.15.78:0/4277371312 wait complete. 2023-10-10T17:42:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 Processor -- start 2023-10-10T17:42:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 -- start start 2023-10-10T17:42:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd264196050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2641003e0 0x7fd264196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd264100d70 0x7fd26419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26419dbd0 con 0x7fd2640ff1f0 2023-10-10T17:42:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26419dd10 con 0x7fd264100d70 2023-10-10T17:42:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd26b911700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd26419dea0 con 0x7fd2641003e0 2023-10-10T17:42:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd2696ad700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd264196050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd2696ad700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd264196050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43634/0 (socket says 172.21.15.78:43634) 2023-10-10T17:42:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd2696ad700 1 -- 172.21.15.78:0/2767670890 learned_addr learned my addr 172.21.15.78:0/2767670890 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.435+0000 7fd268eac700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2641003e0 0x7fd264196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd269eae700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd264100d70 0x7fd26419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2696ad700 1 -- 172.21.15.78:0/2767670890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2641003e0 msgr2=0x7fd264196590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2696ad700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2641003e0 0x7fd264196590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2696ad700 1 -- 172.21.15.78:0/2767670890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd264100d70 msgr2=0x7fd26419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2696ad700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd264100d70 0x7fd26419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2696ad700 1 -- 172.21.15.78:0/2767670890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2600095f0 con 0x7fd2640ff1f0 2023-10-10T17:42:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd268eac700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2641003e0 0x7fd264196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:14.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2696ad700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd264196050 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd260000c00 tx=0x7fd26000f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2567fc700 1 -- 172.21.15.78:0/2767670890 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26001d070 con 0x7fd2640ff1f0 2023-10-10T17:42:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd26410ab80 con 0x7fd2640ff1f0 2023-10-10T17:42:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2567fc700 1 -- 172.21.15.78:0/2767670890 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd26000fc60 con 0x7fd2640ff1f0 2023-10-10T17:42:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd2567fc700 1 -- 172.21.15.78:0/2767670890 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd260017650 con 0x7fd2640ff1f0 2023-10-10T17:42:14.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.436+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd26410a9f0 con 0x7fd2640ff1f0 2023-10-10T17:42:14.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.437+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26419e450 con 0x7fd2640ff1f0 2023-10-10T17:42:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.438+0000 7fd2567fc700 1 -- 172.21.15.78:0/2767670890 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd2600177b0 con 0x7fd2640ff1f0 2023-10-10T17:42:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.438+0000 7fd2567fc700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2500a1150 0x7fd2500a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.438+0000 7fd2567fc700 1 -- 172.21.15.78:0/2767670890 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd2600ce070 con 0x7fd2640ff1f0 2023-10-10T17:42:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.438+0000 7fd268eac700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2500a1150 0x7fd2500a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.439+0000 7fd268eac700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2500a1150 0x7fd2500a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd258005fd0 tx=0x7fd258005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:14.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.443+0000 7fd2567fc700 1 -- 172.21.15.78:0/2767670890 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd26005c730 con 0x7fd2640ff1f0 2023-10-10T17:42:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.634+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd26419eaf0 con 0x7fd2640ff1f0 2023-10-10T17:42:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.634+0000 7fd2567fc700 1 -- 172.21.15.78:0/2767670890 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd26009afd0 con 0x7fd2640ff1f0 2023-10-10T17:42:14.635 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:42:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.636+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2500a1150 msgr2=0x7fd2500a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.636+0000 7fd26b911700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2500a1150 0x7fd2500a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd258005fd0 tx=0x7fd258005e30 comp rx=0 tx=0).stop 2023-10-10T17:42:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.636+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 msgr2=0x7fd264196050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.636+0000 7fd26b911700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd264196050 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd260000c00 tx=0x7fd26000f510 comp rx=0 tx=0).stop 2023-10-10T17:42:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.636+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 shutdown_connections 2023-10-10T17:42:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.637+0000 7fd26b911700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2500a1150 0x7fd2500a3600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.637+0000 7fd26b911700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd264100d70 0x7fd26419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.637+0000 7fd26b911700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2640ff1f0 0x7fd264196050 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.637+0000 7fd26b911700 1 --2- 172.21.15.78:0/2767670890 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2641003e0 0x7fd264196590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.637+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 >> 172.21.15.78:0/2767670890 conn(0x7fd2640fa810 msgr2=0x7fd264068e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.637+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 shutdown_connections 2023-10-10T17:42:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.637+0000 7fd26b911700 1 -- 172.21.15.78:0/2767670890 wait complete. 2023-10-10T17:42:14.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:42:14.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2095802248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 msgr2=0x7f09f4100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2095802248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4100800 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f09f0009990 tx=0x7f09f0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2095802248 shutdown_connections 2023-10-10T17:42:14.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2095802248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f4100d40 0x7f09f4107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2095802248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f40ff1c0 0x7f09f40ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2095802248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4100800 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2095802248 >> 172.21.15.78:0/2095802248 conn(0x7f09f40fa7a0 msgr2=0x7f09f40fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2095802248 shutdown_connections 2023-10-10T17:42:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.773+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2095802248 wait complete. 2023-10-10T17:42:14.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 Processor -- start 2023-10-10T17:42:14.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 -- start start 2023-10-10T17:42:14.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f40ff1c0 0x7f09f4196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f4100d40 0x7f09f419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f419dbc0 con 0x7f09f41003b0 2023-10-10T17:42:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f419dd00 con 0x7f09f40ff1c0 2023-10-10T17:42:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09fb3f0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f419de90 con 0x7f09f4100d40 2023-10-10T17:42:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09f898b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09f898b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43650/0 (socket says 172.21.15.78:43650) 2023-10-10T17:42:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09f898b700 1 -- 172.21.15.78:0/2842158100 learned_addr learned my addr 172.21.15.78:0/2842158100 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.774+0000 7f09f998d700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f4100d40 0x7f09f419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f898b700 1 -- 172.21.15.78:0/2842158100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f4100d40 msgr2=0x7f09f419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f898b700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f4100d40 0x7f09f419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f918c700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f40ff1c0 0x7f09f4196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f898b700 1 -- 172.21.15.78:0/2842158100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f40ff1c0 msgr2=0x7f09f4196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f898b700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f40ff1c0 0x7f09f4196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f898b700 1 -- 172.21.15.78:0/2842158100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f00095f0 con 0x7f09f41003b0 2023-10-10T17:42:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f998d700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f4100d40 0x7f09f419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09f898b700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4196580 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f09f0005910 tx=0x7f09f0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09ea7fc700 1 -- 172.21.15.78:0/2842158100 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f001d070 con 0x7f09f41003b0 2023-10-10T17:42:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09ea7fc700 1 -- 172.21.15.78:0/2842158100 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09f0004060 con 0x7f09f41003b0 2023-10-10T17:42:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f410ab70 con 0x7f09f41003b0 2023-10-10T17:42:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09ea7fc700 1 -- 172.21.15.78:0/2842158100 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f0022e10 con 0x7f09f41003b0 2023-10-10T17:42:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.775+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f410a9e0 con 0x7f09f41003b0 2023-10-10T17:42:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.776+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f4107250 con 0x7f09f41003b0 2023-10-10T17:42:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.777+0000 7f09ea7fc700 1 -- 172.21.15.78:0/2842158100 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f09f0022470 con 0x7f09f41003b0 2023-10-10T17:42:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.778+0000 7f09ea7fc700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f09e00a11d0 0x7f09e00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.778+0000 7f09f918c700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f09e00a11d0 0x7f09e00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.778+0000 7f09ea7fc700 1 -- 172.21.15.78:0/2842158100 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f09f00ce410 con 0x7f09f41003b0 2023-10-10T17:42:14.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.778+0000 7f09f918c700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f09e00a11d0 0x7f09e00a3680 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f09e4005950 tx=0x7f09e40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:14.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.781+0000 7f09ea7fc700 1 -- 172.21.15.78:0/2842158100 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f09f0097970 con 0x7f09f41003b0 2023-10-10T17:42:14.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:14 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3512626392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:14.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:14 smithi078 ceph-mon[99494]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:14.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:14 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1299406379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:14.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:14 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3512626392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:14.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:14 smithi190 ceph-mon[101220]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:14.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:14 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1299406379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.970+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f09f4061700 con 0x7f09f41003b0 2023-10-10T17:42:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.971+0000 7f09ea7fc700 1 -- 172.21.15.78:0/2842158100 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f09f009af90 con 0x7f09f41003b0 2023-10-10T17:42:14.972 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:14.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.972+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f09e00a11d0 msgr2=0x7f09e00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.972+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f09e00a11d0 0x7f09e00a3680 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f09e4005950 tx=0x7f09e40058e0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 msgr2=0x7f09f4196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:14.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4196580 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f09f0005910 tx=0x7f09f0005e80 comp rx=0 tx=0).stop 2023-10-10T17:42:14.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 shutdown_connections 2023-10-10T17:42:14.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f09e00a11d0 0x7f09e00a3680 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09f4100d40 0x7f09f419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09f40ff1c0 0x7f09f4196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 --2- 172.21.15.78:0/2842158100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09f41003b0 0x7f09f4196580 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:14.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 >> 172.21.15.78:0/2842158100 conn(0x7f09f40fa7a0 msgr2=0x7f09f40fc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:14.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 shutdown_connections 2023-10-10T17:42:14.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:14.973+0000 7f09fb3f0700 1 -- 172.21.15.78:0/2842158100 wait complete. 2023-10-10T17:42:14.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:14 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3512626392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:14 smithi157 ceph-mon[100896]: pgmap v310: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:14 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1299406379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:14.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:42:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 -- 172.21.15.78:0/3127055881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 msgr2=0x7fc9d40ff580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 --2- 172.21.15.78:0/3127055881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d40ff580 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8009990 tx=0x7fc9c8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 -- 172.21.15.78:0/3127055881 shutdown_connections 2023-10-10T17:42:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 --2- 172.21.15.78:0/3127055881 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9d4100db0 0x7fc9d4107650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 --2- 172.21.15.78:0/3127055881 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d40ff580 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 --2- 172.21.15.78:0/3127055881 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9d4100400 0x7fc9d4100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 -- 172.21.15.78:0/3127055881 >> 172.21.15.78:0/3127055881 conn(0x7fc9d40fa750 msgr2=0x7fc9d40fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 -- 172.21.15.78:0/3127055881 shutdown_connections 2023-10-10T17:42:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.105+0000 7fc9daa99700 1 -- 172.21.15.78:0/3127055881 wait complete. 2023-10-10T17:42:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 Processor -- start 2023-10-10T17:42:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 -- start start 2023-10-10T17:42:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9d4100400 0x7fc9d419ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9d4100db0 0x7fc9d419d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d419db40 con 0x7fc9d40ff170 2023-10-10T17:42:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d419dcb0 con 0x7fc9d4100400 2023-10-10T17:42:15.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9daa99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9d419de70 con 0x7fc9d4100db0 2023-10-10T17:42:15.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d4068960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43662/0 (socket says 172.21.15.78:43662) 2023-10-10T17:42:15.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 -- 172.21.15.78:0/2519470645 learned_addr learned my addr 172.21.15.78:0/2519470645 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:15.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d9036700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9d4100db0 0x7fc9d419d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 -- 172.21.15.78:0/2519470645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9d4100db0 msgr2=0x7fc9d419d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9d4100db0 0x7fc9d419d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 -- 172.21.15.78:0/2519470645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9d4100400 msgr2=0x7fc9d419ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d3fff700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9d4100400 0x7fc9d419ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9d4100400 0x7fc9d419ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d8835700 1 -- 172.21.15.78:0/2519470645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c80095f0 con 0x7fc9d40ff170 2023-10-10T17:42:15.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.106+0000 7fc9d9036700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9d4100db0 0x7fc9d419d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:15.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.107+0000 7fc9d8835700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d4068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000c00 tx=0x7fc9c800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.107+0000 7fc9d1ffb700 1 -- 172.21.15.78:0/2519470645 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c801d070 con 0x7fc9d40ff170 2023-10-10T17:42:15.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.107+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d410ab70 con 0x7fc9d40ff170 2023-10-10T17:42:15.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.107+0000 7fc9d1ffb700 1 -- 172.21.15.78:0/2519470645 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9c800fc00 con 0x7fc9d40ff170 2023-10-10T17:42:15.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.107+0000 7fc9d1ffb700 1 -- 172.21.15.78:0/2519470645 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c8017650 con 0x7fc9d40ff170 2023-10-10T17:42:15.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.107+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d410a9e0 con 0x7fc9d40ff170 2023-10-10T17:42:15.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.108+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d41061c0 con 0x7fc9d40ff170 2023-10-10T17:42:15.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.109+0000 7fc9d1ffb700 1 -- 172.21.15.78:0/2519470645 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc9c8022470 con 0x7fc9d40ff170 2023-10-10T17:42:15.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.109+0000 7fc9d1ffb700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9c40a1150 0x7fc9c40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.110+0000 7fc9d1ffb700 1 -- 172.21.15.78:0/2519470645 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9c80cde60 con 0x7fc9d40ff170 2023-10-10T17:42:15.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.110+0000 7fc9d3fff700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9c40a1150 0x7fc9c40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.110+0000 7fc9d3fff700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9c40a1150 0x7fc9c40a3600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0006fd0 tx=0x7fc9c0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.114+0000 7fc9d1ffb700 1 -- 172.21.15.78:0/2519470645 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc9c8097330 con 0x7fc9d40ff170 2023-10-10T17:42:15.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.347+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9d402cfe0 con 0x7fc9d40ff170 2023-10-10T17:42:15.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.348+0000 7fc9d1ffb700 1 -- 172.21.15.78:0/2519470645 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7fc9c809a950 con 0x7fc9d40ff170 2023-10-10T17:42:15.349 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:15.349 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:42:15.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9c40a1150 msgr2=0x7fc9c40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9c40a1150 0x7fc9c40a3600 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc9c0006fd0 tx=0x7fc9c0008040 comp rx=0 tx=0).stop 2023-10-10T17:42:15.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 msgr2=0x7fc9d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d4068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc9c8000c00 tx=0x7fc9c800f710 comp rx=0 tx=0).stop 2023-10-10T17:42:15.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 shutdown_connections 2023-10-10T17:42:15.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc9c40a1150 0x7fc9c40a3600 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc9d4100db0 0x7fc9d419d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc9d40ff170 0x7fc9d4068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 --2- 172.21.15.78:0/2519470645 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc9d4100400 0x7fc9d419ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 >> 172.21.15.78:0/2519470645 conn(0x7fc9d40fa750 msgr2=0x7fc9d40fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 shutdown_connections 2023-10-10T17:42:15.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.350+0000 7fc9daa99700 1 -- 172.21.15.78:0/2519470645 wait complete. 2023-10-10T17:42:15.354 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:42:15.363 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:42:15.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 -- 172.21.15.78:0/2544656339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec8100d40 msgr2=0x7f8ec8107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 --2- 172.21.15.78:0/2544656339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec8100d40 0x7f8ec8107600 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000b0a0 tx=0x7f8ec000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 -- 172.21.15.78:0/2544656339 shutdown_connections 2023-10-10T17:42:15.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 --2- 172.21.15.78:0/2544656339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ec80ff1c0 0x7f8ec80ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 --2- 172.21.15.78:0/2544656339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec8100d40 0x7f8ec8107600 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 --2- 172.21.15.78:0/2544656339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ec81003b0 0x7f8ec8100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 -- 172.21.15.78:0/2544656339 >> 172.21.15.78:0/2544656339 conn(0x7f8ec80fa7c0 msgr2=0x7f8ec80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 -- 172.21.15.78:0/2544656339 shutdown_connections 2023-10-10T17:42:15.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.484+0000 7f8ecef16700 1 -- 172.21.15.78:0/2544656339 wait complete. 2023-10-10T17:42:15.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 Processor -- start 2023-10-10T17:42:15.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 -- start start 2023-10-10T17:42:15.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec80ff1c0 0x7f8ec8195fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ec81003b0 0x7f8ec8196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ec8100d40 0x7f8ec819d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec819db40 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec819dc80 con 0x7f8ec81003b0 2023-10-10T17:42:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecef16700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ec819de10 con 0x7f8ec8100d40 2023-10-10T17:42:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecccb2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec80ff1c0 0x7f8ec8195fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecccb2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec80ff1c0 0x7f8ec8195fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43676/0 (socket says 172.21.15.78:43676) 2023-10-10T17:42:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecccb2700 1 -- 172.21.15.78:0/847221071 learned_addr learned my addr 172.21.15.78:0/847221071 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:15.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ecd4b3700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ec8100d40 0x7f8ec819d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.485+0000 7f8ec7fff700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ec81003b0 0x7f8ec8196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecccb2700 1 -- 172.21.15.78:0/847221071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ec8100d40 msgr2=0x7f8ec819d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecccb2700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ec8100d40 0x7f8ec819d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecccb2700 1 -- 172.21.15.78:0/847221071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ec81003b0 msgr2=0x7f8ec8196500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecccb2700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ec81003b0 0x7f8ec8196500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecccb2700 1 -- 172.21.15.78:0/847221071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec0009d00 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecd4b3700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ec8100d40 0x7f8ec819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:15.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ec7fff700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ec81003b0 0x7f8ec8196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:15.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecccb2700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec80ff1c0 0x7f8ec8195fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00c970 tx=0x7f8ebc00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ec5ffb700 1 -- 172.21.15.78:0/847221071 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc004220 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ec5ffb700 1 -- 172.21.15.78:0/847221071 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ebc007d60 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec810ab40 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ec5ffb700 1 -- 172.21.15.78:0/847221071 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ebc003d00 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.486+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec810a9b0 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.487+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec819e560 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.488+0000 7f8ec5ffb700 1 -- 172.21.15.78:0/847221071 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8ebc004380 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.488+0000 7f8ec5ffb700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8eb80a1150 0x7f8eb80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.489+0000 7f8ec5ffb700 1 -- 172.21.15.78:0/847221071 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8ebc0ccbc0 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.489+0000 7f8ec7fff700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8eb80a1150 0x7f8eb80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.489+0000 7f8ec7fff700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8eb80a1150 0x7f8eb80a3600 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400baa0 tx=0x7f8eb4005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.493+0000 7f8ec5ffb700 1 -- 172.21.15.78:0/847221071 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8ebc096000 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.686+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ec819eb10 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.688+0000 7f8ec5ffb700 1 -- 172.21.15.78:0/847221071 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7f8ebc01d090 con 0x7f8ec80ff1c0 2023-10-10T17:42:15.689 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:15.689 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:42:15.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8eb80a1150 msgr2=0x7f8eb80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8eb80a1150 0x7f8eb80a3600 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8eb400baa0 tx=0x7f8eb4005db0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec80ff1c0 msgr2=0x7f8ec8195fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec80ff1c0 0x7f8ec8195fc0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f8ebc00c970 tx=0x7f8ebc00cd30 comp rx=0 tx=0).stop 2023-10-10T17:42:15.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 shutdown_connections 2023-10-10T17:42:15.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8eb80a1150 0x7f8eb80a3600 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ec8100d40 0x7f8ec819d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ec80ff1c0 0x7f8ec8195fc0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 --2- 172.21.15.78:0/847221071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ec81003b0 0x7f8ec8196500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 >> 172.21.15.78:0/847221071 conn(0x7f8ec80fa7c0 msgr2=0x7f8ec8069090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 shutdown_connections 2023-10-10T17:42:15.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.690+0000 7f8ecef16700 1 -- 172.21.15.78:0/847221071 wait complete. 2023-10-10T17:42:15.704 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-10-10T17:40:41.574125+0000', 'modified': '2023-10-10T17:41:01.590420+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24500, 'mds_1': 24490, 'mds_2': 24472}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24437': {'gid': 24437, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.157:6835/2501992479', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6834', 'nonce': 2501992479}, {'type': 'v1', 'addr': '172.21.15.157:6835', 'nonce': 2501992479}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.190:6835/1269112994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6834', 'nonce': 1269112994}, {'type': 'v1', 'addr': '172.21.15.190:6835', 'nonce': 1269112994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24467': {'gid': 24467, 'name': 'h', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.157:6837/1666360072', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6836', 'nonce': 1666360072}, {'type': 'v1', 'addr': '172.21.15.157:6837', 'nonce': 1666360072}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.190:6837/661255149', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6836', 'nonce': 661255149}, {'type': 'v1', 'addr': '172.21.15.190:6837', 'nonce': 661255149}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24490': {'gid': 24490, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.190:6839/486459111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6838', 'nonce': 486459111}, {'type': 'v1', 'addr': '172.21.15.190:6839', 'nonce': 486459111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24500': {'gid': 24500, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.157:6839/2554315829', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.157:6838', 'nonce': 2554315829}, {'type': 'v1', 'addr': '172.21.15.157:6839', 'nonce': 2554315829}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-10T17:42:15.705 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-10T17:42:15.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-10T17:42:15.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 -- 172.21.15.78:0/2241130490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100310 msgr2=0x7f7688100760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 --2- 172.21.15.78:0/2241130490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100310 0x7f7688100760 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f7684009990 tx=0x7f7684009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 -- 172.21.15.78:0/2241130490 shutdown_connections 2023-10-10T17:42:15.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 --2- 172.21.15.78:0/2241130490 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f76880ff120 0x7f76880ff530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 --2- 172.21.15.78:0/2241130490 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7688100ca0 0x7f7688107560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 --2- 172.21.15.78:0/2241130490 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100310 0x7f7688100760 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 -- 172.21.15.78:0/2241130490 >> 172.21.15.78:0/2241130490 conn(0x7f76880fa700 msgr2=0x7f76880fcb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 -- 172.21.15.78:0/2241130490 shutdown_connections 2023-10-10T17:42:15.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 -- 172.21.15.78:0/2241130490 wait complete. 2023-10-10T17:42:15.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.825+0000 7f768f9c2700 1 Processor -- start 2023-10-10T17:42:15.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 -- start start 2023-10-10T17:42:15.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76880ff120 0x7f7688195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7688100310 0x7f7688196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100ca0 0x7f768819d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768819db50 con 0x7f7688100ca0 2023-10-10T17:42:15.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768819dc90 con 0x7f76880ff120 2023-10-10T17:42:15.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768819de20 con 0x7f7688100310 2023-10-10T17:42:15.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100ca0 0x7f768819d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100ca0 0x7f768819d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43694/0 (socket says 172.21.15.78:43694) 2023-10-10T17:42:15.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 -- 172.21.15.78:0/3673800625 learned_addr learned my addr 172.21.15.78:0/3673800625 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:15.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768cf5d700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7688100310 0x7f7688196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768d75e700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76880ff120 0x7f7688195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 -- 172.21.15.78:0/3673800625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7688100310 msgr2=0x7f7688196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7688100310 0x7f7688196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 -- 172.21.15.78:0/3673800625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76880ff120 msgr2=0x7f7688195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76880ff120 0x7f7688195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 -- 172.21.15.78:0/3673800625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840095f0 con 0x7f7688100ca0 2023-10-10T17:42:15.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768d75e700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76880ff120 0x7f7688195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:15.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768df5f700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100ca0 0x7f768819d580 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7680007ee0 tx=0x7f768000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f767e7fc700 1 -- 172.21.15.78:0/3673800625 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680017070 con 0x7f7688100ca0 2023-10-10T17:42:15.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f767e7fc700 1 -- 172.21.15.78:0/3673800625 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f768000f040 con 0x7f7688100ca0 2023-10-10T17:42:15.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768810ab20 con 0x7f7688100ca0 2023-10-10T17:42:15.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.826+0000 7f767e7fc700 1 -- 172.21.15.78:0/3673800625 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680005740 con 0x7f7688100ca0 2023-10-10T17:42:15.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.827+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768810a990 con 0x7f7688100ca0 2023-10-10T17:42:15.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.828+0000 7f767e7fc700 1 -- 172.21.15.78:0/3673800625 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f76800058a0 con 0x7f7688100ca0 2023-10-10T17:42:15.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.829+0000 7f767e7fc700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f76740a1150 0x7f76740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.829+0000 7f768d75e700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f76740a1150 0x7f76740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.829+0000 7f767e7fc700 1 -- 172.21.15.78:0/3673800625 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f76800cc4f0 con 0x7f7688100ca0 2023-10-10T17:42:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f768d75e700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f76740a1150 0x7f76740a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7688100170 tx=0x7f7678008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 Processor -- start 2023-10-10T17:42:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 -- start start 2023-10-10T17:42:15.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e6bb0 0x555e222e6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222e7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222e8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e221ec010 con 0x555e222e80c0 2023-10-10T17:42:15.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e2223e850 con 0x555e222e7500 2023-10-10T17:42:15.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e22237ce0 con 0x555e222e6bb0 2023-10-10T17:42:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f768df5f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222e8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f768df5f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222e8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43702/0 (socket says 172.21.15.78:43702) 2023-10-10T17:42:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f768df5f700 1 -- 172.21.15.78:0/1888148345 learned_addr learned my addr 172.21.15.78:0/1888148345 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f768d75e700 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e6bb0 0x555e222e6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f768cf5d700 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222e7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.830+0000 7f768df5f700 1 -- 172.21.15.78:0/1888148345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e6bb0 msgr2=0x555e222e6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f768df5f700 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e6bb0 0x555e222e6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f768df5f700 1 -- 172.21.15.78:0/1888148345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 msgr2=0x555e222e7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f768df5f700 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222e7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f768df5f700 1 -- 172.21.15.78:0/1888148345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e220b95e0 con 0x555e222e80c0 2023-10-10T17:42:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f768cf5d700 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222e7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f768df5f700 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222e8530 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f76800a14b0 tx=0x7f7680099360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f23708fa2daa868 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f76737fe700 1 -- 172.21.15.78:0/1888148345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76800a19e0 con 0x555e222e80c0 2023-10-10T17:42:15.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f76737fe700 1 -- 172.21.15.78:0/1888148345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76800cb040 con 0x555e222e80c0 2023-10-10T17:42:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f76737fe700 1 -- 172.21.15.78:0/1888148345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768000dbd0 con 0x555e222e80c0 2023-10-10T17:42:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1888148345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 msgr2=0x555e222e8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222e8530 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f76800a14b0 tx=0x7f7680099360 comp rx=0 tx=0).stop 2023-10-10T17:42:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1888148345 shutdown_connections 2023-10-10T17:42:15.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.831+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e6bb0 0x555e222e6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222e8530 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1888148345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222e7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1888148345 >> 172.21.15.78:0/1888148345 conn(0x555e222dc310 msgr2=0x555e222e1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1888148345 shutdown_connections 2023-10-10T17:42:15.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1888148345 wait complete. 2023-10-10T17:42:15.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 -- start start 2023-10-10T17:42:15.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 Processor -- start 2023-10-10T17:42:15.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f0620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e7500 0x555e222f0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222f7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e221d9890 con 0x555e222e80c0 2023-10-10T17:42:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e221d9530 con 0x555e222e6bb0 2023-10-10T17:42:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e220bb8d0 con 0x555e222e7500 2023-10-10T17:42:15.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f769bb4eb80 10 client.0 _ll_register_callbacks cb 0x555e221f7730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f768df5f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222f7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f768df5f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222f7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43708/0 (socket says 172.21.15.78:43708) 2023-10-10T17:42:15.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.832+0000 7f768df5f700 1 -- 172.21.15.78:0/2189768756 learned_addr learned my addr 172.21.15.78:0/2189768756 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768d75e700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768cf5d700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e7500 0x555e222f0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768df5f700 1 -- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e7500 msgr2=0x555e222f0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768df5f700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e7500 0x555e222f0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768df5f700 1 -- 172.21.15.78:0/2189768756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 msgr2=0x555e222f0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768df5f700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768df5f700 1 -- 172.21.15.78:0/2189768756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768810a990 con 0x555e222e80c0 2023-10-10T17:42:15.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768cf5d700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e7500 0x555e222f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:15.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768d75e700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:15.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f768df5f700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222f7ba0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7680097b80 tx=0x7f76800b9180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9118f34c35b89858 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f7670ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:15.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f7670ff9700 1 -- 172.21.15.78:0/2189768756 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76800b5070 con 0x555e222e80c0 2023-10-10T17:42:15.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f7670ff9700 1 -- 172.21.15.78:0/2189768756 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f768009d380 con 0x555e222e80c0 2023-10-10T17:42:15.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e222d9ad0 con 0x555e222e80c0 2023-10-10T17:42:15.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f7670ff9700 1 -- 172.21.15.78:0/2189768756 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76800a19e0 con 0x555e222e80c0 2023-10-10T17:42:15.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 10 client.15363 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:15.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555e221fe140 con 0x555e222e80c0 2023-10-10T17:42:15.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f7670ff9700 1 -- 172.21.15.78:0/2189768756 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e222d9ad0 con 0x555e222e80c0 2023-10-10T17:42:15.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f7670ff9700 1 -- 172.21.15.78:0/2189768756 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7680098af0 con 0x555e222e80c0 2023-10-10T17:42:15.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 10 client.15363 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:15.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 10 client.15363 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-10T17:42:15.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 20 client.15363 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:15.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e222fb2b0 0x555e222faa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.833+0000 7f769bb4eb80 4 client.15363 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:15.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.834+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e221fa050 con 0x555e222fb2b0 2023-10-10T17:42:15.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.834+0000 7f768d75e700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e222fb2b0 0x555e222faa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.834+0000 7f7670ff9700 10 client.15363 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:15.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.834+0000 7f768d75e700 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e222fb2b0 0x555e222faa40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9a7b5bfd47bdb9c8 server_cookie=53f1a85bed031b4c in_seq=0 out_seq=0 2023-10-10T17:42:15.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f7670ff9700 1 -- 172.21.15.78:0/2189768756 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555e221fa050 con 0x555e222fb2b0 2023-10-10T17:42:15.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f7670ff9700 10 client.15363 handle_command_reply: tid=0 2023-10-10T17:42:15.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 client.15363 shutdown 2023-10-10T17:42:15.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 msgr2=0x555e222f7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222f7ba0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7680097b80 tx=0x7f76800b9180 comp rx=0 tx=0).stop 2023-10-10T17:42:15.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 shutdown_connections 2023-10-10T17:42:15.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e80c0 0x555e222f7ba0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e222fb2b0 0x555e222faa40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2189768756 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e7500 0x555e222f0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f7670ff9700 0 client.15363 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:15.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 >> 172.21.15.78:0/2189768756 conn(0x555e222dc310 msgr2=0x555e222e0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 shutdown_connections 2023-10-10T17:42:15.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2189768756 wait complete. 2023-10-10T17:42:15.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.835+0000 7f769bb4eb80 20 client.15363 trim_cache size 0 max 16384 2023-10-10T17:42:15.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 Processor -- start 2023-10-10T17:42:15.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 -- start start 2023-10-10T17:42:15.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e6bb0 0x555e222eb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222eb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 0x555e222f03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e222f13b0 con 0x555e222e6bb0 2023-10-10T17:42:15.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e222fc590 con 0x555e222e7500 2023-10-10T17:42:15.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e221fa050 con 0x555e222e80c0 2023-10-10T17:42:15.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e6bb0 0x555e222eb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e6bb0 0x555e222eb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43712/0 (socket says 172.21.15.78:43712) 2023-10-10T17:42:15.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 -- 172.21.15.78:0/2358664101 learned_addr learned my addr 172.21.15.78:0/2358664101 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:15.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 -- 172.21.15.78:0/2358664101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 msgr2=0x555e222f03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 --2- 172.21.15.78:0/2358664101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 0x555e222f03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 -- 172.21.15.78:0/2358664101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 msgr2=0x555e222eb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 --2- 172.21.15.78:0/2358664101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222eb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 -- 172.21.15.78:0/2358664101 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e222ddab0 con 0x555e222e6bb0 2023-10-10T17:42:15.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f768d75e700 1 --2- 172.21.15.78:0/2358664101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e6bb0 0x555e222eb0b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x555e221d95f0 tx=0x7f767800f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6291fcb9328765d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f7673fff700 1 -- 172.21.15.78:0/2358664101 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767800cce0 con 0x555e222e6bb0 2023-10-10T17:42:15.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f7673fff700 1 -- 172.21.15.78:0/2358664101 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f767800ce40 con 0x555e222e6bb0 2023-10-10T17:42:15.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.853+0000 7f7673fff700 1 -- 172.21.15.78:0/2358664101 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767802db50 con 0x555e222e6bb0 2023-10-10T17:42:15.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2358664101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e6bb0 msgr2=0x555e222eb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2358664101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e6bb0 0x555e222eb0b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x555e221d95f0 tx=0x7f767800f000 comp rx=0 tx=0).stop 2023-10-10T17:42:15.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2358664101 shutdown_connections 2023-10-10T17:42:15.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2358664101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 0x555e222f03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2358664101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e6bb0 0x555e222eb0b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/2358664101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e7500 0x555e222eb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2358664101 >> 172.21.15.78:0/2358664101 conn(0x555e222dc310 msgr2=0x555e222ee5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2358664101 shutdown_connections 2023-10-10T17:42:15.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 -- 172.21.15.78:0/2358664101 wait complete. 2023-10-10T17:42:15.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 -- start start 2023-10-10T17:42:15.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.854+0000 7f769bb4eb80 1 Processor -- start 2023-10-10T17:42:15.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e7500 0x555e222f7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 0x555e223533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e22353900 con 0x555e222e7500 2023-10-10T17:42:15.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e22353a40 con 0x555e222e6bb0 2023-10-10T17:42:15.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e22353c00 con 0x555e222e80c0 2023-10-10T17:42:15.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 10 client.0 _ll_register_callbacks cb 0x7f7680098fa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:15.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e7500 0x555e222f7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e7500 0x555e222f7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43720/0 (socket says 172.21.15.78:43720) 2023-10-10T17:42:15.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 -- 172.21.15.78:0/1399909287 learned_addr learned my addr 172.21.15.78:0/1399909287 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:15.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768d75e700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768df5f700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 0x555e223533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 -- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 msgr2=0x555e223533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 0x555e223533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 -- 172.21.15.78:0/1399909287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 msgr2=0x555e222f6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:15.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 -- 172.21.15.78:0/1399909287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f767801b040 con 0x555e222e7500 2023-10-10T17:42:15.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768d75e700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:15.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f768cf5d700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e7500 0x555e222f7450 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7684009990 tx=0x7f7684005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3656722bf505f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:15.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f7670ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:15.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f7670ff9700 1 -- 172.21.15.78:0/1399909287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768400bc00 con 0x555e222e7500 2023-10-10T17:42:15.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f7670ff9700 1 -- 172.21.15.78:0/1399909287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f768400bd60 con 0x555e222e7500 2023-10-10T17:42:15.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.855+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76800cb040 con 0x555e222e7500 2023-10-10T17:42:15.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f7670ff9700 1 -- 172.21.15.78:0/1399909287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7684004500 con 0x555e222e7500 2023-10-10T17:42:15.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 10 client.15369 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:15.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555e22065d90 con 0x555e222e7500 2023-10-10T17:42:15.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f7670ff9700 1 -- 172.21.15.78:0/1399909287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f76800cb040 con 0x555e222e7500 2023-10-10T17:42:15.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f7670ff9700 1 -- 172.21.15.78:0/1399909287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7684003b90 con 0x555e222e7500 2023-10-10T17:42:15.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 10 client.15369 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:15.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 10 client.15369 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-10T17:42:15.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 20 client.15369 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:15.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e223588b0 0x555e223552a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:15.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 4 client.15369 mds_command: new command op to 24472 tid=0[{"prefix": "status"}] 2023-10-10T17:42:15.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555e22206610 con 0x555e223588b0 2023-10-10T17:42:15.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.856+0000 7f768d75e700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e223588b0 0x555e223552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:15.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f7670ff9700 10 client.15369 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:15.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f768d75e700 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e223588b0 0x555e223552a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e810a44818e414fd server_cookie=df9692c7aea63ba6 in_seq=0 out_seq=0 2023-10-10T17:42:15.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f7670ff9700 1 -- 172.21.15.78:0/1399909287 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x555e22206610 con 0x555e223588b0 2023-10-10T17:42:15.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f7670ff9700 10 client.15369 handle_command_reply: tid=0 2023-10-10T17:42:15.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 client.15369 shutdown 2023-10-10T17:42:15.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e7500 msgr2=0x555e222f7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e7500 0x555e222f7450 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7684009990 tx=0x7f7684005c00 comp rx=0 tx=0).stop 2023-10-10T17:42:15.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 shutdown_connections 2023-10-10T17:42:15.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x555e223588b0 0x555e223552a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555e222e80c0 0x555e223533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f7670ff9700 0 client.15369 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:15.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555e222e6bb0 0x555e222f6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 --2- 172.21.15.78:0/1399909287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555e222e7500 0x555e222f7450 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 >> 172.21.15.78:0/1399909287 conn(0x555e222dc310 msgr2=0x555e222eb0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 shutdown_connections 2023-10-10T17:42:15.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 1 -- 172.21.15.78:0/1399909287 wait complete. 2023-10-10T17:42:15.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.857+0000 7f769bb4eb80 20 client.15369 trim_cache size 0 max 16384 2023-10-10T17:42:15.885 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:42:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:42:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2023-10-10T17:42:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24472, 2023-10-10T17:42:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-10T17:42:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-10T17:42:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-10T17:42:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 75.271503148999997, 2023-10-10T17:42:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2023-10-10T17:42:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2023-10-10T17:42:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2023-10-10T17:42:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 111.02283373100001 2023-10-10T17:42:15.889 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:42:15.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f76740a1150 msgr2=0x7f76740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f76740a1150 0x7f76740a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7688100170 tx=0x7f7678008040 comp rx=0 tx=0).stop 2023-10-10T17:42:15.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100ca0 msgr2=0x7f768819d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:15.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100ca0 0x7f768819d580 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7680007ee0 tx=0x7f768000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 shutdown_connections 2023-10-10T17:42:15.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f76740a1150 0x7f76740a3600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7688100ca0 0x7f768819d580 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f76880ff120 0x7f7688195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 --2- 172.21.15.78:0/3673800625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7688100310 0x7f7688196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:15.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 >> 172.21.15.78:0/3673800625 conn(0x7f76880fa700 msgr2=0x7f76880fc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:15.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 shutdown_connections 2023-10-10T17:42:15.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:15.858+0000 7f768f9c2700 1 -- 172.21.15.78:0/3673800625 wait complete. 2023-10-10T17:42:15.896 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-10-10T17:42:15.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:15 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2767670890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:15.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:15 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2842158100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:15.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:15 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2519470645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:15.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:15 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2767670890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:15.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:15 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2842158100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:15.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:15 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2519470645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:15 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2767670890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:15 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2842158100' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:15.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:15 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2519470645' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:16.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 -- 172.21.15.78:0/3025282755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 msgr2=0x7f34b4100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/3025282755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4100750 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f34a4009990 tx=0x7f34a4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 -- 172.21.15.78:0/3025282755 shutdown_connections 2023-10-10T17:42:16.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/3025282755 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34b4100c90 0x7f34b4107550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/3025282755 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b40ff110 0x7f34b40ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/3025282755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4100750 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 -- 172.21.15.78:0/3025282755 >> 172.21.15.78:0/3025282755 conn(0x7f34b40fa6f0 msgr2=0x7f34b40fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 -- 172.21.15.78:0/3025282755 shutdown_connections 2023-10-10T17:42:16.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.017+0000 7f34bb5f7700 1 -- 172.21.15.78:0/3025282755 wait complete. 2023-10-10T17:42:16.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 Processor -- start 2023-10-10T17:42:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 -- start start 2023-10-10T17:42:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34b40ff110 0x7f34b4195ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b4100c90 0x7f34b419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b419da70 con 0x7f34b4100300 2023-10-10T17:42:16.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b419dbb0 con 0x7f34b4100c90 2023-10-10T17:42:16.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b419dd40 con 0x7f34b40ff110 2023-10-10T17:42:16.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43734/0 (socket says 172.21.15.78:43734) 2023-10-10T17:42:16.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 -- 172.21.15.78:0/532040315 learned_addr learned my addr 172.21.15.78:0/532040315 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b9393700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34b40ff110 0x7f34b4195ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 -- 172.21.15.78:0/532040315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34b40ff110 msgr2=0x7f34b4195ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34b40ff110 0x7f34b4195ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 -- 172.21.15.78:0/532040315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b4100c90 msgr2=0x7f34b419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b4100c90 0x7f34b419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 -- 172.21.15.78:0/532040315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a40095f0 con 0x7f34b4100300 2023-10-10T17:42:16.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34b8b92700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4196430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f34a4005f50 tx=0x7f34a40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34aa7fc700 1 -- 172.21.15.78:0/532040315 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a401d070 con 0x7f34b4100300 2023-10-10T17:42:16.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34aa7fc700 1 -- 172.21.15.78:0/532040315 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34a400fbf0 con 0x7f34b4100300 2023-10-10T17:42:16.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b410aa80 con 0x7f34b4100300 2023-10-10T17:42:16.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34aa7fc700 1 -- 172.21.15.78:0/532040315 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a4017650 con 0x7f34b4100300 2023-10-10T17:42:16.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.018+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b410a8f0 con 0x7f34b4100300 2023-10-10T17:42:16.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.020+0000 7f34aa7fc700 1 -- 172.21.15.78:0/532040315 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f34a4021410 con 0x7f34b4100300 2023-10-10T17:42:16.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.021+0000 7f34aa7fc700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f34a00a1150 0x7f34a00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.021+0000 7f34aa7fc700 1 -- 172.21.15.78:0/532040315 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f34a40ce140 con 0x7f34b4100300 2023-10-10T17:42:16.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.021+0000 7f34b9393700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f34a00a1150 0x7f34a00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.021+0000 7f34b9393700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f34a00a1150 0x7f34a00a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f34b4100160 tx=0x7f34b0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 Processor -- start 2023-10-10T17:42:16.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 -- start start 2023-10-10T17:42:16.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f78fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f79ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f7a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8e7e010 con 0x5569a8f78bb0 2023-10-10T17:42:16.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8ed0850 con 0x5569a8f7a0c0 2023-10-10T17:42:16.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34c7783b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8ec9ce0 con 0x5569a8f79500 2023-10-10T17:42:16.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34b9393700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f78fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34b9393700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f78fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43740/0 (socket says 172.21.15.78:43740) 2023-10-10T17:42:16.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34b9393700 1 -- 172.21.15.78:0/2308063192 learned_addr learned my addr 172.21.15.78:0/2308063192 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34b8b92700 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f79ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.022+0000 7f34b9b94700 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f7a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34b9393700 1 -- 172.21.15.78:0/2308063192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 msgr2=0x5569a8f79ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34b9393700 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f79ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34b9393700 1 -- 172.21.15.78:0/2308063192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 msgr2=0x5569a8f7a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34b9393700 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f7a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34b9393700 1 -- 172.21.15.78:0/2308063192 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569a8d4b5e0 con 0x5569a8f78bb0 2023-10-10T17:42:16.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34b9b94700 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f7a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:16.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34b9393700 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f78fc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f34b0011d50 tx=0x7f34b0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9cea04dd1c5c0dd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f349f7fe700 1 -- 172.21.15.78:0/2308063192 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b000edd0 con 0x5569a8f78bb0 2023-10-10T17:42:16.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f349f7fe700 1 -- 172.21.15.78:0/2308063192 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34b0021070 con 0x5569a8f78bb0 2023-10-10T17:42:16.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f349f7fe700 1 -- 172.21.15.78:0/2308063192 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b001d670 con 0x5569a8f78bb0 2023-10-10T17:42:16.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34c7783b80 1 -- 172.21.15.78:0/2308063192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 msgr2=0x5569a8f78fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.023+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f78fc0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f34b0011d50 tx=0x7f34b0014000 comp rx=0 tx=0).stop 2023-10-10T17:42:16.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- 172.21.15.78:0/2308063192 shutdown_connections 2023-10-10T17:42:16.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f7a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f78fc0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2308063192 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f79ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- 172.21.15.78:0/2308063192 >> 172.21.15.78:0/2308063192 conn(0x5569a8f6e310 msgr2=0x5569a8f73620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- 172.21.15.78:0/2308063192 shutdown_connections 2023-10-10T17:42:16.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- 172.21.15.78:0/2308063192 wait complete. 2023-10-10T17:42:16.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- start start 2023-10-10T17:42:16.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 Processor -- start 2023-10-10T17:42:16.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f78bb0 0x5569a8f82590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f82ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8e6b890 con 0x5569a8f7a0c0 2023-10-10T17:42:16.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8e6b530 con 0x5569a8f78bb0 2023-10-10T17:42:16.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8d4d8d0 con 0x5569a8f79500 2023-10-10T17:42:16.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.024+0000 7f34c7783b80 10 client.0 _ll_register_callbacks cb 0x5569a8e89730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:16.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43750/0 (socket says 172.21.15.78:43750) 2023-10-10T17:42:16.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 -- 172.21.15.78:0/652888515 learned_addr learned my addr 172.21.15.78:0/652888515 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9393700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f78bb0 0x5569a8f82590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b8b92700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f82ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 -- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 msgr2=0x5569a8f82ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f82ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 -- 172.21.15.78:0/652888515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f78bb0 msgr2=0x5569a8f82590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f78bb0 0x5569a8f82590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 -- 172.21.15.78:0/652888515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b0017070 con 0x5569a8f7a0c0 2023-10-10T17:42:16.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b8b92700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f82ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:16.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34b9b94700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89b10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00dca0 tx=0x7f34ac008d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3da09630fd82be2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f349cff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:16.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f349cff9700 1 -- 172.21.15.78:0/652888515 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac00b9c0 con 0x5569a8f7a0c0 2023-10-10T17:42:16.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5569a8f6bad0 con 0x5569a8f7a0c0 2023-10-10T17:42:16.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f349cff9700 1 -- 172.21.15.78:0/652888515 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f34ac00bb20 con 0x5569a8f7a0c0 2023-10-10T17:42:16.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f349cff9700 1 -- 172.21.15.78:0/652888515 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34ac00bce0 con 0x5569a8f7a0c0 2023-10-10T17:42:16.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34c7783b80 10 client.15381 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:16.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5569a8e90140 con 0x5569a8f7a0c0 2023-10-10T17:42:16.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.025+0000 7f349cff9700 1 -- 172.21.15.78:0/652888515 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5569a8f6bad0 con 0x5569a8f7a0c0 2023-10-10T17:42:16.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f349cff9700 1 -- 172.21.15.78:0/652888515 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f34ac0123c0 con 0x5569a8f7a0c0 2023-10-10T17:42:16.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34c7783b80 10 client.15381 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:16.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34c7783b80 10 client.15381 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-10T17:42:16.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34c7783b80 20 client.15381 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:16.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34c7783b80 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8f8d8b0 0x5569a8f8cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34c7783b80 4 client.15381 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:16.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569a8e8c050 con 0x5569a8f8d8b0 2023-10-10T17:42:16.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34b9393700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8f8d8b0 0x5569a8f8cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f34b9393700 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8f8d8b0 0x5569a8f8cc30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=299e2efcf9c5ba1f server_cookie=ab66cf8f27f0e957 in_seq=0 out_seq=0 2023-10-10T17:42:16.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.026+0000 7f349cff9700 10 client.15381 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:16.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f349cff9700 1 -- 172.21.15.78:0/652888515 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5569a8e8c050 con 0x5569a8f8d8b0 2023-10-10T17:42:16.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f349cff9700 10 client.15381 handle_command_reply: tid=0 2023-10-10T17:42:16.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 client.15381 shutdown 2023-10-10T17:42:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f7a0c0 msgr2=0x5569a8f89b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89b10 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00dca0 tx=0x7f34ac008d00 comp rx=0 tx=0).stop 2023-10-10T17:42:16.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 shutdown_connections 2023-10-10T17:42:16.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89b10 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f78bb0 0x5569a8f82590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8f8d8b0 0x5569a8f8cc30 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 --2- 172.21.15.78:0/652888515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f82ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 >> 172.21.15.78:0/652888515 conn(0x5569a8f6e310 msgr2=0x5569a8f72d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.027+0000 7f349cff9700 0 client.15381 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:16.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.028+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 shutdown_connections 2023-10-10T17:42:16.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.028+0000 7f34c7783b80 1 -- 172.21.15.78:0/652888515 wait complete. 2023-10-10T17:42:16.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.028+0000 7f34c7783b80 20 client.15381 trim_cache size 0 max 16384 2023-10-10T17:42:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 Processor -- start 2023-10-10T17:42:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 -- start start 2023-10-10T17:42:16.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f812d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f89a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f8cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8e8c050 con 0x5569a8f78bb0 2023-10-10T17:42:16.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8dd5ed0 con 0x5569a8f7a0c0 2023-10-10T17:42:16.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34c7783b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8e65e10 con 0x5569a8f79500 2023-10-10T17:42:16.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34b9393700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f812d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34b9393700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f812d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43754/0 (socket says 172.21.15.78:43754) 2023-10-10T17:42:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.045+0000 7f34b9393700 1 -- 172.21.15.78:0/2577796110 learned_addr learned my addr 172.21.15.78:0/2577796110 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b9b94700 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f8cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b8b92700 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f89a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b9393700 1 -- 172.21.15.78:0/2577796110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 msgr2=0x5569a8f89a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b9393700 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f89a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b9393700 1 -- 172.21.15.78:0/2577796110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 msgr2=0x5569a8f8cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b9393700 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f8cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b9393700 1 -- 172.21.15.78:0/2577796110 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569a8e8fe90 con 0x5569a8f78bb0 2023-10-10T17:42:16.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34b9393700 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f812d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f34b0011d50 tx=0x7f34b000ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a9d7436e402938 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f349ffff700 1 -- 172.21.15.78:0/2577796110 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b001cd50 con 0x5569a8f78bb0 2023-10-10T17:42:16.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f349ffff700 1 -- 172.21.15.78:0/2577796110 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34b001ceb0 con 0x5569a8f78bb0 2023-10-10T17:42:16.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f349ffff700 1 -- 172.21.15.78:0/2577796110 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b00324f0 con 0x5569a8f78bb0 2023-10-10T17:42:16.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.046+0000 7f34c7783b80 1 -- 172.21.15.78:0/2577796110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 msgr2=0x5569a8f812d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f812d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f34b0011d50 tx=0x7f34b000ead0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- 172.21.15.78:0/2577796110 shutdown_connections 2023-10-10T17:42:16.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f8cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f78bb0 0x5569a8f812d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 --2- 172.21.15.78:0/2577796110 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f79500 0x5569a8f89a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- 172.21.15.78:0/2577796110 >> 172.21.15.78:0/2577796110 conn(0x5569a8f6e310 msgr2=0x5569a8f965e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- 172.21.15.78:0/2577796110 shutdown_connections 2023-10-10T17:42:16.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- 172.21.15.78:0/2577796110 wait complete. 2023-10-10T17:42:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- start start 2023-10-10T17:42:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 Processor -- start 2023-10-10T17:42:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f78bb0 0x5569a8f83e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f79500 0x5569a8f84350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8f898c0 con 0x5569a8f79500 2023-10-10T17:42:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8f89a30 con 0x5569a8f7a0c0 2023-10-10T17:42:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569a8f89bf0 con 0x5569a8f78bb0 2023-10-10T17:42:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34c7783b80 10 client.0 _ll_register_callbacks cb 0x7f34ac00d4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34b8b92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f79500 0x5569a8f84350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34b8b92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f79500 0x5569a8f84350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43762/0 (socket says 172.21.15.78:43762) 2023-10-10T17:42:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.047+0000 7f34b8b92700 1 -- 172.21.15.78:0/1489494870 learned_addr learned my addr 172.21.15.78:0/1489494870 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b9b94700 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b8b92700 1 -- 172.21.15.78:0/1489494870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f78bb0 msgr2=0x5569a8f83e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b8b92700 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f78bb0 0x5569a8f83e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b8b92700 1 -- 172.21.15.78:0/1489494870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 msgr2=0x5569a8f89380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b8b92700 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b8b92700 1 -- 172.21.15.78:0/1489494870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b002a040 con 0x5569a8f79500 2023-10-10T17:42:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b9b94700 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34b8b92700 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f79500 0x5569a8f84350 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c010 tx=0x7f34a409b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=526cffb61a5d67c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f349cff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:16.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f349cff9700 1 -- 172.21.15.78:0/1489494870 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a40a3690 con 0x5569a8f79500 2023-10-10T17:42:16.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f349cff9700 1 -- 172.21.15.78:0/1489494870 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f34a400fd60 con 0x5569a8f79500 2023-10-10T17:42:16.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34b0021070 con 0x5569a8f79500 2023-10-10T17:42:16.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f349cff9700 1 -- 172.21.15.78:0/1489494870 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34a40a10b0 con 0x5569a8f79500 2023-10-10T17:42:16.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 10 client.15387 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:16.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5569a8ce0c10 con 0x5569a8f79500 2023-10-10T17:42:16.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f349cff9700 1 -- 172.21.15.78:0/1489494870 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f34a40b7080 con 0x5569a8f79500 2023-10-10T17:42:16.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f349cff9700 1 -- 172.21.15.78:0/1489494870 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f34a4097150 con 0x5569a8f79500 2023-10-10T17:42:16.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 10 client.15387 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:16.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 10 client.15387 resolve_mds: resolved name 'i' to daemon mds.i 2023-10-10T17:42:16.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 20 client.15387 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:16.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8fe5ba0 0x5569a8fe24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 4 client.15387 mds_command: new command op to 24490 tid=0[{"prefix": "status"}] 2023-10-10T17:42:16.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.048+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5569a8e98610 con 0x5569a8fe5ba0 2023-10-10T17:42:16.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f34b9393700 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8fe5ba0 0x5569a8fe24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f349cff9700 10 client.15387 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:16.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f34b9393700 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8fe5ba0 0x5569a8fe24f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7a6647b43843dd79 server_cookie=e134c7b49b76d0d1 in_seq=0 out_seq=0 2023-10-10T17:42:16.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f349cff9700 1 -- 172.21.15.78:0/1489494870 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5569a8e98610 con 0x5569a8fe5ba0 2023-10-10T17:42:16.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f349cff9700 10 client.15387 handle_command_reply: tid=0 2023-10-10T17:42:16.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f34c7783b80 1 client.15387 shutdown 2023-10-10T17:42:16.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f79500 msgr2=0x5569a8f84350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.049+0000 7f34c7783b80 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f79500 0x5569a8f84350 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f34a400c010 tx=0x7f34a409b810 comp rx=0 tx=0).stop 2023-10-10T17:42:16.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 shutdown_connections 2023-10-10T17:42:16.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5569a8fe5ba0 0x5569a8fe24f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569a8f78bb0 0x5569a8f83e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569a8f7a0c0 0x5569a8f89380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 --2- 172.21.15.78:0/1489494870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569a8f79500 0x5569a8f84350 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f349cff9700 0 client.15387 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:16.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 >> 172.21.15.78:0/1489494870 conn(0x5569a8f6e310 msgr2=0x5569a8f812d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 shutdown_connections 2023-10-10T17:42:16.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 1 -- 172.21.15.78:0/1489494870 wait complete. 2023-10-10T17:42:16.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34c7783b80 20 client.15387 trim_cache size 0 max 16384 2023-10-10T17:42:16.086 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:42:16.087 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:42:16.087 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2023-10-10T17:42:16.087 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24490, 2023-10-10T17:42:16.088 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-10T17:42:16.088 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-10T17:42:16.088 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-10T17:42:16.089 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 76.471271653000002, 2023-10-10T17:42:16.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2023-10-10T17:42:16.089 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2023-10-10T17:42:16.090 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2023-10-10T17:42:16.090 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 106.96526852700001 2023-10-10T17:42:16.090 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:42:16.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f34a00a1150 msgr2=0x7f34a00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f34a00a1150 0x7f34a00a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f34b4100160 tx=0x7f34b0009260 comp rx=0 tx=0).stop 2023-10-10T17:42:16.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 msgr2=0x7f34b4196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4196430 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f34a4005f50 tx=0x7f34a40057b0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 shutdown_connections 2023-10-10T17:42:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f34a00a1150 0x7f34a00a3600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f34b40ff110 0x7f34b4195ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f34b4100c90 0x7f34b419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 --2- 172.21.15.78:0/532040315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100300 0x7f34b4196430 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 >> 172.21.15.78:0/532040315 conn(0x7f34b40fa6f0 msgr2=0x7f34b40fc910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 shutdown_connections 2023-10-10T17:42:16.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.050+0000 7f34bb5f7700 1 -- 172.21.15.78:0/532040315 wait complete. 2023-10-10T17:42:16.098 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-10T17:42:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 -- 172.21.15.78:0/429920616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 msgr2=0x7f0920107550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 --2- 172.21.15.78:0/429920616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f0920107550 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f091800b0a0 tx=0x7f091800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 -- 172.21.15.78:0/429920616 shutdown_connections 2023-10-10T17:42:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 --2- 172.21.15.78:0/429920616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09200ff110 0x7f09200ff520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 --2- 172.21.15.78:0/429920616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f0920107550 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 --2- 172.21.15.78:0/429920616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0920100300 0x7f0920100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 -- 172.21.15.78:0/429920616 >> 172.21.15.78:0/429920616 conn(0x7f09200fa6f0 msgr2=0x7f09200fcb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 -- 172.21.15.78:0/429920616 shutdown_connections 2023-10-10T17:42:16.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.218+0000 7f0927ce8700 1 -- 172.21.15.78:0/429920616 wait complete. 2023-10-10T17:42:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.219+0000 7f0927ce8700 1 Processor -- start 2023-10-10T17:42:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.219+0000 7f0927ce8700 1 -- start start 2023-10-10T17:42:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.219+0000 7f0927ce8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09200ff110 0x7f0920194f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.219+0000 7f0927ce8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0920100300 0x7f09201954b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0927ce8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f092019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0927ce8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092019caf0 con 0x7f0920100c90 2023-10-10T17:42:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0927ce8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092019cc30 con 0x7f0920100300 2023-10-10T17:42:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0927ce8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f092019cdc0 con 0x7f09200ff110 2023-10-10T17:42:16.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f092019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f092019c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43776/0 (socket says 172.21.15.78:43776) 2023-10-10T17:42:16.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 -- 172.21.15.78:0/1702826866 learned_addr learned my addr 172.21.15.78:0/1702826866 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0925a84700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09200ff110 0x7f0920194f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0925283700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0920100300 0x7f09201954b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 -- 172.21.15.78:0/1702826866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09200ff110 msgr2=0x7f0920194f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09200ff110 0x7f0920194f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 -- 172.21.15.78:0/1702826866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0920100300 msgr2=0x7f09201954b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0920100300 0x7f09201954b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 -- 172.21.15.78:0/1702826866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918009d00 con 0x7f0920100c90 2023-10-10T17:42:16.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0925283700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0920100300 0x7f09201954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0926285700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f092019c520 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f091800b830 tx=0x7f0918004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0916ffd700 1 -- 172.21.15.78:0/1702826866 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091800e040 con 0x7f0920100c90 2023-10-10T17:42:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f092010aa90 con 0x7f0920100c90 2023-10-10T17:42:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0916ffd700 1 -- 172.21.15.78:0/1702826866 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f091801ab90 con 0x7f0920100c90 2023-10-10T17:42:16.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0916ffd700 1 -- 172.21.15.78:0/1702826866 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09180057a0 con 0x7f0920100c90 2023-10-10T17:42:16.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.220+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f092010a900 con 0x7f0920100c90 2023-10-10T17:42:16.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.223+0000 7f0916ffd700 1 -- 172.21.15.78:0/1702826866 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f091801ad00 con 0x7f0920100c90 2023-10-10T17:42:16.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0916ffd700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f090c0a1150 0x7f090c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0916ffd700 1 -- 172.21.15.78:0/1702826866 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f09180d01b0 con 0x7f0920100c90 2023-10-10T17:42:16.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0925a84700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f090c0a1150 0x7f090c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0925a84700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f090c0a1150 0x7f090c0a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0920100160 tx=0x7f091000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0933e74b80 1 Processor -- start 2023-10-10T17:42:16.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0933e74b80 1 -- start start 2023-10-10T17:42:16.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dddbb0 0x55b0e1dddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dde500 0x55b0e1ddeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1ddf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.224+0000 7f0933e74b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1ce3010 con 0x55b0e1dde500 2023-10-10T17:42:16.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0933e74b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1d35850 con 0x55b0e1dddbb0 2023-10-10T17:42:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0933e74b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1d2ece0 con 0x55b0e1ddf0c0 2023-10-10T17:42:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dde500 0x55b0e1ddeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dde500 0x55b0e1ddeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43782/0 (socket says 172.21.15.78:43782) 2023-10-10T17:42:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 -- 172.21.15.78:0/2064387779 learned_addr learned my addr 172.21.15.78:0/2064387779 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925a84700 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dddbb0 0x55b0e1dddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0926285700 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1ddf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 -- 172.21.15.78:0/2064387779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 msgr2=0x55b0e1ddf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1ddf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 -- 172.21.15.78:0/2064387779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dddbb0 msgr2=0x55b0e1dddfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dddbb0 0x55b0e1dddfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 -- 172.21.15.78:0/2064387779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0e1bb05e0 con 0x55b0e1dde500 2023-10-10T17:42:16.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0926285700 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1ddf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:16.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925a84700 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dddbb0 0x55b0e1dddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:16.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f0925283700 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dde500 0x55b0e1ddeae0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f091c00bd90 tx=0x7f091c00f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36e8ff9b1a5b3fc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f090bfff700 1 -- 172.21.15.78:0/2064387779 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c00fe50 con 0x55b0e1dde500 2023-10-10T17:42:16.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f090bfff700 1 -- 172.21.15.78:0/2064387779 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f091c011900 con 0x55b0e1dde500 2023-10-10T17:42:16.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.225+0000 7f090bfff700 1 -- 172.21.15.78:0/2064387779 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c010740 con 0x55b0e1dde500 2023-10-10T17:42:16.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 -- 172.21.15.78:0/2064387779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dde500 msgr2=0x55b0e1ddeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dde500 0x55b0e1ddeae0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f091c00bd90 tx=0x7f091c00f640 comp rx=0 tx=0).stop 2023-10-10T17:42:16.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 -- 172.21.15.78:0/2064387779 shutdown_connections 2023-10-10T17:42:16.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1ddf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dddbb0 0x55b0e1dddfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2064387779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dde500 0x55b0e1ddeae0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 -- 172.21.15.78:0/2064387779 >> 172.21.15.78:0/2064387779 conn(0x55b0e1dd3310 msgr2=0x55b0e1dd8620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 -- 172.21.15.78:0/2064387779 shutdown_connections 2023-10-10T17:42:16.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 -- 172.21.15.78:0/2064387779 wait complete. 2023-10-10T17:42:16.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 -- start start 2023-10-10T17:42:16.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.226+0000 7f0933e74b80 1 Processor -- start 2023-10-10T17:42:16.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0933e74b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1cd0890 con 0x55b0e1dddbb0 2023-10-10T17:42:16.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0933e74b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1cd0530 con 0x55b0e1ddf0c0 2023-10-10T17:42:16.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0933e74b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1bb28d0 con 0x55b0e1dde500 2023-10-10T17:42:16.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0933e74b80 10 client.0 _ll_register_callbacks cb 0x55b0e1cee730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:16.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43786/0 (socket says 172.21.15.78:43786) 2023-10-10T17:42:16.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 -- 172.21.15.78:0/2954021401 learned_addr learned my addr 172.21.15.78:0/2954021401 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925283700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0926285700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 -- 172.21.15.78:0/2954021401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 msgr2=0x55b0e1de7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 -- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 msgr2=0x55b0e1deea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 -- 172.21.15.78:0/2954021401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f091c00b9c0 con 0x55b0e1dddbb0 2023-10-10T17:42:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925283700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f0925a84700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de7510 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f09100116f0 tx=0x7f09100047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0e508005571f2e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.227+0000 7f09097fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f09097fa700 1 -- 172.21.15.78:0/2954021401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091001e830 con 0x55b0e1dddbb0 2023-10-10T17:42:16.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f09097fa700 1 -- 172.21.15.78:0/2954021401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0910017070 con 0x55b0e1dddbb0 2023-10-10T17:42:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0e1dd0ad0 con 0x55b0e1dddbb0 2023-10-10T17:42:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f09097fa700 1 -- 172.21.15.78:0/2954021401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f091001e990 con 0x55b0e1dddbb0 2023-10-10T17:42:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 10 client.15399 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b0e1cf5140 con 0x55b0e1dddbb0 2023-10-10T17:42:16.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f09097fa700 1 -- 172.21.15.78:0/2954021401 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b0e1dd0ad0 con 0x55b0e1dddbb0 2023-10-10T17:42:16.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f09097fa700 1 -- 172.21.15.78:0/2954021401 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f091001bc90 con 0x55b0e1dddbb0 2023-10-10T17:42:16.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 10 client.15399 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:16.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 10 client.15399 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T17:42:16.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 20 client.15399 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:16.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1df2780 0x55b0e1df1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 4 client.15399 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:16.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0e1cf1050 con 0x55b0e1df2780 2023-10-10T17:42:16.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.228+0000 7f0925283700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1df2780 0x55b0e1df1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.229+0000 7f09097fa700 10 client.15399 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:16.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.229+0000 7f0925283700 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1df2780 0x55b0e1df1bb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f92e3d14b0a7119 server_cookie=64f5b8f2ecad7f0c in_seq=0 out_seq=0 2023-10-10T17:42:16.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.229+0000 7f09097fa700 1 -- 172.21.15.78:0/2954021401 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b0e1cf1050 con 0x55b0e1df2780 2023-10-10T17:42:16.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.229+0000 7f09097fa700 10 client.15399 handle_command_reply: tid=0 2023-10-10T17:42:16.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.229+0000 7f0933e74b80 1 client.15399 shutdown 2023-10-10T17:42:16.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 msgr2=0x55b0e1de7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de7510 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f09100116f0 tx=0x7f09100047c0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 shutdown_connections 2023-10-10T17:42:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1df2780 0x55b0e1df1bb0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de7510 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f09097fa700 0 client.15399 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:16.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2954021401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 >> 172.21.15.78:0/2954021401 conn(0x55b0e1dd3310 msgr2=0x55b0e1dd7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 shutdown_connections 2023-10-10T17:42:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 1 -- 172.21.15.78:0/2954021401 wait complete. 2023-10-10T17:42:16.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.230+0000 7f0933e74b80 20 client.15399 trim_cache size 0 max 16384 2023-10-10T17:42:16.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 Processor -- start 2023-10-10T17:42:16.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 -- start start 2023-10-10T17:42:16.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1bb4a30 con 0x55b0e1dddbb0 2023-10-10T17:42:16.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1ddef40 con 0x55b0e1ddf0c0 2023-10-10T17:42:16.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0933e74b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1df3a70 con 0x55b0e1dde500 2023-10-10T17:42:16.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43794/0 (socket says 172.21.15.78:43794) 2023-10-10T17:42:16.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 -- 172.21.15.78:0/4117961449 learned_addr learned my addr 172.21.15.78:0/4117961449 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0926285700 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 -- 172.21.15.78:0/4117961449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 msgr2=0x55b0e1de5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 -- 172.21.15.78:0/4117961449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 msgr2=0x55b0e1deea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 -- 172.21.15.78:0/4117961449 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0e1cf4e90 con 0x55b0e1dddbb0 2023-10-10T17:42:16.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0926285700 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:16.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.248+0000 7f0925a84700 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de5440 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0910004730 tx=0x7f091001d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58be691916fe1f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0914ff9700 1 -- 172.21.15.78:0/4117961449 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091001a370 con 0x55b0e1dddbb0 2023-10-10T17:42:16.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0914ff9700 1 -- 172.21.15.78:0/4117961449 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0910015070 con 0x55b0e1dddbb0 2023-10-10T17:42:16.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0914ff9700 1 -- 172.21.15.78:0/4117961449 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091001ebd0 con 0x55b0e1dddbb0 2023-10-10T17:42:16.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 -- 172.21.15.78:0/4117961449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 msgr2=0x55b0e1de5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de5440 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0910004730 tx=0x7f091001d560 comp rx=0 tx=0).stop 2023-10-10T17:42:16.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 -- 172.21.15.78:0/4117961449 shutdown_connections 2023-10-10T17:42:16.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1deea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de5440 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 --2- 172.21.15.78:0/4117961449 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1dde500 0x55b0e1de5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 -- 172.21.15.78:0/4117961449 >> 172.21.15.78:0/4117961449 conn(0x55b0e1dd3310 msgr2=0x55b0e1df1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 -- 172.21.15.78:0/4117961449 shutdown_connections 2023-10-10T17:42:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.249+0000 7f0933e74b80 1 -- 172.21.15.78:0/4117961449 wait complete. 2023-10-10T17:42:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 -- start start 2023-10-10T17:42:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 Processor -- start 2023-10-10T17:42:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dde500 0x55b0e1de97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1dee700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1deec40 con 0x55b0e1dddbb0 2023-10-10T17:42:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1deedb0 con 0x55b0e1dde500 2023-10-10T17:42:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0e1e46640 con 0x55b0e1ddf0c0 2023-10-10T17:42:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0933e74b80 10 client.0 _ll_register_callbacks cb 0x7f0910010d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:16.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43800/0 (socket says 172.21.15.78:43800) 2023-10-10T17:42:16.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 -- 172.21.15.78:0/2510695365 learned_addr learned my addr 172.21.15.78:0/2510695365 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:16.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925283700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dde500 0x55b0e1de97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0926285700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1dee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 -- 172.21.15.78:0/2510695365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 msgr2=0x55b0e1dee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1dee700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 -- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dde500 msgr2=0x55b0e1de97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:16.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dde500 0x55b0e1de97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.250+0000 7f0925a84700 1 -- 172.21.15.78:0/2510695365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0e1cf5140 con 0x55b0e1dddbb0 2023-10-10T17:42:16.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0925283700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dde500 0x55b0e1de97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:16.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0925a84700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de9290 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f091001ffd0 tx=0x7f091001d4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0e8c712a12bbbaf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:16.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f09097fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:16.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f09097fa700 1 -- 172.21.15.78:0/2510695365 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091001dd00 con 0x55b0e1dddbb0 2023-10-10T17:42:16.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f09097fa700 1 -- 172.21.15.78:0/2510695365 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0910017070 con 0x55b0e1dddbb0 2023-10-10T17:42:16.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f091c011900 con 0x55b0e1dddbb0 2023-10-10T17:42:16.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f09097fa700 1 -- 172.21.15.78:0/2510695365 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f091000b9e0 con 0x55b0e1dddbb0 2023-10-10T17:42:16.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 10 client.15405 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:16.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b0e1b45c10 con 0x55b0e1dddbb0 2023-10-10T17:42:16.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f09097fa700 1 -- 172.21.15.78:0/2510695365 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f091c011900 con 0x55b0e1dddbb0 2023-10-10T17:42:16.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f09097fa700 1 -- 172.21.15.78:0/2510695365 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f091003b040 con 0x55b0e1dddbb0 2023-10-10T17:42:16.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 10 client.15405 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:16.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 10 client.15405 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-10T17:42:16.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 20 client.15405 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:16.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1e4aeb0 0x55b0e1e47860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:16.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 4 client.15405 mds_command: new command op to 24500 tid=0[{"prefix": "status"}] 2023-10-10T17:42:16.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.251+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b0e1cfd610 con 0x55b0e1e4aeb0 2023-10-10T17:42:16.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0925283700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1e4aeb0 0x55b0e1e47860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:16.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f09097fa700 10 client.15405 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:16.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0925283700 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1e4aeb0 0x55b0e1e47860 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=678dae80ef4226c1 server_cookie=fec6ac34cf90bdcb in_seq=0 out_seq=0 2023-10-10T17:42:16.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f09097fa700 1 -- 172.21.15.78:0/2510695365 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55b0e1cfd610 con 0x55b0e1e4aeb0 2023-10-10T17:42:16.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f09097fa700 10 client.15405 handle_command_reply: tid=0 2023-10-10T17:42:16.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 client.15405 shutdown 2023-10-10T17:42:16.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 msgr2=0x55b0e1de9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de9290 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f091001ffd0 tx=0x7f091001d4c0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 shutdown_connections 2023-10-10T17:42:16.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0e1ddf0c0 0x55b0e1dee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0e1e4aeb0 0x55b0e1e47860 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0e1dddbb0 0x55b0e1de9290 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f09097fa700 0 client.15405 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:16.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 --2- 172.21.15.78:0/2510695365 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0e1dde500 0x55b0e1de97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 >> 172.21.15.78:0/2510695365 conn(0x55b0e1dd3310 msgr2=0x55b0e1de5440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 shutdown_connections 2023-10-10T17:42:16.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.252+0000 7f0933e74b80 1 -- 172.21.15.78:0/2510695365 wait complete. 2023-10-10T17:42:16.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0933e74b80 20 client.15405 trim_cache size 0 max 16384 2023-10-10T17:42:16.276 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:42:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "0ba2616a-6793-11ee-8db6-212e2dc638e7", 2023-10-10T17:42:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-10-10T17:42:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24500, 2023-10-10T17:42:16.276 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-10-10T17:42:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-10-10T17:42:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-10-10T17:42:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 94.676987629999999, 2023-10-10T17:42:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 24, 2023-10-10T17:42:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 91, 2023-10-10T17:42:16.277 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 89, 2023-10-10T17:42:16.278 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 100.325520755 2023-10-10T17:42:16.278 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:42:16.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f090c0a1150 msgr2=0x7f090c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f090c0a1150 0x7f090c0a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0920100160 tx=0x7f091000b410 comp rx=0 tx=0).stop 2023-10-10T17:42:16.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 msgr2=0x7f092019c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:16.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f092019c520 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f091800b830 tx=0x7f0918004c80 comp rx=0 tx=0).stop 2023-10-10T17:42:16.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 shutdown_connections 2023-10-10T17:42:16.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f090c0a1150 0x7f090c0a3600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09200ff110 0x7f0920194f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0920100c90 0x7f092019c520 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 --2- 172.21.15.78:0/1702826866 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0920100300 0x7f09201954b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:16.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 >> 172.21.15.78:0/1702826866 conn(0x7f09200fa6f0 msgr2=0x7f09200fc8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:16.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 shutdown_connections 2023-10-10T17:42:16.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:16.253+0000 7f0927ce8700 1 -- 172.21.15.78:0/1702826866 wait complete. 2023-10-10T17:42:16.282 INFO:teuthology.run_tasks:Running task kclient... 2023-10-10T17:42:16.354 INFO:tasks.kclient:Mounting kernel clients... 2023-10-10T17:42:16.354 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-10T17:42:16.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:16.355 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-10-10T17:42:16.372 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-10-10T17:42:16.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:16.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-10-10T17:42:16.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:16.451 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2023-10-10T17:42:16.550 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v2'} 2023-10-10T17:42:16.550 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-10T17:42:16.550 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:16.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph fs ls 2023-10-10T17:42:16.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:16 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/847221071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:16.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:16 smithi078 ceph-mon[99494]: pgmap v311: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:16.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:16 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/847221071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:16.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:16 smithi190 ceph-mon[101220]: pgmap v311: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:16.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:16 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/847221071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:16.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:16 smithi157 ceph-mon[100896]: pgmap v311: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:17.195 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:42:18.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 -- 172.21.15.78:0/2540463303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe084104500 msgr2=0x7fe08410c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:18.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2540463303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe084104500 0x7fe08410c080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00b0d0 tx=0x7fe07c00b3e0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 -- 172.21.15.78:0/2540463303 shutdown_connections 2023-10-10T17:42:18.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2540463303 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe084104500 0x7fe08410c080 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2540463303 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0840ff460 0x7fe0840ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2540463303 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe0840ffe00 0x7fe084103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 -- 172.21.15.78:0/2540463303 >> 172.21.15.78:0/2540463303 conn(0x7fe0840770a0 msgr2=0x7fe0840774a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:18.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 -- 172.21.15.78:0/2540463303 shutdown_connections 2023-10-10T17:42:18.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.483+0000 7fe0890a5700 1 -- 172.21.15.78:0/2540463303 wait complete. 2023-10-10T17:42:18.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 Processor -- start 2023-10-10T17:42:18.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 -- start start 2023-10-10T17:42:18.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0840ff460 0x7fe084195460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:18.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0840ffe00 0x7fe0841959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:18.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084104500 0x7fe084199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:18.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08419a420 con 0x7fe0840ff460 2023-10-10T17:42:18.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08419a590 con 0x7fe0840ffe00 2023-10-10T17:42:18.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe0890a5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08419a750 con 0x7fe084104500 2023-10-10T17:42:18.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe082d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0840ff460 0x7fe084195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:18.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe082d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0840ff460 0x7fe084195460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43810/0 (socket says 172.21.15.78:43810) 2023-10-10T17:42:18.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe082d9d700 1 -- 172.21.15.78:0/2837163823 learned_addr learned my addr 172.21.15.78:0/2837163823 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:18.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe08259c700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0840ffe00 0x7fe0841959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:18.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.484+0000 7fe08359e700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084104500 0x7fe084199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:18.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe082d9d700 1 -- 172.21.15.78:0/2837163823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084104500 msgr2=0x7fe084199e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:18.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe082d9d700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084104500 0x7fe084199e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe082d9d700 1 -- 172.21.15.78:0/2837163823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0840ffe00 msgr2=0x7fe0841959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:18.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe082d9d700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0840ffe00 0x7fe0841959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe082d9d700 1 -- 172.21.15.78:0/2837163823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c009d00 con 0x7fe0840ff460 2023-10-10T17:42:18.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe082d9d700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0840ff460 0x7fe084195460 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00d9e0 tx=0x7fe06c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:18.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe07bfff700 1 -- 172.21.15.78:0/2837163823 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c00eb70 con 0x7fe0840ff460 2023-10-10T17:42:18.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08410f2c0 con 0x7fe0840ff460 2023-10-10T17:42:18.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe07bfff700 1 -- 172.21.15.78:0/2837163823 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe06c00ecd0 con 0x7fe0840ff460 2023-10-10T17:42:18.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe07bfff700 1 -- 172.21.15.78:0/2837163823 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c0146c0 con 0x7fe0840ff460 2023-10-10T17:42:18.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.485+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08410f130 con 0x7fe0840ff460 2023-10-10T17:42:18.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.486+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe08410f450 con 0x7fe0840ff460 2023-10-10T17:42:18.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.487+0000 7fe07bfff700 1 -- 172.21.15.78:0/2837163823 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe06c014820 con 0x7fe0840ff460 2023-10-10T17:42:18.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.488+0000 7fe07bfff700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe0700a1150 0x7fe0700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:18.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.488+0000 7fe08259c700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe0700a1150 0x7fe0700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:18.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.488+0000 7fe07bfff700 1 -- 172.21.15.78:0/2837163823 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe06c0ce930 con 0x7fe0840ff460 2023-10-10T17:42:18.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.488+0000 7fe08259c700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe0700a1150 0x7fe0700a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe084196a80 tx=0x7fe07400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:18.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.491+0000 7fe07bfff700 1 -- 172.21.15.78:0/2837163823 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe06c097d10 con 0x7fe0840ff460 2023-10-10T17:42:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.722+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe084066ea0 con 0x7fe0840ff460 2023-10-10T17:42:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.722+0000 7fe07bfff700 1 -- 172.21.15.78:0/2837163823 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe06c09b330 con 0x7fe0840ff460 2023-10-10T17:42:18.725 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-10-10T17:42:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe0700a1150 msgr2=0x7fe0700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe0700a1150 0x7fe0700a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe084196a80 tx=0x7fe07400b410 comp rx=0 tx=0).stop 2023-10-10T17:42:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0840ff460 msgr2=0x7fe084195460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0840ff460 0x7fe084195460 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00d9e0 tx=0x7fe06c00dcf0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 shutdown_connections 2023-10-10T17:42:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe0700a1150 0x7fe0700a3600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe084104500 0x7fe084199e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe0840ff460 0x7fe084195460 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 --2- 172.21.15.78:0/2837163823 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe0840ffe00 0x7fe0841959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.724+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 >> 172.21.15.78:0/2837163823 conn(0x7fe0840770a0 msgr2=0x7fe0840fcca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.725+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 shutdown_connections 2023-10-10T17:42:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:18.725+0000 7fe0890a5700 1 -- 172.21.15.78:0/2837163823 wait complete. 2023-10-10T17:42:19.138 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-10T17:42:19.139 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-10T17:42:19.140 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2023-10-10T17:42:19.140 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-10T17:42:19.140 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:19.140 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-10T17:42:19.141 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-10T17:42:19.141 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-10T17:42:19.141 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-10T17:42:19.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.142 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2023-10-10T17:42:19.161 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-10T17:42:19.161 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-10T17:42:19.161 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2023-10-10T17:42:19.162 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-10T17:42:19.162 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2023-10-10T17:42:19.162 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-10T17:42:19.162 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T17:42:19.162 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2023-10-10T17:42:19.163 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T17:42:19.163 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2023-10-10T17:42:19.163 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-10T17:42:19.163 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2023-10-10T17:42:19.164 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-10T17:42:19.164 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2023-10-10T17:42:19.164 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-10T17:42:19.164 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 41797sec preferred_lft 41797sec 2023-10-10T17:42:19.164 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::d6a8:fe40:6759:a046/64 scope link noprefixroute 2023-10-10T17:42:19.165 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-10-10T17:42:19.165 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-10T17:42:19.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.166 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:42:19.166 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:42:19.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2023-10-10T17:42:19.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2023-10-10T17:42:19.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2023-10-10T17:42:19.167 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-10T17:42:19.168 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:42:19.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.281 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-10T17:42:19.307 INFO:teuthology.orchestra.run.smithi078.stdout:1 2023-10-10T17:42:19.308 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.309 DEBUG:teuthology.orchestra.run.smithi078:> route 2023-10-10T17:42:19.367 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2023-10-10T17:42:19.367 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-10T17:42:19.368 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-10T17:42:19.368 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-10T17:42:19.368 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-10T17:42:19.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.369 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:42:19.369 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:42:19.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-10T17:42:19.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-10T17:42:19.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-10T17:42:19.370 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:42:19.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.473 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-10-10T17:42:19.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.490 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2023-10-10T17:42:19.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.547 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:42:19.547 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:42:19.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-10T17:42:19.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-10T17:42:19.548 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:42:19.641 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-10T17:42:19.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.641 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:42:19.642 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:42:19.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-10T17:42:19.642 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-10T17:42:19.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-10T17:42:19.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-10T17:42:19.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-10T17:42:19.644 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:42:19.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:19 smithi078 ceph-mon[99494]: pgmap v312: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:42:19.696 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:19 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2837163823' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-10T17:42:19.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:19.836 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-10-10T17:42:19.837 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-10-10T17:42:19.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2023-10-10T17:42:19.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2023-10-10T17:42:19.837 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-10-10T17:42:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:19 smithi190 ceph-mon[101220]: pgmap v312: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:42:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:19 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2837163823' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-10T17:42:19.954 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-10T17:42:19.954 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:19.975 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-10T17:42:19.975 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-10T17:42:19.976 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:19.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:19 smithi157 ceph-mon[100896]: pgmap v312: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:42:19.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:19 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2837163823' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-10-10T17:42:20.032 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2023-10-10T17:42:20.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:20.033 DEBUG:teuthology.orchestra.run.smithi078:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-10-10T17:42:20.118 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-10-10T17:42:20.118 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-10-10T17:42:20.289 INFO:teuthology.orchestra.run.smithi078.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,wsync,nofallback 2023-10-10T17:42:20.382 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync". 2023-10-10T17:42:20.382 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: resolved to: "172.21.15.78:3300,172.21.15.157:3300,172.21.15.190:3300" 2023-10-10T17:42:20.383 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with new device syntax: 0@0ba2616a-6793-11ee-8db6-212e2dc638e7.cephfs=/ 2023-10-10T17:42:20.383 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mon_addr=172.21.15.78:3300/172.21.15.157:3300/172.21.15.190:3300" will pass to kernel 2023-10-10T17:42:20.383 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: v2 mount support check returned 2 2023-10-10T17:42:20.383 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: kernel does not support v2 syntax 2023-10-10T17:42:20.383 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.78:3300,172.21.15.157:3300,172.21.15.190:3300:/ 2023-10-10T17:42:20.384 INFO:teuthology.orchestra.run.smithi078.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,wsync,name=0,key=0,mds_namespace=cephfs,fsid=0ba2616a-6793-11ee-8db6-212e2dc638e7" will pass to kernel 2023-10-10T17:42:20.385 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-10T17:42:20.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:20.386 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:20.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:20.414 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-10T17:42:20.487 INFO:teuthology.orchestra.run.smithi078.stdout:client15414 2023-10-10T17:42:20.488 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:42:20.489 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-10T17:42:20.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:20.548 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-10T17:42:20.620 INFO:teuthology.orchestra.run.smithi078.stdout:client15414 2023-10-10T17:42:20.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:20.621 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-10T17:42:20.692 INFO:teuthology.orchestra.run.smithi078.stdout:client15414 2023-10-10T17:42:20.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:42:20.693 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7.client15414/status') 2023-10-10T17:42:20.774 INFO:teuthology.orchestra.run.smithi078.stdout:instance: client.15414 (3)192.168.0.1:0/795040048 2023-10-10T17:42:20.775 INFO:teuthology.orchestra.run.smithi078.stdout:blocklisted: false 2023-10-10T17:42:20.777 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records in 2023-10-10T17:42:20.778 INFO:teuthology.orchestra.run.smithi078.stderr:0+1 records out 2023-10-10T17:42:20.778 INFO:teuthology.orchestra.run.smithi078.stderr:69 bytes copied, 2.3199e-05 s, 3.0 MB/s 2023-10-10T17:42:20.778 INFO:teuthology.run_tasks:Running task check-counter... 2023-10-10T17:42:20.811 INFO:teuthology.run_tasks:Running task fwd_scrub... 2023-10-10T17:42:20.838 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:42:20.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 -- 172.21.15.78:0/2389302979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 msgr2=0x7fd1a00ff8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:20.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/2389302979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00ff8d0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd1900099c0 tx=0x7fd190009cd0 comp rx=0 tx=0).stop 2023-10-10T17:42:20.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 -- 172.21.15.78:0/2389302979 shutdown_connections 2023-10-10T17:42:20.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/2389302979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a0100770 0x7fd1a0100b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:20.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/2389302979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a00ffe10 0x7fd1a01076c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:20.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/2389302979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00ff8d0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:20.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 -- 172.21.15.78:0/2389302979 >> 172.21.15.78:0/2389302979 conn(0x7fd1a00f9ee0 msgr2=0x7fd1a00fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:20.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 -- 172.21.15.78:0/2389302979 shutdown_connections 2023-10-10T17:42:20.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.959+0000 7fd1a5f41700 1 -- 172.21.15.78:0/2389302979 wait complete. 2023-10-10T17:42:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 Processor -- start 2023-10-10T17:42:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 -- start start 2023-10-10T17:42:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00feec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:20.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a00ffe10 0x7fd1a019ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a0100770 0x7fd1a019d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a019db40 con 0x7fd1a00ff460 2023-10-10T17:42:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a019dcb0 con 0x7fd1a00ffe10 2023-10-10T17:42:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd1a5f41700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a019de70 con 0x7fd1a0100770 2023-10-10T17:42:20.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd19f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00feec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd19f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00feec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43824/0 (socket says 172.21.15.78:43824) 2023-10-10T17:42:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd19f7fe700 1 -- 172.21.15.78:0/481047370 learned_addr learned my addr 172.21.15.78:0/481047370 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd19f7fe700 1 -- 172.21.15.78:0/481047370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a0100770 msgr2=0x7fd1a019d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd19f7fe700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a0100770 0x7fd1a019d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd19f7fe700 1 -- 172.21.15.78:0/481047370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a00ffe10 msgr2=0x7fd1a019ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:20.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.960+0000 7fd19f7fe700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a00ffe10 0x7fd1a019ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.961+0000 7fd19f7fe700 1 -- 172.21.15.78:0/481047370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1900095f0 con 0x7fd1a00ff460 2023-10-10T17:42:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.961+0000 7fd19f7fe700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00feec0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c970 tx=0x7fd18800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.961+0000 7fd19cff9700 1 -- 172.21.15.78:0/481047370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188007aa0 con 0x7fd1a00ff460 2023-10-10T17:42:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.961+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a010aba0 con 0x7fd1a00ff460 2023-10-10T17:42:20.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.961+0000 7fd19cff9700 1 -- 172.21.15.78:0/481047370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd188004cf0 con 0x7fd1a00ff460 2023-10-10T17:42:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.961+0000 7fd19cff9700 1 -- 172.21.15.78:0/481047370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd188005600 con 0x7fd1a00ff460 2023-10-10T17:42:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.961+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a010aa10 con 0x7fd1a00ff460 2023-10-10T17:42:20.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.962+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1a0106870 con 0x7fd1a00ff460 2023-10-10T17:42:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.963+0000 7fd19cff9700 1 -- 172.21.15.78:0/481047370 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd188004750 con 0x7fd1a00ff460 2023-10-10T17:42:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.963+0000 7fd19cff9700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd18c0a1150 0x7fd18c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:20.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.964+0000 7fd19effd700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd18c0a1150 0x7fd18c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:20.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.964+0000 7fd19cff9700 1 -- 172.21.15.78:0/481047370 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd188013070 con 0x7fd1a00ff460 2023-10-10T17:42:20.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.964+0000 7fd19effd700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd18c0a1150 0x7fd18c0a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd1900036f0 tx=0x7fd190009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:20.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:20.967+0000 7fd19cff9700 1 -- 172.21.15.78:0/481047370 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd188096530 con 0x7fd1a00ff460 2023-10-10T17:42:21.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.159+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1a002cfe0 con 0x7fd1a00ff460 2023-10-10T17:42:21.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.161+0000 7fd19cff9700 1 -- 172.21.15.78:0/481047370 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7fd188099b50 con 0x7fd1a00ff460 2023-10-10T17:42:21.164 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:21.164 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:42:21.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd18c0a1150 msgr2=0x7fd18c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd18c0a1150 0x7fd18c0a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd1900036f0 tx=0x7fd190009f10 comp rx=0 tx=0).stop 2023-10-10T17:42:21.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 msgr2=0x7fd1a00feec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00feec0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fd18800c970 tx=0x7fd18800cd30 comp rx=0 tx=0).stop 2023-10-10T17:42:21.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 shutdown_connections 2023-10-10T17:42:21.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd18c0a1150 0x7fd18c0a3600 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd1a0100770 0x7fd1a019d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1a00ff460 0x7fd1a00feec0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 --2- 172.21.15.78:0/481047370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1a00ffe10 0x7fd1a019ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.163+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 >> 172.21.15.78:0/481047370 conn(0x7fd1a00f9ee0 msgr2=0x7fd1a01039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.164+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 shutdown_connections 2023-10-10T17:42:21.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.164+0000 7fd1a5f41700 1 -- 172.21.15.78:0/481047370 wait complete. 2023-10-10T17:42:21.178 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:42:21.180 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:42:21.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 -- 172.21.15.78:0/2466686206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 msgr2=0x7f632c100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 --2- 172.21.15.78:0/2466686206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c100bd0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f631c009990 tx=0x7f631c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 -- 172.21.15.78:0/2466686206 shutdown_connections 2023-10-10T17:42:21.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 --2- 172.21.15.78:0/2466686206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c0ffdc0 0x7f632c107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 --2- 172.21.15.78:0/2466686206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c100bd0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 --2- 172.21.15.78:0/2466686206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f632c0ff410 0x7f632c0ff880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 -- 172.21.15.78:0/2466686206 >> 172.21.15.78:0/2466686206 conn(0x7f632c0f9f30 msgr2=0x7f632c0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:21.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 -- 172.21.15.78:0/2466686206 shutdown_connections 2023-10-10T17:42:21.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.303+0000 7f6332921700 1 -- 172.21.15.78:0/2466686206 wait complete. 2023-10-10T17:42:21.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 Processor -- start 2023-10-10T17:42:21.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 -- start start 2023-10-10T17:42:21.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c0ff410 0x7f632c194f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f632c0ffdc0 0x7f632c1954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19cae0 con 0x7f632c1007c0 2023-10-10T17:42:21.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19cc20 con 0x7f632c0ffdc0 2023-10-10T17:42:21.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.304+0000 7f6332921700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c19cdb0 con 0x7f632c0ff410 2023-10-10T17:42:21.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43842/0 (socket says 172.21.15.78:43842) 2023-10-10T17:42:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 -- 172.21.15.78:0/1895413924 learned_addr learned my addr 172.21.15.78:0/1895413924 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6323fff700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f632c0ffdc0 0x7f632c1954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f632bfff700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c0ff410 0x7f632c194f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 -- 172.21.15.78:0/1895413924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c0ff410 msgr2=0x7f632c194f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c0ff410 0x7f632c194f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 -- 172.21.15.78:0/1895413924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f632c0ffdc0 msgr2=0x7f632c1954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f632c0ffdc0 0x7f632c1954a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 -- 172.21.15.78:0/1895413924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c0095f0 con 0x7f632c1007c0 2023-10-10T17:42:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6323fff700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f632c0ffdc0 0x7f632c1954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:21.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6330ebe700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c19c510 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6324007ee0 tx=0x7f632400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6329ffb700 1 -- 172.21.15.78:0/1895413924 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632400ddd0 con 0x7f632c1007c0 2023-10-10T17:42:21.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6329ffb700 1 -- 172.21.15.78:0/1895413924 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f632400f040 con 0x7f632c1007c0 2023-10-10T17:42:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6329ffb700 1 -- 172.21.15.78:0/1895413924 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324015400 con 0x7f632c1007c0 2023-10-10T17:42:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c10ab30 con 0x7f632c1007c0 2023-10-10T17:42:21.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.305+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c10a9a0 con 0x7f632c1007c0 2023-10-10T17:42:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.306+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c19d450 con 0x7f632c1007c0 2023-10-10T17:42:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.307+0000 7f6329ffb700 1 -- 172.21.15.78:0/1895413924 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6324004020 con 0x7f632c1007c0 2023-10-10T17:42:21.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.308+0000 7f6329ffb700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63180a1150 0x7f63180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.308+0000 7f632bfff700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63180a1150 0x7f63180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.308+0000 7f6329ffb700 1 -- 172.21.15.78:0/1895413924 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f63240ccdb0 con 0x7f632c1007c0 2023-10-10T17:42:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.308+0000 7f632bfff700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63180a1150 0x7f63180a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f631c0095c0 tx=0x7f631c009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.312+0000 7f6329ffb700 1 -- 172.21.15.78:0/1895413924 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f63240961f0 con 0x7f632c1007c0 2023-10-10T17:42:21.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.502+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f632c106160 con 0x7f632c1007c0 2023-10-10T17:42:21.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.502+0000 7f6329ffb700 1 -- 172.21.15.78:0/1895413924 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63240230a0 con 0x7f632c1007c0 2023-10-10T17:42:21.504 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:42:21.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63180a1150 msgr2=0x7f63180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63180a1150 0x7f63180a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f631c0095c0 tx=0x7f631c009530 comp rx=0 tx=0).stop 2023-10-10T17:42:21.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 msgr2=0x7f632c19c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c19c510 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f6324007ee0 tx=0x7f632400d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 shutdown_connections 2023-10-10T17:42:21.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f63180a1150 0x7f63180a3600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f632c0ff410 0x7f632c194f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f632c1007c0 0x7f632c19c510 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 --2- 172.21.15.78:0/1895413924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f632c0ffdc0 0x7f632c1954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 >> 172.21.15.78:0/1895413924 conn(0x7f632c0f9f30 msgr2=0x7f632c0fc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:21.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 shutdown_connections 2023-10-10T17:42:21.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.504+0000 7f6332921700 1 -- 172.21.15.78:0/1895413924 wait complete. 2023-10-10T17:42:21.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:42:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 -- 172.21.15.78:0/884093341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c1003b0 msgr2=0x7f7d3c100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 --2- 172.21.15.78:0/884093341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c1003b0 0x7f7d3c100800 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009990 tx=0x7f7d30009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 -- 172.21.15.78:0/884093341 shutdown_connections 2023-10-10T17:42:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 --2- 172.21.15.78:0/884093341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d3c0ff1c0 0x7f7d3c0ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 --2- 172.21.15.78:0/884093341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d3c100d40 0x7f7d3c107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 --2- 172.21.15.78:0/884093341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c1003b0 0x7f7d3c100800 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 -- 172.21.15.78:0/884093341 >> 172.21.15.78:0/884093341 conn(0x7f7d3c0fa7a0 msgr2=0x7f7d3c0fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 -- 172.21.15.78:0/884093341 shutdown_connections 2023-10-10T17:42:21.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 -- 172.21.15.78:0/884093341 wait complete. 2023-10-10T17:42:21.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 Processor -- start 2023-10-10T17:42:21.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.644+0000 7f7d42441700 1 -- start start 2023-10-10T17:42:21.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d3c0ff1c0 0x7f7d3c069a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d3c1003b0 0x7f7d3c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c100d40 0x7f7d3c19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c19dbd0 con 0x7f7d3c100d40 2023-10-10T17:42:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c19dd40 con 0x7f7d3c0ff1c0 2023-10-10T17:42:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3c19df00 con 0x7f7d3c1003b0 2023-10-10T17:42:21.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c100d40 0x7f7d3c19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c100d40 0x7f7d3c19d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43856/0 (socket says 172.21.15.78:43856) 2023-10-10T17:42:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 -- 172.21.15.78:0/522174493 learned_addr learned my addr 172.21.15.78:0/522174493 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:21.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d3b7fe700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d3c1003b0 0x7f7d3c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d3bfff700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d3c0ff1c0 0x7f7d3c069a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 -- 172.21.15.78:0/522174493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d3c1003b0 msgr2=0x7f7d3c19adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d3c1003b0 0x7f7d3c19adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 -- 172.21.15.78:0/522174493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d3c0ff1c0 msgr2=0x7f7d3c069a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d3c0ff1c0 0x7f7d3c069a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 -- 172.21.15.78:0/522174493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300095f0 con 0x7f7d3c100d40 2023-10-10T17:42:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d409de700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c100d40 0x7f7d3c19d400 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400ada0 tx=0x7f7d3400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:21.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d397fa700 1 -- 172.21.15.78:0/522174493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3400f070 con 0x7f7d3c100d40 2023-10-10T17:42:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3c10ab50 con 0x7f7d3c100d40 2023-10-10T17:42:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d397fa700 1 -- 172.21.15.78:0/522174493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7d34007b30 con 0x7f7d3c100d40 2023-10-10T17:42:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d397fa700 1 -- 172.21.15.78:0/522174493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d34005440 con 0x7f7d3c100d40 2023-10-10T17:42:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.645+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3c10a9c0 con 0x7f7d3c100d40 2023-10-10T17:42:21.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.646+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3c106110 con 0x7f7d3c100d40 2023-10-10T17:42:21.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.647+0000 7f7d397fa700 1 -- 172.21.15.78:0/522174493 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7d34004ad0 con 0x7f7d3c100d40 2023-10-10T17:42:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.648+0000 7f7d397fa700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7d240a1150 0x7f7d240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.648+0000 7f7d397fa700 1 -- 172.21.15.78:0/522174493 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7d340cc8b0 con 0x7f7d3c100d40 2023-10-10T17:42:21.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.648+0000 7f7d3bfff700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7d240a1150 0x7f7d240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.648+0000 7f7d3bfff700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7d240a1150 0x7f7d240a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c100210 tx=0x7f7d2c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:21.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.651+0000 7f7d397fa700 1 -- 172.21.15.78:0/522174493 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f7d34095d80 con 0x7f7d3c100d40 2023-10-10T17:42:21.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.846+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d3c061700 con 0x7f7d3c100d40 2023-10-10T17:42:21.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.846+0000 7f7d397fa700 1 -- 172.21.15.78:0/522174493 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d340993a0 con 0x7f7d3c100d40 2023-10-10T17:42:21.848 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:21.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7d240a1150 msgr2=0x7f7d240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7d240a1150 0x7f7d240a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c100210 tx=0x7f7d2c009260 comp rx=0 tx=0).stop 2023-10-10T17:42:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c100d40 msgr2=0x7f7d3c19d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c100d40 0x7f7d3c19d400 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7d3400ada0 tx=0x7f7d3400c330 comp rx=0 tx=0).stop 2023-10-10T17:42:21.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 shutdown_connections 2023-10-10T17:42:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f7d240a1150 0x7f7d240a3600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7d3c100d40 0x7f7d3c19d400 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7d3c0ff1c0 0x7f7d3c069a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 --2- 172.21.15.78:0/522174493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7d3c1003b0 0x7f7d3c19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 >> 172.21.15.78:0/522174493 conn(0x7f7d3c0fa7a0 msgr2=0x7f7d3c0fca80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:21.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 shutdown_connections 2023-10-10T17:42:21.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.848+0000 7f7d42441700 1 -- 172.21.15.78:0/522174493 wait complete. 2023-10-10T17:42:21.861 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:21 smithi078 ceph-mon[99494]: pgmap v313: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:21.861 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:21 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/481047370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:21.862 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:21 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1895413924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:21.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:42:21.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:21 smithi190 ceph-mon[101220]: pgmap v313: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:21 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/481047370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:21.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:21 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1895413924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:21.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:21 smithi157 ceph-mon[100896]: pgmap v313: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:42:21.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:21 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/481047370' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:21.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:21 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1895413924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:21.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- 172.21.15.78:0/4161819218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c101e50 msgr2=0x7fa54c108710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 --2- 172.21.15.78:0/4161819218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c101e50 0x7fa54c108710 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa54400b0a0 tx=0x7fa54400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- 172.21.15.78:0/4161819218 shutdown_connections 2023-10-10T17:42:21.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 --2- 172.21.15.78:0/4161819218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c101e50 0x7fa54c108710 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 --2- 172.21.15.78:0/4161819218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa54c1002d0 0x7fa54c1006e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 --2- 172.21.15.78:0/4161819218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa54c1014c0 0x7fa54c101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- 172.21.15.78:0/4161819218 >> 172.21.15.78:0/4161819218 conn(0x7fa54c0fb8b0 msgr2=0x7fa54c0fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:21.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- 172.21.15.78:0/4161819218 shutdown_connections 2023-10-10T17:42:21.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- 172.21.15.78:0/4161819218 wait complete. 2023-10-10T17:42:21.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 Processor -- start 2023-10-10T17:42:21.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- start start 2023-10-10T17:42:21.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c1002d0 0x7fa54c196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa54c1014c0 0x7fa54c196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa54c101e50 0x7fa54c19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:21.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c19dc90 con 0x7fa54c1002d0 2023-10-10T17:42:21.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c19ddd0 con 0x7fa54c1014c0 2023-10-10T17:42:21.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.987+0000 7fa552bb6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c19df60 con 0x7fa54c101e50 2023-10-10T17:42:21.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c1002d0 0x7fa54c196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c1002d0 0x7fa54c196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43866/0 (socket says 172.21.15.78:43866) 2023-10-10T17:42:21.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 -- 172.21.15.78:0/1863060749 learned_addr learned my addr 172.21.15.78:0/1863060749 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:21.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa54bfff700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa54c1014c0 0x7fa54c196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa551153700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa54c101e50 0x7fa54c19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:21.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 -- 172.21.15.78:0/1863060749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa54c101e50 msgr2=0x7fa54c19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa54c101e50 0x7fa54c19d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 -- 172.21.15.78:0/1863060749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa54c1014c0 msgr2=0x7fa54c196650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa54c1014c0 0x7fa54c196650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 -- 172.21.15.78:0/1863060749 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa544009d00 con 0x7fa54c1002d0 2023-10-10T17:42:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa550952700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c1002d0 0x7fa54c196110 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa54000d980 tx=0x7fa54000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:21.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa549ffb700 1 -- 172.21.15.78:0/1863060749 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540004020 con 0x7fa54c1002d0 2023-10-10T17:42:21.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa549ffb700 1 -- 172.21.15.78:0/1863060749 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa54000f690 con 0x7fa54c1002d0 2023-10-10T17:42:21.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c10bc60 con 0x7fa54c1002d0 2023-10-10T17:42:21.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa549ffb700 1 -- 172.21.15.78:0/1863060749 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa540003a60 con 0x7fa54c1002d0 2023-10-10T17:42:21.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.988+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c10bad0 con 0x7fa54c1002d0 2023-10-10T17:42:21.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.989+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c1077f0 con 0x7fa54c1002d0 2023-10-10T17:42:21.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.990+0000 7fa549ffb700 1 -- 172.21.15.78:0/1863060749 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa54000fb00 con 0x7fa54c1002d0 2023-10-10T17:42:22.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.991+0000 7fa549ffb700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa53c0a1150 0x7fa53c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.991+0000 7fa54bfff700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa53c0a1150 0x7fa53c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.991+0000 7fa549ffb700 1 -- 172.21.15.78:0/1863060749 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa5400cc810 con 0x7fa54c1002d0 2023-10-10T17:42:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.991+0000 7fa54bfff700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa53c0a1150 0x7fa53c0a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa538005950 tx=0x7fa53800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:22.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:21.995+0000 7fa549ffb700 1 -- 172.21.15.78:0/1863060749 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa540095c50 con 0x7fa54c1002d0 2023-10-10T17:42:22.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.184+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa54c061700 con 0x7fa54c1002d0 2023-10-10T17:42:22.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.185+0000 7fa549ffb700 1 -- 172.21.15.78:0/1863060749 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa540099270 con 0x7fa54c1002d0 2023-10-10T17:42:22.187 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:42:22.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa53c0a1150 msgr2=0x7fa53c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa53c0a1150 0x7fa53c0a3600 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa538005950 tx=0x7fa53800b410 comp rx=0 tx=0).stop 2023-10-10T17:42:22.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c1002d0 msgr2=0x7fa54c196110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c1002d0 0x7fa54c196110 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fa54000d980 tx=0x7fa54000dc90 comp rx=0 tx=0).stop 2023-10-10T17:42:22.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 shutdown_connections 2023-10-10T17:42:22.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fa53c0a1150 0x7fa53c0a3600 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa54c101e50 0x7fa54c19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa54c1002d0 0x7fa54c196110 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 --2- 172.21.15.78:0/1863060749 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa54c1014c0 0x7fa54c196650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 >> 172.21.15.78:0/1863060749 conn(0x7fa54c0fb8b0 msgr2=0x7fa54c104930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:22.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 shutdown_connections 2023-10-10T17:42:22.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.187+0000 7fa552bb6700 1 -- 172.21.15.78:0/1863060749 wait complete. 2023-10-10T17:42:22.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:42:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 -- 172.21.15.78:0/1996070069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2414100d40 msgr2=0x7f2414107600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 --2- 172.21.15.78:0/1996070069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2414100d40 0x7f2414107600 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f240c00b0a0 tx=0x7f240c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 -- 172.21.15.78:0/1996070069 shutdown_connections 2023-10-10T17:42:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 --2- 172.21.15.78:0/1996070069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24140ff1c0 0x7f24140ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 --2- 172.21.15.78:0/1996070069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2414100d40 0x7f2414107600 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 --2- 172.21.15.78:0/1996070069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24141003b0 0x7f2414100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 -- 172.21.15.78:0/1996070069 >> 172.21.15.78:0/1996070069 conn(0x7f24140fa7c0 msgr2=0x7f24140fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.323+0000 7f241894a700 1 -- 172.21.15.78:0/1996070069 shutdown_connections 2023-10-10T17:42:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 -- 172.21.15.78:0/1996070069 wait complete. 2023-10-10T17:42:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 Processor -- start 2023-10-10T17:42:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 -- start start 2023-10-10T17:42:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24140ff1c0 0x7f24140689a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24141003b0 0x7f241419add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2414100d40 0x7f241419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241419dc00 con 0x7f24140ff1c0 2023-10-10T17:42:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241419dd70 con 0x7f2414100d40 2023-10-10T17:42:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241894a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241419df30 con 0x7f24141003b0 2023-10-10T17:42:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.324+0000 7f241259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24140ff1c0 0x7f24140689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24140ff1c0 0x7f24140689a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43880/0 (socket says 172.21.15.78:43880) 2023-10-10T17:42:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 -- 172.21.15.78:0/641987370 learned_addr learned my addr 172.21.15.78:0/641987370 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f2412d9d700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2414100d40 0x7f241419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f2411d9b700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24141003b0 0x7f241419add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 -- 172.21.15.78:0/641987370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24141003b0 msgr2=0x7f241419add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24141003b0 0x7f241419add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 -- 172.21.15.78:0/641987370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2414100d40 msgr2=0x7f241419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2414100d40 0x7f241419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 -- 172.21.15.78:0/641987370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c009d00 con 0x7f24140ff1c0 2023-10-10T17:42:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241259c700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24140ff1c0 0x7f24140689a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f240400c990 tx=0x7f240400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f24037fe700 1 -- 172.21.15.78:0/641987370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404007b00 con 0x7f24140ff1c0 2023-10-10T17:42:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f24037fe700 1 -- 172.21.15.78:0/641987370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2404007c60 con 0x7f24140ff1c0 2023-10-10T17:42:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241410ab30 con 0x7f24140ff1c0 2023-10-10T17:42:22.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f24037fe700 1 -- 172.21.15.78:0/641987370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404018660 con 0x7f24140ff1c0 2023-10-10T17:42:22.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.325+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241410a9a0 con 0x7f24140ff1c0 2023-10-10T17:42:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.326+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241419e680 con 0x7f24140ff1c0 2023-10-10T17:42:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.327+0000 7f24037fe700 1 -- 172.21.15.78:0/641987370 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f240400f450 con 0x7f24140ff1c0 2023-10-10T17:42:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.328+0000 7f24037fe700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23fc0a1150 0x7f23fc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.328+0000 7f2411d9b700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23fc0a1150 0x7f23fc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.328+0000 7f24037fe700 1 -- 172.21.15.78:0/641987370 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f24040ccca0 con 0x7f24140ff1c0 2023-10-10T17:42:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.328+0000 7f2411d9b700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23fc0a1150 0x7f23fc0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2408005950 tx=0x7f24080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.331+0000 7f24037fe700 1 -- 172.21.15.78:0/641987370 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2404096110 con 0x7f24140ff1c0 2023-10-10T17:42:22.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.519+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2414105fd0 con 0x7f24140ff1c0 2023-10-10T17:42:22.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.519+0000 7f24037fe700 1 -- 172.21.15.78:0/641987370 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2404099730 con 0x7f24140ff1c0 2023-10-10T17:42:22.521 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:22.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23fc0a1150 msgr2=0x7f23fc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23fc0a1150 0x7f23fc0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2408005950 tx=0x7f24080058e0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24140ff1c0 msgr2=0x7f24140689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24140ff1c0 0x7f24140689a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f240400c990 tx=0x7f240400cca0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 shutdown_connections 2023-10-10T17:42:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f23fc0a1150 0x7f23fc0a3600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2414100d40 0x7f241419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24140ff1c0 0x7f24140689a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 --2- 172.21.15.78:0/641987370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24141003b0 0x7f241419add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 >> 172.21.15.78:0/641987370 conn(0x7f24140fa7c0 msgr2=0x7f24140fc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 shutdown_connections 2023-10-10T17:42:22.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.521+0000 7f241894a700 1 -- 172.21.15.78:0/641987370 wait complete. 2023-10-10T17:42:22.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:42:22.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 -- 172.21.15.78:0/790655263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0fe900 msgr2=0x7fdc4c0fed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 --2- 172.21.15.78:0/790655263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0fe900 0x7fdc4c0fed10 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009990 tx=0x7fdc40009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 -- 172.21.15.78:0/790655263 shutdown_connections 2023-10-10T17:42:22.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 --2- 172.21.15.78:0/790655263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc4c100540 0x7fdc4c106de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 --2- 172.21.15.78:0/790655263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c0fe900 0x7fdc4c0fed10 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 --2- 172.21.15.78:0/790655263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c0ffb90 0x7fdc4c100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 -- 172.21.15.78:0/790655263 >> 172.21.15.78:0/790655263 conn(0x7fdc4c0f9ee0 msgr2=0x7fdc4c0fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:22.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 -- 172.21.15.78:0/790655263 shutdown_connections 2023-10-10T17:42:22.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 -- 172.21.15.78:0/790655263 wait complete. 2023-10-10T17:42:22.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.656+0000 7fdc52010700 1 Processor -- start 2023-10-10T17:42:22.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 -- start start 2023-10-10T17:42:22.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c0fe900 0x7fdc4c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc4c0ffb90 0x7fdc4c198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c100540 0x7fdc4c19f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c19fc60 con 0x7fdc4c100540 2023-10-10T17:42:22.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c19fda0 con 0x7fdc4c0fe900 2023-10-10T17:42:22.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c19ff30 con 0x7fdc4c0ffb90 2023-10-10T17:42:22.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c100540 0x7fdc4c19f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c100540 0x7fdc4c19f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43896/0 (socket says 172.21.15.78:43896) 2023-10-10T17:42:22.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 -- 172.21.15.78:0/452435726 learned_addr learned my addr 172.21.15.78:0/452435726 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:22.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4affd700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc4c0ffb90 0x7fdc4c198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4b7fe700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c0fe900 0x7fdc4c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 -- 172.21.15.78:0/452435726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc4c0ffb90 msgr2=0x7fdc4c198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc4c0ffb90 0x7fdc4c198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 -- 172.21.15.78:0/452435726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c0fe900 msgr2=0x7fdc4c1980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c0fe900 0x7fdc4c1980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 -- 172.21.15.78:0/452435726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc400095f0 con 0x7fdc4c100540 2023-10-10T17:42:22.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4b7fe700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c0fe900 0x7fdc4c1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:22.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc4bfff700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c100540 0x7fdc4c19f690 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400ef30 tx=0x7fdc4400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:22.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc48ff9700 1 -- 172.21.15.78:0/452435726 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44018070 con 0x7fdc4c100540 2023-10-10T17:42:22.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.657+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c10ab20 con 0x7fdc4c100540 2023-10-10T17:42:22.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.658+0000 7fdc48ff9700 1 -- 172.21.15.78:0/452435726 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdc4400f040 con 0x7fdc4c100540 2023-10-10T17:42:22.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.658+0000 7fdc48ff9700 1 -- 172.21.15.78:0/452435726 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400cdb0 con 0x7fdc4c100540 2023-10-10T17:42:22.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.658+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c10a990 con 0x7fdc4c100540 2023-10-10T17:42:22.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.658+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c0671d0 con 0x7fdc4c100540 2023-10-10T17:42:22.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.660+0000 7fdc48ff9700 1 -- 172.21.15.78:0/452435726 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fdc44003a00 con 0x7fdc4c100540 2023-10-10T17:42:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.660+0000 7fdc48ff9700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc340a1150 0x7fdc340a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.660+0000 7fdc4b7fe700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc340a1150 0x7fdc340a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.660+0000 7fdc48ff9700 1 -- 172.21.15.78:0/452435726 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdc440cd170 con 0x7fdc4c100540 2023-10-10T17:42:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.661+0000 7fdc4b7fe700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc340a1150 0x7fdc340a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000b2e0 tx=0x7fdc40005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.663+0000 7fdc48ff9700 1 -- 172.21.15.78:0/452435726 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdc44096640 con 0x7fdc4c100540 2023-10-10T17:42:22.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.854+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc4c1a0b20 con 0x7fdc4c100540 2023-10-10T17:42:22.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.856+0000 7fdc48ff9700 1 -- 172.21.15.78:0/452435726 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7fdc44099c60 con 0x7fdc4c100540 2023-10-10T17:42:22.858 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:22.858 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:42:22.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc340a1150 msgr2=0x7fdc340a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc340a1150 0x7fdc340a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdc4000b2e0 tx=0x7fdc40005e80 comp rx=0 tx=0).stop 2023-10-10T17:42:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c100540 msgr2=0x7fdc4c19f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c100540 0x7fdc4c19f690 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400ef30 tx=0x7fdc4400c3a0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 shutdown_connections 2023-10-10T17:42:22.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fdc340a1150 0x7fdc340a3600 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdc4c100540 0x7fdc4c19f690 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdc4c0fe900 0x7fdc4c1980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 --2- 172.21.15.78:0/452435726 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdc4c0ffb90 0x7fdc4c198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 >> 172.21.15.78:0/452435726 conn(0x7fdc4c0f9ee0 msgr2=0x7fdc4c107650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:22.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 shutdown_connections 2023-10-10T17:42:22.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.858+0000 7fdc52010700 1 -- 172.21.15.78:0/452435726 wait complete. 2023-10-10T17:42:22.870 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:22 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/522174493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.871 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:22 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1863060749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.871 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:22 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/641987370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.873 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:42:22.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:42:22.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:22 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/522174493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:22 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1863060749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:22 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/641987370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/522174493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1863060749' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:22 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/641987370' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:22.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 -- 172.21.15.78:0/2242973414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 msgr2=0x7fee000fed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:22.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 --2- 172.21.15.78:0/2242973414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fed40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fede8009990 tx=0x7fede8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:22.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 -- 172.21.15.78:0/2242973414 shutdown_connections 2023-10-10T17:42:23.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 --2- 172.21.15.78:0/2242973414 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee001004b0 0x7fee00106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 --2- 172.21.15.78:0/2242973414 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fed40 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 --2- 172.21.15.78:0/2242973414 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee000ffb20 0x7fee000fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 -- 172.21.15.78:0/2242973414 >> 172.21.15.78:0/2242973414 conn(0x7fee000f9f30 msgr2=0x7fee000fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:23.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 -- 172.21.15.78:0/2242973414 shutdown_connections 2023-10-10T17:42:23.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.997+0000 7fee05152700 1 -- 172.21.15.78:0/2242973414 wait complete. 2023-10-10T17:42:23.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 Processor -- start 2023-10-10T17:42:23.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 -- start start 2023-10-10T17:42:23.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee000ffb20 0x7fee0019ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee001004b0 0x7fee0019d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0019db60 con 0x7fee000fe930 2023-10-10T17:42:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0019dcd0 con 0x7fee000ffb20 2023-10-10T17:42:23.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fee05152700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0019de90 con 0x7fee001004b0 2023-10-10T17:42:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fedfed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fedfed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43904/0 (socket says 172.21.15.78:43904) 2023-10-10T17:42:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fedfed9d700 1 -- 172.21.15.78:0/1366553496 learned_addr learned my addr 172.21.15.78:0/1366553496 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fedff59e700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee001004b0 0x7fee0019d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.998+0000 7fedfe59c700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee000ffb20 0x7fee0019ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedfed9d700 1 -- 172.21.15.78:0/1366553496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee001004b0 msgr2=0x7fee0019d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedfed9d700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee001004b0 0x7fee0019d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedfed9d700 1 -- 172.21.15.78:0/1366553496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee000ffb20 msgr2=0x7fee0019ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedfed9d700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee000ffb20 0x7fee0019ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedfed9d700 1 -- 172.21.15.78:0/1366553496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede80095f0 con 0x7fee000fe930 2023-10-10T17:42:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedfed9d700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fc340 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fede8000c00 tx=0x7fede80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedf7fff700 1 -- 172.21.15.78:0/1366553496 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede801d070 con 0x7fee000fe930 2023-10-10T17:42:23.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedf7fff700 1 -- 172.21.15.78:0/1366553496 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fede8004060 con 0x7fee000fe930 2023-10-10T17:42:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0010a290 con 0x7fee000fe930 2023-10-10T17:42:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fedf7fff700 1 -- 172.21.15.78:0/1366553496 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8022d30 con 0x7fee000fe930 2023-10-10T17:42:23.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:22.999+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0010a100 con 0x7fee000fe930 2023-10-10T17:42:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.000+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee001064a0 con 0x7fee000fe930 2023-10-10T17:42:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.001+0000 7fedf7fff700 1 -- 172.21.15.78:0/1366553496 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fede8003ba0 con 0x7fee000fe930 2023-10-10T17:42:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.001+0000 7fedf7fff700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fedec0a1150 0x7fedec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.002+0000 7fedfe59c700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fedec0a1150 0x7fedec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.002+0000 7fedf7fff700 1 -- 172.21.15.78:0/1366553496 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fede80ce300 con 0x7fee000fe930 2023-10-10T17:42:23.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.002+0000 7fedfe59c700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fedec0a1150 0x7fedec0a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fedf0005950 tx=0x7fedf00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:23.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.005+0000 7fedf7fff700 1 -- 172.21.15.78:0/1366553496 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fede8097770 con 0x7fee000fe930 2023-10-10T17:42:23.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.192+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee00061700 con 0x7fee000fe930 2023-10-10T17:42:23.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.192+0000 7fedf7fff700 1 -- 172.21.15.78:0/1366553496 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fede809ad90 con 0x7fee000fe930 2023-10-10T17:42:23.194 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:42:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fedec0a1150 msgr2=0x7fedec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fedec0a1150 0x7fedec0a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fedf0005950 tx=0x7fedf00058e0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 msgr2=0x7fee000fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fc340 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fede8000c00 tx=0x7fede80047b0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 shutdown_connections 2023-10-10T17:42:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fedec0a1150 0x7fedec0a3600 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fee001004b0 0x7fee0019d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fee000fe930 0x7fee000fc340 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 --2- 172.21.15.78:0/1366553496 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fee000ffb20 0x7fee0019ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 >> 172.21.15.78:0/1366553496 conn(0x7fee000f9f30 msgr2=0x7fee001035e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:23.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 shutdown_connections 2023-10-10T17:42:23.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.194+0000 7fee05152700 1 -- 172.21.15.78:0/1366553496 wait complete. 2023-10-10T17:42:23.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:42:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.331+0000 7faa735f2700 1 -- 172.21.15.78:0/1395087499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c1018d0 msgr2=0x7faa6c101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.331+0000 7faa735f2700 1 --2- 172.21.15.78:0/1395087499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c1018d0 0x7faa6c101ce0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7faa68009990 tx=0x7faa68009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.331+0000 7faa735f2700 1 -- 172.21.15.78:0/1395087499 shutdown_connections 2023-10-10T17:42:23.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.331+0000 7faa735f2700 1 --2- 172.21.15.78:0/1395087499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa6c0fecd0 0x7faa6c1088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.331+0000 7faa735f2700 1 --2- 172.21.15.78:0/1395087499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c1018d0 0x7faa6c101ce0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.331+0000 7faa735f2700 1 --2- 172.21.15.78:0/1395087499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa6c0fe340 0x7faa6c0fe790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.331+0000 7faa735f2700 1 -- 172.21.15.78:0/1395087499 >> 172.21.15.78:0/1395087499 conn(0x7faa6c0f9f30 msgr2=0x7faa6c0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:23.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 -- 172.21.15.78:0/1395087499 shutdown_connections 2023-10-10T17:42:23.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 -- 172.21.15.78:0/1395087499 wait complete. 2023-10-10T17:42:23.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 Processor -- start 2023-10-10T17:42:23.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 -- start start 2023-10-10T17:42:23.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c0fe340 0x7faa6c196010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa6c0fecd0 0x7faa6c196550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa6c1018d0 0x7faa6c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c19db90 con 0x7faa6c0fe340 2023-10-10T17:42:23.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c19dcd0 con 0x7faa6c0fecd0 2023-10-10T17:42:23.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa735f2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c19de60 con 0x7faa6c1018d0 2023-10-10T17:42:23.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa7138e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c0fe340 0x7faa6c196010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa7138e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c0fe340 0x7faa6c196010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43918/0 (socket says 172.21.15.78:43918) 2023-10-10T17:42:23.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa7138e700 1 -- 172.21.15.78:0/1847908729 learned_addr learned my addr 172.21.15.78:0/1847908729 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:23.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.332+0000 7faa70b8d700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa6c0fecd0 0x7faa6c196550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa71b8f700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa6c1018d0 0x7faa6c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa7138e700 1 -- 172.21.15.78:0/1847908729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa6c1018d0 msgr2=0x7faa6c19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa7138e700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa6c1018d0 0x7faa6c19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa7138e700 1 -- 172.21.15.78:0/1847908729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa6c0fecd0 msgr2=0x7faa6c196550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa7138e700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa6c0fecd0 0x7faa6c196550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa7138e700 1 -- 172.21.15.78:0/1847908729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa680095f0 con 0x7faa6c0fe340 2023-10-10T17:42:23.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa70b8d700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa6c0fecd0 0x7faa6c196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:23.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa71b8f700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa6c1018d0 0x7faa6c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:23.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa7138e700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c0fe340 0x7faa6c196010 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7faa6800b680 tx=0x7faa68005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:23.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa627fc700 1 -- 172.21.15.78:0/1847908729 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6801d070 con 0x7faa6c0fe340 2023-10-10T17:42:23.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c10bc10 con 0x7faa6c0fe340 2023-10-10T17:42:23.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa627fc700 1 -- 172.21.15.78:0/1847908729 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa68004ca0 con 0x7faa6c0fe340 2023-10-10T17:42:23.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa627fc700 1 -- 172.21.15.78:0/1847908729 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa680055f0 con 0x7faa6c0fe340 2023-10-10T17:42:23.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.333+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c10ba80 con 0x7faa6c0fe340 2023-10-10T17:42:23.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.334+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c067270 con 0x7faa6c0fe340 2023-10-10T17:42:23.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.335+0000 7faa627fc700 1 -- 172.21.15.78:0/1847908729 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faa6800fc70 con 0x7faa6c0fe340 2023-10-10T17:42:23.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.335+0000 7faa627fc700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.335+0000 7faa627fc700 1 -- 172.21.15.78:0/1847908729 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faa680ce4e0 con 0x7faa6c0fe340 2023-10-10T17:42:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.335+0000 7faa70b8d700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.336+0000 7faa70b8d700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faa5c005fd0 tx=0x7faa5c005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:23.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.339+0000 7faa627fc700 1 -- 172.21.15.78:0/1847908729 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7faa680979b0 con 0x7faa6c0fe340 2023-10-10T17:42:23.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.527+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa6c1073a0 con 0x7faa6c0fe340 2023-10-10T17:42:23.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.527+0000 7faa627fc700 1 -- 172.21.15.78:0/1847908729 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa6809afd0 con 0x7faa6c0fe340 2023-10-10T17:42:23.529 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:23.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 msgr2=0x7faa580a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7faa5c005fd0 tx=0x7faa5c005f60 comp rx=0 tx=0).stop 2023-10-10T17:42:23.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c0fe340 msgr2=0x7faa6c196010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c0fe340 0x7faa6c196010 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7faa6800b680 tx=0x7faa68005ef0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 shutdown_connections 2023-10-10T17:42:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7faa580a1150 0x7faa580a3600 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7faa6c1018d0 0x7faa6c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faa6c0fe340 0x7faa6c196010 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 --2- 172.21.15.78:0/1847908729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7faa6c0fecd0 0x7faa6c196550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 >> 172.21.15.78:0/1847908729 conn(0x7faa6c0f9f30 msgr2=0x7faa6c0fc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:23.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 shutdown_connections 2023-10-10T17:42:23.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.529+0000 7faa735f2700 1 -- 172.21.15.78:0/1847908729 wait complete. 2023-10-10T17:42:23.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:42:23.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 -- 172.21.15.78:0/3571111832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880ffb20 msgr2=0x7fe1880fff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 --2- 172.21.15.78:0/3571111832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880ffb20 0x7fe1880fff70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe178009990 tx=0x7fe178009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 -- 172.21.15.78:0/3571111832 shutdown_connections 2023-10-10T17:42:23.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 --2- 172.21.15.78:0/3571111832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1880fe930 0x7fe1880fed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 --2- 172.21.15.78:0/3571111832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1881004b0 0x7fe188106d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 --2- 172.21.15.78:0/3571111832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880ffb20 0x7fe1880fff70 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 -- 172.21.15.78:0/3571111832 >> 172.21.15.78:0/3571111832 conn(0x7fe1880f9f30 msgr2=0x7fe1880fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:23.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 -- 172.21.15.78:0/3571111832 shutdown_connections 2023-10-10T17:42:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 -- 172.21.15.78:0/3571111832 wait complete. 2023-10-10T17:42:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.667+0000 7fe18d192700 1 Processor -- start 2023-10-10T17:42:23.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18d192700 1 -- start start 2023-10-10T17:42:23.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18d192700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880fe930 0x7fe188194e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18d192700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1880ffb20 0x7fe1881953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18d192700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1881004b0 0x7fe18819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18d192700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18819ca10 con 0x7fe1880fe930 2023-10-10T17:42:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18d192700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18819cb50 con 0x7fe1880ffb20 2023-10-10T17:42:23.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18d192700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18819cce0 con 0x7fe1881004b0 2023-10-10T17:42:23.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880fe930 0x7fe188194e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880fe930 0x7fe188194e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43932/0 (socket says 172.21.15.78:43932) 2023-10-10T17:42:23.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 -- 172.21.15.78:0/2412951531 learned_addr learned my addr 172.21.15.78:0/2412951531 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:23.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18659c700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1880ffb20 0x7fe1881953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18759e700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1881004b0 0x7fe18819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 -- 172.21.15.78:0/2412951531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1881004b0 msgr2=0x7fe18819c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1881004b0 0x7fe18819c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 -- 172.21.15.78:0/2412951531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1880ffb20 msgr2=0x7fe1881953d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1880ffb20 0x7fe1881953d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 -- 172.21.15.78:0/2412951531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1780095f0 con 0x7fe1880fe930 2023-10-10T17:42:23.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe18759e700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1881004b0 0x7fe18819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:23.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.668+0000 7fe186d9d700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880fe930 0x7fe188194e90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00d980 tx=0x7fe17c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:23.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.669+0000 7fe16ffff700 1 -- 172.21.15.78:0/2412951531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c004020 con 0x7fe1880fe930 2023-10-10T17:42:23.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.669+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18810ab10 con 0x7fe1880fe930 2023-10-10T17:42:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.669+0000 7fe16ffff700 1 -- 172.21.15.78:0/2412951531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe17c009730 con 0x7fe1880fe930 2023-10-10T17:42:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.669+0000 7fe16ffff700 1 -- 172.21.15.78:0/2412951531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe17c003a80 con 0x7fe1880fe930 2023-10-10T17:42:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.669+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18810a980 con 0x7fe1880fe930 2023-10-10T17:42:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.669+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1881057e0 con 0x7fe1880fe930 2023-10-10T17:42:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.671+0000 7fe16ffff700 1 -- 172.21.15.78:0/2412951531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe17c0098a0 con 0x7fe1880fe930 2023-10-10T17:42:23.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.671+0000 7fe16ffff700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe1700a1150 0x7fe1700a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.671+0000 7fe18659c700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe1700a1150 0x7fe1700a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.671+0000 7fe16ffff700 1 -- 172.21.15.78:0/2412951531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe17c0ccc80 con 0x7fe1880fe930 2023-10-10T17:42:23.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.672+0000 7fe18659c700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe1700a1150 0x7fe1700a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe17800b5c0 tx=0x7fe178005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:23.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.675+0000 7fe16ffff700 1 -- 172.21.15.78:0/2412951531 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe17c0960c0 con 0x7fe1880fe930 2023-10-10T17:42:23.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.865+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe188061940 con 0x7fe1880fe930 2023-10-10T17:42:23.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.865+0000 7fe16ffff700 1 -- 172.21.15.78:0/2412951531 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe17c0996e0 con 0x7fe1880fe930 2023-10-10T17:42:23.867 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:42:23.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe1700a1150 msgr2=0x7fe1700a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe1700a1150 0x7fe1700a3600 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe17800b5c0 tx=0x7fe178005960 comp rx=0 tx=0).stop 2023-10-10T17:42:23.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880fe930 msgr2=0x7fe188194e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:23.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880fe930 0x7fe188194e90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fe17c00d980 tx=0x7fe17c00dc90 comp rx=0 tx=0).stop 2023-10-10T17:42:23.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 shutdown_connections 2023-10-10T17:42:23.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fe1700a1150 0x7fe1700a3600 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1881004b0 0x7fe18819c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1880fe930 0x7fe188194e90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 --2- 172.21.15.78:0/2412951531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1880ffb20 0x7fe1881953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:23.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 >> 172.21.15.78:0/2412951531 conn(0x7fe1880f9f30 msgr2=0x7fe1880fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:23.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 shutdown_connections 2023-10-10T17:42:23.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:23.867+0000 7fe18d192700 1 -- 172.21.15.78:0/2412951531 wait complete. 2023-10-10T17:42:23.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:23 smithi078 ceph-mon[99494]: pgmap v314: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:23.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:23 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/452435726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:23 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1366553496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:23.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:23 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1847908729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:23.881 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:42:23.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:23 smithi190 ceph-mon[101220]: pgmap v314: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:23.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:23 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/452435726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:23.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:23 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1366553496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:23.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:23 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1847908729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:23.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:23 smithi157 ceph-mon[100896]: pgmap v314: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-10-10T17:42:23.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:23 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/452435726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:23.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:23 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1366553496' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:23.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:23 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1847908729' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:24.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 -- 172.21.15.78:0/801418938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae01015a0 msgr2=0x7f1ae01019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/801418938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae01015a0 0x7f1ae01019f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f1acc009990 tx=0x7f1acc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 -- 172.21.15.78:0/801418938 shutdown_connections 2023-10-10T17:42:24.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/801418938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae00fe5d0 0x7f1ae00fe9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/801418938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ae0101f30 0x7f1ae01087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/801418938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae01015a0 0x7f1ae01019f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 -- 172.21.15.78:0/801418938 >> 172.21.15.78:0/801418938 conn(0x7f1ae00f9fb0 msgr2=0x7f1ae00fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:24.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 -- 172.21.15.78:0/801418938 shutdown_connections 2023-10-10T17:42:24.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.003+0000 7f1ae6c29700 1 -- 172.21.15.78:0/801418938 wait complete. 2023-10-10T17:42:24.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 Processor -- start 2023-10-10T17:42:24.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 -- start start 2023-10-10T17:42:24.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae00fe5d0 0x7f1ae0195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ae01015a0 0x7f1ae01964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae0101f30 0x7f1ae019d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae019daf0 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae019dc30 con 0x7f1ae01015a0 2023-10-10T17:42:24.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae6c29700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae019ddc0 con 0x7f1ae0101f30 2023-10-10T17:42:24.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae49c5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae00fe5d0 0x7f1ae0195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae49c5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae00fe5d0 0x7f1ae0195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43944/0 (socket says 172.21.15.78:43944) 2023-10-10T17:42:24.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae49c5700 1 -- 172.21.15.78:0/606056069 learned_addr learned my addr 172.21.15.78:0/606056069 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:24.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae51c6700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae0101f30 0x7f1ae019d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1adffff700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ae01015a0 0x7f1ae01964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae49c5700 1 -- 172.21.15.78:0/606056069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae0101f30 msgr2=0x7f1ae019d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.004+0000 7f1ae49c5700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae0101f30 0x7f1ae019d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1ae49c5700 1 -- 172.21.15.78:0/606056069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ae01015a0 msgr2=0x7f1ae01964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1ae49c5700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ae01015a0 0x7f1ae01964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1ae49c5700 1 -- 172.21.15.78:0/606056069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc0095f0 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1ae49c5700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae00fe5d0 0x7f1ae0195f70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400d980 tx=0x7f1ad400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:24.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1addffb700 1 -- 172.21.15.78:0/606056069 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad400de60 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1addffb700 1 -- 172.21.15.78:0/606056069 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ad40044e0 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1addffb700 1 -- 172.21.15.78:0/606056069 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4003be0 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae010bc70 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.005+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae010bae0 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.006+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae0107210 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.007+0000 7f1addffb700 1 -- 172.21.15.78:0/606056069 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1ad4009bb0 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.007+0000 7f1addffb700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1ad00a11d0 0x7f1ad00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.008+0000 7f1addffb700 1 -- 172.21.15.78:0/606056069 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1ad40cca10 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.008+0000 7f1adffff700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1ad00a11d0 0x7f1ad00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.008+0000 7f1adffff700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1ad00a11d0 0x7f1ad00a3680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1acc006010 tx=0x7f1acc00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:24.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.012+0000 7f1addffb700 1 -- 172.21.15.78:0/606056069 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1ad4095df0 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.200+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1ae0061700 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.201+0000 7f1addffb700 1 -- 172.21.15.78:0/606056069 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1ad4009940 con 0x7f1ae00fe5d0 2023-10-10T17:42:24.203 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1ad00a11d0 msgr2=0x7f1ad00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1ad00a11d0 0x7f1ad00a3680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f1acc006010 tx=0x7f1acc00b540 comp rx=0 tx=0).stop 2023-10-10T17:42:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae00fe5d0 msgr2=0x7f1ae0195f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae00fe5d0 0x7f1ae0195f70 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f1ad400d980 tx=0x7f1ad400dd40 comp rx=0 tx=0).stop 2023-10-10T17:42:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 shutdown_connections 2023-10-10T17:42:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f1ad00a11d0 0x7f1ad00a3680 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1ae0101f30 0x7f1ae019d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae00fe5d0 0x7f1ae0195f70 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 --2- 172.21.15.78:0/606056069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1ae01015a0 0x7f1ae01964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 >> 172.21.15.78:0/606056069 conn(0x7f1ae00f9fb0 msgr2=0x7f1ae00fc160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:24.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 shutdown_connections 2023-10-10T17:42:24.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.203+0000 7f1ae6c29700 1 -- 172.21.15.78:0/606056069 wait complete. 2023-10-10T17:42:24.217 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2023-10-10T17:42:24.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:42:24.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 -- 172.21.15.78:0/2352698584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 msgr2=0x7ff2b00fe950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2352698584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b00fe950 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009990 tx=0x7ff2a0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 -- 172.21.15.78:0/2352698584 shutdown_connections 2023-10-10T17:42:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2352698584 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b0101c40 0x7ff2b0108800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2352698584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b00fe950 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2352698584 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2b00fee90 0x7ff2b01015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 -- 172.21.15.78:0/2352698584 >> 172.21.15.78:0/2352698584 conn(0x7ff2b00f9ee0 msgr2=0x7ff2b00fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 -- 172.21.15.78:0/2352698584 shutdown_connections 2023-10-10T17:42:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.337+0000 7ff2b7450700 1 -- 172.21.15.78:0/2352698584 wait complete. 2023-10-10T17:42:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 Processor -- start 2023-10-10T17:42:24.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 -- start start 2023-10-10T17:42:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b0196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b00fee90 0x7ff2b0196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2b0101c40 0x7ff2b019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b019dbc0 con 0x7ff2b00fe540 2023-10-10T17:42:24.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b019dd00 con 0x7ff2b0101c40 2023-10-10T17:42:24.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b7450700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b019de90 con 0x7ff2b00fee90 2023-10-10T17:42:24.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b51ec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b0196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b51ec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b0196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43958/0 (socket says 172.21.15.78:43958) 2023-10-10T17:42:24.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b51ec700 1 -- 172.21.15.78:0/2270935002 learned_addr learned my addr 172.21.15.78:0/2270935002 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:24.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.338+0000 7ff2b49eb700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b00fee90 0x7ff2b0196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b59ed700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2b0101c40 0x7ff2b019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b51ec700 1 -- 172.21.15.78:0/2270935002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b00fee90 msgr2=0x7ff2b0196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b51ec700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b00fee90 0x7ff2b0196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b51ec700 1 -- 172.21.15.78:0/2270935002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2b0101c40 msgr2=0x7ff2b019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b51ec700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2b0101c40 0x7ff2b019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b51ec700 1 -- 172.21.15.78:0/2270935002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00095f0 con 0x7ff2b00fe540 2023-10-10T17:42:24.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b59ed700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2b0101c40 0x7ff2b019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:24.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b51ec700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b0196040 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005910 tx=0x7ff2a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:24.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2a67fc700 1 -- 172.21.15.78:0/2270935002 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a001d070 con 0x7ff2b00fe540 2023-10-10T17:42:24.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2a67fc700 1 -- 172.21.15.78:0/2270935002 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2a0003f60 con 0x7ff2b00fe540 2023-10-10T17:42:24.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b010bc80 con 0x7ff2b00fe540 2023-10-10T17:42:24.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2a67fc700 1 -- 172.21.15.78:0/2270935002 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0022da0 con 0x7ff2b00fe540 2023-10-10T17:42:24.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.339+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b010baf0 con 0x7ff2b00fe540 2023-10-10T17:42:24.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.340+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b01079a0 con 0x7ff2b00fe540 2023-10-10T17:42:24.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.341+0000 7ff2a67fc700 1 -- 172.21.15.78:0/2270935002 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff2a0003aa0 con 0x7ff2b00fe540 2023-10-10T17:42:24.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.342+0000 7ff2a67fc700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff29c0a11d0 0x7ff29c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.342+0000 7ff2b49eb700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff29c0a11d0 0x7ff29c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.342+0000 7ff2a67fc700 1 -- 172.21.15.78:0/2270935002 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2a00ce7b0 con 0x7ff2b00fe540 2023-10-10T17:42:24.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.342+0000 7ff2b49eb700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff29c0a11d0 0x7ff29c0a3680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac005950 tx=0x7ff2ac00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:24.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.345+0000 7ff2a67fc700 1 -- 172.21.15.78:0/2270935002 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff2a0097c20 con 0x7ff2b00fe540 2023-10-10T17:42:24.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.574+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2b002cfe0 con 0x7ff2b00fe540 2023-10-10T17:42:24.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.575+0000 7ff2a67fc700 1 -- 172.21.15.78:0/2270935002 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7ff2a009b240 con 0x7ff2b00fe540 2023-10-10T17:42:24.578 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:24.578 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:42:24.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff29c0a11d0 msgr2=0x7ff29c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff29c0a11d0 0x7ff29c0a3680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff2ac005950 tx=0x7ff2ac00a300 comp rx=0 tx=0).stop 2023-10-10T17:42:24.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 msgr2=0x7ff2b0196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b0196040 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0005910 tx=0x7ff2a0004730 comp rx=0 tx=0).stop 2023-10-10T17:42:24.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 shutdown_connections 2023-10-10T17:42:24.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff29c0a11d0 0x7ff29c0a3680 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2b0101c40 0x7ff2b019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2b00fe540 0x7ff2b0196040 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 --2- 172.21.15.78:0/2270935002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2b00fee90 0x7ff2b0196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 >> 172.21.15.78:0/2270935002 conn(0x7ff2b00f9ee0 msgr2=0x7ff2b0104ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:24.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 shutdown_connections 2023-10-10T17:42:24.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.577+0000 7ff2b7450700 1 -- 172.21.15.78:0/2270935002 wait complete. 2023-10-10T17:42:24.581 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:42:24.591 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2023-10-10T17:42:24.592 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2023-10-10T17:42:24.592 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T17:42:24.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- 172.21.15.78:0/4047114685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37681036e0 msgr2=0x7f3768105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 --2- 172.21.15.78:0/4047114685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37681036e0 0x7f3768105ac0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3754009990 tx=0x7f3754009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- 172.21.15.78:0/4047114685 shutdown_connections 2023-10-10T17:42:24.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 --2- 172.21.15.78:0/4047114685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3768106000 0x7f37681083e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 --2- 172.21.15.78:0/4047114685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3768100dc0 0x7f37681031a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 --2- 172.21.15.78:0/4047114685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37681036e0 0x7f3768105ac0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- 172.21.15.78:0/4047114685 >> 172.21.15.78:0/4047114685 conn(0x7f37680fa7a0 msgr2=0x7f37680fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:24.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- 172.21.15.78:0/4047114685 shutdown_connections 2023-10-10T17:42:24.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- 172.21.15.78:0/4047114685 wait complete. 2023-10-10T17:42:24.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 Processor -- start 2023-10-10T17:42:24.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- start start 2023-10-10T17:42:24.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3768100dc0 0x7f3768195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37681036e0 0x7f37681964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3768106000 0x7f376819d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376819daf0 con 0x7f3768106000 2023-10-10T17:42:24.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376819dc30 con 0x7f3768100dc0 2023-10-10T17:42:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.713+0000 7f376ef09700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376819ddc0 con 0x7f37681036e0 2023-10-10T17:42:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3768106000 0x7f376819d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376cca5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3768100dc0 0x7f3768195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376cca5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3768100dc0 0x7f3768195f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:45694/0 (socket says 172.21.15.78:45694) 2023-10-10T17:42:24.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376cca5700 1 -- 172.21.15.78:0/2642345825 learned_addr learned my addr 172.21.15.78:0/2642345825 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:24.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3768106000 0x7f376819d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43976/0 (socket says 172.21.15.78:43976) 2023-10-10T17:42:24.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f3767fff700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37681036e0 0x7f37681964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 -- 172.21.15.78:0/2642345825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37681036e0 msgr2=0x7f37681964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37681036e0 0x7f37681964b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 -- 172.21.15.78:0/2642345825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3768100dc0 msgr2=0x7f3768195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3768100dc0 0x7f3768195f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 -- 172.21.15.78:0/2642345825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540095f0 con 0x7f3768106000 2023-10-10T17:42:24.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376d4a6700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3768106000 0x7f376819d520 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3760007f10 tx=0x7f376000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:24.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f3765ffb700 1 -- 172.21.15.78:0/2642345825 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760004bb0 con 0x7f3768106000 2023-10-10T17:42:24.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f3765ffb700 1 -- 172.21.15.78:0/2642345825 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f376000f040 con 0x7f3768106000 2023-10-10T17:42:24.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376810bc40 con 0x7f3768106000 2023-10-10T17:42:24.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f3765ffb700 1 -- 172.21.15.78:0/2642345825 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37600059d0 con 0x7f3768106000 2023-10-10T17:42:24.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.714+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376810bab0 con 0x7f3768106000 2023-10-10T17:42:24.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.715+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37680fc310 con 0x7f3768106000 2023-10-10T17:42:24.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.716+0000 7f3765ffb700 1 -- 172.21.15.78:0/2642345825 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3760004fa0 con 0x7f3768106000 2023-10-10T17:42:24.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.717+0000 7f3765ffb700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f37580a11d0 0x7f37580a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:24.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.717+0000 7f376cca5700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f37580a11d0 0x7f37580a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:24.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.717+0000 7f3765ffb700 1 -- 172.21.15.78:0/2642345825 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f37600cc3a0 con 0x7f3768106000 2023-10-10T17:42:24.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.717+0000 7f376cca5700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f37580a11d0 0x7f37580a3680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f375c005c50 tx=0x7f375c007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:24.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.721+0000 7f3765ffb700 1 -- 172.21.15.78:0/2642345825 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f376004e730 con 0x7f3768106000 2023-10-10T17:42:24.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:24 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2412951531' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:24.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:24 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/606056069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:24.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:24 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2270935002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:24.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.906+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3768061c00 con 0x7f37580a11d0 2023-10-10T17:42:24.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:24 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2412951531' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:24.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:24 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/606056069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:24.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:24 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2270935002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:24.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.918+0000 7f3765ffb700 1 -- 172.21.15.78:0/2642345825 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427292 (secure 0 0 0) 0x7f3768061c00 con 0x7f37580a11d0 2023-10-10T17:42:24.924 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:24.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f37580a11d0 msgr2=0x7f37580a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f37580a11d0 0x7f37580a3680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f375c005c50 tx=0x7f375c007480 comp rx=0 tx=0).stop 2023-10-10T17:42:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3768106000 msgr2=0x7f376819d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3768106000 0x7f376819d520 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3760007f10 tx=0x7f376000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 shutdown_connections 2023-10-10T17:42:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f37580a11d0 0x7f37580a3680 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3768106000 0x7f376819d520 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3768100dc0 0x7f3768195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 --2- 172.21.15.78:0/2642345825 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37681036e0 0x7f37681964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 >> 172.21.15.78:0/2642345825 conn(0x7f37680fa7a0 msgr2=0x7f3768101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 shutdown_connections 2023-10-10T17:42:24.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:24.922+0000 7f376ef09700 1 -- 172.21.15.78:0/2642345825 wait complete. 2023-10-10T17:42:24.931 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-10T17:42:24.936 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-10-10T17:42:24.243511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597936,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":648,"num_read_kb":389,"num_write":312,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80844,"kb_used_data":11884,"kb_used_omap":0,"kb_used_meta":68864,"kb_avail":1124860980,"statfs":{"total":1151940427776,"available":1151857643520,"internally_reserved":0,"allocated":12169216,"data_stored":7693629,"data_compressed":319365,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70516736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":260,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":46,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003374"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788069+0000","last_change":"2023-10-10T17:40:41.604268+0000","last_active":"2023-10-10T17:41:37.788069+0000","last_peered":"2023-10-10T17:41:37.788069+0000","last_clean":"2023-10-10T17:41:37.788069+0000","last_became_active":"2023-10-10T17:40:41.603839+0000","last_became_peered":"2023-10-10T17:40:41.603839+0000","last_unstale":"2023-10-10T17:41:37.788069+0000","last_undegraded":"2023-10-10T17:41:37.788069+0000","last_fullsized":"2023-10-10T17:41:37.788069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:00:22.805728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599055+0000","last_change":"2023-10-10T17:40:41.603642+0000","last_active":"2023-10-10T17:41:37.599055+0000","last_peered":"2023-10-10T17:41:37.599055+0000","last_clean":"2023-10-10T17:41:37.599055+0000","last_became_active":"2023-10-10T17:40:41.603091+0000","last_became_peered":"2023-10-10T17:40:41.603091+0000","last_unstale":"2023-10-10T17:41:37.599055+0000","last_undegraded":"2023-10-10T17:41:37.599055+0000","last_fullsized":"2023-10-10T17:41:37.599055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:10:38.807138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598740+0000","last_change":"2023-10-10T17:40:41.610647+0000","last_active":"2023-10-10T17:41:37.598740+0000","last_peered":"2023-10-10T17:41:37.598740+0000","last_clean":"2023-10-10T17:41:37.598740+0000","last_became_active":"2023-10-10T17:40:41.610338+0000","last_became_peered":"2023-10-10T17:40:41.610338+0000","last_unstale":"2023-10-10T17:41:37.598740+0000","last_undegraded":"2023-10-10T17:41:37.598740+0000","last_fullsized":"2023-10-10T17:41:37.598740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:16:20.110335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602832+0000","last_change":"2023-10-10T17:40:41.607832+0000","last_active":"2023-10-10T17:41:37.602832+0000","last_peered":"2023-10-10T17:41:37.602832+0000","last_clean":"2023-10-10T17:41:37.602832+0000","last_became_active":"2023-10-10T17:40:41.607492+0000","last_became_peered":"2023-10-10T17:40:41.607492+0000","last_unstale":"2023-10-10T17:41:37.602832+0000","last_undegraded":"2023-10-10T17:41:37.602832+0000","last_fullsized":"2023-10-10T17:41:37.602832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:43.514129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600669+0000","last_change":"2023-10-10T17:40:41.601975+0000","last_active":"2023-10-10T17:41:37.600669+0000","last_peered":"2023-10-10T17:41:37.600669+0000","last_clean":"2023-10-10T17:41:37.600669+0000","last_became_active":"2023-10-10T17:40:41.601668+0000","last_became_peered":"2023-10-10T17:40:41.601668+0000","last_unstale":"2023-10-10T17:41:37.600669+0000","last_undegraded":"2023-10-10T17:41:37.600669+0000","last_fullsized":"2023-10-10T17:41:37.600669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:25.433615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.604098+0000","last_change":"2023-10-10T17:41:37.604098+0000","last_active":"2023-10-10T17:41:37.604098+0000","last_peered":"2023-10-10T17:41:37.604098+0000","last_clean":"2023-10-10T17:41:37.604098+0000","last_became_active":"2023-10-10T17:41:37.603750+0000","last_became_peered":"2023-10-10T17:41:37.603750+0000","last_unstale":"2023-10-10T17:41:37.604098+0000","last_undegraded":"2023-10-10T17:41:37.604098+0000","last_fullsized":"2023-10-10T17:41:37.604098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:52:52.491557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602564+0000","last_change":"2023-10-10T17:40:41.594628+0000","last_active":"2023-10-10T17:41:37.602564+0000","last_peered":"2023-10-10T17:41:37.602564+0000","last_clean":"2023-10-10T17:41:37.602564+0000","last_became_active":"2023-10-10T17:40:41.594325+0000","last_became_peered":"2023-10-10T17:40:41.594325+0000","last_unstale":"2023-10-10T17:41:37.602564+0000","last_undegraded":"2023-10-10T17:41:37.602564+0000","last_fullsized":"2023-10-10T17:41:37.602564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:09:20.447064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978940+0000","last_change":"2023-10-10T17:40:41.598032+0000","last_active":"2023-10-10T17:41:37.978940+0000","last_peered":"2023-10-10T17:41:37.978940+0000","last_clean":"2023-10-10T17:41:37.978940+0000","last_became_active":"2023-10-10T17:40:41.597671+0000","last_became_peered":"2023-10-10T17:40:41.597671+0000","last_unstale":"2023-10-10T17:41:37.978940+0000","last_undegraded":"2023-10-10T17:41:37.978940+0000","last_fullsized":"2023-10-10T17:41:37.978940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:56:22.583298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598085+0000","last_change":"2023-10-10T17:40:41.599770+0000","last_active":"2023-10-10T17:41:37.598085+0000","last_peered":"2023-10-10T17:41:37.598085+0000","last_clean":"2023-10-10T17:41:37.598085+0000","last_became_active":"2023-10-10T17:40:41.599449+0000","last_became_peered":"2023-10-10T17:40:41.599449+0000","last_unstale":"2023-10-10T17:41:37.598085+0000","last_undegraded":"2023-10-10T17:41:37.598085+0000","last_fullsized":"2023-10-10T17:41:37.598085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:25:18.482096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598837+0000","last_change":"2023-10-10T17:40:41.610655+0000","last_active":"2023-10-10T17:41:37.598837+0000","last_peered":"2023-10-10T17:41:37.598837+0000","last_clean":"2023-10-10T17:41:37.598837+0000","last_became_active":"2023-10-10T17:40:41.610133+0000","last_became_peered":"2023-10-10T17:40:41.610133+0000","last_unstale":"2023-10-10T17:41:37.598837+0000","last_undegraded":"2023-10-10T17:41:37.598837+0000","last_fullsized":"2023-10-10T17:41:37.598837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:32:12.198400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602157+0000","last_change":"2023-10-10T17:40:41.601065+0000","last_active":"2023-10-10T17:41:37.602157+0000","last_peered":"2023-10-10T17:41:37.602157+0000","last_clean":"2023-10-10T17:41:37.602157+0000","last_became_active":"2023-10-10T17:40:41.600530+0000","last_became_peered":"2023-10-10T17:40:41.600530+0000","last_unstale":"2023-10-10T17:41:37.602157+0000","last_undegraded":"2023-10-10T17:41:37.602157+0000","last_fullsized":"2023-10-10T17:41:37.602157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:48:30.086020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602647+0000","last_change":"2023-10-10T17:40:41.596067+0000","last_active":"2023-10-10T17:41:37.602647+0000","last_peered":"2023-10-10T17:41:37.602647+0000","last_clean":"2023-10-10T17:41:37.602647+0000","last_became_active":"2023-10-10T17:40:41.595703+0000","last_became_peered":"2023-10-10T17:40:41.595703+0000","last_unstale":"2023-10-10T17:41:37.602647+0000","last_undegraded":"2023-10-10T17:41:37.602647+0000","last_fullsized":"2023-10-10T17:41:37.602647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:13.451772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598863+0000","last_change":"2023-10-10T17:40:41.593296+0000","last_active":"2023-10-10T17:41:37.598863+0000","last_peered":"2023-10-10T17:41:37.598863+0000","last_clean":"2023-10-10T17:41:37.598863+0000","last_became_active":"2023-10-10T17:40:41.592965+0000","last_became_peered":"2023-10-10T17:40:41.592965+0000","last_unstale":"2023-10-10T17:41:37.598863+0000","last_undegraded":"2023-10-10T17:41:37.598863+0000","last_fullsized":"2023-10-10T17:41:37.598863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:30:41.515067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598115+0000","last_change":"2023-10-10T17:40:41.604542+0000","last_active":"2023-10-10T17:41:37.598115+0000","last_peered":"2023-10-10T17:41:37.598115+0000","last_clean":"2023-10-10T17:41:37.598115+0000","last_became_active":"2023-10-10T17:40:41.604000+0000","last_became_peered":"2023-10-10T17:40:41.604000+0000","last_unstale":"2023-10-10T17:41:37.598115+0000","last_undegraded":"2023-10-10T17:41:37.598115+0000","last_fullsized":"2023-10-10T17:41:37.598115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:42:32.658592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602108+0000","last_change":"2023-10-10T17:41:37.602108+0000","last_active":"2023-10-10T17:41:37.602108+0000","last_peered":"2023-10-10T17:41:37.602108+0000","last_clean":"2023-10-10T17:41:37.602108+0000","last_became_active":"2023-10-10T17:41:37.601778+0000","last_became_peered":"2023-10-10T17:41:37.601778+0000","last_unstale":"2023-10-10T17:41:37.602108+0000","last_undegraded":"2023-10-10T17:41:37.602108+0000","last_fullsized":"2023-10-10T17:41:37.602108+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:48:51.539525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595527+0000","last_change":"2023-10-10T17:40:41.606659+0000","last_active":"2023-10-10T17:41:37.595527+0000","last_peered":"2023-10-10T17:41:37.595527+0000","last_clean":"2023-10-10T17:41:37.595527+0000","last_became_active":"2023-10-10T17:40:41.606061+0000","last_became_peered":"2023-10-10T17:40:41.606061+0000","last_unstale":"2023-10-10T17:41:37.595527+0000","last_undegraded":"2023-10-10T17:41:37.595527+0000","last_fullsized":"2023-10-10T17:41:37.595527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:04:19.222774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788308+0000","last_change":"2023-10-10T17:40:41.607630+0000","last_active":"2023-10-10T17:41:37.788308+0000","last_peered":"2023-10-10T17:41:37.788308+0000","last_clean":"2023-10-10T17:41:37.788308+0000","last_became_active":"2023-10-10T17:40:41.607297+0000","last_became_peered":"2023-10-10T17:40:41.607297+0000","last_unstale":"2023-10-10T17:41:37.788308+0000","last_undegraded":"2023-10-10T17:41:37.788308+0000","last_fullsized":"2023-10-10T17:41:37.788308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:15:06.512286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595355+0000","last_change":"2023-10-10T17:40:41.594082+0000","last_active":"2023-10-10T17:41:37.595355+0000","last_peered":"2023-10-10T17:41:37.595355+0000","last_clean":"2023-10-10T17:41:37.595355+0000","last_became_active":"2023-10-10T17:40:41.593787+0000","last_became_peered":"2023-10-10T17:40:41.593787+0000","last_unstale":"2023-10-10T17:41:37.595355+0000","last_undegraded":"2023-10-10T17:41:37.595355+0000","last_fullsized":"2023-10-10T17:41:37.595355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:10:48.548705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596760+0000","last_change":"2023-10-10T17:40:41.608529+0000","last_active":"2023-10-10T17:41:37.596760+0000","last_peered":"2023-10-10T17:41:37.596760+0000","last_clean":"2023-10-10T17:41:37.596760+0000","last_became_active":"2023-10-10T17:40:41.608247+0000","last_became_peered":"2023-10-10T17:40:41.608247+0000","last_unstale":"2023-10-10T17:41:37.596760+0000","last_undegraded":"2023-10-10T17:41:37.596760+0000","last_fullsized":"2023-10-10T17:41:37.596760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:20:05.900726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596528+0000","last_change":"2023-10-10T17:40:45.707455+0000","last_active":"2023-10-10T17:41:37.596528+0000","last_peered":"2023-10-10T17:41:37.596528+0000","last_clean":"2023-10-10T17:41:37.596528+0000","last_became_active":"2023-10-10T17:40:45.707024+0000","last_became_peered":"2023-10-10T17:40:45.707024+0000","last_unstale":"2023-10-10T17:41:37.596528+0000","last_undegraded":"2023-10-10T17:41:37.596528+0000","last_fullsized":"2023-10-10T17:41:37.596528+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:24.408486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595807+0000","last_change":"2023-10-10T17:40:41.165983+0000","last_active":"2023-10-10T17:41:37.595807+0000","last_peered":"2023-10-10T17:41:37.595807+0000","last_clean":"2023-10-10T17:41:37.595807+0000","last_became_active":"2023-10-10T17:40:41.165487+0000","last_became_peered":"2023-10-10T17:40:41.165487+0000","last_unstale":"2023-10-10T17:41:37.595807+0000","last_undegraded":"2023-10-10T17:41:37.595807+0000","last_fullsized":"2023-10-10T17:41:37.595807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:08:45.809159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979220+0000","last_change":"2023-10-10T17:40:41.614258+0000","last_active":"2023-10-10T17:41:37.979220+0000","last_peered":"2023-10-10T17:41:37.979220+0000","last_clean":"2023-10-10T17:41:37.979220+0000","last_became_active":"2023-10-10T17:40:41.613951+0000","last_became_peered":"2023-10-10T17:40:41.613951+0000","last_unstale":"2023-10-10T17:41:37.979220+0000","last_undegraded":"2023-10-10T17:41:37.979220+0000","last_fullsized":"2023-10-10T17:41:37.979220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:43:34.983899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:45.708568+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:45.708108+0000","last_became_peered":"2023-10-10T17:40:45.708108+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:30:24.153573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980255+0000","last_change":"2023-10-10T17:40:41.167744+0000","last_active":"2023-10-10T17:41:37.980255+0000","last_peered":"2023-10-10T17:41:37.980255+0000","last_clean":"2023-10-10T17:41:37.980255+0000","last_became_active":"2023-10-10T17:40:41.167189+0000","last_became_peered":"2023-10-10T17:40:41.167189+0000","last_unstale":"2023-10-10T17:41:37.980255+0000","last_undegraded":"2023-10-10T17:41:37.980255+0000","last_fullsized":"2023-10-10T17:41:37.980255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:47.667198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598064+0000","last_change":"2023-10-10T17:40:41.608514+0000","last_active":"2023-10-10T17:41:37.598064+0000","last_peered":"2023-10-10T17:41:37.598064+0000","last_clean":"2023-10-10T17:41:37.598064+0000","last_became_active":"2023-10-10T17:40:41.608208+0000","last_became_peered":"2023-10-10T17:40:41.608208+0000","last_unstale":"2023-10-10T17:41:37.598064+0000","last_undegraded":"2023-10-10T17:41:37.598064+0000","last_fullsized":"2023-10-10T17:41:37.598064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:48:25.411862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.981315+0000","last_change":"2023-10-10T17:41:37.981315+0000","last_active":"2023-10-10T17:41:37.981315+0000","last_peered":"2023-10-10T17:41:37.981315+0000","last_clean":"2023-10-10T17:41:37.981315+0000","last_became_active":"2023-10-10T17:41:37.980691+0000","last_became_peered":"2023-10-10T17:41:37.980691+0000","last_unstale":"2023-10-10T17:41:37.981315+0000","last_undegraded":"2023-10-10T17:41:37.981315+0000","last_fullsized":"2023-10-10T17:41:37.981315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:36:12.102836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601523+0000","last_change":"2023-10-10T17:40:41.169821+0000","last_active":"2023-10-10T17:41:37.601523+0000","last_peered":"2023-10-10T17:41:37.601523+0000","last_clean":"2023-10-10T17:41:37.601523+0000","last_became_active":"2023-10-10T17:40:41.169502+0000","last_became_peered":"2023-10-10T17:40:41.169502+0000","last_unstale":"2023-10-10T17:41:37.601523+0000","last_undegraded":"2023-10-10T17:41:37.601523+0000","last_fullsized":"2023-10-10T17:41:37.601523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:32:01.106471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595719+0000","last_change":"2023-10-10T17:40:41.602720+0000","last_active":"2023-10-10T17:41:37.595719+0000","last_peered":"2023-10-10T17:41:37.595719+0000","last_clean":"2023-10-10T17:41:37.595719+0000","last_became_active":"2023-10-10T17:40:41.602187+0000","last_became_peered":"2023-10-10T17:40:41.602187+0000","last_unstale":"2023-10-10T17:41:37.595719+0000","last_undegraded":"2023-10-10T17:41:37.595719+0000","last_fullsized":"2023-10-10T17:41:37.595719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:18:35.228329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601501+0000","last_change":"2023-10-10T17:40:45.706946+0000","last_active":"2023-10-10T17:41:37.601501+0000","last_peered":"2023-10-10T17:41:37.601501+0000","last_clean":"2023-10-10T17:41:37.601501+0000","last_became_active":"2023-10-10T17:40:45.706495+0000","last_became_peered":"2023-10-10T17:40:45.706495+0000","last_unstale":"2023-10-10T17:41:37.601501+0000","last_undegraded":"2023-10-10T17:41:37.601501+0000","last_fullsized":"2023-10-10T17:41:37.601501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:11.140718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601201+0000","last_change":"2023-10-10T17:40:41.171855+0000","last_active":"2023-10-10T17:41:37.601201+0000","last_peered":"2023-10-10T17:41:37.601201+0000","last_clean":"2023-10-10T17:41:37.601201+0000","last_became_active":"2023-10-10T17:40:41.171515+0000","last_became_peered":"2023-10-10T17:40:41.171515+0000","last_unstale":"2023-10-10T17:41:37.601201+0000","last_undegraded":"2023-10-10T17:41:37.601201+0000","last_fullsized":"2023-10-10T17:41:37.601201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:15.556629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652087+0000","last_change":"2023-10-10T17:40:41.587083+0000","last_active":"2023-10-10T17:41:37.652087+0000","last_peered":"2023-10-10T17:41:37.652087+0000","last_clean":"2023-10-10T17:41:37.652087+0000","last_became_active":"2023-10-10T17:40:41.586758+0000","last_became_peered":"2023-10-10T17:40:41.586758+0000","last_unstale":"2023-10-10T17:41:37.652087+0000","last_undegraded":"2023-10-10T17:41:37.652087+0000","last_fullsized":"2023-10-10T17:41:37.652087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:31:30.273631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596265+0000","last_change":"2023-10-10T17:40:45.710492+0000","last_active":"2023-10-10T17:41:37.596265+0000","last_peered":"2023-10-10T17:41:37.596265+0000","last_clean":"2023-10-10T17:41:37.596265+0000","last_became_active":"2023-10-10T17:40:45.710069+0000","last_became_peered":"2023-10-10T17:40:45.710069+0000","last_unstale":"2023-10-10T17:41:37.596265+0000","last_undegraded":"2023-10-10T17:41:37.596265+0000","last_fullsized":"2023-10-10T17:41:37.596265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:58:28.507189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:19.708911+0000","last_change":"2023-10-10T17:40:41.167449+0000","last_active":"2023-10-10T17:42:19.708911+0000","last_peered":"2023-10-10T17:42:19.708911+0000","last_clean":"2023-10-10T17:42:19.708911+0000","last_became_active":"2023-10-10T17:40:41.167038+0000","last_became_peered":"2023-10-10T17:40:41.167038+0000","last_unstale":"2023-10-10T17:42:19.708911+0000","last_undegraded":"2023-10-10T17:42:19.708911+0000","last_fullsized":"2023-10-10T17:42:19.708911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:48:38.144880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599067+0000","last_change":"2023-10-10T17:40:41.594327+0000","last_active":"2023-10-10T17:41:37.599067+0000","last_peered":"2023-10-10T17:41:37.599067+0000","last_clean":"2023-10-10T17:41:37.599067+0000","last_became_active":"2023-10-10T17:40:41.594008+0000","last_became_peered":"2023-10-10T17:40:41.594008+0000","last_unstale":"2023-10-10T17:41:37.599067+0000","last_undegraded":"2023-10-10T17:41:37.599067+0000","last_fullsized":"2023-10-10T17:41:37.599067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:56:08.300633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.653403+0000","last_change":"2023-10-10T17:41:37.653403+0000","last_active":"2023-10-10T17:41:37.653403+0000","last_peered":"2023-10-10T17:41:37.653403+0000","last_clean":"2023-10-10T17:41:37.653403+0000","last_became_active":"2023-10-10T17:41:37.652946+0000","last_became_peered":"2023-10-10T17:41:37.652946+0000","last_unstale":"2023-10-10T17:41:37.653403+0000","last_undegraded":"2023-10-10T17:41:37.653403+0000","last_fullsized":"2023-10-10T17:41:37.653403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:16:29.417131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598235+0000","last_change":"2023-10-10T17:40:41.171926+0000","last_active":"2023-10-10T17:41:37.598235+0000","last_peered":"2023-10-10T17:41:37.598235+0000","last_clean":"2023-10-10T17:41:37.598235+0000","last_became_active":"2023-10-10T17:40:41.171627+0000","last_became_peered":"2023-10-10T17:40:41.171627+0000","last_unstale":"2023-10-10T17:41:37.598235+0000","last_undegraded":"2023-10-10T17:41:37.598235+0000","last_fullsized":"2023-10-10T17:41:37.598235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:00:51.367484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651905+0000","last_change":"2023-10-10T17:40:41.593135+0000","last_active":"2023-10-10T17:41:37.651905+0000","last_peered":"2023-10-10T17:41:37.651905+0000","last_clean":"2023-10-10T17:41:37.651905+0000","last_became_active":"2023-10-10T17:40:41.592871+0000","last_became_peered":"2023-10-10T17:40:41.592871+0000","last_unstale":"2023-10-10T17:41:37.651905+0000","last_undegraded":"2023-10-10T17:41:37.651905+0000","last_fullsized":"2023-10-10T17:41:37.651905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:32.143194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788755+0000","last_change":"2023-10-10T17:40:45.703557+0000","last_active":"2023-10-10T17:41:37.788755+0000","last_peered":"2023-10-10T17:41:37.788755+0000","last_clean":"2023-10-10T17:41:37.788755+0000","last_became_active":"2023-10-10T17:40:45.703085+0000","last_became_peered":"2023-10-10T17:40:45.703085+0000","last_unstale":"2023-10-10T17:41:37.788755+0000","last_undegraded":"2023-10-10T17:41:37.788755+0000","last_fullsized":"2023-10-10T17:41:37.788755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:09:11.130227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598088+0000","last_change":"2023-10-10T17:40:41.162829+0000","last_active":"2023-10-10T17:41:37.598088+0000","last_peered":"2023-10-10T17:41:37.598088+0000","last_clean":"2023-10-10T17:41:37.598088+0000","last_became_active":"2023-10-10T17:40:41.162516+0000","last_became_peered":"2023-10-10T17:40:41.162516+0000","last_unstale":"2023-10-10T17:41:37.598088+0000","last_undegraded":"2023-10-10T17:41:37.598088+0000","last_fullsized":"2023-10-10T17:41:37.598088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:07:29.590599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.789909+0000","last_change":"2023-10-10T17:41:37.789510+0000","last_active":"2023-10-10T17:41:37.789909+0000","last_peered":"2023-10-10T17:41:37.789909+0000","last_clean":"2023-10-10T17:41:37.789909+0000","last_became_active":"2023-10-10T17:41:37.789166+0000","last_became_peered":"2023-10-10T17:41:37.789166+0000","last_unstale":"2023-10-10T17:41:37.789909+0000","last_undegraded":"2023-10-10T17:41:37.789909+0000","last_fullsized":"2023-10-10T17:41:37.789909+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:33:55.993371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598938+0000","last_change":"2023-10-10T17:40:45.707139+0000","last_active":"2023-10-10T17:41:37.598938+0000","last_peered":"2023-10-10T17:41:37.598938+0000","last_clean":"2023-10-10T17:41:37.598938+0000","last_became_active":"2023-10-10T17:40:45.706708+0000","last_became_peered":"2023-10-10T17:40:45.706708+0000","last_unstale":"2023-10-10T17:41:37.598938+0000","last_undegraded":"2023-10-10T17:41:37.598938+0000","last_fullsized":"2023-10-10T17:41:37.598938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:46:42.381782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651470+0000","last_change":"2023-10-10T17:40:41.170512+0000","last_active":"2023-10-10T17:41:37.651470+0000","last_peered":"2023-10-10T17:41:37.651470+0000","last_clean":"2023-10-10T17:41:37.651470+0000","last_became_active":"2023-10-10T17:40:41.170264+0000","last_became_peered":"2023-10-10T17:40:41.170264+0000","last_unstale":"2023-10-10T17:41:37.651470+0000","last_undegraded":"2023-10-10T17:41:37.651470+0000","last_fullsized":"2023-10-10T17:41:37.651470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:21:55.138815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594921+0000","last_change":"2023-10-10T17:40:41.597647+0000","last_active":"2023-10-10T17:41:37.594921+0000","last_peered":"2023-10-10T17:41:37.594921+0000","last_clean":"2023-10-10T17:41:37.594921+0000","last_became_active":"2023-10-10T17:40:41.597245+0000","last_became_peered":"2023-10-10T17:40:41.597245+0000","last_unstale":"2023-10-10T17:41:37.594921+0000","last_undegraded":"2023-10-10T17:41:37.594921+0000","last_fullsized":"2023-10-10T17:41:37.594921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:02.702932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979667+0000","last_change":"2023-10-10T17:40:45.705089+0000","last_active":"2023-10-10T17:41:37.979667+0000","last_peered":"2023-10-10T17:41:37.979667+0000","last_clean":"2023-10-10T17:41:37.979667+0000","last_became_active":"2023-10-10T17:40:45.704475+0000","last_became_peered":"2023-10-10T17:40:45.704475+0000","last_unstale":"2023-10-10T17:41:37.979667+0000","last_undegraded":"2023-10-10T17:41:37.979667+0000","last_fullsized":"2023-10-10T17:41:37.979667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:26.896709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:21.710912+0000","last_change":"2023-10-10T17:40:41.167409+0000","last_active":"2023-10-10T17:42:21.710912+0000","last_peered":"2023-10-10T17:42:21.710912+0000","last_clean":"2023-10-10T17:42:21.710912+0000","last_became_active":"2023-10-10T17:40:41.167125+0000","last_became_peered":"2023-10-10T17:40:41.167125+0000","last_unstale":"2023-10-10T17:42:21.710912+0000","last_undegraded":"2023-10-10T17:42:21.710912+0000","last_fullsized":"2023-10-10T17:42:21.710912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:12:36.905039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598010+0000","last_change":"2023-10-10T17:40:41.606216+0000","last_active":"2023-10-10T17:41:37.598010+0000","last_peered":"2023-10-10T17:41:37.598010+0000","last_clean":"2023-10-10T17:41:37.598010+0000","last_became_active":"2023-10-10T17:40:41.603258+0000","last_became_peered":"2023-10-10T17:40:41.603258+0000","last_unstale":"2023-10-10T17:41:37.598010+0000","last_undegraded":"2023-10-10T17:41:37.598010+0000","last_fullsized":"2023-10-10T17:41:37.598010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:30:40.923843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594917+0000","last_change":"2023-10-10T17:40:45.705623+0000","last_active":"2023-10-10T17:41:37.594917+0000","last_peered":"2023-10-10T17:41:37.594917+0000","last_clean":"2023-10-10T17:41:37.594917+0000","last_became_active":"2023-10-10T17:40:45.705242+0000","last_became_peered":"2023-10-10T17:40:45.705242+0000","last_unstale":"2023-10-10T17:41:37.594917+0000","last_undegraded":"2023-10-10T17:41:37.594917+0000","last_fullsized":"2023-10-10T17:41:37.594917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:43:18.391641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601242+0000","last_change":"2023-10-10T17:40:41.171552+0000","last_active":"2023-10-10T17:41:37.601242+0000","last_peered":"2023-10-10T17:41:37.601242+0000","last_clean":"2023-10-10T17:41:37.601242+0000","last_became_active":"2023-10-10T17:40:41.171207+0000","last_became_peered":"2023-10-10T17:40:41.171207+0000","last_unstale":"2023-10-10T17:41:37.601242+0000","last_undegraded":"2023-10-10T17:41:37.601242+0000","last_fullsized":"2023-10-10T17:41:37.601242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:35.436090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599824+0000","last_change":"2023-10-10T17:40:41.609655+0000","last_active":"2023-10-10T17:41:37.599824+0000","last_peered":"2023-10-10T17:41:37.599824+0000","last_clean":"2023-10-10T17:41:37.599824+0000","last_became_active":"2023-10-10T17:40:41.609318+0000","last_became_peered":"2023-10-10T17:40:41.609318+0000","last_unstale":"2023-10-10T17:41:37.599824+0000","last_undegraded":"2023-10-10T17:41:37.599824+0000","last_fullsized":"2023-10-10T17:41:37.599824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:06:22.359139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602379+0000","last_change":"2023-10-10T17:40:45.709513+0000","last_active":"2023-10-10T17:41:37.602379+0000","last_peered":"2023-10-10T17:41:37.602379+0000","last_clean":"2023-10-10T17:41:37.602379+0000","last_became_active":"2023-10-10T17:40:45.708963+0000","last_became_peered":"2023-10-10T17:40:45.708963+0000","last_unstale":"2023-10-10T17:41:37.602379+0000","last_undegraded":"2023-10-10T17:41:37.602379+0000","last_fullsized":"2023-10-10T17:41:37.602379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:15:02.687427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652007+0000","last_change":"2023-10-10T17:40:41.163234+0000","last_active":"2023-10-10T17:41:37.652007+0000","last_peered":"2023-10-10T17:41:37.652007+0000","last_clean":"2023-10-10T17:41:37.652007+0000","last_became_active":"2023-10-10T17:40:41.162916+0000","last_became_peered":"2023-10-10T17:40:41.162916+0000","last_unstale":"2023-10-10T17:41:37.652007+0000","last_undegraded":"2023-10-10T17:41:37.652007+0000","last_fullsized":"2023-10-10T17:41:37.652007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:09:06.358768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598352+0000","last_change":"2023-10-10T17:40:41.602852+0000","last_active":"2023-10-10T17:41:37.598352+0000","last_peered":"2023-10-10T17:41:37.598352+0000","last_clean":"2023-10-10T17:41:37.598352+0000","last_became_active":"2023-10-10T17:40:41.602530+0000","last_became_peered":"2023-10-10T17:40:41.602530+0000","last_unstale":"2023-10-10T17:41:37.598352+0000","last_undegraded":"2023-10-10T17:41:37.598352+0000","last_fullsized":"2023-10-10T17:41:37.598352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:58:41.949886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595188+0000","last_change":"2023-10-10T17:40:45.713714+0000","last_active":"2023-10-10T17:41:37.595188+0000","last_peered":"2023-10-10T17:41:37.595188+0000","last_clean":"2023-10-10T17:41:37.595188+0000","last_became_active":"2023-10-10T17:40:45.712953+0000","last_became_peered":"2023-10-10T17:40:45.712953+0000","last_unstale":"2023-10-10T17:41:37.595188+0000","last_undegraded":"2023-10-10T17:41:37.595188+0000","last_fullsized":"2023-10-10T17:41:37.595188+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:15.222779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":204,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:22.711933+0000","last_change":"2023-10-10T17:40:41.168328+0000","last_active":"2023-10-10T17:42:22.711933+0000","last_peered":"2023-10-10T17:42:22.711933+0000","last_clean":"2023-10-10T17:42:22.711933+0000","last_became_active":"2023-10-10T17:40:41.168044+0000","last_became_peered":"2023-10-10T17:40:41.168044+0000","last_unstale":"2023-10-10T17:42:22.711933+0000","last_undegraded":"2023-10-10T17:42:22.711933+0000","last_fullsized":"2023-10-10T17:42:22.711933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:52:24.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":166,"num_read_kb":87,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599535+0000","last_change":"2023-10-10T17:40:41.597795+0000","last_active":"2023-10-10T17:41:37.599535+0000","last_peered":"2023-10-10T17:41:37.599535+0000","last_clean":"2023-10-10T17:41:37.599535+0000","last_became_active":"2023-10-10T17:40:41.597537+0000","last_became_peered":"2023-10-10T17:40:41.597537+0000","last_unstale":"2023-10-10T17:41:37.599535+0000","last_undegraded":"2023-10-10T17:41:37.599535+0000","last_fullsized":"2023-10-10T17:41:37.599535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:16:34.371564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652449+0000","last_change":"2023-10-10T17:40:45.709827+0000","last_active":"2023-10-10T17:41:37.652449+0000","last_peered":"2023-10-10T17:41:37.652449+0000","last_clean":"2023-10-10T17:41:37.652449+0000","last_became_active":"2023-10-10T17:40:45.709373+0000","last_became_peered":"2023-10-10T17:40:45.709373+0000","last_unstale":"2023-10-10T17:41:37.652449+0000","last_undegraded":"2023-10-10T17:41:37.652449+0000","last_fullsized":"2023-10-10T17:41:37.652449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:35:42.826536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596527+0000","last_change":"2023-10-10T17:40:41.160804+0000","last_active":"2023-10-10T17:41:37.596527+0000","last_peered":"2023-10-10T17:41:37.596527+0000","last_clean":"2023-10-10T17:41:37.596527+0000","last_became_active":"2023-10-10T17:40:41.160493+0000","last_became_peered":"2023-10-10T17:40:41.160493+0000","last_unstale":"2023-10-10T17:41:37.596527+0000","last_undegraded":"2023-10-10T17:41:37.596527+0000","last_fullsized":"2023-10-10T17:41:37.596527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:59.561663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598334+0000","last_change":"2023-10-10T17:40:41.601357+0000","last_active":"2023-10-10T17:41:37.598334+0000","last_peered":"2023-10-10T17:41:37.598334+0000","last_clean":"2023-10-10T17:41:37.598334+0000","last_became_active":"2023-10-10T17:40:41.601057+0000","last_became_peered":"2023-10-10T17:40:41.601057+0000","last_unstale":"2023-10-10T17:41:37.598334+0000","last_undegraded":"2023-10-10T17:41:37.598334+0000","last_fullsized":"2023-10-10T17:41:37.598334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:07.286077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788393+0000","last_change":"2023-10-10T17:40:45.708067+0000","last_active":"2023-10-10T17:41:37.788393+0000","last_peered":"2023-10-10T17:41:37.788393+0000","last_clean":"2023-10-10T17:41:37.788393+0000","last_became_active":"2023-10-10T17:40:45.707686+0000","last_became_peered":"2023-10-10T17:40:45.707686+0000","last_unstale":"2023-10-10T17:41:37.788393+0000","last_undegraded":"2023-10-10T17:41:37.788393+0000","last_fullsized":"2023-10-10T17:41:37.788393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:08.956407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651829+0000","last_change":"2023-10-10T17:40:41.168006+0000","last_active":"2023-10-10T17:41:37.651829+0000","last_peered":"2023-10-10T17:41:37.651829+0000","last_clean":"2023-10-10T17:41:37.651829+0000","last_became_active":"2023-10-10T17:40:41.167777+0000","last_became_peered":"2023-10-10T17:40:41.167777+0000","last_unstale":"2023-10-10T17:41:37.651829+0000","last_undegraded":"2023-10-10T17:41:37.651829+0000","last_fullsized":"2023-10-10T17:41:37.651829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:30:32.283329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601103+0000","last_change":"2023-10-10T17:40:41.591215+0000","last_active":"2023-10-10T17:41:37.601103+0000","last_peered":"2023-10-10T17:41:37.601103+0000","last_clean":"2023-10-10T17:41:37.601103+0000","last_became_active":"2023-10-10T17:40:41.590929+0000","last_became_peered":"2023-10-10T17:40:41.590929+0000","last_unstale":"2023-10-10T17:41:37.601103+0000","last_undegraded":"2023-10-10T17:41:37.601103+0000","last_fullsized":"2023-10-10T17:41:37.601103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:11:13.144245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595485+0000","last_change":"2023-10-10T17:40:45.709038+0000","last_active":"2023-10-10T17:41:37.595485+0000","last_peered":"2023-10-10T17:41:37.595485+0000","last_clean":"2023-10-10T17:41:37.595485+0000","last_became_active":"2023-10-10T17:40:45.708683+0000","last_became_peered":"2023-10-10T17:40:45.708683+0000","last_unstale":"2023-10-10T17:41:37.595485+0000","last_undegraded":"2023-10-10T17:41:37.595485+0000","last_fullsized":"2023-10-10T17:41:37.595485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:50:01.174506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:22.722538+0000","last_change":"2023-10-10T17:40:41.163817+0000","last_active":"2023-10-10T17:42:22.722538+0000","last_peered":"2023-10-10T17:42:22.722538+0000","last_clean":"2023-10-10T17:42:22.722538+0000","last_became_active":"2023-10-10T17:40:41.163497+0000","last_became_peered":"2023-10-10T17:40:41.163497+0000","last_unstale":"2023-10-10T17:42:22.722538+0000","last_undegraded":"2023-10-10T17:42:22.722538+0000","last_fullsized":"2023-10-10T17:42:22.722538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:54:45.382546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":800,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.607553+0000","last_change":"2023-10-10T17:41:37.607553+0000","last_active":"2023-10-10T17:41:37.607553+0000","last_peered":"2023-10-10T17:41:37.607553+0000","last_clean":"2023-10-10T17:41:37.607553+0000","last_became_active":"2023-10-10T17:41:37.606536+0000","last_became_peered":"2023-10-10T17:41:37.606536+0000","last_unstale":"2023-10-10T17:41:37.607553+0000","last_undegraded":"2023-10-10T17:41:37.607553+0000","last_fullsized":"2023-10-10T17:41:37.607553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:58:09.182227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600059+0000","last_change":"2023-10-10T17:40:45.714900+0000","last_active":"2023-10-10T17:41:37.600059+0000","last_peered":"2023-10-10T17:41:37.600059+0000","last_clean":"2023-10-10T17:41:37.600059+0000","last_became_active":"2023-10-10T17:40:45.714525+0000","last_became_peered":"2023-10-10T17:40:45.714525+0000","last_unstale":"2023-10-10T17:41:37.600059+0000","last_undegraded":"2023-10-10T17:41:37.600059+0000","last_fullsized":"2023-10-10T17:41:37.600059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:01:36.283546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:38.795173+0000","last_change":"2023-10-10T17:41:38.795173+0000","last_active":"2023-10-10T17:41:38.795173+0000","last_peered":"2023-10-10T17:41:38.795173+0000","last_clean":"2023-10-10T17:41:38.795173+0000","last_became_active":"2023-10-10T17:41:37.603917+0000","last_became_peered":"2023-10-10T17:41:37.603917+0000","last_unstale":"2023-10-10T17:41:38.795173+0000","last_undegraded":"2023-10-10T17:41:38.795173+0000","last_fullsized":"2023-10-10T17:41:38.795173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:32:38.418130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651410+0000","last_change":"2023-10-10T17:40:41.586591+0000","last_active":"2023-10-10T17:41:37.651410+0000","last_peered":"2023-10-10T17:41:37.651410+0000","last_clean":"2023-10-10T17:41:37.651410+0000","last_became_active":"2023-10-10T17:40:41.586293+0000","last_became_peered":"2023-10-10T17:40:41.586293+0000","last_unstale":"2023-10-10T17:41:37.651410+0000","last_undegraded":"2023-10-10T17:41:37.651410+0000","last_fullsized":"2023-10-10T17:41:37.651410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:39:26.663874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651375+0000","last_change":"2023-10-10T17:40:45.708348+0000","last_active":"2023-10-10T17:41:37.651375+0000","last_peered":"2023-10-10T17:41:37.651375+0000","last_clean":"2023-10-10T17:41:37.651375+0000","last_became_active":"2023-10-10T17:40:45.707899+0000","last_became_peered":"2023-10-10T17:40:45.707899+0000","last_unstale":"2023-10-10T17:41:37.651375+0000","last_undegraded":"2023-10-10T17:41:37.651375+0000","last_fullsized":"2023-10-10T17:41:37.651375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:29:39.345572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601729+0000","last_change":"2023-10-10T17:40:41.168209+0000","last_active":"2023-10-10T17:41:37.601729+0000","last_peered":"2023-10-10T17:41:37.601729+0000","last_clean":"2023-10-10T17:41:37.601729+0000","last_became_active":"2023-10-10T17:40:41.167907+0000","last_became_peered":"2023-10-10T17:40:41.167907+0000","last_unstale":"2023-10-10T17:41:37.601729+0000","last_undegraded":"2023-10-10T17:41:37.601729+0000","last_fullsized":"2023-10-10T17:41:37.601729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:35:19.344734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.594848+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.594562+0000","last_became_peered":"2023-10-10T17:40:41.594562+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:15:21.805794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600519+0000","last_change":"2023-10-10T17:40:45.711043+0000","last_active":"2023-10-10T17:41:37.600519+0000","last_peered":"2023-10-10T17:41:37.600519+0000","last_clean":"2023-10-10T17:41:37.600519+0000","last_became_active":"2023-10-10T17:40:45.710617+0000","last_became_peered":"2023-10-10T17:40:45.710617+0000","last_unstale":"2023-10-10T17:41:37.600519+0000","last_undegraded":"2023-10-10T17:41:37.600519+0000","last_fullsized":"2023-10-10T17:41:37.600519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:00:48.152401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595133+0000","last_change":"2023-10-10T17:40:41.163238+0000","last_active":"2023-10-10T17:41:37.595133+0000","last_peered":"2023-10-10T17:41:37.595133+0000","last_clean":"2023-10-10T17:41:37.595133+0000","last_became_active":"2023-10-10T17:40:41.162934+0000","last_became_peered":"2023-10-10T17:40:41.162934+0000","last_unstale":"2023-10-10T17:41:37.595133+0000","last_undegraded":"2023-10-10T17:41:37.595133+0000","last_fullsized":"2023-10-10T17:41:37.595133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:23:10.440706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598371+0000","last_change":"2023-10-10T17:40:41.597026+0000","last_active":"2023-10-10T17:41:37.598371+0000","last_peered":"2023-10-10T17:41:37.598371+0000","last_clean":"2023-10-10T17:41:37.598371+0000","last_became_active":"2023-10-10T17:40:41.596747+0000","last_became_peered":"2023-10-10T17:40:41.596747+0000","last_unstale":"2023-10-10T17:41:37.598371+0000","last_undegraded":"2023-10-10T17:41:37.598371+0000","last_fullsized":"2023-10-10T17:41:37.598371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:52.010575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599056+0000","last_change":"2023-10-10T17:40:45.707218+0000","last_active":"2023-10-10T17:41:37.599056+0000","last_peered":"2023-10-10T17:41:37.599056+0000","last_clean":"2023-10-10T17:41:37.599056+0000","last_became_active":"2023-10-10T17:40:45.706850+0000","last_became_peered":"2023-10-10T17:40:45.706850+0000","last_unstale":"2023-10-10T17:41:37.599056+0000","last_undegraded":"2023-10-10T17:41:37.599056+0000","last_fullsized":"2023-10-10T17:41:37.599056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:36:44.235237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:41.166713+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:41.165951+0000","last_became_peered":"2023-10-10T17:40:41.165951+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:56:00.129165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597783+0000","last_change":"2023-10-10T17:40:41.605860+0000","last_active":"2023-10-10T17:41:37.597783+0000","last_peered":"2023-10-10T17:41:37.597783+0000","last_clean":"2023-10-10T17:41:37.597783+0000","last_became_active":"2023-10-10T17:40:41.605556+0000","last_became_peered":"2023-10-10T17:40:41.605556+0000","last_unstale":"2023-10-10T17:41:37.597783+0000","last_undegraded":"2023-10-10T17:41:37.597783+0000","last_fullsized":"2023-10-10T17:41:37.597783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:07.438058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599757+0000","last_change":"2023-10-10T17:40:45.711844+0000","last_active":"2023-10-10T17:41:37.599757+0000","last_peered":"2023-10-10T17:41:37.599757+0000","last_clean":"2023-10-10T17:41:37.599757+0000","last_became_active":"2023-10-10T17:40:45.711413+0000","last_became_peered":"2023-10-10T17:40:45.711413+0000","last_unstale":"2023-10-10T17:41:37.599757+0000","last_undegraded":"2023-10-10T17:41:37.599757+0000","last_fullsized":"2023-10-10T17:41:37.599757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:13:03.476627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.650650+0000","last_change":"2023-10-10T17:40:41.608625+0000","last_active":"2023-10-10T17:41:37.650650+0000","last_peered":"2023-10-10T17:41:37.650650+0000","last_clean":"2023-10-10T17:41:37.650650+0000","last_became_active":"2023-10-10T17:40:41.608332+0000","last_became_peered":"2023-10-10T17:40:41.608332+0000","last_unstale":"2023-10-10T17:41:37.650650+0000","last_undegraded":"2023-10-10T17:41:37.650650+0000","last_fullsized":"2023-10-10T17:41:37.650650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:07:55.183440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:21.720594+0000","last_change":"2023-10-10T17:40:41.169807+0000","last_active":"2023-10-10T17:42:21.720594+0000","last_peered":"2023-10-10T17:42:21.720594+0000","last_clean":"2023-10-10T17:42:21.720594+0000","last_became_active":"2023-10-10T17:40:41.169475+0000","last_became_peered":"2023-10-10T17:40:41.169475+0000","last_unstale":"2023-10-10T17:42:21.720594+0000","last_undegraded":"2023-10-10T17:42:21.720594+0000","last_fullsized":"2023-10-10T17:42:21.720594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:22.887502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601878+0000","last_change":"2023-10-10T17:40:45.710544+0000","last_active":"2023-10-10T17:41:37.601878+0000","last_peered":"2023-10-10T17:41:37.601878+0000","last_clean":"2023-10-10T17:41:37.601878+0000","last_became_active":"2023-10-10T17:40:45.710084+0000","last_became_peered":"2023-10-10T17:40:45.710084+0000","last_unstale":"2023-10-10T17:41:37.601878+0000","last_undegraded":"2023-10-10T17:41:37.601878+0000","last_fullsized":"2023-10-10T17:41:37.601878+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:44:01.303666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979130+0000","last_change":"2023-10-10T17:40:41.599028+0000","last_active":"2023-10-10T17:41:37.979130+0000","last_peered":"2023-10-10T17:41:37.979130+0000","last_clean":"2023-10-10T17:41:37.979130+0000","last_became_active":"2023-10-10T17:40:41.598684+0000","last_became_peered":"2023-10-10T17:40:41.598684+0000","last_unstale":"2023-10-10T17:41:37.979130+0000","last_undegraded":"2023-10-10T17:41:37.979130+0000","last_fullsized":"2023-10-10T17:41:37.979130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:16:57.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602213+0000","last_change":"2023-10-10T17:40:41.165756+0000","last_active":"2023-10-10T17:41:37.602213+0000","last_peered":"2023-10-10T17:41:37.602213+0000","last_clean":"2023-10-10T17:41:37.602213+0000","last_became_active":"2023-10-10T17:40:41.165368+0000","last_became_peered":"2023-10-10T17:40:41.165368+0000","last_unstale":"2023-10-10T17:41:37.602213+0000","last_undegraded":"2023-10-10T17:41:37.602213+0000","last_fullsized":"2023-10-10T17:41:37.602213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:16.089880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600889+0000","last_change":"2023-10-10T17:39:42.508828+0000","last_active":"2023-10-10T17:41:37.600889+0000","last_peered":"2023-10-10T17:41:37.600889+0000","last_clean":"2023-10-10T17:41:37.600889+0000","last_became_active":"2023-10-10T17:39:42.508475+0000","last_became_peered":"2023-10-10T17:39:42.508475+0000","last_unstale":"2023-10-10T17:41:37.600889+0000","last_undegraded":"2023-10-10T17:41:37.600889+0000","last_fullsized":"2023-10-10T17:41:37.600889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_clean_scrub_stamp":"2023-10-10T17:36:46.553699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:08:38.008210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652108+0000","last_change":"2023-10-10T17:40:45.714604+0000","last_active":"2023-10-10T17:41:37.652108+0000","last_peered":"2023-10-10T17:41:37.652108+0000","last_clean":"2023-10-10T17:41:37.652108+0000","last_became_active":"2023-10-10T17:40:45.714185+0000","last_became_peered":"2023-10-10T17:40:45.714185+0000","last_unstale":"2023-10-10T17:41:37.652108+0000","last_undegraded":"2023-10-10T17:41:37.652108+0000","last_fullsized":"2023-10-10T17:41:37.652108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:55:26.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597853+0000","last_change":"2023-10-10T17:40:41.610002+0000","last_active":"2023-10-10T17:41:37.597853+0000","last_peered":"2023-10-10T17:41:37.597853+0000","last_clean":"2023-10-10T17:41:37.597853+0000","last_became_active":"2023-10-10T17:40:41.609708+0000","last_became_peered":"2023-10-10T17:40:41.609708+0000","last_unstale":"2023-10-10T17:41:37.597853+0000","last_undegraded":"2023-10-10T17:41:37.597853+0000","last_fullsized":"2023-10-10T17:41:37.597853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:34.611184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601055+0000","last_change":"2023-10-10T17:40:41.167715+0000","last_active":"2023-10-10T17:41:37.601055+0000","last_peered":"2023-10-10T17:41:37.601055+0000","last_clean":"2023-10-10T17:41:37.601055+0000","last_became_active":"2023-10-10T17:40:41.166385+0000","last_became_peered":"2023-10-10T17:40:41.166385+0000","last_unstale":"2023-10-10T17:41:37.601055+0000","last_undegraded":"2023-10-10T17:41:37.601055+0000","last_fullsized":"2023-10-10T17:41:37.601055+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:49:05.385383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788299+0000","last_change":"2023-10-10T17:40:45.707236+0000","last_active":"2023-10-10T17:41:37.788299+0000","last_peered":"2023-10-10T17:41:37.788299+0000","last_clean":"2023-10-10T17:41:37.788299+0000","last_became_active":"2023-10-10T17:40:45.706827+0000","last_became_peered":"2023-10-10T17:40:45.706827+0000","last_unstale":"2023-10-10T17:41:37.788299+0000","last_undegraded":"2023-10-10T17:41:37.788299+0000","last_fullsized":"2023-10-10T17:41:37.788299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:58.930376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601396+0000","last_change":"2023-10-10T17:40:41.170523+0000","last_active":"2023-10-10T17:41:37.601396+0000","last_peered":"2023-10-10T17:41:37.601396+0000","last_clean":"2023-10-10T17:41:37.601396+0000","last_became_active":"2023-10-10T17:40:41.170243+0000","last_became_peered":"2023-10-10T17:40:41.170243+0000","last_unstale":"2023-10-10T17:41:37.601396+0000","last_undegraded":"2023-10-10T17:41:37.601396+0000","last_fullsized":"2023-10-10T17:41:37.601396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:40.239129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599195+0000","last_change":"2023-10-10T17:40:41.610472+0000","last_active":"2023-10-10T17:41:37.599195+0000","last_peered":"2023-10-10T17:41:37.599195+0000","last_clean":"2023-10-10T17:41:37.599195+0000","last_became_active":"2023-10-10T17:40:41.610139+0000","last_became_peered":"2023-10-10T17:40:41.610139+0000","last_unstale":"2023-10-10T17:41:37.599195+0000","last_undegraded":"2023-10-10T17:41:37.599195+0000","last_fullsized":"2023-10-10T17:41:37.599195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:22:52.414485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979558+0000","last_change":"2023-10-10T17:40:45.712179+0000","last_active":"2023-10-10T17:41:37.979558+0000","last_peered":"2023-10-10T17:41:37.979558+0000","last_clean":"2023-10-10T17:41:37.979558+0000","last_became_active":"2023-10-10T17:40:45.711708+0000","last_became_peered":"2023-10-10T17:40:45.711708+0000","last_unstale":"2023-10-10T17:41:37.979558+0000","last_undegraded":"2023-10-10T17:41:37.979558+0000","last_fullsized":"2023-10-10T17:41:37.979558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:25:37.789886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595330+0000","last_change":"2023-10-10T17:40:41.164882+0000","last_active":"2023-10-10T17:41:37.595330+0000","last_peered":"2023-10-10T17:41:37.595330+0000","last_clean":"2023-10-10T17:41:37.595330+0000","last_became_active":"2023-10-10T17:40:41.164536+0000","last_became_peered":"2023-10-10T17:40:41.164536+0000","last_unstale":"2023-10-10T17:41:37.595330+0000","last_undegraded":"2023-10-10T17:41:37.595330+0000","last_fullsized":"2023-10-10T17:41:37.595330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:20:38.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980304+0000","last_change":"2023-10-10T17:40:41.595043+0000","last_active":"2023-10-10T17:41:37.980304+0000","last_peered":"2023-10-10T17:41:37.980304+0000","last_clean":"2023-10-10T17:41:37.980304+0000","last_became_active":"2023-10-10T17:40:41.594703+0000","last_became_peered":"2023-10-10T17:40:41.594703+0000","last_unstale":"2023-10-10T17:41:37.980304+0000","last_undegraded":"2023-10-10T17:41:37.980304+0000","last_fullsized":"2023-10-10T17:41:37.980304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:29:21.277302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596130+0000","last_change":"2023-10-10T17:40:45.705232+0000","last_active":"2023-10-10T17:41:37.596130+0000","last_peered":"2023-10-10T17:41:37.596130+0000","last_clean":"2023-10-10T17:41:37.596130+0000","last_became_active":"2023-10-10T17:40:45.704811+0000","last_became_peered":"2023-10-10T17:40:45.704811+0000","last_unstale":"2023-10-10T17:41:37.596130+0000","last_undegraded":"2023-10-10T17:41:37.596130+0000","last_fullsized":"2023-10-10T17:41:37.596130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:50.034871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602441+0000","last_change":"2023-10-10T17:40:41.165620+0000","last_active":"2023-10-10T17:41:37.602441+0000","last_peered":"2023-10-10T17:41:37.602441+0000","last_clean":"2023-10-10T17:41:37.602441+0000","last_became_active":"2023-10-10T17:40:41.165262+0000","last_became_peered":"2023-10-10T17:40:41.165262+0000","last_unstale":"2023-10-10T17:41:37.602441+0000","last_undegraded":"2023-10-10T17:41:37.602441+0000","last_fullsized":"2023-10-10T17:41:37.602441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:19.126274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599185+0000","last_change":"2023-10-10T17:40:41.603549+0000","last_active":"2023-10-10T17:41:37.599185+0000","last_peered":"2023-10-10T17:41:37.599185+0000","last_clean":"2023-10-10T17:41:37.599185+0000","last_became_active":"2023-10-10T17:40:41.602991+0000","last_became_peered":"2023-10-10T17:40:41.602991+0000","last_unstale":"2023-10-10T17:41:37.599185+0000","last_undegraded":"2023-10-10T17:41:37.599185+0000","last_fullsized":"2023-10-10T17:41:37.599185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:30.283350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.606557+0000","last_change":"2023-10-10T17:41:37.606557+0000","last_active":"2023-10-10T17:41:37.606557+0000","last_peered":"2023-10-10T17:41:37.606557+0000","last_clean":"2023-10-10T17:41:37.606557+0000","last_became_active":"2023-10-10T17:41:37.606070+0000","last_became_peered":"2023-10-10T17:41:37.606070+0000","last_unstale":"2023-10-10T17:41:37.606557+0000","last_undegraded":"2023-10-10T17:41:37.606557+0000","last_fullsized":"2023-10-10T17:41:37.606557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:09:50.492126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595549+0000","last_change":"2023-10-10T17:40:41.169859+0000","last_active":"2023-10-10T17:41:37.595549+0000","last_peered":"2023-10-10T17:41:37.595549+0000","last_clean":"2023-10-10T17:41:37.595549+0000","last_became_active":"2023-10-10T17:40:41.169507+0000","last_became_peered":"2023-10-10T17:40:41.169507+0000","last_unstale":"2023-10-10T17:41:37.595549+0000","last_undegraded":"2023-10-10T17:41:37.595549+0000","last_fullsized":"2023-10-10T17:41:37.595549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:29:19.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600382+0000","last_change":"2023-10-10T17:40:41.593929+0000","last_active":"2023-10-10T17:41:37.600382+0000","last_peered":"2023-10-10T17:41:37.600382+0000","last_clean":"2023-10-10T17:41:37.600382+0000","last_became_active":"2023-10-10T17:40:41.593565+0000","last_became_peered":"2023-10-10T17:40:41.593565+0000","last_unstale":"2023-10-10T17:41:37.600382+0000","last_undegraded":"2023-10-10T17:41:37.600382+0000","last_fullsized":"2023-10-10T17:41:37.600382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:52.950921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595626+0000","last_change":"2023-10-10T17:40:45.710759+0000","last_active":"2023-10-10T17:41:37.595626+0000","last_peered":"2023-10-10T17:41:37.595626+0000","last_clean":"2023-10-10T17:41:37.595626+0000","last_became_active":"2023-10-10T17:40:45.710118+0000","last_became_peered":"2023-10-10T17:40:45.710118+0000","last_unstale":"2023-10-10T17:41:37.595626+0000","last_undegraded":"2023-10-10T17:41:37.595626+0000","last_fullsized":"2023-10-10T17:41:37.595626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:03:22.069829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979760+0000","last_change":"2023-10-10T17:40:41.165988+0000","last_active":"2023-10-10T17:41:37.979760+0000","last_peered":"2023-10-10T17:41:37.979760+0000","last_clean":"2023-10-10T17:41:37.979760+0000","last_became_active":"2023-10-10T17:40:41.165407+0000","last_became_peered":"2023-10-10T17:40:41.165407+0000","last_unstale":"2023-10-10T17:41:37.979760+0000","last_undegraded":"2023-10-10T17:41:37.979760+0000","last_fullsized":"2023-10-10T17:41:37.979760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:16:55.568098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787946+0000","last_change":"2023-10-10T17:40:41.609834+0000","last_active":"2023-10-10T17:41:37.787946+0000","last_peered":"2023-10-10T17:41:37.787946+0000","last_clean":"2023-10-10T17:41:37.787946+0000","last_became_active":"2023-10-10T17:40:41.609509+0000","last_became_peered":"2023-10-10T17:40:41.609509+0000","last_unstale":"2023-10-10T17:41:37.787946+0000","last_undegraded":"2023-10-10T17:41:37.787946+0000","last_fullsized":"2023-10-10T17:41:37.787946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:26:49.739028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602247+0000","last_change":"2023-10-10T17:40:45.709003+0000","last_active":"2023-10-10T17:41:37.602247+0000","last_peered":"2023-10-10T17:41:37.602247+0000","last_clean":"2023-10-10T17:41:37.602247+0000","last_became_active":"2023-10-10T17:40:45.708610+0000","last_became_peered":"2023-10-10T17:40:45.708610+0000","last_unstale":"2023-10-10T17:41:37.602247+0000","last_undegraded":"2023-10-10T17:41:37.602247+0000","last_fullsized":"2023-10-10T17:41:37.602247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:04:55.725864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596673+0000","last_change":"2023-10-10T17:40:41.168104+0000","last_active":"2023-10-10T17:41:37.596673+0000","last_peered":"2023-10-10T17:41:37.596673+0000","last_clean":"2023-10-10T17:41:37.596673+0000","last_became_active":"2023-10-10T17:40:41.167802+0000","last_became_peered":"2023-10-10T17:40:41.167802+0000","last_unstale":"2023-10-10T17:41:37.596673+0000","last_undegraded":"2023-10-10T17:41:37.596673+0000","last_fullsized":"2023-10-10T17:41:37.596673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:22:40.050459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979300+0000","last_change":"2023-10-10T17:40:41.606553+0000","last_active":"2023-10-10T17:41:37.979300+0000","last_peered":"2023-10-10T17:41:37.979300+0000","last_clean":"2023-10-10T17:41:37.979300+0000","last_became_active":"2023-10-10T17:40:41.606087+0000","last_became_peered":"2023-10-10T17:40:41.606087+0000","last_unstale":"2023-10-10T17:41:37.979300+0000","last_undegraded":"2023-10-10T17:41:37.979300+0000","last_fullsized":"2023-10-10T17:41:37.979300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:44:40.056635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602105+0000","last_change":"2023-10-10T17:40:45.712028+0000","last_active":"2023-10-10T17:41:37.602105+0000","last_peered":"2023-10-10T17:41:37.602105+0000","last_clean":"2023-10-10T17:41:37.602105+0000","last_became_active":"2023-10-10T17:40:45.711534+0000","last_became_peered":"2023-10-10T17:40:45.711534+0000","last_unstale":"2023-10-10T17:41:37.602105+0000","last_undegraded":"2023-10-10T17:41:37.602105+0000","last_fullsized":"2023-10-10T17:41:37.602105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:51:36.426627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600532+0000","last_change":"2023-10-10T17:40:41.164456+0000","last_active":"2023-10-10T17:41:37.600532+0000","last_peered":"2023-10-10T17:41:37.600532+0000","last_clean":"2023-10-10T17:41:37.600532+0000","last_became_active":"2023-10-10T17:40:41.164018+0000","last_became_peered":"2023-10-10T17:40:41.164018+0000","last_unstale":"2023-10-10T17:41:37.600532+0000","last_undegraded":"2023-10-10T17:41:37.600532+0000","last_fullsized":"2023-10-10T17:41:37.600532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:54:18.517460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601647+0000","last_change":"2023-10-10T17:40:41.599724+0000","last_active":"2023-10-10T17:41:37.601647+0000","last_peered":"2023-10-10T17:41:37.601647+0000","last_clean":"2023-10-10T17:41:37.601647+0000","last_became_active":"2023-10-10T17:40:41.599356+0000","last_became_peered":"2023-10-10T17:40:41.599356+0000","last_unstale":"2023-10-10T17:41:37.601647+0000","last_undegraded":"2023-10-10T17:41:37.601647+0000","last_fullsized":"2023-10-10T17:41:37.601647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:57.775792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596111+0000","last_change":"2023-10-10T17:40:45.704911+0000","last_active":"2023-10-10T17:41:37.596111+0000","last_peered":"2023-10-10T17:41:37.596111+0000","last_clean":"2023-10-10T17:41:37.596111+0000","last_became_active":"2023-10-10T17:40:45.704299+0000","last_became_peered":"2023-10-10T17:40:45.704299+0000","last_unstale":"2023-10-10T17:41:37.596111+0000","last_undegraded":"2023-10-10T17:41:37.596111+0000","last_fullsized":"2023-10-10T17:41:37.596111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:13:42.759241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597150+0000","last_change":"2023-10-10T17:40:41.166091+0000","last_active":"2023-10-10T17:41:37.597150+0000","last_peered":"2023-10-10T17:41:37.597150+0000","last_clean":"2023-10-10T17:41:37.597150+0000","last_became_active":"2023-10-10T17:40:41.165610+0000","last_became_peered":"2023-10-10T17:40:41.165610+0000","last_unstale":"2023-10-10T17:41:37.597150+0000","last_undegraded":"2023-10-10T17:41:37.597150+0000","last_fullsized":"2023-10-10T17:41:37.597150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:31:09.299708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602252+0000","last_change":"2023-10-10T17:40:41.600837+0000","last_active":"2023-10-10T17:41:37.602252+0000","last_peered":"2023-10-10T17:41:37.602252+0000","last_clean":"2023-10-10T17:41:37.602252+0000","last_became_active":"2023-10-10T17:40:41.600090+0000","last_became_peered":"2023-10-10T17:40:41.600090+0000","last_unstale":"2023-10-10T17:41:37.602252+0000","last_undegraded":"2023-10-10T17:41:37.602252+0000","last_fullsized":"2023-10-10T17:41:37.602252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:15:21.942555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603744+0000","last_change":"2023-10-10T17:41:37.603744+0000","last_active":"2023-10-10T17:41:37.603744+0000","last_peered":"2023-10-10T17:41:37.603744+0000","last_clean":"2023-10-10T17:41:37.603744+0000","last_became_active":"2023-10-10T17:41:37.603376+0000","last_became_peered":"2023-10-10T17:41:37.603376+0000","last_unstale":"2023-10-10T17:41:37.603744+0000","last_undegraded":"2023-10-10T17:41:37.603744+0000","last_fullsized":"2023-10-10T17:41:37.603744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:59:51.305467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652254+0000","last_change":"2023-10-10T17:40:41.169820+0000","last_active":"2023-10-10T17:41:37.652254+0000","last_peered":"2023-10-10T17:41:37.652254+0000","last_clean":"2023-10-10T17:41:37.652254+0000","last_became_active":"2023-10-10T17:40:41.168767+0000","last_became_peered":"2023-10-10T17:40:41.168767+0000","last_unstale":"2023-10-10T17:41:37.652254+0000","last_undegraded":"2023-10-10T17:41:37.652254+0000","last_fullsized":"2023-10-10T17:41:37.652254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:22:55.616442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603091+0000","last_change":"2023-10-10T17:40:41.597482+0000","last_active":"2023-10-10T17:41:37.603091+0000","last_peered":"2023-10-10T17:41:37.603091+0000","last_clean":"2023-10-10T17:41:37.603091+0000","last_became_active":"2023-10-10T17:40:41.597166+0000","last_became_peered":"2023-10-10T17:40:41.597166+0000","last_unstale":"2023-10-10T17:41:37.603091+0000","last_undegraded":"2023-10-10T17:41:37.603091+0000","last_fullsized":"2023-10-10T17:41:37.603091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:11.444261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651261+0000","last_change":"2023-10-10T17:40:45.707175+0000","last_active":"2023-10-10T17:41:37.651261+0000","last_peered":"2023-10-10T17:41:37.651261+0000","last_clean":"2023-10-10T17:41:37.651261+0000","last_became_active":"2023-10-10T17:40:45.706448+0000","last_became_peered":"2023-10-10T17:40:45.706448+0000","last_unstale":"2023-10-10T17:41:37.651261+0000","last_undegraded":"2023-10-10T17:41:37.651261+0000","last_fullsized":"2023-10-10T17:41:37.651261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:33.970113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600136+0000","last_change":"2023-10-10T17:40:41.169806+0000","last_active":"2023-10-10T17:41:37.600136+0000","last_peered":"2023-10-10T17:41:37.600136+0000","last_clean":"2023-10-10T17:41:37.600136+0000","last_became_active":"2023-10-10T17:40:41.169496+0000","last_became_peered":"2023-10-10T17:40:41.169496+0000","last_unstale":"2023-10-10T17:41:37.600136+0000","last_undegraded":"2023-10-10T17:41:37.600136+0000","last_fullsized":"2023-10-10T17:41:37.600136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:28:16.024298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603755+0000","last_change":"2023-10-10T17:40:41.598366+0000","last_active":"2023-10-10T17:41:37.603755+0000","last_peered":"2023-10-10T17:41:37.603755+0000","last_clean":"2023-10-10T17:41:37.603755+0000","last_became_active":"2023-10-10T17:40:41.598036+0000","last_became_peered":"2023-10-10T17:40:41.598036+0000","last_unstale":"2023-10-10T17:41:37.603755+0000","last_undegraded":"2023-10-10T17:41:37.603755+0000","last_fullsized":"2023-10-10T17:41:37.603755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:59:22.632882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787912+0000","last_change":"2023-10-10T17:40:41.609923+0000","last_active":"2023-10-10T17:41:37.787912+0000","last_peered":"2023-10-10T17:41:37.787912+0000","last_clean":"2023-10-10T17:41:37.787912+0000","last_became_active":"2023-10-10T17:40:41.609575+0000","last_became_peered":"2023-10-10T17:40:41.609575+0000","last_unstale":"2023-10-10T17:41:37.787912+0000","last_undegraded":"2023-10-10T17:41:37.787912+0000","last_fullsized":"2023-10-10T17:41:37.787912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:14:10.560266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596871+0000","last_change":"2023-10-10T17:40:41.609012+0000","last_active":"2023-10-10T17:41:37.596871+0000","last_peered":"2023-10-10T17:41:37.596871+0000","last_clean":"2023-10-10T17:41:37.596871+0000","last_became_active":"2023-10-10T17:40:41.608738+0000","last_became_peered":"2023-10-10T17:40:41.608738+0000","last_unstale":"2023-10-10T17:41:37.596871+0000","last_undegraded":"2023-10-10T17:41:37.596871+0000","last_fullsized":"2023-10-10T17:41:37.596871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:11:39.107240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596125+0000","last_change":"2023-10-10T17:40:41.608801+0000","last_active":"2023-10-10T17:41:37.596125+0000","last_peered":"2023-10-10T17:41:37.596125+0000","last_clean":"2023-10-10T17:41:37.596125+0000","last_became_active":"2023-10-10T17:40:41.608510+0000","last_became_peered":"2023-10-10T17:40:41.608510+0000","last_unstale":"2023-10-10T17:41:37.596125+0000","last_undegraded":"2023-10-10T17:41:37.596125+0000","last_fullsized":"2023-10-10T17:41:37.596125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:16:53.945359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598418+0000","last_change":"2023-10-10T17:40:41.603516+0000","last_active":"2023-10-10T17:41:37.598418+0000","last_peered":"2023-10-10T17:41:37.598418+0000","last_clean":"2023-10-10T17:41:37.598418+0000","last_became_active":"2023-10-10T17:40:41.602948+0000","last_became_peered":"2023-10-10T17:40:41.602948+0000","last_unstale":"2023-10-10T17:41:37.598418+0000","last_undegraded":"2023-10-10T17:41:37.598418+0000","last_fullsized":"2023-10-10T17:41:37.598418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:04.694328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602112+0000","last_change":"2023-10-10T17:41:37.602112+0000","last_active":"2023-10-10T17:41:37.602112+0000","last_peered":"2023-10-10T17:41:37.602112+0000","last_clean":"2023-10-10T17:41:37.602112+0000","last_became_active":"2023-10-10T17:41:37.601781+0000","last_became_peered":"2023-10-10T17:41:37.601781+0000","last_unstale":"2023-10-10T17:41:37.602112+0000","last_undegraded":"2023-10-10T17:41:37.602112+0000","last_fullsized":"2023-10-10T17:41:37.602112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:00.565768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.597595+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.597301+0000","last_became_peered":"2023-10-10T17:40:41.597301+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:51:11.601251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596929+0000","last_change":"2023-10-10T17:40:41.611841+0000","last_active":"2023-10-10T17:41:37.596929+0000","last_peered":"2023-10-10T17:41:37.596929+0000","last_clean":"2023-10-10T17:41:37.596929+0000","last_became_active":"2023-10-10T17:40:41.611438+0000","last_became_peered":"2023-10-10T17:40:41.611438+0000","last_unstale":"2023-10-10T17:41:37.596929+0000","last_undegraded":"2023-10-10T17:41:37.596929+0000","last_fullsized":"2023-10-10T17:41:37.596929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:29:05.260738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979112+0000","last_change":"2023-10-10T17:40:41.607707+0000","last_active":"2023-10-10T17:41:37.979112+0000","last_peered":"2023-10-10T17:41:37.979112+0000","last_clean":"2023-10-10T17:41:37.979112+0000","last_became_active":"2023-10-10T17:40:41.607412+0000","last_became_peered":"2023-10-10T17:40:41.607412+0000","last_unstale":"2023-10-10T17:41:37.979112+0000","last_undegraded":"2023-10-10T17:41:37.979112+0000","last_fullsized":"2023-10-10T17:41:37.979112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:09:49.015272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651033+0000","last_change":"2023-10-10T17:40:41.593674+0000","last_active":"2023-10-10T17:41:37.651033+0000","last_peered":"2023-10-10T17:41:37.651033+0000","last_clean":"2023-10-10T17:41:37.651033+0000","last_became_active":"2023-10-10T17:40:41.593339+0000","last_became_peered":"2023-10-10T17:40:41.593339+0000","last_unstale":"2023-10-10T17:41:37.651033+0000","last_undegraded":"2023-10-10T17:41:37.651033+0000","last_fullsized":"2023-10-10T17:41:37.651033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:39:37.705884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978977+0000","last_change":"2023-10-10T17:40:41.610075+0000","last_active":"2023-10-10T17:41:37.978977+0000","last_peered":"2023-10-10T17:41:37.978977+0000","last_clean":"2023-10-10T17:41:37.978977+0000","last_became_active":"2023-10-10T17:40:41.609669+0000","last_became_peered":"2023-10-10T17:40:41.609669+0000","last_unstale":"2023-10-10T17:41:37.978977+0000","last_undegraded":"2023-10-10T17:41:37.978977+0000","last_fullsized":"2023-10-10T17:41:37.978977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:42.281285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596551+0000","last_change":"2023-10-10T17:40:41.596780+0000","last_active":"2023-10-10T17:41:37.596551+0000","last_peered":"2023-10-10T17:41:37.596551+0000","last_clean":"2023-10-10T17:41:37.596551+0000","last_became_active":"2023-10-10T17:40:41.596477+0000","last_became_peered":"2023-10-10T17:40:41.596477+0000","last_unstale":"2023-10-10T17:41:37.596551+0000","last_undegraded":"2023-10-10T17:41:37.596551+0000","last_fullsized":"2023-10-10T17:41:37.596551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:51:14.200732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598899+0000","last_change":"2023-10-10T17:40:41.606226+0000","last_active":"2023-10-10T17:41:37.598899+0000","last_peered":"2023-10-10T17:41:37.598899+0000","last_clean":"2023-10-10T17:41:37.598899+0000","last_became_active":"2023-10-10T17:40:41.605843+0000","last_became_peered":"2023-10-10T17:40:41.605843+0000","last_unstale":"2023-10-10T17:41:37.598899+0000","last_undegraded":"2023-10-10T17:41:37.598899+0000","last_fullsized":"2023-10-10T17:41:37.598899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:44.165622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788525+0000","last_change":"2023-10-10T17:40:41.596122+0000","last_active":"2023-10-10T17:41:37.788525+0000","last_peered":"2023-10-10T17:41:37.788525+0000","last_clean":"2023-10-10T17:41:37.788525+0000","last_became_active":"2023-10-10T17:40:41.595818+0000","last_became_peered":"2023-10-10T17:40:41.595818+0000","last_unstale":"2023-10-10T17:41:37.788525+0000","last_undegraded":"2023-10-10T17:41:37.788525+0000","last_fullsized":"2023-10-10T17:41:37.788525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:24:32.658064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250871,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7568,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":512,"num_read_kb":270,"num_write":131,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":22704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":987136,"data_stored":492386,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995}]},{"osd":1,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.75800000000000001,"15min":0.77700000000000002},"min":{"1min":0.46899999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90200000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.68600000000000005,"15min":0.69599999999999995},"min":{"1min":0.45700000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.49299999999999999,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66000000000000003,"5min":1.026,"15min":1.026},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.80600000000000005,"15min":0.82599999999999996},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.95799999999999996,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97499999999999998,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.80300000000000005}]},{"osd":5,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.628,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81599999999999995,"5min":1.129,"15min":1.129},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66800000000000004},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85999999999999999,"5min":1.107,"15min":1.107},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80900000000000005,"15min":0.81999999999999995},"min":{"1min":0.51300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.01,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.81200000000000006,"15min":0.82799999999999996},"min":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0309999999999999,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.85299999999999998}]},{"osd":8,"last update":"Tue Oct 10 17:41:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.75800000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.72699999999999998,"15min":0.72899999999999998},"min":{"1min":0.47599999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.93600000000000005,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.752,"15min":0.753},"min":{"1min":0.52100000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.05,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.83799999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.97599999999999998,"15min":1.01},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.117,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84599999999999997,"15min":0.85899999999999999},"min":{"1min":0.61399999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0720000000000001,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.82699999999999996}]}]},{"osd":10,"up_from":73,"seq":313532612643,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6432,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738720,"statfs":{"total":95995035648,"available":95988449280,"internally_reserved":0,"allocated":1073152,"data_stored":1087425,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.435}]},{"osd":1,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.32700000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.47399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78200000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.51500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.80200000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68600000000000005}]},{"osd":3,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76700000000000002,"5min":1.071,"15min":1.071},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.501}]},{"osd":4,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.63}]},{"osd":5,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.65400000000000003},"min":{"1min":0.52900000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.81599999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73499999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.56699999999999995,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90100000000000002,"5min":1.05,"15min":1.05},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.70899999999999996}]},{"osd":8,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70399999999999996,"15min":0.69999999999999996},"min":{"1min":0.60299999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.85499999999999998,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.753},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91900000000000004,"5min":1.157,"15min":1.157},"last":0.76300000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":1.2,"15min":1.2},"last":0.80800000000000005}]},{"osd":11,"last update":"Tue Oct 10 17:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76300000000000001,"15min":0.76100000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77100000000000002,"15min":0.77000000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.94299999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.78300000000000003}]}]},{"osd":9,"up_from":68,"seq":292057776169,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":1003520,"data_stored":492622,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.373},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48199999999999998,"15min":0.502},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54000000000000004,"5min":1.016,"15min":1.016},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85799999999999998,"5min":1.103,"15min":1.103},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.755,"5min":1.03,"15min":1.03},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59899999999999998,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69199999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70199999999999996,"5min":1.151,"15min":1.151},"last":0.5}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.54000000000000004,"15min":0.56299999999999994},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.62,"5min":0.748,"15min":0.748},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.54600000000000004,"15min":0.56899999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64300000000000002,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.38700000000000001}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73799999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68999999999999995,"15min":0.70999999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.69599999999999995,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.436,"15min":0.436},"max":{"1min":0.77000000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69899999999999995,"15min":0.72099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.71299999999999997,"5min":1.417,"15min":1.417},"last":0.48099999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.52600000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.76700000000000002,"15min":0.79100000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.80900000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67000000000000004}]},{"osd":8,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.753,"15min":0.77500000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77600000000000002,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.78900000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79400000000000004,"5min":1.377,"15min":1.377},"last":0.67800000000000005}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69799999999999995,"15min":0.70999999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.83399999999999996,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71699999999999997,"15min":0.72299999999999998},"min":{"1min":0.52900000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.876,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64500000000000002}]},{"osd":11,"last update":"Tue Oct 10 17:41:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.65000000000000002}]}]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6492,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738660,"statfs":{"total":95995035648,"available":95988387840,"internally_reserved":0,"allocated":1003520,"data_stored":492542,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.436,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.64400000000000002,"15min":0.58399999999999996},"min":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.61899999999999999}]},{"osd":1,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58199999999999996,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84599999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.624,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86099999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.42799999999999999,"15min":0.40000000000000002},"min":{"1min":0.29399999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64100000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52200000000000002}]},{"osd":3,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.57999999999999996},"min":{"1min":0.439,"5min":0.37,"15min":0.37},"max":{"1min":0.94799999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.012,"5min":0.65500000000000003,"15min":0.58999999999999997},"min":{"1min":0.52100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.68799999999999994}]},{"osd":4,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.60999999999999999,"15min":0.58799999999999997},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.63100000000000001,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72999999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.33600000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.56499999999999995,"15min":0.53100000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93799999999999994,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.69899999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.93799999999999994,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.70899999999999996,"15min":0.69399999999999995},"min":{"1min":0.56299999999999994,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90400000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.77200000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.71599999999999997,"15min":0.68700000000000006},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.999,"5min":1.022,"15min":1.022},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.69899999999999995,"15min":0.67300000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91900000000000004,"5min":1.101,"15min":1.101},"last":0.74299999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79500000000000004,"15min":0.72499999999999998},"min":{"1min":0.58499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.012,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.76400000000000001,"15min":0.73999999999999999},"min":{"1min":0.58499999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.036,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.83499999999999996}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.751,"15min":0.73199999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.87},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.73699999999999999,"15min":0.71499999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.88600000000000001}]},{"osd":11,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.76000000000000001}]}]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1044480,"data_stored":1083855,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72099999999999997,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.374}]},{"osd":1,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40899999999999997,"15min":0.39300000000000002},"min":{"1min":0.28699999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72099999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.78000000000000003,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70099999999999996}]},{"osd":3,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.59799999999999998,"15min":0.58899999999999997},"min":{"1min":0.48399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.755,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46200000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.312,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.495},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48099999999999998,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.432}]},{"osd":5,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.442,"5min":0.441,"15min":0.441},"max":{"1min":0.85499999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64000000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.374}]},{"osd":8,"last update":"Tue Oct 10 17:42:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.70699999999999996,"15min":0.69099999999999995},"min":{"1min":0.623,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68200000000000005,"15min":0.67200000000000004},"min":{"1min":0.58799999999999997,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.625,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.754,"15min":0.74399999999999999},"min":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.77300000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.751,"15min":0.745},"min":{"1min":0.55600000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92300000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48199999999999998}]},{"osd":11,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66800000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":987136,"data_stored":491894,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.375,"15min":0.375},"min":{"1min":0.27700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.41299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.38500000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83099999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54100000000000004}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.442,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.33600000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.441},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57399999999999995,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.50800000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.57799999999999996},"min":{"1min":0.46400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89500000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.627}]},{"osd":8,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65800000000000003,"15min":0.66400000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.60399999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76500000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.42899999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:41:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.73199999999999998,"15min":0.74299999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.501},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.73899999999999999,"15min":0.753},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.875,"5min":1.042,"15min":1.042},"last":0.59099999999999997}]},{"osd":11,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":995328,"data_stored":495598,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57999999999999996,"15min":0.67700000000000005},"min":{"1min":0.39900000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80400000000000005,"5min":5.859,"15min":5.859},"last":0.44400000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.58099999999999996,"15min":0.51900000000000002},"min":{"1min":0.46200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.88300000000000001,"5min":1,"15min":1},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.623,"15min":0.60899999999999999},"min":{"1min":0.47299999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86899999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.84499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.45500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.72699999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80500000000000005,"5min":0.875,"15min":0.875},"last":0.627}]},{"osd":4,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.45800000000000002,"15min":0.40400000000000003},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.46999999999999997,"15min":0.41499999999999998},"min":{"1min":0.38700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.501}]},{"osd":5,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59499999999999997,"15min":0.55700000000000005},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.60799999999999998,"15min":0.55600000000000005},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.70199999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:42:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59299999999999997},"min":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63900000000000001,"15min":0.61299999999999999},"min":{"1min":0.45200000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88800000000000001,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.59999999999999998,"15min":0.55700000000000005},"min":{"1min":0.42199999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.495,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.76700000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67300000000000004,"15min":0.65100000000000002},"min":{"1min":0.54500000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.67700000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.99099999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.85799999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.76300000000000001,"15min":0.749},"min":{"1min":0.60999999999999999,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.91900000000000004,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73399999999999999,"15min":0.71399999999999997},"min":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.01,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.871}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75800000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.77500000000000002,"15min":0.76300000000000001},"min":{"1min":0.57599999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.008,"15min":1.008},"last":0.754}]},{"osd":11,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.89000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7396,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737756,"statfs":{"total":95995035648,"available":95987462144,"internally_reserved":0,"allocated":1011712,"data_stored":493282,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 17:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54500000000000004,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.49199999999999999}]},{"osd":3,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.274,"5min":7.274,"15min":7.274},"last":0.57799999999999996}]},{"osd":4,"last update":"Tue Oct 10 17:41:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55600000000000005,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54500000000000004,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.53100000000000003},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.75800000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63100000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":0.65200000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.622}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68000000000000005}]},{"osd":10,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.75},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.72799999999999998},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.59799999999999998}]},{"osd":11,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67200000000000004,"15min":0.66700000000000004},"min":{"1min":0.53000000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.70599999999999996,"15min":0.70199999999999996},"min":{"1min":0.51800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.86599999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7456,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737696,"statfs":{"total":95995035648,"available":95987400704,"internally_reserved":0,"allocated":1007616,"data_stored":496169,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":0.88700000000000001,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.49199999999999999,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68300000000000005,"5min":1.177,"15min":1.177},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60099999999999998,"15min":0.53100000000000003},"min":{"1min":0.50900000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81399999999999995,"5min":5.484,"15min":5.484},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.56699999999999995,"15min":0.53400000000000003},"min":{"1min":0.46500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56999999999999995,"15min":0.52500000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.58599999999999997,"15min":0.54400000000000004},"min":{"1min":0.51300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84799999999999998,"5min":1.099,"15min":1.099},"last":0.56699999999999995}]},{"osd":4,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.64300000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.48599999999999999,"15min":0.42299999999999999},"min":{"1min":0.39900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.435},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.52000000000000002,"15min":0.46800000000000003},"min":{"1min":0.38300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.41499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.97899999999999998,"5min":1.129,"15min":1.129},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92300000000000004,"5min":1.075,"15min":1.075},"last":0.504}]},{"osd":7,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.755},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.84499999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.627}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93600000000000005,"5min":1.087,"15min":1.087},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.996,"5min":1.226,"15min":1.226},"last":0.79100000000000004}]},{"osd":10,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.66100000000000003,"15min":0.64400000000000002},"min":{"1min":0.56599999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.70599999999999996,"15min":0.68300000000000005},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.83099999999999996}]},{"osd":11,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.879},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.80900000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6740,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738412,"statfs":{"total":95995035648,"available":95988133888,"internally_reserved":0,"allocated":995328,"data_stored":491757,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35499999999999998,"15min":0.34799999999999998},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.621,"5min":0.752,"15min":0.752},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56499999999999995,"15min":0.53900000000000003},"min":{"1min":0.39200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55800000000000005,"15min":0.52500000000000002},"min":{"1min":0.40300000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.60799999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56599999999999995,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73099999999999998,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57099999999999995,"15min":0.54100000000000004},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80000000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Tue Oct 10 17:41:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.41099999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84099999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.627},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61299999999999999,"15min":0.59299999999999997},"min":{"1min":0.46000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.746,"5min":1.214,"15min":1.214},"last":0.753}]},{"osd":5,"last update":"Tue Oct 10 17:42:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58699999999999997,"15min":0.59799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78900000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76100000000000001,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.67600000000000005}]},{"osd":6,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.5,"15min":0.50800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.621,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.80200000000000005}]},{"osd":8,"last update":"Tue Oct 10 17:39:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53700000000000003}]},{"osd":9,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.624,"15min":0.65400000000000003},"min":{"1min":0.377,"5min":0.317,"15min":0.317},"max":{"1min":0.69899999999999995,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64400000000000002,"15min":0.68200000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68999999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54200000000000004}]},{"osd":10,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.76300000000000001}]},{"osd":11,"last update":"Tue Oct 10 17:41:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.70799999999999996}]}]},{"osd":4,"up_from":34,"seq":146028888127,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1003520,"data_stored":492772,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.432,"5min":0.372,"15min":0.372},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.36599999999999999},"min":{"1min":0.31,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65200000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.40799999999999997,"15min":0.372},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.63400000000000001,"15min":0.57499999999999996},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85099999999999998}]},{"osd":3,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.60899999999999999,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.1289999999999996,"5min":6.1289999999999996,"15min":6.1289999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.75900000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.626,"15min":0.58599999999999997},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":1.589,"15min":1.589},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.56100000000000005,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.69499999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.499,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95399999999999996,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.57299999999999995,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.61799999999999999,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.76900000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.46899999999999997,"5min":0.432,"15min":0.432},"max":{"1min":0.93400000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76100000000000001,"15min":0.71799999999999997},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.006,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.83899999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.73199999999999998,"15min":0.71499999999999997},"min":{"1min":0.627,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.95899999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.78100000000000003,"15min":0.76600000000000001},"min":{"1min":0.61699999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.98999999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.93799999999999994}]},{"osd":10,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68200000000000005,"15min":0.67400000000000004},"min":{"1min":0.52300000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.80400000000000005,"15min":0.79600000000000004},"min":{"1min":0.56100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.038,"5min":1.169,"15min":1.169},"last":0.89900000000000002}]},{"osd":11,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.88600000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6736,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738416,"statfs":{"total":95995035648,"available":95988137984,"internally_reserved":0,"allocated":1056768,"data_stored":1083327,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.57599999999999996,"15min":0.52200000000000002},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.79600000000000004,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69999999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68700000000000006,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.55400000000000005},"min":{"1min":0.497,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71599999999999997,"5min":7.0570000000000004,"15min":7.0570000000000004},"last":0.61699999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.40899999999999997,"15min":0.40100000000000002},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46100000000000002,"15min":0.42499999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56899999999999995,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.438},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.56999999999999995,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83799999999999997,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65100000000000002,"15min":0.59199999999999997},"min":{"1min":0.438,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76300000000000001,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73499999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.70499999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.432,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.504,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.624},"min":{"1min":0.54300000000000004,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66000000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.55600000000000005},"min":{"1min":0.47399999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":6.7229999999999999,"15min":6.7229999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81899999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64700000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70899999999999996,"15min":0.71299999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90300000000000002,"5min":1.034,"15min":1.034},"last":0.751},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.71299999999999997,"15min":0.69899999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":0.76500000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.56200000000000006,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88100000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70799999999999996,"15min":0.70299999999999996},"min":{"1min":0.65200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T17:42:24.945 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T17:42:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:24 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2412951531' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:24 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/606056069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:24.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:24 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2270935002' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:25.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 -- 172.21.15.78:0/3863069691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0068b10 msgr2=0x7f48b0102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3863069691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0068b10 0x7f48b0102e00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f48a4009990 tx=0x7f48a4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 -- 172.21.15.78:0/3863069691 shutdown_connections 2023-10-10T17:42:25.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3863069691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48b0105e50 0x7f48b0108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3863069691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0068b10 0x7f48b0102e00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3863069691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48b0103340 0x7f48b01057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 -- 172.21.15.78:0/3863069691 >> 172.21.15.78:0/3863069691 conn(0x7f48b00fa750 msgr2=0x7f48b00fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:25.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 -- 172.21.15.78:0/3863069691 shutdown_connections 2023-10-10T17:42:25.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.095+0000 7f48b4d44700 1 -- 172.21.15.78:0/3863069691 wait complete. 2023-10-10T17:42:25.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 Processor -- start 2023-10-10T17:42:25.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 -- start start 2023-10-10T17:42:25.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48b0068b10 0x7f48b0195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0103340 0x7f48b0196440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48b0105e50 0x7f48b019d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019da80 con 0x7f48b0103340 2023-10-10T17:42:25.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019dbc0 con 0x7f48b0105e50 2023-10-10T17:42:25.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48b4d44700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48b019dd50 con 0x7f48b0068b10 2023-10-10T17:42:25.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0103340 0x7f48b0196440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0103340 0x7f48b0196440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43986/0 (socket says 172.21.15.78:43986) 2023-10-10T17:42:25.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 -- 172.21.15.78:0/3141035406 learned_addr learned my addr 172.21.15.78:0/3141035406 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:25.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48ae59c700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48b0068b10 0x7f48b0195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48aed9d700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48b0105e50 0x7f48b019d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 -- 172.21.15.78:0/3141035406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48b0068b10 msgr2=0x7f48b0195f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48b0068b10 0x7f48b0195f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 -- 172.21.15.78:0/3141035406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48b0105e50 msgr2=0x7f48b019d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48b0105e50 0x7f48b019d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 -- 172.21.15.78:0/3141035406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a40095f0 con 0x7f48b0103340 2023-10-10T17:42:25.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48aed9d700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48b0105e50 0x7f48b019d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:25.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.096+0000 7f48add9b700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0103340 0x7f48b0196440 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c9b0 tx=0x7f48a000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:25.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.097+0000 7f489f7fe700 1 -- 172.21.15.78:0/3141035406 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0007b00 con 0x7f48b0103340 2023-10-10T17:42:25.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.097+0000 7f489f7fe700 1 -- 172.21.15.78:0/3141035406 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48a0007c60 con 0x7f48b0103340 2023-10-10T17:42:25.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.097+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b010bbd0 con 0x7f48b0103340 2023-10-10T17:42:25.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.097+0000 7f489f7fe700 1 -- 172.21.15.78:0/3141035406 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a0018660 con 0x7f48b0103340 2023-10-10T17:42:25.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.097+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b010ba40 con 0x7f48b0103340 2023-10-10T17:42:25.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.098+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b00fc2c0 con 0x7f48b0103340 2023-10-10T17:42:25.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.099+0000 7f489f7fe700 1 -- 172.21.15.78:0/3141035406 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f48a0018880 con 0x7f48b0103340 2023-10-10T17:42:25.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.099+0000 7f489f7fe700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f48980a1150 0x7f48980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.099+0000 7f489f7fe700 1 -- 172.21.15.78:0/3141035406 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48a00cccc0 con 0x7f48b0103340 2023-10-10T17:42:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.099+0000 7f48ae59c700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f48980a1150 0x7f48980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.100+0000 7f48ae59c700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f48980a1150 0x7f48980a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f48a4006010 tx=0x7f48a400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:25.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.103+0000 7f489f7fe700 1 -- 172.21.15.78:0/3141035406 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f48a0096100 con 0x7f48b0103340 2023-10-10T17:42:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.292+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f48b0061c60 con 0x7f48980a1150 2023-10-10T17:42:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.303+0000 7f489f7fe700 1 -- 172.21.15.78:0/3141035406 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427292 (secure 0 0 0) 0x7f48b0061c60 con 0x7f48980a1150 2023-10-10T17:42:25.311 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f48980a1150 msgr2=0x7f48980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f48980a1150 0x7f48980a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f48a4006010 tx=0x7f48a400b260 comp rx=0 tx=0).stop 2023-10-10T17:42:25.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0103340 msgr2=0x7f48b0196440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0103340 0x7f48b0196440 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f48a000c9b0 tx=0x7f48a000cd70 comp rx=0 tx=0).stop 2023-10-10T17:42:25.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 shutdown_connections 2023-10-10T17:42:25.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f48980a1150 0x7f48980a3600 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f48b0068b10 0x7f48b0195f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f48b0105e50 0x7f48b019d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 --2- 172.21.15.78:0/3141035406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48b0103340 0x7f48b0196440 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 >> 172.21.15.78:0/3141035406 conn(0x7f48b00fa750 msgr2=0x7f48b0101680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:25.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 shutdown_connections 2023-10-10T17:42:25.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.307+0000 7f48b4d44700 1 -- 172.21.15.78:0/3141035406 wait complete. 2023-10-10T17:42:25.321 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-10T17:42:25.323 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-10-10T17:42:24.243511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597936,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":648,"num_read_kb":389,"num_write":312,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80844,"kb_used_data":11884,"kb_used_omap":0,"kb_used_meta":68864,"kb_avail":1124860980,"statfs":{"total":1151940427776,"available":1151857643520,"internally_reserved":0,"allocated":12169216,"data_stored":7693629,"data_compressed":319365,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70516736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":260,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":46,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003374"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788069+0000","last_change":"2023-10-10T17:40:41.604268+0000","last_active":"2023-10-10T17:41:37.788069+0000","last_peered":"2023-10-10T17:41:37.788069+0000","last_clean":"2023-10-10T17:41:37.788069+0000","last_became_active":"2023-10-10T17:40:41.603839+0000","last_became_peered":"2023-10-10T17:40:41.603839+0000","last_unstale":"2023-10-10T17:41:37.788069+0000","last_undegraded":"2023-10-10T17:41:37.788069+0000","last_fullsized":"2023-10-10T17:41:37.788069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:00:22.805728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599055+0000","last_change":"2023-10-10T17:40:41.603642+0000","last_active":"2023-10-10T17:41:37.599055+0000","last_peered":"2023-10-10T17:41:37.599055+0000","last_clean":"2023-10-10T17:41:37.599055+0000","last_became_active":"2023-10-10T17:40:41.603091+0000","last_became_peered":"2023-10-10T17:40:41.603091+0000","last_unstale":"2023-10-10T17:41:37.599055+0000","last_undegraded":"2023-10-10T17:41:37.599055+0000","last_fullsized":"2023-10-10T17:41:37.599055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:10:38.807138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598740+0000","last_change":"2023-10-10T17:40:41.610647+0000","last_active":"2023-10-10T17:41:37.598740+0000","last_peered":"2023-10-10T17:41:37.598740+0000","last_clean":"2023-10-10T17:41:37.598740+0000","last_became_active":"2023-10-10T17:40:41.610338+0000","last_became_peered":"2023-10-10T17:40:41.610338+0000","last_unstale":"2023-10-10T17:41:37.598740+0000","last_undegraded":"2023-10-10T17:41:37.598740+0000","last_fullsized":"2023-10-10T17:41:37.598740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:16:20.110335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602832+0000","last_change":"2023-10-10T17:40:41.607832+0000","last_active":"2023-10-10T17:41:37.602832+0000","last_peered":"2023-10-10T17:41:37.602832+0000","last_clean":"2023-10-10T17:41:37.602832+0000","last_became_active":"2023-10-10T17:40:41.607492+0000","last_became_peered":"2023-10-10T17:40:41.607492+0000","last_unstale":"2023-10-10T17:41:37.602832+0000","last_undegraded":"2023-10-10T17:41:37.602832+0000","last_fullsized":"2023-10-10T17:41:37.602832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:43.514129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600669+0000","last_change":"2023-10-10T17:40:41.601975+0000","last_active":"2023-10-10T17:41:37.600669+0000","last_peered":"2023-10-10T17:41:37.600669+0000","last_clean":"2023-10-10T17:41:37.600669+0000","last_became_active":"2023-10-10T17:40:41.601668+0000","last_became_peered":"2023-10-10T17:40:41.601668+0000","last_unstale":"2023-10-10T17:41:37.600669+0000","last_undegraded":"2023-10-10T17:41:37.600669+0000","last_fullsized":"2023-10-10T17:41:37.600669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:25.433615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.604098+0000","last_change":"2023-10-10T17:41:37.604098+0000","last_active":"2023-10-10T17:41:37.604098+0000","last_peered":"2023-10-10T17:41:37.604098+0000","last_clean":"2023-10-10T17:41:37.604098+0000","last_became_active":"2023-10-10T17:41:37.603750+0000","last_became_peered":"2023-10-10T17:41:37.603750+0000","last_unstale":"2023-10-10T17:41:37.604098+0000","last_undegraded":"2023-10-10T17:41:37.604098+0000","last_fullsized":"2023-10-10T17:41:37.604098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:52:52.491557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602564+0000","last_change":"2023-10-10T17:40:41.594628+0000","last_active":"2023-10-10T17:41:37.602564+0000","last_peered":"2023-10-10T17:41:37.602564+0000","last_clean":"2023-10-10T17:41:37.602564+0000","last_became_active":"2023-10-10T17:40:41.594325+0000","last_became_peered":"2023-10-10T17:40:41.594325+0000","last_unstale":"2023-10-10T17:41:37.602564+0000","last_undegraded":"2023-10-10T17:41:37.602564+0000","last_fullsized":"2023-10-10T17:41:37.602564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:09:20.447064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978940+0000","last_change":"2023-10-10T17:40:41.598032+0000","last_active":"2023-10-10T17:41:37.978940+0000","last_peered":"2023-10-10T17:41:37.978940+0000","last_clean":"2023-10-10T17:41:37.978940+0000","last_became_active":"2023-10-10T17:40:41.597671+0000","last_became_peered":"2023-10-10T17:40:41.597671+0000","last_unstale":"2023-10-10T17:41:37.978940+0000","last_undegraded":"2023-10-10T17:41:37.978940+0000","last_fullsized":"2023-10-10T17:41:37.978940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:56:22.583298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598085+0000","last_change":"2023-10-10T17:40:41.599770+0000","last_active":"2023-10-10T17:41:37.598085+0000","last_peered":"2023-10-10T17:41:37.598085+0000","last_clean":"2023-10-10T17:41:37.598085+0000","last_became_active":"2023-10-10T17:40:41.599449+0000","last_became_peered":"2023-10-10T17:40:41.599449+0000","last_unstale":"2023-10-10T17:41:37.598085+0000","last_undegraded":"2023-10-10T17:41:37.598085+0000","last_fullsized":"2023-10-10T17:41:37.598085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:25:18.482096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598837+0000","last_change":"2023-10-10T17:40:41.610655+0000","last_active":"2023-10-10T17:41:37.598837+0000","last_peered":"2023-10-10T17:41:37.598837+0000","last_clean":"2023-10-10T17:41:37.598837+0000","last_became_active":"2023-10-10T17:40:41.610133+0000","last_became_peered":"2023-10-10T17:40:41.610133+0000","last_unstale":"2023-10-10T17:41:37.598837+0000","last_undegraded":"2023-10-10T17:41:37.598837+0000","last_fullsized":"2023-10-10T17:41:37.598837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:32:12.198400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602157+0000","last_change":"2023-10-10T17:40:41.601065+0000","last_active":"2023-10-10T17:41:37.602157+0000","last_peered":"2023-10-10T17:41:37.602157+0000","last_clean":"2023-10-10T17:41:37.602157+0000","last_became_active":"2023-10-10T17:40:41.600530+0000","last_became_peered":"2023-10-10T17:40:41.600530+0000","last_unstale":"2023-10-10T17:41:37.602157+0000","last_undegraded":"2023-10-10T17:41:37.602157+0000","last_fullsized":"2023-10-10T17:41:37.602157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:48:30.086020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602647+0000","last_change":"2023-10-10T17:40:41.596067+0000","last_active":"2023-10-10T17:41:37.602647+0000","last_peered":"2023-10-10T17:41:37.602647+0000","last_clean":"2023-10-10T17:41:37.602647+0000","last_became_active":"2023-10-10T17:40:41.595703+0000","last_became_peered":"2023-10-10T17:40:41.595703+0000","last_unstale":"2023-10-10T17:41:37.602647+0000","last_undegraded":"2023-10-10T17:41:37.602647+0000","last_fullsized":"2023-10-10T17:41:37.602647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:13.451772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598863+0000","last_change":"2023-10-10T17:40:41.593296+0000","last_active":"2023-10-10T17:41:37.598863+0000","last_peered":"2023-10-10T17:41:37.598863+0000","last_clean":"2023-10-10T17:41:37.598863+0000","last_became_active":"2023-10-10T17:40:41.592965+0000","last_became_peered":"2023-10-10T17:40:41.592965+0000","last_unstale":"2023-10-10T17:41:37.598863+0000","last_undegraded":"2023-10-10T17:41:37.598863+0000","last_fullsized":"2023-10-10T17:41:37.598863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:30:41.515067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598115+0000","last_change":"2023-10-10T17:40:41.604542+0000","last_active":"2023-10-10T17:41:37.598115+0000","last_peered":"2023-10-10T17:41:37.598115+0000","last_clean":"2023-10-10T17:41:37.598115+0000","last_became_active":"2023-10-10T17:40:41.604000+0000","last_became_peered":"2023-10-10T17:40:41.604000+0000","last_unstale":"2023-10-10T17:41:37.598115+0000","last_undegraded":"2023-10-10T17:41:37.598115+0000","last_fullsized":"2023-10-10T17:41:37.598115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:42:32.658592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602108+0000","last_change":"2023-10-10T17:41:37.602108+0000","last_active":"2023-10-10T17:41:37.602108+0000","last_peered":"2023-10-10T17:41:37.602108+0000","last_clean":"2023-10-10T17:41:37.602108+0000","last_became_active":"2023-10-10T17:41:37.601778+0000","last_became_peered":"2023-10-10T17:41:37.601778+0000","last_unstale":"2023-10-10T17:41:37.602108+0000","last_undegraded":"2023-10-10T17:41:37.602108+0000","last_fullsized":"2023-10-10T17:41:37.602108+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:48:51.539525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595527+0000","last_change":"2023-10-10T17:40:41.606659+0000","last_active":"2023-10-10T17:41:37.595527+0000","last_peered":"2023-10-10T17:41:37.595527+0000","last_clean":"2023-10-10T17:41:37.595527+0000","last_became_active":"2023-10-10T17:40:41.606061+0000","last_became_peered":"2023-10-10T17:40:41.606061+0000","last_unstale":"2023-10-10T17:41:37.595527+0000","last_undegraded":"2023-10-10T17:41:37.595527+0000","last_fullsized":"2023-10-10T17:41:37.595527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:04:19.222774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788308+0000","last_change":"2023-10-10T17:40:41.607630+0000","last_active":"2023-10-10T17:41:37.788308+0000","last_peered":"2023-10-10T17:41:37.788308+0000","last_clean":"2023-10-10T17:41:37.788308+0000","last_became_active":"2023-10-10T17:40:41.607297+0000","last_became_peered":"2023-10-10T17:40:41.607297+0000","last_unstale":"2023-10-10T17:41:37.788308+0000","last_undegraded":"2023-10-10T17:41:37.788308+0000","last_fullsized":"2023-10-10T17:41:37.788308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:15:06.512286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595355+0000","last_change":"2023-10-10T17:40:41.594082+0000","last_active":"2023-10-10T17:41:37.595355+0000","last_peered":"2023-10-10T17:41:37.595355+0000","last_clean":"2023-10-10T17:41:37.595355+0000","last_became_active":"2023-10-10T17:40:41.593787+0000","last_became_peered":"2023-10-10T17:40:41.593787+0000","last_unstale":"2023-10-10T17:41:37.595355+0000","last_undegraded":"2023-10-10T17:41:37.595355+0000","last_fullsized":"2023-10-10T17:41:37.595355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:10:48.548705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596760+0000","last_change":"2023-10-10T17:40:41.608529+0000","last_active":"2023-10-10T17:41:37.596760+0000","last_peered":"2023-10-10T17:41:37.596760+0000","last_clean":"2023-10-10T17:41:37.596760+0000","last_became_active":"2023-10-10T17:40:41.608247+0000","last_became_peered":"2023-10-10T17:40:41.608247+0000","last_unstale":"2023-10-10T17:41:37.596760+0000","last_undegraded":"2023-10-10T17:41:37.596760+0000","last_fullsized":"2023-10-10T17:41:37.596760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:20:05.900726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596528+0000","last_change":"2023-10-10T17:40:45.707455+0000","last_active":"2023-10-10T17:41:37.596528+0000","last_peered":"2023-10-10T17:41:37.596528+0000","last_clean":"2023-10-10T17:41:37.596528+0000","last_became_active":"2023-10-10T17:40:45.707024+0000","last_became_peered":"2023-10-10T17:40:45.707024+0000","last_unstale":"2023-10-10T17:41:37.596528+0000","last_undegraded":"2023-10-10T17:41:37.596528+0000","last_fullsized":"2023-10-10T17:41:37.596528+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:24.408486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595807+0000","last_change":"2023-10-10T17:40:41.165983+0000","last_active":"2023-10-10T17:41:37.595807+0000","last_peered":"2023-10-10T17:41:37.595807+0000","last_clean":"2023-10-10T17:41:37.595807+0000","last_became_active":"2023-10-10T17:40:41.165487+0000","last_became_peered":"2023-10-10T17:40:41.165487+0000","last_unstale":"2023-10-10T17:41:37.595807+0000","last_undegraded":"2023-10-10T17:41:37.595807+0000","last_fullsized":"2023-10-10T17:41:37.595807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:08:45.809159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979220+0000","last_change":"2023-10-10T17:40:41.614258+0000","last_active":"2023-10-10T17:41:37.979220+0000","last_peered":"2023-10-10T17:41:37.979220+0000","last_clean":"2023-10-10T17:41:37.979220+0000","last_became_active":"2023-10-10T17:40:41.613951+0000","last_became_peered":"2023-10-10T17:40:41.613951+0000","last_unstale":"2023-10-10T17:41:37.979220+0000","last_undegraded":"2023-10-10T17:41:37.979220+0000","last_fullsized":"2023-10-10T17:41:37.979220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:43:34.983899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:45.708568+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:45.708108+0000","last_became_peered":"2023-10-10T17:40:45.708108+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:30:24.153573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980255+0000","last_change":"2023-10-10T17:40:41.167744+0000","last_active":"2023-10-10T17:41:37.980255+0000","last_peered":"2023-10-10T17:41:37.980255+0000","last_clean":"2023-10-10T17:41:37.980255+0000","last_became_active":"2023-10-10T17:40:41.167189+0000","last_became_peered":"2023-10-10T17:40:41.167189+0000","last_unstale":"2023-10-10T17:41:37.980255+0000","last_undegraded":"2023-10-10T17:41:37.980255+0000","last_fullsized":"2023-10-10T17:41:37.980255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:47.667198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598064+0000","last_change":"2023-10-10T17:40:41.608514+0000","last_active":"2023-10-10T17:41:37.598064+0000","last_peered":"2023-10-10T17:41:37.598064+0000","last_clean":"2023-10-10T17:41:37.598064+0000","last_became_active":"2023-10-10T17:40:41.608208+0000","last_became_peered":"2023-10-10T17:40:41.608208+0000","last_unstale":"2023-10-10T17:41:37.598064+0000","last_undegraded":"2023-10-10T17:41:37.598064+0000","last_fullsized":"2023-10-10T17:41:37.598064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:48:25.411862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.981315+0000","last_change":"2023-10-10T17:41:37.981315+0000","last_active":"2023-10-10T17:41:37.981315+0000","last_peered":"2023-10-10T17:41:37.981315+0000","last_clean":"2023-10-10T17:41:37.981315+0000","last_became_active":"2023-10-10T17:41:37.980691+0000","last_became_peered":"2023-10-10T17:41:37.980691+0000","last_unstale":"2023-10-10T17:41:37.981315+0000","last_undegraded":"2023-10-10T17:41:37.981315+0000","last_fullsized":"2023-10-10T17:41:37.981315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:36:12.102836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601523+0000","last_change":"2023-10-10T17:40:41.169821+0000","last_active":"2023-10-10T17:41:37.601523+0000","last_peered":"2023-10-10T17:41:37.601523+0000","last_clean":"2023-10-10T17:41:37.601523+0000","last_became_active":"2023-10-10T17:40:41.169502+0000","last_became_peered":"2023-10-10T17:40:41.169502+0000","last_unstale":"2023-10-10T17:41:37.601523+0000","last_undegraded":"2023-10-10T17:41:37.601523+0000","last_fullsized":"2023-10-10T17:41:37.601523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:32:01.106471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595719+0000","last_change":"2023-10-10T17:40:41.602720+0000","last_active":"2023-10-10T17:41:37.595719+0000","last_peered":"2023-10-10T17:41:37.595719+0000","last_clean":"2023-10-10T17:41:37.595719+0000","last_became_active":"2023-10-10T17:40:41.602187+0000","last_became_peered":"2023-10-10T17:40:41.602187+0000","last_unstale":"2023-10-10T17:41:37.595719+0000","last_undegraded":"2023-10-10T17:41:37.595719+0000","last_fullsized":"2023-10-10T17:41:37.595719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:18:35.228329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601501+0000","last_change":"2023-10-10T17:40:45.706946+0000","last_active":"2023-10-10T17:41:37.601501+0000","last_peered":"2023-10-10T17:41:37.601501+0000","last_clean":"2023-10-10T17:41:37.601501+0000","last_became_active":"2023-10-10T17:40:45.706495+0000","last_became_peered":"2023-10-10T17:40:45.706495+0000","last_unstale":"2023-10-10T17:41:37.601501+0000","last_undegraded":"2023-10-10T17:41:37.601501+0000","last_fullsized":"2023-10-10T17:41:37.601501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:11.140718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601201+0000","last_change":"2023-10-10T17:40:41.171855+0000","last_active":"2023-10-10T17:41:37.601201+0000","last_peered":"2023-10-10T17:41:37.601201+0000","last_clean":"2023-10-10T17:41:37.601201+0000","last_became_active":"2023-10-10T17:40:41.171515+0000","last_became_peered":"2023-10-10T17:40:41.171515+0000","last_unstale":"2023-10-10T17:41:37.601201+0000","last_undegraded":"2023-10-10T17:41:37.601201+0000","last_fullsized":"2023-10-10T17:41:37.601201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:15.556629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652087+0000","last_change":"2023-10-10T17:40:41.587083+0000","last_active":"2023-10-10T17:41:37.652087+0000","last_peered":"2023-10-10T17:41:37.652087+0000","last_clean":"2023-10-10T17:41:37.652087+0000","last_became_active":"2023-10-10T17:40:41.586758+0000","last_became_peered":"2023-10-10T17:40:41.586758+0000","last_unstale":"2023-10-10T17:41:37.652087+0000","last_undegraded":"2023-10-10T17:41:37.652087+0000","last_fullsized":"2023-10-10T17:41:37.652087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:31:30.273631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596265+0000","last_change":"2023-10-10T17:40:45.710492+0000","last_active":"2023-10-10T17:41:37.596265+0000","last_peered":"2023-10-10T17:41:37.596265+0000","last_clean":"2023-10-10T17:41:37.596265+0000","last_became_active":"2023-10-10T17:40:45.710069+0000","last_became_peered":"2023-10-10T17:40:45.710069+0000","last_unstale":"2023-10-10T17:41:37.596265+0000","last_undegraded":"2023-10-10T17:41:37.596265+0000","last_fullsized":"2023-10-10T17:41:37.596265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:58:28.507189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:19.708911+0000","last_change":"2023-10-10T17:40:41.167449+0000","last_active":"2023-10-10T17:42:19.708911+0000","last_peered":"2023-10-10T17:42:19.708911+0000","last_clean":"2023-10-10T17:42:19.708911+0000","last_became_active":"2023-10-10T17:40:41.167038+0000","last_became_peered":"2023-10-10T17:40:41.167038+0000","last_unstale":"2023-10-10T17:42:19.708911+0000","last_undegraded":"2023-10-10T17:42:19.708911+0000","last_fullsized":"2023-10-10T17:42:19.708911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:48:38.144880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599067+0000","last_change":"2023-10-10T17:40:41.594327+0000","last_active":"2023-10-10T17:41:37.599067+0000","last_peered":"2023-10-10T17:41:37.599067+0000","last_clean":"2023-10-10T17:41:37.599067+0000","last_became_active":"2023-10-10T17:40:41.594008+0000","last_became_peered":"2023-10-10T17:40:41.594008+0000","last_unstale":"2023-10-10T17:41:37.599067+0000","last_undegraded":"2023-10-10T17:41:37.599067+0000","last_fullsized":"2023-10-10T17:41:37.599067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:56:08.300633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.653403+0000","last_change":"2023-10-10T17:41:37.653403+0000","last_active":"2023-10-10T17:41:37.653403+0000","last_peered":"2023-10-10T17:41:37.653403+0000","last_clean":"2023-10-10T17:41:37.653403+0000","last_became_active":"2023-10-10T17:41:37.652946+0000","last_became_peered":"2023-10-10T17:41:37.652946+0000","last_unstale":"2023-10-10T17:41:37.653403+0000","last_undegraded":"2023-10-10T17:41:37.653403+0000","last_fullsized":"2023-10-10T17:41:37.653403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:16:29.417131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598235+0000","last_change":"2023-10-10T17:40:41.171926+0000","last_active":"2023-10-10T17:41:37.598235+0000","last_peered":"2023-10-10T17:41:37.598235+0000","last_clean":"2023-10-10T17:41:37.598235+0000","last_became_active":"2023-10-10T17:40:41.171627+0000","last_became_peered":"2023-10-10T17:40:41.171627+0000","last_unstale":"2023-10-10T17:41:37.598235+0000","last_undegraded":"2023-10-10T17:41:37.598235+0000","last_fullsized":"2023-10-10T17:41:37.598235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:00:51.367484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651905+0000","last_change":"2023-10-10T17:40:41.593135+0000","last_active":"2023-10-10T17:41:37.651905+0000","last_peered":"2023-10-10T17:41:37.651905+0000","last_clean":"2023-10-10T17:41:37.651905+0000","last_became_active":"2023-10-10T17:40:41.592871+0000","last_became_peered":"2023-10-10T17:40:41.592871+0000","last_unstale":"2023-10-10T17:41:37.651905+0000","last_undegraded":"2023-10-10T17:41:37.651905+0000","last_fullsized":"2023-10-10T17:41:37.651905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:32.143194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788755+0000","last_change":"2023-10-10T17:40:45.703557+0000","last_active":"2023-10-10T17:41:37.788755+0000","last_peered":"2023-10-10T17:41:37.788755+0000","last_clean":"2023-10-10T17:41:37.788755+0000","last_became_active":"2023-10-10T17:40:45.703085+0000","last_became_peered":"2023-10-10T17:40:45.703085+0000","last_unstale":"2023-10-10T17:41:37.788755+0000","last_undegraded":"2023-10-10T17:41:37.788755+0000","last_fullsized":"2023-10-10T17:41:37.788755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:09:11.130227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598088+0000","last_change":"2023-10-10T17:40:41.162829+0000","last_active":"2023-10-10T17:41:37.598088+0000","last_peered":"2023-10-10T17:41:37.598088+0000","last_clean":"2023-10-10T17:41:37.598088+0000","last_became_active":"2023-10-10T17:40:41.162516+0000","last_became_peered":"2023-10-10T17:40:41.162516+0000","last_unstale":"2023-10-10T17:41:37.598088+0000","last_undegraded":"2023-10-10T17:41:37.598088+0000","last_fullsized":"2023-10-10T17:41:37.598088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:07:29.590599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.789909+0000","last_change":"2023-10-10T17:41:37.789510+0000","last_active":"2023-10-10T17:41:37.789909+0000","last_peered":"2023-10-10T17:41:37.789909+0000","last_clean":"2023-10-10T17:41:37.789909+0000","last_became_active":"2023-10-10T17:41:37.789166+0000","last_became_peered":"2023-10-10T17:41:37.789166+0000","last_unstale":"2023-10-10T17:41:37.789909+0000","last_undegraded":"2023-10-10T17:41:37.789909+0000","last_fullsized":"2023-10-10T17:41:37.789909+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:33:55.993371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598938+0000","last_change":"2023-10-10T17:40:45.707139+0000","last_active":"2023-10-10T17:41:37.598938+0000","last_peered":"2023-10-10T17:41:37.598938+0000","last_clean":"2023-10-10T17:41:37.598938+0000","last_became_active":"2023-10-10T17:40:45.706708+0000","last_became_peered":"2023-10-10T17:40:45.706708+0000","last_unstale":"2023-10-10T17:41:37.598938+0000","last_undegraded":"2023-10-10T17:41:37.598938+0000","last_fullsized":"2023-10-10T17:41:37.598938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:46:42.381782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651470+0000","last_change":"2023-10-10T17:40:41.170512+0000","last_active":"2023-10-10T17:41:37.651470+0000","last_peered":"2023-10-10T17:41:37.651470+0000","last_clean":"2023-10-10T17:41:37.651470+0000","last_became_active":"2023-10-10T17:40:41.170264+0000","last_became_peered":"2023-10-10T17:40:41.170264+0000","last_unstale":"2023-10-10T17:41:37.651470+0000","last_undegraded":"2023-10-10T17:41:37.651470+0000","last_fullsized":"2023-10-10T17:41:37.651470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:21:55.138815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594921+0000","last_change":"2023-10-10T17:40:41.597647+0000","last_active":"2023-10-10T17:41:37.594921+0000","last_peered":"2023-10-10T17:41:37.594921+0000","last_clean":"2023-10-10T17:41:37.594921+0000","last_became_active":"2023-10-10T17:40:41.597245+0000","last_became_peered":"2023-10-10T17:40:41.597245+0000","last_unstale":"2023-10-10T17:41:37.594921+0000","last_undegraded":"2023-10-10T17:41:37.594921+0000","last_fullsized":"2023-10-10T17:41:37.594921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:02.702932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979667+0000","last_change":"2023-10-10T17:40:45.705089+0000","last_active":"2023-10-10T17:41:37.979667+0000","last_peered":"2023-10-10T17:41:37.979667+0000","last_clean":"2023-10-10T17:41:37.979667+0000","last_became_active":"2023-10-10T17:40:45.704475+0000","last_became_peered":"2023-10-10T17:40:45.704475+0000","last_unstale":"2023-10-10T17:41:37.979667+0000","last_undegraded":"2023-10-10T17:41:37.979667+0000","last_fullsized":"2023-10-10T17:41:37.979667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:26.896709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:21.710912+0000","last_change":"2023-10-10T17:40:41.167409+0000","last_active":"2023-10-10T17:42:21.710912+0000","last_peered":"2023-10-10T17:42:21.710912+0000","last_clean":"2023-10-10T17:42:21.710912+0000","last_became_active":"2023-10-10T17:40:41.167125+0000","last_became_peered":"2023-10-10T17:40:41.167125+0000","last_unstale":"2023-10-10T17:42:21.710912+0000","last_undegraded":"2023-10-10T17:42:21.710912+0000","last_fullsized":"2023-10-10T17:42:21.710912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:12:36.905039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598010+0000","last_change":"2023-10-10T17:40:41.606216+0000","last_active":"2023-10-10T17:41:37.598010+0000","last_peered":"2023-10-10T17:41:37.598010+0000","last_clean":"2023-10-10T17:41:37.598010+0000","last_became_active":"2023-10-10T17:40:41.603258+0000","last_became_peered":"2023-10-10T17:40:41.603258+0000","last_unstale":"2023-10-10T17:41:37.598010+0000","last_undegraded":"2023-10-10T17:41:37.598010+0000","last_fullsized":"2023-10-10T17:41:37.598010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:30:40.923843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594917+0000","last_change":"2023-10-10T17:40:45.705623+0000","last_active":"2023-10-10T17:41:37.594917+0000","last_peered":"2023-10-10T17:41:37.594917+0000","last_clean":"2023-10-10T17:41:37.594917+0000","last_became_active":"2023-10-10T17:40:45.705242+0000","last_became_peered":"2023-10-10T17:40:45.705242+0000","last_unstale":"2023-10-10T17:41:37.594917+0000","last_undegraded":"2023-10-10T17:41:37.594917+0000","last_fullsized":"2023-10-10T17:41:37.594917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:43:18.391641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601242+0000","last_change":"2023-10-10T17:40:41.171552+0000","last_active":"2023-10-10T17:41:37.601242+0000","last_peered":"2023-10-10T17:41:37.601242+0000","last_clean":"2023-10-10T17:41:37.601242+0000","last_became_active":"2023-10-10T17:40:41.171207+0000","last_became_peered":"2023-10-10T17:40:41.171207+0000","last_unstale":"2023-10-10T17:41:37.601242+0000","last_undegraded":"2023-10-10T17:41:37.601242+0000","last_fullsized":"2023-10-10T17:41:37.601242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:35.436090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599824+0000","last_change":"2023-10-10T17:40:41.609655+0000","last_active":"2023-10-10T17:41:37.599824+0000","last_peered":"2023-10-10T17:41:37.599824+0000","last_clean":"2023-10-10T17:41:37.599824+0000","last_became_active":"2023-10-10T17:40:41.609318+0000","last_became_peered":"2023-10-10T17:40:41.609318+0000","last_unstale":"2023-10-10T17:41:37.599824+0000","last_undegraded":"2023-10-10T17:41:37.599824+0000","last_fullsized":"2023-10-10T17:41:37.599824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:06:22.359139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602379+0000","last_change":"2023-10-10T17:40:45.709513+0000","last_active":"2023-10-10T17:41:37.602379+0000","last_peered":"2023-10-10T17:41:37.602379+0000","last_clean":"2023-10-10T17:41:37.602379+0000","last_became_active":"2023-10-10T17:40:45.708963+0000","last_became_peered":"2023-10-10T17:40:45.708963+0000","last_unstale":"2023-10-10T17:41:37.602379+0000","last_undegraded":"2023-10-10T17:41:37.602379+0000","last_fullsized":"2023-10-10T17:41:37.602379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:15:02.687427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652007+0000","last_change":"2023-10-10T17:40:41.163234+0000","last_active":"2023-10-10T17:41:37.652007+0000","last_peered":"2023-10-10T17:41:37.652007+0000","last_clean":"2023-10-10T17:41:37.652007+0000","last_became_active":"2023-10-10T17:40:41.162916+0000","last_became_peered":"2023-10-10T17:40:41.162916+0000","last_unstale":"2023-10-10T17:41:37.652007+0000","last_undegraded":"2023-10-10T17:41:37.652007+0000","last_fullsized":"2023-10-10T17:41:37.652007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:09:06.358768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598352+0000","last_change":"2023-10-10T17:40:41.602852+0000","last_active":"2023-10-10T17:41:37.598352+0000","last_peered":"2023-10-10T17:41:37.598352+0000","last_clean":"2023-10-10T17:41:37.598352+0000","last_became_active":"2023-10-10T17:40:41.602530+0000","last_became_peered":"2023-10-10T17:40:41.602530+0000","last_unstale":"2023-10-10T17:41:37.598352+0000","last_undegraded":"2023-10-10T17:41:37.598352+0000","last_fullsized":"2023-10-10T17:41:37.598352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:58:41.949886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595188+0000","last_change":"2023-10-10T17:40:45.713714+0000","last_active":"2023-10-10T17:41:37.595188+0000","last_peered":"2023-10-10T17:41:37.595188+0000","last_clean":"2023-10-10T17:41:37.595188+0000","last_became_active":"2023-10-10T17:40:45.712953+0000","last_became_peered":"2023-10-10T17:40:45.712953+0000","last_unstale":"2023-10-10T17:41:37.595188+0000","last_undegraded":"2023-10-10T17:41:37.595188+0000","last_fullsized":"2023-10-10T17:41:37.595188+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:15.222779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":204,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:22.711933+0000","last_change":"2023-10-10T17:40:41.168328+0000","last_active":"2023-10-10T17:42:22.711933+0000","last_peered":"2023-10-10T17:42:22.711933+0000","last_clean":"2023-10-10T17:42:22.711933+0000","last_became_active":"2023-10-10T17:40:41.168044+0000","last_became_peered":"2023-10-10T17:40:41.168044+0000","last_unstale":"2023-10-10T17:42:22.711933+0000","last_undegraded":"2023-10-10T17:42:22.711933+0000","last_fullsized":"2023-10-10T17:42:22.711933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:52:24.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":166,"num_read_kb":87,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599535+0000","last_change":"2023-10-10T17:40:41.597795+0000","last_active":"2023-10-10T17:41:37.599535+0000","last_peered":"2023-10-10T17:41:37.599535+0000","last_clean":"2023-10-10T17:41:37.599535+0000","last_became_active":"2023-10-10T17:40:41.597537+0000","last_became_peered":"2023-10-10T17:40:41.597537+0000","last_unstale":"2023-10-10T17:41:37.599535+0000","last_undegraded":"2023-10-10T17:41:37.599535+0000","last_fullsized":"2023-10-10T17:41:37.599535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:16:34.371564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652449+0000","last_change":"2023-10-10T17:40:45.709827+0000","last_active":"2023-10-10T17:41:37.652449+0000","last_peered":"2023-10-10T17:41:37.652449+0000","last_clean":"2023-10-10T17:41:37.652449+0000","last_became_active":"2023-10-10T17:40:45.709373+0000","last_became_peered":"2023-10-10T17:40:45.709373+0000","last_unstale":"2023-10-10T17:41:37.652449+0000","last_undegraded":"2023-10-10T17:41:37.652449+0000","last_fullsized":"2023-10-10T17:41:37.652449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:35:42.826536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596527+0000","last_change":"2023-10-10T17:40:41.160804+0000","last_active":"2023-10-10T17:41:37.596527+0000","last_peered":"2023-10-10T17:41:37.596527+0000","last_clean":"2023-10-10T17:41:37.596527+0000","last_became_active":"2023-10-10T17:40:41.160493+0000","last_became_peered":"2023-10-10T17:40:41.160493+0000","last_unstale":"2023-10-10T17:41:37.596527+0000","last_undegraded":"2023-10-10T17:41:37.596527+0000","last_fullsized":"2023-10-10T17:41:37.596527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:59.561663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598334+0000","last_change":"2023-10-10T17:40:41.601357+0000","last_active":"2023-10-10T17:41:37.598334+0000","last_peered":"2023-10-10T17:41:37.598334+0000","last_clean":"2023-10-10T17:41:37.598334+0000","last_became_active":"2023-10-10T17:40:41.601057+0000","last_became_peered":"2023-10-10T17:40:41.601057+0000","last_unstale":"2023-10-10T17:41:37.598334+0000","last_undegraded":"2023-10-10T17:41:37.598334+0000","last_fullsized":"2023-10-10T17:41:37.598334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:07.286077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788393+0000","last_change":"2023-10-10T17:40:45.708067+0000","last_active":"2023-10-10T17:41:37.788393+0000","last_peered":"2023-10-10T17:41:37.788393+0000","last_clean":"2023-10-10T17:41:37.788393+0000","last_became_active":"2023-10-10T17:40:45.707686+0000","last_became_peered":"2023-10-10T17:40:45.707686+0000","last_unstale":"2023-10-10T17:41:37.788393+0000","last_undegraded":"2023-10-10T17:41:37.788393+0000","last_fullsized":"2023-10-10T17:41:37.788393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:08.956407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651829+0000","last_change":"2023-10-10T17:40:41.168006+0000","last_active":"2023-10-10T17:41:37.651829+0000","last_peered":"2023-10-10T17:41:37.651829+0000","last_clean":"2023-10-10T17:41:37.651829+0000","last_became_active":"2023-10-10T17:40:41.167777+0000","last_became_peered":"2023-10-10T17:40:41.167777+0000","last_unstale":"2023-10-10T17:41:37.651829+0000","last_undegraded":"2023-10-10T17:41:37.651829+0000","last_fullsized":"2023-10-10T17:41:37.651829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:30:32.283329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601103+0000","last_change":"2023-10-10T17:40:41.591215+0000","last_active":"2023-10-10T17:41:37.601103+0000","last_peered":"2023-10-10T17:41:37.601103+0000","last_clean":"2023-10-10T17:41:37.601103+0000","last_became_active":"2023-10-10T17:40:41.590929+0000","last_became_peered":"2023-10-10T17:40:41.590929+0000","last_unstale":"2023-10-10T17:41:37.601103+0000","last_undegraded":"2023-10-10T17:41:37.601103+0000","last_fullsized":"2023-10-10T17:41:37.601103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:11:13.144245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595485+0000","last_change":"2023-10-10T17:40:45.709038+0000","last_active":"2023-10-10T17:41:37.595485+0000","last_peered":"2023-10-10T17:41:37.595485+0000","last_clean":"2023-10-10T17:41:37.595485+0000","last_became_active":"2023-10-10T17:40:45.708683+0000","last_became_peered":"2023-10-10T17:40:45.708683+0000","last_unstale":"2023-10-10T17:41:37.595485+0000","last_undegraded":"2023-10-10T17:41:37.595485+0000","last_fullsized":"2023-10-10T17:41:37.595485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:50:01.174506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:22.722538+0000","last_change":"2023-10-10T17:40:41.163817+0000","last_active":"2023-10-10T17:42:22.722538+0000","last_peered":"2023-10-10T17:42:22.722538+0000","last_clean":"2023-10-10T17:42:22.722538+0000","last_became_active":"2023-10-10T17:40:41.163497+0000","last_became_peered":"2023-10-10T17:40:41.163497+0000","last_unstale":"2023-10-10T17:42:22.722538+0000","last_undegraded":"2023-10-10T17:42:22.722538+0000","last_fullsized":"2023-10-10T17:42:22.722538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:54:45.382546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":800,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.607553+0000","last_change":"2023-10-10T17:41:37.607553+0000","last_active":"2023-10-10T17:41:37.607553+0000","last_peered":"2023-10-10T17:41:37.607553+0000","last_clean":"2023-10-10T17:41:37.607553+0000","last_became_active":"2023-10-10T17:41:37.606536+0000","last_became_peered":"2023-10-10T17:41:37.606536+0000","last_unstale":"2023-10-10T17:41:37.607553+0000","last_undegraded":"2023-10-10T17:41:37.607553+0000","last_fullsized":"2023-10-10T17:41:37.607553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:58:09.182227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600059+0000","last_change":"2023-10-10T17:40:45.714900+0000","last_active":"2023-10-10T17:41:37.600059+0000","last_peered":"2023-10-10T17:41:37.600059+0000","last_clean":"2023-10-10T17:41:37.600059+0000","last_became_active":"2023-10-10T17:40:45.714525+0000","last_became_peered":"2023-10-10T17:40:45.714525+0000","last_unstale":"2023-10-10T17:41:37.600059+0000","last_undegraded":"2023-10-10T17:41:37.600059+0000","last_fullsized":"2023-10-10T17:41:37.600059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:01:36.283546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:38.795173+0000","last_change":"2023-10-10T17:41:38.795173+0000","last_active":"2023-10-10T17:41:38.795173+0000","last_peered":"2023-10-10T17:41:38.795173+0000","last_clean":"2023-10-10T17:41:38.795173+0000","last_became_active":"2023-10-10T17:41:37.603917+0000","last_became_peered":"2023-10-10T17:41:37.603917+0000","last_unstale":"2023-10-10T17:41:38.795173+0000","last_undegraded":"2023-10-10T17:41:38.795173+0000","last_fullsized":"2023-10-10T17:41:38.795173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:32:38.418130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651410+0000","last_change":"2023-10-10T17:40:41.586591+0000","last_active":"2023-10-10T17:41:37.651410+0000","last_peered":"2023-10-10T17:41:37.651410+0000","last_clean":"2023-10-10T17:41:37.651410+0000","last_became_active":"2023-10-10T17:40:41.586293+0000","last_became_peered":"2023-10-10T17:40:41.586293+0000","last_unstale":"2023-10-10T17:41:37.651410+0000","last_undegraded":"2023-10-10T17:41:37.651410+0000","last_fullsized":"2023-10-10T17:41:37.651410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:39:26.663874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651375+0000","last_change":"2023-10-10T17:40:45.708348+0000","last_active":"2023-10-10T17:41:37.651375+0000","last_peered":"2023-10-10T17:41:37.651375+0000","last_clean":"2023-10-10T17:41:37.651375+0000","last_became_active":"2023-10-10T17:40:45.707899+0000","last_became_peered":"2023-10-10T17:40:45.707899+0000","last_unstale":"2023-10-10T17:41:37.651375+0000","last_undegraded":"2023-10-10T17:41:37.651375+0000","last_fullsized":"2023-10-10T17:41:37.651375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:29:39.345572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601729+0000","last_change":"2023-10-10T17:40:41.168209+0000","last_active":"2023-10-10T17:41:37.601729+0000","last_peered":"2023-10-10T17:41:37.601729+0000","last_clean":"2023-10-10T17:41:37.601729+0000","last_became_active":"2023-10-10T17:40:41.167907+0000","last_became_peered":"2023-10-10T17:40:41.167907+0000","last_unstale":"2023-10-10T17:41:37.601729+0000","last_undegraded":"2023-10-10T17:41:37.601729+0000","last_fullsized":"2023-10-10T17:41:37.601729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:35:19.344734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.594848+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.594562+0000","last_became_peered":"2023-10-10T17:40:41.594562+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:15:21.805794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600519+0000","last_change":"2023-10-10T17:40:45.711043+0000","last_active":"2023-10-10T17:41:37.600519+0000","last_peered":"2023-10-10T17:41:37.600519+0000","last_clean":"2023-10-10T17:41:37.600519+0000","last_became_active":"2023-10-10T17:40:45.710617+0000","last_became_peered":"2023-10-10T17:40:45.710617+0000","last_unstale":"2023-10-10T17:41:37.600519+0000","last_undegraded":"2023-10-10T17:41:37.600519+0000","last_fullsized":"2023-10-10T17:41:37.600519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:00:48.152401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595133+0000","last_change":"2023-10-10T17:40:41.163238+0000","last_active":"2023-10-10T17:41:37.595133+0000","last_peered":"2023-10-10T17:41:37.595133+0000","last_clean":"2023-10-10T17:41:37.595133+0000","last_became_active":"2023-10-10T17:40:41.162934+0000","last_became_peered":"2023-10-10T17:40:41.162934+0000","last_unstale":"2023-10-10T17:41:37.595133+0000","last_undegraded":"2023-10-10T17:41:37.595133+0000","last_fullsized":"2023-10-10T17:41:37.595133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:23:10.440706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598371+0000","last_change":"2023-10-10T17:40:41.597026+0000","last_active":"2023-10-10T17:41:37.598371+0000","last_peered":"2023-10-10T17:41:37.598371+0000","last_clean":"2023-10-10T17:41:37.598371+0000","last_became_active":"2023-10-10T17:40:41.596747+0000","last_became_peered":"2023-10-10T17:40:41.596747+0000","last_unstale":"2023-10-10T17:41:37.598371+0000","last_undegraded":"2023-10-10T17:41:37.598371+0000","last_fullsized":"2023-10-10T17:41:37.598371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:52.010575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599056+0000","last_change":"2023-10-10T17:40:45.707218+0000","last_active":"2023-10-10T17:41:37.599056+0000","last_peered":"2023-10-10T17:41:37.599056+0000","last_clean":"2023-10-10T17:41:37.599056+0000","last_became_active":"2023-10-10T17:40:45.706850+0000","last_became_peered":"2023-10-10T17:40:45.706850+0000","last_unstale":"2023-10-10T17:41:37.599056+0000","last_undegraded":"2023-10-10T17:41:37.599056+0000","last_fullsized":"2023-10-10T17:41:37.599056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:36:44.235237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:41.166713+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:41.165951+0000","last_became_peered":"2023-10-10T17:40:41.165951+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:56:00.129165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597783+0000","last_change":"2023-10-10T17:40:41.605860+0000","last_active":"2023-10-10T17:41:37.597783+0000","last_peered":"2023-10-10T17:41:37.597783+0000","last_clean":"2023-10-10T17:41:37.597783+0000","last_became_active":"2023-10-10T17:40:41.605556+0000","last_became_peered":"2023-10-10T17:40:41.605556+0000","last_unstale":"2023-10-10T17:41:37.597783+0000","last_undegraded":"2023-10-10T17:41:37.597783+0000","last_fullsized":"2023-10-10T17:41:37.597783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:07.438058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599757+0000","last_change":"2023-10-10T17:40:45.711844+0000","last_active":"2023-10-10T17:41:37.599757+0000","last_peered":"2023-10-10T17:41:37.599757+0000","last_clean":"2023-10-10T17:41:37.599757+0000","last_became_active":"2023-10-10T17:40:45.711413+0000","last_became_peered":"2023-10-10T17:40:45.711413+0000","last_unstale":"2023-10-10T17:41:37.599757+0000","last_undegraded":"2023-10-10T17:41:37.599757+0000","last_fullsized":"2023-10-10T17:41:37.599757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:13:03.476627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.650650+0000","last_change":"2023-10-10T17:40:41.608625+0000","last_active":"2023-10-10T17:41:37.650650+0000","last_peered":"2023-10-10T17:41:37.650650+0000","last_clean":"2023-10-10T17:41:37.650650+0000","last_became_active":"2023-10-10T17:40:41.608332+0000","last_became_peered":"2023-10-10T17:40:41.608332+0000","last_unstale":"2023-10-10T17:41:37.650650+0000","last_undegraded":"2023-10-10T17:41:37.650650+0000","last_fullsized":"2023-10-10T17:41:37.650650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:07:55.183440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:21.720594+0000","last_change":"2023-10-10T17:40:41.169807+0000","last_active":"2023-10-10T17:42:21.720594+0000","last_peered":"2023-10-10T17:42:21.720594+0000","last_clean":"2023-10-10T17:42:21.720594+0000","last_became_active":"2023-10-10T17:40:41.169475+0000","last_became_peered":"2023-10-10T17:40:41.169475+0000","last_unstale":"2023-10-10T17:42:21.720594+0000","last_undegraded":"2023-10-10T17:42:21.720594+0000","last_fullsized":"2023-10-10T17:42:21.720594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:22.887502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601878+0000","last_change":"2023-10-10T17:40:45.710544+0000","last_active":"2023-10-10T17:41:37.601878+0000","last_peered":"2023-10-10T17:41:37.601878+0000","last_clean":"2023-10-10T17:41:37.601878+0000","last_became_active":"2023-10-10T17:40:45.710084+0000","last_became_peered":"2023-10-10T17:40:45.710084+0000","last_unstale":"2023-10-10T17:41:37.601878+0000","last_undegraded":"2023-10-10T17:41:37.601878+0000","last_fullsized":"2023-10-10T17:41:37.601878+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:44:01.303666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979130+0000","last_change":"2023-10-10T17:40:41.599028+0000","last_active":"2023-10-10T17:41:37.979130+0000","last_peered":"2023-10-10T17:41:37.979130+0000","last_clean":"2023-10-10T17:41:37.979130+0000","last_became_active":"2023-10-10T17:40:41.598684+0000","last_became_peered":"2023-10-10T17:40:41.598684+0000","last_unstale":"2023-10-10T17:41:37.979130+0000","last_undegraded":"2023-10-10T17:41:37.979130+0000","last_fullsized":"2023-10-10T17:41:37.979130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:16:57.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602213+0000","last_change":"2023-10-10T17:40:41.165756+0000","last_active":"2023-10-10T17:41:37.602213+0000","last_peered":"2023-10-10T17:41:37.602213+0000","last_clean":"2023-10-10T17:41:37.602213+0000","last_became_active":"2023-10-10T17:40:41.165368+0000","last_became_peered":"2023-10-10T17:40:41.165368+0000","last_unstale":"2023-10-10T17:41:37.602213+0000","last_undegraded":"2023-10-10T17:41:37.602213+0000","last_fullsized":"2023-10-10T17:41:37.602213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:16.089880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600889+0000","last_change":"2023-10-10T17:39:42.508828+0000","last_active":"2023-10-10T17:41:37.600889+0000","last_peered":"2023-10-10T17:41:37.600889+0000","last_clean":"2023-10-10T17:41:37.600889+0000","last_became_active":"2023-10-10T17:39:42.508475+0000","last_became_peered":"2023-10-10T17:39:42.508475+0000","last_unstale":"2023-10-10T17:41:37.600889+0000","last_undegraded":"2023-10-10T17:41:37.600889+0000","last_fullsized":"2023-10-10T17:41:37.600889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_clean_scrub_stamp":"2023-10-10T17:36:46.553699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:08:38.008210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652108+0000","last_change":"2023-10-10T17:40:45.714604+0000","last_active":"2023-10-10T17:41:37.652108+0000","last_peered":"2023-10-10T17:41:37.652108+0000","last_clean":"2023-10-10T17:41:37.652108+0000","last_became_active":"2023-10-10T17:40:45.714185+0000","last_became_peered":"2023-10-10T17:40:45.714185+0000","last_unstale":"2023-10-10T17:41:37.652108+0000","last_undegraded":"2023-10-10T17:41:37.652108+0000","last_fullsized":"2023-10-10T17:41:37.652108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:55:26.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597853+0000","last_change":"2023-10-10T17:40:41.610002+0000","last_active":"2023-10-10T17:41:37.597853+0000","last_peered":"2023-10-10T17:41:37.597853+0000","last_clean":"2023-10-10T17:41:37.597853+0000","last_became_active":"2023-10-10T17:40:41.609708+0000","last_became_peered":"2023-10-10T17:40:41.609708+0000","last_unstale":"2023-10-10T17:41:37.597853+0000","last_undegraded":"2023-10-10T17:41:37.597853+0000","last_fullsized":"2023-10-10T17:41:37.597853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:34.611184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601055+0000","last_change":"2023-10-10T17:40:41.167715+0000","last_active":"2023-10-10T17:41:37.601055+0000","last_peered":"2023-10-10T17:41:37.601055+0000","last_clean":"2023-10-10T17:41:37.601055+0000","last_became_active":"2023-10-10T17:40:41.166385+0000","last_became_peered":"2023-10-10T17:40:41.166385+0000","last_unstale":"2023-10-10T17:41:37.601055+0000","last_undegraded":"2023-10-10T17:41:37.601055+0000","last_fullsized":"2023-10-10T17:41:37.601055+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:49:05.385383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788299+0000","last_change":"2023-10-10T17:40:45.707236+0000","last_active":"2023-10-10T17:41:37.788299+0000","last_peered":"2023-10-10T17:41:37.788299+0000","last_clean":"2023-10-10T17:41:37.788299+0000","last_became_active":"2023-10-10T17:40:45.706827+0000","last_became_peered":"2023-10-10T17:40:45.706827+0000","last_unstale":"2023-10-10T17:41:37.788299+0000","last_undegraded":"2023-10-10T17:41:37.788299+0000","last_fullsized":"2023-10-10T17:41:37.788299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:58.930376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601396+0000","last_change":"2023-10-10T17:40:41.170523+0000","last_active":"2023-10-10T17:41:37.601396+0000","last_peered":"2023-10-10T17:41:37.601396+0000","last_clean":"2023-10-10T17:41:37.601396+0000","last_became_active":"2023-10-10T17:40:41.170243+0000","last_became_peered":"2023-10-10T17:40:41.170243+0000","last_unstale":"2023-10-10T17:41:37.601396+0000","last_undegraded":"2023-10-10T17:41:37.601396+0000","last_fullsized":"2023-10-10T17:41:37.601396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:40.239129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599195+0000","last_change":"2023-10-10T17:40:41.610472+0000","last_active":"2023-10-10T17:41:37.599195+0000","last_peered":"2023-10-10T17:41:37.599195+0000","last_clean":"2023-10-10T17:41:37.599195+0000","last_became_active":"2023-10-10T17:40:41.610139+0000","last_became_peered":"2023-10-10T17:40:41.610139+0000","last_unstale":"2023-10-10T17:41:37.599195+0000","last_undegraded":"2023-10-10T17:41:37.599195+0000","last_fullsized":"2023-10-10T17:41:37.599195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:22:52.414485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979558+0000","last_change":"2023-10-10T17:40:45.712179+0000","last_active":"2023-10-10T17:41:37.979558+0000","last_peered":"2023-10-10T17:41:37.979558+0000","last_clean":"2023-10-10T17:41:37.979558+0000","last_became_active":"2023-10-10T17:40:45.711708+0000","last_became_peered":"2023-10-10T17:40:45.711708+0000","last_unstale":"2023-10-10T17:41:37.979558+0000","last_undegraded":"2023-10-10T17:41:37.979558+0000","last_fullsized":"2023-10-10T17:41:37.979558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:25:37.789886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595330+0000","last_change":"2023-10-10T17:40:41.164882+0000","last_active":"2023-10-10T17:41:37.595330+0000","last_peered":"2023-10-10T17:41:37.595330+0000","last_clean":"2023-10-10T17:41:37.595330+0000","last_became_active":"2023-10-10T17:40:41.164536+0000","last_became_peered":"2023-10-10T17:40:41.164536+0000","last_unstale":"2023-10-10T17:41:37.595330+0000","last_undegraded":"2023-10-10T17:41:37.595330+0000","last_fullsized":"2023-10-10T17:41:37.595330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:20:38.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980304+0000","last_change":"2023-10-10T17:40:41.595043+0000","last_active":"2023-10-10T17:41:37.980304+0000","last_peered":"2023-10-10T17:41:37.980304+0000","last_clean":"2023-10-10T17:41:37.980304+0000","last_became_active":"2023-10-10T17:40:41.594703+0000","last_became_peered":"2023-10-10T17:40:41.594703+0000","last_unstale":"2023-10-10T17:41:37.980304+0000","last_undegraded":"2023-10-10T17:41:37.980304+0000","last_fullsized":"2023-10-10T17:41:37.980304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:29:21.277302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596130+0000","last_change":"2023-10-10T17:40:45.705232+0000","last_active":"2023-10-10T17:41:37.596130+0000","last_peered":"2023-10-10T17:41:37.596130+0000","last_clean":"2023-10-10T17:41:37.596130+0000","last_became_active":"2023-10-10T17:40:45.704811+0000","last_became_peered":"2023-10-10T17:40:45.704811+0000","last_unstale":"2023-10-10T17:41:37.596130+0000","last_undegraded":"2023-10-10T17:41:37.596130+0000","last_fullsized":"2023-10-10T17:41:37.596130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:50.034871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602441+0000","last_change":"2023-10-10T17:40:41.165620+0000","last_active":"2023-10-10T17:41:37.602441+0000","last_peered":"2023-10-10T17:41:37.602441+0000","last_clean":"2023-10-10T17:41:37.602441+0000","last_became_active":"2023-10-10T17:40:41.165262+0000","last_became_peered":"2023-10-10T17:40:41.165262+0000","last_unstale":"2023-10-10T17:41:37.602441+0000","last_undegraded":"2023-10-10T17:41:37.602441+0000","last_fullsized":"2023-10-10T17:41:37.602441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:19.126274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599185+0000","last_change":"2023-10-10T17:40:41.603549+0000","last_active":"2023-10-10T17:41:37.599185+0000","last_peered":"2023-10-10T17:41:37.599185+0000","last_clean":"2023-10-10T17:41:37.599185+0000","last_became_active":"2023-10-10T17:40:41.602991+0000","last_became_peered":"2023-10-10T17:40:41.602991+0000","last_unstale":"2023-10-10T17:41:37.599185+0000","last_undegraded":"2023-10-10T17:41:37.599185+0000","last_fullsized":"2023-10-10T17:41:37.599185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:30.283350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.606557+0000","last_change":"2023-10-10T17:41:37.606557+0000","last_active":"2023-10-10T17:41:37.606557+0000","last_peered":"2023-10-10T17:41:37.606557+0000","last_clean":"2023-10-10T17:41:37.606557+0000","last_became_active":"2023-10-10T17:41:37.606070+0000","last_became_peered":"2023-10-10T17:41:37.606070+0000","last_unstale":"2023-10-10T17:41:37.606557+0000","last_undegraded":"2023-10-10T17:41:37.606557+0000","last_fullsized":"2023-10-10T17:41:37.606557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:09:50.492126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595549+0000","last_change":"2023-10-10T17:40:41.169859+0000","last_active":"2023-10-10T17:41:37.595549+0000","last_peered":"2023-10-10T17:41:37.595549+0000","last_clean":"2023-10-10T17:41:37.595549+0000","last_became_active":"2023-10-10T17:40:41.169507+0000","last_became_peered":"2023-10-10T17:40:41.169507+0000","last_unstale":"2023-10-10T17:41:37.595549+0000","last_undegraded":"2023-10-10T17:41:37.595549+0000","last_fullsized":"2023-10-10T17:41:37.595549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:29:19.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600382+0000","last_change":"2023-10-10T17:40:41.593929+0000","last_active":"2023-10-10T17:41:37.600382+0000","last_peered":"2023-10-10T17:41:37.600382+0000","last_clean":"2023-10-10T17:41:37.600382+0000","last_became_active":"2023-10-10T17:40:41.593565+0000","last_became_peered":"2023-10-10T17:40:41.593565+0000","last_unstale":"2023-10-10T17:41:37.600382+0000","last_undegraded":"2023-10-10T17:41:37.600382+0000","last_fullsized":"2023-10-10T17:41:37.600382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:52.950921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595626+0000","last_change":"2023-10-10T17:40:45.710759+0000","last_active":"2023-10-10T17:41:37.595626+0000","last_peered":"2023-10-10T17:41:37.595626+0000","last_clean":"2023-10-10T17:41:37.595626+0000","last_became_active":"2023-10-10T17:40:45.710118+0000","last_became_peered":"2023-10-10T17:40:45.710118+0000","last_unstale":"2023-10-10T17:41:37.595626+0000","last_undegraded":"2023-10-10T17:41:37.595626+0000","last_fullsized":"2023-10-10T17:41:37.595626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:03:22.069829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979760+0000","last_change":"2023-10-10T17:40:41.165988+0000","last_active":"2023-10-10T17:41:37.979760+0000","last_peered":"2023-10-10T17:41:37.979760+0000","last_clean":"2023-10-10T17:41:37.979760+0000","last_became_active":"2023-10-10T17:40:41.165407+0000","last_became_peered":"2023-10-10T17:40:41.165407+0000","last_unstale":"2023-10-10T17:41:37.979760+0000","last_undegraded":"2023-10-10T17:41:37.979760+0000","last_fullsized":"2023-10-10T17:41:37.979760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:16:55.568098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787946+0000","last_change":"2023-10-10T17:40:41.609834+0000","last_active":"2023-10-10T17:41:37.787946+0000","last_peered":"2023-10-10T17:41:37.787946+0000","last_clean":"2023-10-10T17:41:37.787946+0000","last_became_active":"2023-10-10T17:40:41.609509+0000","last_became_peered":"2023-10-10T17:40:41.609509+0000","last_unstale":"2023-10-10T17:41:37.787946+0000","last_undegraded":"2023-10-10T17:41:37.787946+0000","last_fullsized":"2023-10-10T17:41:37.787946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:26:49.739028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602247+0000","last_change":"2023-10-10T17:40:45.709003+0000","last_active":"2023-10-10T17:41:37.602247+0000","last_peered":"2023-10-10T17:41:37.602247+0000","last_clean":"2023-10-10T17:41:37.602247+0000","last_became_active":"2023-10-10T17:40:45.708610+0000","last_became_peered":"2023-10-10T17:40:45.708610+0000","last_unstale":"2023-10-10T17:41:37.602247+0000","last_undegraded":"2023-10-10T17:41:37.602247+0000","last_fullsized":"2023-10-10T17:41:37.602247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:04:55.725864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596673+0000","last_change":"2023-10-10T17:40:41.168104+0000","last_active":"2023-10-10T17:41:37.596673+0000","last_peered":"2023-10-10T17:41:37.596673+0000","last_clean":"2023-10-10T17:41:37.596673+0000","last_became_active":"2023-10-10T17:40:41.167802+0000","last_became_peered":"2023-10-10T17:40:41.167802+0000","last_unstale":"2023-10-10T17:41:37.596673+0000","last_undegraded":"2023-10-10T17:41:37.596673+0000","last_fullsized":"2023-10-10T17:41:37.596673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:22:40.050459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979300+0000","last_change":"2023-10-10T17:40:41.606553+0000","last_active":"2023-10-10T17:41:37.979300+0000","last_peered":"2023-10-10T17:41:37.979300+0000","last_clean":"2023-10-10T17:41:37.979300+0000","last_became_active":"2023-10-10T17:40:41.606087+0000","last_became_peered":"2023-10-10T17:40:41.606087+0000","last_unstale":"2023-10-10T17:41:37.979300+0000","last_undegraded":"2023-10-10T17:41:37.979300+0000","last_fullsized":"2023-10-10T17:41:37.979300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:44:40.056635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602105+0000","last_change":"2023-10-10T17:40:45.712028+0000","last_active":"2023-10-10T17:41:37.602105+0000","last_peered":"2023-10-10T17:41:37.602105+0000","last_clean":"2023-10-10T17:41:37.602105+0000","last_became_active":"2023-10-10T17:40:45.711534+0000","last_became_peered":"2023-10-10T17:40:45.711534+0000","last_unstale":"2023-10-10T17:41:37.602105+0000","last_undegraded":"2023-10-10T17:41:37.602105+0000","last_fullsized":"2023-10-10T17:41:37.602105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:51:36.426627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600532+0000","last_change":"2023-10-10T17:40:41.164456+0000","last_active":"2023-10-10T17:41:37.600532+0000","last_peered":"2023-10-10T17:41:37.600532+0000","last_clean":"2023-10-10T17:41:37.600532+0000","last_became_active":"2023-10-10T17:40:41.164018+0000","last_became_peered":"2023-10-10T17:40:41.164018+0000","last_unstale":"2023-10-10T17:41:37.600532+0000","last_undegraded":"2023-10-10T17:41:37.600532+0000","last_fullsized":"2023-10-10T17:41:37.600532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:54:18.517460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601647+0000","last_change":"2023-10-10T17:40:41.599724+0000","last_active":"2023-10-10T17:41:37.601647+0000","last_peered":"2023-10-10T17:41:37.601647+0000","last_clean":"2023-10-10T17:41:37.601647+0000","last_became_active":"2023-10-10T17:40:41.599356+0000","last_became_peered":"2023-10-10T17:40:41.599356+0000","last_unstale":"2023-10-10T17:41:37.601647+0000","last_undegraded":"2023-10-10T17:41:37.601647+0000","last_fullsized":"2023-10-10T17:41:37.601647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:57.775792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596111+0000","last_change":"2023-10-10T17:40:45.704911+0000","last_active":"2023-10-10T17:41:37.596111+0000","last_peered":"2023-10-10T17:41:37.596111+0000","last_clean":"2023-10-10T17:41:37.596111+0000","last_became_active":"2023-10-10T17:40:45.704299+0000","last_became_peered":"2023-10-10T17:40:45.704299+0000","last_unstale":"2023-10-10T17:41:37.596111+0000","last_undegraded":"2023-10-10T17:41:37.596111+0000","last_fullsized":"2023-10-10T17:41:37.596111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:13:42.759241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597150+0000","last_change":"2023-10-10T17:40:41.166091+0000","last_active":"2023-10-10T17:41:37.597150+0000","last_peered":"2023-10-10T17:41:37.597150+0000","last_clean":"2023-10-10T17:41:37.597150+0000","last_became_active":"2023-10-10T17:40:41.165610+0000","last_became_peered":"2023-10-10T17:40:41.165610+0000","last_unstale":"2023-10-10T17:41:37.597150+0000","last_undegraded":"2023-10-10T17:41:37.597150+0000","last_fullsized":"2023-10-10T17:41:37.597150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:31:09.299708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602252+0000","last_change":"2023-10-10T17:40:41.600837+0000","last_active":"2023-10-10T17:41:37.602252+0000","last_peered":"2023-10-10T17:41:37.602252+0000","last_clean":"2023-10-10T17:41:37.602252+0000","last_became_active":"2023-10-10T17:40:41.600090+0000","last_became_peered":"2023-10-10T17:40:41.600090+0000","last_unstale":"2023-10-10T17:41:37.602252+0000","last_undegraded":"2023-10-10T17:41:37.602252+0000","last_fullsized":"2023-10-10T17:41:37.602252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:15:21.942555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603744+0000","last_change":"2023-10-10T17:41:37.603744+0000","last_active":"2023-10-10T17:41:37.603744+0000","last_peered":"2023-10-10T17:41:37.603744+0000","last_clean":"2023-10-10T17:41:37.603744+0000","last_became_active":"2023-10-10T17:41:37.603376+0000","last_became_peered":"2023-10-10T17:41:37.603376+0000","last_unstale":"2023-10-10T17:41:37.603744+0000","last_undegraded":"2023-10-10T17:41:37.603744+0000","last_fullsized":"2023-10-10T17:41:37.603744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:59:51.305467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652254+0000","last_change":"2023-10-10T17:40:41.169820+0000","last_active":"2023-10-10T17:41:37.652254+0000","last_peered":"2023-10-10T17:41:37.652254+0000","last_clean":"2023-10-10T17:41:37.652254+0000","last_became_active":"2023-10-10T17:40:41.168767+0000","last_became_peered":"2023-10-10T17:40:41.168767+0000","last_unstale":"2023-10-10T17:41:37.652254+0000","last_undegraded":"2023-10-10T17:41:37.652254+0000","last_fullsized":"2023-10-10T17:41:37.652254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:22:55.616442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603091+0000","last_change":"2023-10-10T17:40:41.597482+0000","last_active":"2023-10-10T17:41:37.603091+0000","last_peered":"2023-10-10T17:41:37.603091+0000","last_clean":"2023-10-10T17:41:37.603091+0000","last_became_active":"2023-10-10T17:40:41.597166+0000","last_became_peered":"2023-10-10T17:40:41.597166+0000","last_unstale":"2023-10-10T17:41:37.603091+0000","last_undegraded":"2023-10-10T17:41:37.603091+0000","last_fullsized":"2023-10-10T17:41:37.603091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:11.444261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651261+0000","last_change":"2023-10-10T17:40:45.707175+0000","last_active":"2023-10-10T17:41:37.651261+0000","last_peered":"2023-10-10T17:41:37.651261+0000","last_clean":"2023-10-10T17:41:37.651261+0000","last_became_active":"2023-10-10T17:40:45.706448+0000","last_became_peered":"2023-10-10T17:40:45.706448+0000","last_unstale":"2023-10-10T17:41:37.651261+0000","last_undegraded":"2023-10-10T17:41:37.651261+0000","last_fullsized":"2023-10-10T17:41:37.651261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:33.970113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600136+0000","last_change":"2023-10-10T17:40:41.169806+0000","last_active":"2023-10-10T17:41:37.600136+0000","last_peered":"2023-10-10T17:41:37.600136+0000","last_clean":"2023-10-10T17:41:37.600136+0000","last_became_active":"2023-10-10T17:40:41.169496+0000","last_became_peered":"2023-10-10T17:40:41.169496+0000","last_unstale":"2023-10-10T17:41:37.600136+0000","last_undegraded":"2023-10-10T17:41:37.600136+0000","last_fullsized":"2023-10-10T17:41:37.600136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:28:16.024298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603755+0000","last_change":"2023-10-10T17:40:41.598366+0000","last_active":"2023-10-10T17:41:37.603755+0000","last_peered":"2023-10-10T17:41:37.603755+0000","last_clean":"2023-10-10T17:41:37.603755+0000","last_became_active":"2023-10-10T17:40:41.598036+0000","last_became_peered":"2023-10-10T17:40:41.598036+0000","last_unstale":"2023-10-10T17:41:37.603755+0000","last_undegraded":"2023-10-10T17:41:37.603755+0000","last_fullsized":"2023-10-10T17:41:37.603755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:59:22.632882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787912+0000","last_change":"2023-10-10T17:40:41.609923+0000","last_active":"2023-10-10T17:41:37.787912+0000","last_peered":"2023-10-10T17:41:37.787912+0000","last_clean":"2023-10-10T17:41:37.787912+0000","last_became_active":"2023-10-10T17:40:41.609575+0000","last_became_peered":"2023-10-10T17:40:41.609575+0000","last_unstale":"2023-10-10T17:41:37.787912+0000","last_undegraded":"2023-10-10T17:41:37.787912+0000","last_fullsized":"2023-10-10T17:41:37.787912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:14:10.560266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596871+0000","last_change":"2023-10-10T17:40:41.609012+0000","last_active":"2023-10-10T17:41:37.596871+0000","last_peered":"2023-10-10T17:41:37.596871+0000","last_clean":"2023-10-10T17:41:37.596871+0000","last_became_active":"2023-10-10T17:40:41.608738+0000","last_became_peered":"2023-10-10T17:40:41.608738+0000","last_unstale":"2023-10-10T17:41:37.596871+0000","last_undegraded":"2023-10-10T17:41:37.596871+0000","last_fullsized":"2023-10-10T17:41:37.596871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:11:39.107240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596125+0000","last_change":"2023-10-10T17:40:41.608801+0000","last_active":"2023-10-10T17:41:37.596125+0000","last_peered":"2023-10-10T17:41:37.596125+0000","last_clean":"2023-10-10T17:41:37.596125+0000","last_became_active":"2023-10-10T17:40:41.608510+0000","last_became_peered":"2023-10-10T17:40:41.608510+0000","last_unstale":"2023-10-10T17:41:37.596125+0000","last_undegraded":"2023-10-10T17:41:37.596125+0000","last_fullsized":"2023-10-10T17:41:37.596125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:16:53.945359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598418+0000","last_change":"2023-10-10T17:40:41.603516+0000","last_active":"2023-10-10T17:41:37.598418+0000","last_peered":"2023-10-10T17:41:37.598418+0000","last_clean":"2023-10-10T17:41:37.598418+0000","last_became_active":"2023-10-10T17:40:41.602948+0000","last_became_peered":"2023-10-10T17:40:41.602948+0000","last_unstale":"2023-10-10T17:41:37.598418+0000","last_undegraded":"2023-10-10T17:41:37.598418+0000","last_fullsized":"2023-10-10T17:41:37.598418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:04.694328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602112+0000","last_change":"2023-10-10T17:41:37.602112+0000","last_active":"2023-10-10T17:41:37.602112+0000","last_peered":"2023-10-10T17:41:37.602112+0000","last_clean":"2023-10-10T17:41:37.602112+0000","last_became_active":"2023-10-10T17:41:37.601781+0000","last_became_peered":"2023-10-10T17:41:37.601781+0000","last_unstale":"2023-10-10T17:41:37.602112+0000","last_undegraded":"2023-10-10T17:41:37.602112+0000","last_fullsized":"2023-10-10T17:41:37.602112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:00.565768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.597595+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.597301+0000","last_became_peered":"2023-10-10T17:40:41.597301+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:51:11.601251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596929+0000","last_change":"2023-10-10T17:40:41.611841+0000","last_active":"2023-10-10T17:41:37.596929+0000","last_peered":"2023-10-10T17:41:37.596929+0000","last_clean":"2023-10-10T17:41:37.596929+0000","last_became_active":"2023-10-10T17:40:41.611438+0000","last_became_peered":"2023-10-10T17:40:41.611438+0000","last_unstale":"2023-10-10T17:41:37.596929+0000","last_undegraded":"2023-10-10T17:41:37.596929+0000","last_fullsized":"2023-10-10T17:41:37.596929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:29:05.260738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979112+0000","last_change":"2023-10-10T17:40:41.607707+0000","last_active":"2023-10-10T17:41:37.979112+0000","last_peered":"2023-10-10T17:41:37.979112+0000","last_clean":"2023-10-10T17:41:37.979112+0000","last_became_active":"2023-10-10T17:40:41.607412+0000","last_became_peered":"2023-10-10T17:40:41.607412+0000","last_unstale":"2023-10-10T17:41:37.979112+0000","last_undegraded":"2023-10-10T17:41:37.979112+0000","last_fullsized":"2023-10-10T17:41:37.979112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:09:49.015272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651033+0000","last_change":"2023-10-10T17:40:41.593674+0000","last_active":"2023-10-10T17:41:37.651033+0000","last_peered":"2023-10-10T17:41:37.651033+0000","last_clean":"2023-10-10T17:41:37.651033+0000","last_became_active":"2023-10-10T17:40:41.593339+0000","last_became_peered":"2023-10-10T17:40:41.593339+0000","last_unstale":"2023-10-10T17:41:37.651033+0000","last_undegraded":"2023-10-10T17:41:37.651033+0000","last_fullsized":"2023-10-10T17:41:37.651033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:39:37.705884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978977+0000","last_change":"2023-10-10T17:40:41.610075+0000","last_active":"2023-10-10T17:41:37.978977+0000","last_peered":"2023-10-10T17:41:37.978977+0000","last_clean":"2023-10-10T17:41:37.978977+0000","last_became_active":"2023-10-10T17:40:41.609669+0000","last_became_peered":"2023-10-10T17:40:41.609669+0000","last_unstale":"2023-10-10T17:41:37.978977+0000","last_undegraded":"2023-10-10T17:41:37.978977+0000","last_fullsized":"2023-10-10T17:41:37.978977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:42.281285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596551+0000","last_change":"2023-10-10T17:40:41.596780+0000","last_active":"2023-10-10T17:41:37.596551+0000","last_peered":"2023-10-10T17:41:37.596551+0000","last_clean":"2023-10-10T17:41:37.596551+0000","last_became_active":"2023-10-10T17:40:41.596477+0000","last_became_peered":"2023-10-10T17:40:41.596477+0000","last_unstale":"2023-10-10T17:41:37.596551+0000","last_undegraded":"2023-10-10T17:41:37.596551+0000","last_fullsized":"2023-10-10T17:41:37.596551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:51:14.200732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598899+0000","last_change":"2023-10-10T17:40:41.606226+0000","last_active":"2023-10-10T17:41:37.598899+0000","last_peered":"2023-10-10T17:41:37.598899+0000","last_clean":"2023-10-10T17:41:37.598899+0000","last_became_active":"2023-10-10T17:40:41.605843+0000","last_became_peered":"2023-10-10T17:40:41.605843+0000","last_unstale":"2023-10-10T17:41:37.598899+0000","last_undegraded":"2023-10-10T17:41:37.598899+0000","last_fullsized":"2023-10-10T17:41:37.598899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:44.165622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788525+0000","last_change":"2023-10-10T17:40:41.596122+0000","last_active":"2023-10-10T17:41:37.788525+0000","last_peered":"2023-10-10T17:41:37.788525+0000","last_clean":"2023-10-10T17:41:37.788525+0000","last_became_active":"2023-10-10T17:40:41.595818+0000","last_became_peered":"2023-10-10T17:40:41.595818+0000","last_unstale":"2023-10-10T17:41:37.788525+0000","last_undegraded":"2023-10-10T17:41:37.788525+0000","last_fullsized":"2023-10-10T17:41:37.788525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:24:32.658064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250871,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7568,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":512,"num_read_kb":270,"num_write":131,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":22704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":987136,"data_stored":492386,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995}]},{"osd":1,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.75800000000000001,"15min":0.77700000000000002},"min":{"1min":0.46899999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90200000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.68600000000000005,"15min":0.69599999999999995},"min":{"1min":0.45700000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.49299999999999999,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66000000000000003,"5min":1.026,"15min":1.026},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.80600000000000005,"15min":0.82599999999999996},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.95799999999999996,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97499999999999998,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.80300000000000005}]},{"osd":5,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.628,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81599999999999995,"5min":1.129,"15min":1.129},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66800000000000004},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85999999999999999,"5min":1.107,"15min":1.107},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80900000000000005,"15min":0.81999999999999995},"min":{"1min":0.51300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.01,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.81200000000000006,"15min":0.82799999999999996},"min":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0309999999999999,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.85299999999999998}]},{"osd":8,"last update":"Tue Oct 10 17:41:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.75800000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.72699999999999998,"15min":0.72899999999999998},"min":{"1min":0.47599999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.93600000000000005,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.752,"15min":0.753},"min":{"1min":0.52100000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.05,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.83799999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.97599999999999998,"15min":1.01},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.117,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84599999999999997,"15min":0.85899999999999999},"min":{"1min":0.61399999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0720000000000001,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.82699999999999996}]}]},{"osd":10,"up_from":73,"seq":313532612643,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6432,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738720,"statfs":{"total":95995035648,"available":95988449280,"internally_reserved":0,"allocated":1073152,"data_stored":1087425,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.435}]},{"osd":1,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.32700000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.47399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78200000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.51500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.80200000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68600000000000005}]},{"osd":3,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76700000000000002,"5min":1.071,"15min":1.071},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.501}]},{"osd":4,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.63}]},{"osd":5,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.65400000000000003},"min":{"1min":0.52900000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.81599999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73499999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.56699999999999995,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90100000000000002,"5min":1.05,"15min":1.05},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.70899999999999996}]},{"osd":8,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70399999999999996,"15min":0.69999999999999996},"min":{"1min":0.60299999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.85499999999999998,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.753},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91900000000000004,"5min":1.157,"15min":1.157},"last":0.76300000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":1.2,"15min":1.2},"last":0.80800000000000005}]},{"osd":11,"last update":"Tue Oct 10 17:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76300000000000001,"15min":0.76100000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77100000000000002,"15min":0.77000000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.94299999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.78300000000000003}]}]},{"osd":9,"up_from":68,"seq":292057776169,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":1003520,"data_stored":492622,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.373},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48199999999999998,"15min":0.502},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54000000000000004,"5min":1.016,"15min":1.016},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85799999999999998,"5min":1.103,"15min":1.103},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.755,"5min":1.03,"15min":1.03},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59899999999999998,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69199999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70199999999999996,"5min":1.151,"15min":1.151},"last":0.5}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.54000000000000004,"15min":0.56299999999999994},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.62,"5min":0.748,"15min":0.748},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.54600000000000004,"15min":0.56899999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64300000000000002,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.38700000000000001}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73799999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68999999999999995,"15min":0.70999999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.69599999999999995,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.436,"15min":0.436},"max":{"1min":0.77000000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69899999999999995,"15min":0.72099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.71299999999999997,"5min":1.417,"15min":1.417},"last":0.48099999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.52600000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.76700000000000002,"15min":0.79100000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.80900000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67000000000000004}]},{"osd":8,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.753,"15min":0.77500000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77600000000000002,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.78900000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79400000000000004,"5min":1.377,"15min":1.377},"last":0.67800000000000005}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69799999999999995,"15min":0.70999999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.83399999999999996,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71699999999999997,"15min":0.72299999999999998},"min":{"1min":0.52900000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.876,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64500000000000002}]},{"osd":11,"last update":"Tue Oct 10 17:41:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.65000000000000002}]}]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6492,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738660,"statfs":{"total":95995035648,"available":95988387840,"internally_reserved":0,"allocated":1003520,"data_stored":492542,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.436,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.64400000000000002,"15min":0.58399999999999996},"min":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.61899999999999999}]},{"osd":1,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58199999999999996,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84599999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.624,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86099999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.42799999999999999,"15min":0.40000000000000002},"min":{"1min":0.29399999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64100000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52200000000000002}]},{"osd":3,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.57999999999999996},"min":{"1min":0.439,"5min":0.37,"15min":0.37},"max":{"1min":0.94799999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.012,"5min":0.65500000000000003,"15min":0.58999999999999997},"min":{"1min":0.52100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.68799999999999994}]},{"osd":4,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.60999999999999999,"15min":0.58799999999999997},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.63100000000000001,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72999999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.33600000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.56499999999999995,"15min":0.53100000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93799999999999994,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.69899999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.93799999999999994,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.70899999999999996,"15min":0.69399999999999995},"min":{"1min":0.56299999999999994,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90400000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.77200000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.71599999999999997,"15min":0.68700000000000006},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.999,"5min":1.022,"15min":1.022},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.69899999999999995,"15min":0.67300000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91900000000000004,"5min":1.101,"15min":1.101},"last":0.74299999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79500000000000004,"15min":0.72499999999999998},"min":{"1min":0.58499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.012,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.76400000000000001,"15min":0.73999999999999999},"min":{"1min":0.58499999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.036,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.83499999999999996}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.751,"15min":0.73199999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.87},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.73699999999999999,"15min":0.71499999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.88600000000000001}]},{"osd":11,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.76000000000000001}]}]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1044480,"data_stored":1083855,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72099999999999997,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.374}]},{"osd":1,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40899999999999997,"15min":0.39300000000000002},"min":{"1min":0.28699999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72099999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.78000000000000003,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70099999999999996}]},{"osd":3,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.59799999999999998,"15min":0.58899999999999997},"min":{"1min":0.48399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.755,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46200000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.312,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.495},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48099999999999998,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.432}]},{"osd":5,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.442,"5min":0.441,"15min":0.441},"max":{"1min":0.85499999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64000000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.374}]},{"osd":8,"last update":"Tue Oct 10 17:42:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.70699999999999996,"15min":0.69099999999999995},"min":{"1min":0.623,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68200000000000005,"15min":0.67200000000000004},"min":{"1min":0.58799999999999997,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.625,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.754,"15min":0.74399999999999999},"min":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.77300000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.751,"15min":0.745},"min":{"1min":0.55600000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92300000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48199999999999998}]},{"osd":11,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66800000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":987136,"data_stored":491894,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.375,"15min":0.375},"min":{"1min":0.27700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.41299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.38500000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83099999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54100000000000004}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.442,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.33600000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.441},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57399999999999995,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.50800000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.57799999999999996},"min":{"1min":0.46400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89500000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.627}]},{"osd":8,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65800000000000003,"15min":0.66400000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.60399999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76500000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.42899999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:41:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.73199999999999998,"15min":0.74299999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.501},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.73899999999999999,"15min":0.753},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.875,"5min":1.042,"15min":1.042},"last":0.59099999999999997}]},{"osd":11,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":995328,"data_stored":495598,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57999999999999996,"15min":0.67700000000000005},"min":{"1min":0.39900000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80400000000000005,"5min":5.859,"15min":5.859},"last":0.44400000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.58099999999999996,"15min":0.51900000000000002},"min":{"1min":0.46200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.88300000000000001,"5min":1,"15min":1},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.623,"15min":0.60899999999999999},"min":{"1min":0.47299999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86899999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.84499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.45500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.72699999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80500000000000005,"5min":0.875,"15min":0.875},"last":0.627}]},{"osd":4,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.45800000000000002,"15min":0.40400000000000003},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.46999999999999997,"15min":0.41499999999999998},"min":{"1min":0.38700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.501}]},{"osd":5,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59499999999999997,"15min":0.55700000000000005},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.60799999999999998,"15min":0.55600000000000005},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.70199999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:42:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59299999999999997},"min":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63900000000000001,"15min":0.61299999999999999},"min":{"1min":0.45200000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88800000000000001,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.59999999999999998,"15min":0.55700000000000005},"min":{"1min":0.42199999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.495,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.76700000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67300000000000004,"15min":0.65100000000000002},"min":{"1min":0.54500000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.67700000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.99099999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.85799999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.76300000000000001,"15min":0.749},"min":{"1min":0.60999999999999999,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.91900000000000004,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73399999999999999,"15min":0.71399999999999997},"min":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.01,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.871}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75800000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.77500000000000002,"15min":0.76300000000000001},"min":{"1min":0.57599999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.008,"15min":1.008},"last":0.754}]},{"osd":11,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.89000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7396,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737756,"statfs":{"total":95995035648,"available":95987462144,"internally_reserved":0,"allocated":1011712,"data_stored":493282,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 17:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54500000000000004,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.49199999999999999}]},{"osd":3,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.274,"5min":7.274,"15min":7.274},"last":0.57799999999999996}]},{"osd":4,"last update":"Tue Oct 10 17:41:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55600000000000005,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54500000000000004,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.53100000000000003},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.75800000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63100000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":0.65200000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.622}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68000000000000005}]},{"osd":10,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.75},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.72799999999999998},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.59799999999999998}]},{"osd":11,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67200000000000004,"15min":0.66700000000000004},"min":{"1min":0.53000000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.70599999999999996,"15min":0.70199999999999996},"min":{"1min":0.51800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.86599999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7456,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737696,"statfs":{"total":95995035648,"available":95987400704,"internally_reserved":0,"allocated":1007616,"data_stored":496169,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":0.88700000000000001,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.49199999999999999,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68300000000000005,"5min":1.177,"15min":1.177},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60099999999999998,"15min":0.53100000000000003},"min":{"1min":0.50900000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81399999999999995,"5min":5.484,"15min":5.484},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.56699999999999995,"15min":0.53400000000000003},"min":{"1min":0.46500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56999999999999995,"15min":0.52500000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.58599999999999997,"15min":0.54400000000000004},"min":{"1min":0.51300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84799999999999998,"5min":1.099,"15min":1.099},"last":0.56699999999999995}]},{"osd":4,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.64300000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.48599999999999999,"15min":0.42299999999999999},"min":{"1min":0.39900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.435},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.52000000000000002,"15min":0.46800000000000003},"min":{"1min":0.38300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.41499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.97899999999999998,"5min":1.129,"15min":1.129},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92300000000000004,"5min":1.075,"15min":1.075},"last":0.504}]},{"osd":7,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.755},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.84499999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.627}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93600000000000005,"5min":1.087,"15min":1.087},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.996,"5min":1.226,"15min":1.226},"last":0.79100000000000004}]},{"osd":10,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.66100000000000003,"15min":0.64400000000000002},"min":{"1min":0.56599999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.70599999999999996,"15min":0.68300000000000005},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.83099999999999996}]},{"osd":11,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.879},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.80900000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6740,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738412,"statfs":{"total":95995035648,"available":95988133888,"internally_reserved":0,"allocated":995328,"data_stored":491757,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35499999999999998,"15min":0.34799999999999998},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.621,"5min":0.752,"15min":0.752},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56499999999999995,"15min":0.53900000000000003},"min":{"1min":0.39200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55800000000000005,"15min":0.52500000000000002},"min":{"1min":0.40300000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.60799999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56599999999999995,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73099999999999998,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57099999999999995,"15min":0.54100000000000004},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80000000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Tue Oct 10 17:41:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.41099999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84099999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.627},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61299999999999999,"15min":0.59299999999999997},"min":{"1min":0.46000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.746,"5min":1.214,"15min":1.214},"last":0.753}]},{"osd":5,"last update":"Tue Oct 10 17:42:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58699999999999997,"15min":0.59799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78900000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76100000000000001,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.67600000000000005}]},{"osd":6,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.5,"15min":0.50800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.621,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.80200000000000005}]},{"osd":8,"last update":"Tue Oct 10 17:39:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53700000000000003}]},{"osd":9,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.624,"15min":0.65400000000000003},"min":{"1min":0.377,"5min":0.317,"15min":0.317},"max":{"1min":0.69899999999999995,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64400000000000002,"15min":0.68200000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68999999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54200000000000004}]},{"osd":10,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.76300000000000001}]},{"osd":11,"last update":"Tue Oct 10 17:41:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.70799999999999996}]}]},{"osd":4,"up_from":34,"seq":146028888127,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1003520,"data_stored":492772,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.432,"5min":0.372,"15min":0.372},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.36599999999999999},"min":{"1min":0.31,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65200000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.40799999999999997,"15min":0.372},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.63400000000000001,"15min":0.57499999999999996},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85099999999999998}]},{"osd":3,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.60899999999999999,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.1289999999999996,"5min":6.1289999999999996,"15min":6.1289999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.75900000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.626,"15min":0.58599999999999997},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":1.589,"15min":1.589},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.56100000000000005,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.69499999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.499,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95399999999999996,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.57299999999999995,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.61799999999999999,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.76900000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.46899999999999997,"5min":0.432,"15min":0.432},"max":{"1min":0.93400000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76100000000000001,"15min":0.71799999999999997},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.006,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.83899999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.73199999999999998,"15min":0.71499999999999997},"min":{"1min":0.627,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.95899999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.78100000000000003,"15min":0.76600000000000001},"min":{"1min":0.61699999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.98999999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.93799999999999994}]},{"osd":10,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68200000000000005,"15min":0.67400000000000004},"min":{"1min":0.52300000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.80400000000000005,"15min":0.79600000000000004},"min":{"1min":0.56100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.038,"5min":1.169,"15min":1.169},"last":0.89900000000000002}]},{"osd":11,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.88600000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6736,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738416,"statfs":{"total":95995035648,"available":95988137984,"internally_reserved":0,"allocated":1056768,"data_stored":1083327,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.57599999999999996,"15min":0.52200000000000002},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.79600000000000004,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69999999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68700000000000006,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.55400000000000005},"min":{"1min":0.497,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71599999999999997,"5min":7.0570000000000004,"15min":7.0570000000000004},"last":0.61699999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.40899999999999997,"15min":0.40100000000000002},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46100000000000002,"15min":0.42499999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56899999999999995,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.438},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.56999999999999995,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83799999999999997,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65100000000000002,"15min":0.59199999999999997},"min":{"1min":0.438,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76300000000000001,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73499999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.70499999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.432,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.504,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.624},"min":{"1min":0.54300000000000004,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66000000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.55600000000000005},"min":{"1min":0.47399999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":6.7229999999999999,"15min":6.7229999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81899999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64700000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70899999999999996,"15min":0.71299999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90300000000000002,"5min":1.034,"15min":1.034},"last":0.751},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.71299999999999997,"15min":0.69899999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":0.76500000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.56200000000000006,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88100000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70799999999999996,"15min":0.70299999999999996},"min":{"1min":0.65200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T17:42:25.332 INFO:tasks.fwd_scrub.ceph_manager:clean! 2023-10-10T17:42:25.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-10T17:42:25.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 -- 172.21.15.78:0/196467030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 msgr2=0x7f33100ff8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 --2- 172.21.15.78:0/196467030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100ff8f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f330c009990 tx=0x7f330c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 -- 172.21.15.78:0/196467030 shutdown_connections 2023-10-10T17:42:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 --2- 172.21.15.78:0/196467030 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33100ffe30 0x7f33101076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 --2- 172.21.15.78:0/196467030 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3310100790 0x7f3310100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 --2- 172.21.15.78:0/196467030 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100ff8f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 -- 172.21.15.78:0/196467030 >> 172.21.15.78:0/196467030 conn(0x7f33100f9ee0 msgr2=0x7f33100fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 -- 172.21.15.78:0/196467030 shutdown_connections 2023-10-10T17:42:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.484+0000 7f3317a82700 1 -- 172.21.15.78:0/196467030 wait complete. 2023-10-10T17:42:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 Processor -- start 2023-10-10T17:42:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 -- start start 2023-10-10T17:42:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100feee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33100ffe30 0x7f331019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3310100790 0x7f331019d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331019db30 con 0x7f33100ff480 2023-10-10T17:42:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f331581e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100feee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331019dca0 con 0x7f3310100790 2023-10-10T17:42:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f3317a82700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331019de60 con 0x7f33100ffe30 2023-10-10T17:42:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f331581e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100feee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:43998/0 (socket says 172.21.15.78:43998) 2023-10-10T17:42:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f331581e700 1 -- 172.21.15.78:0/1987231445 learned_addr learned my addr 172.21.15.78:0/1987231445 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f331501d700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33100ffe30 0x7f331019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.485+0000 7f331601f700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3310100790 0x7f331019d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331581e700 1 -- 172.21.15.78:0/1987231445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33100ffe30 msgr2=0x7f331019ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331581e700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33100ffe30 0x7f331019ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331581e700 1 -- 172.21.15.78:0/1987231445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3310100790 msgr2=0x7f331019d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331581e700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3310100790 0x7f331019d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331581e700 1 -- 172.21.15.78:0/1987231445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0095f0 con 0x7f33100ff480 2023-10-10T17:42:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331501d700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33100ffe30 0x7f331019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331601f700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3310100790 0x7f331019d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f331581e700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100feee0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f330000d980 tx=0x7f330000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f3306ffd700 1 -- 172.21.15.78:0/1987231445 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300009b40 con 0x7f33100ff480 2023-10-10T17:42:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f3306ffd700 1 -- 172.21.15.78:0/1987231445 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f330000de60 con 0x7f33100ff480 2023-10-10T17:42:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f331010ab30 con 0x7f33100ff480 2023-10-10T17:42:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f3306ffd700 1 -- 172.21.15.78:0/1987231445 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330000f620 con 0x7f33100ff480 2023-10-10T17:42:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.486+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331010a9a0 con 0x7f33100ff480 2023-10-10T17:42:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.487+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3310106880 con 0x7f33100ff480 2023-10-10T17:42:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.488+0000 7f3306ffd700 1 -- 172.21.15.78:0/1987231445 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3300010460 con 0x7f33100ff480 2023-10-10T17:42:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.489+0000 7f3306ffd700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f32fc0a1150 0x7f32fc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.489+0000 7f331501d700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f32fc0a1150 0x7f32fc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.489+0000 7f3306ffd700 1 -- 172.21.15.78:0/1987231445 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f33000cc7c0 con 0x7f33100ff480 2023-10-10T17:42:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.489+0000 7f331501d700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f32fc0a1150 0x7f32fc0a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f330c00b2e0 tx=0x7f330c005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.492+0000 7f3306ffd700 1 -- 172.21.15.78:0/1987231445 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3300095c00 con 0x7f33100ff480 2023-10-10T17:42:25.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.683+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3310061c70 con 0x7f32fc0a1150 2023-10-10T17:42:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.695+0000 7f3306ffd700 1 -- 172.21.15.78:0/1987231445 <== mgr.14152 v2:172.21.15.78:6800/321866266 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427292 (secure 0 0 0) 0x7f3310061c70 con 0x7f32fc0a1150 2023-10-10T17:42:25.705 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:25.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.698+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f32fc0a1150 msgr2=0x7f32fc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f32fc0a1150 0x7f32fc0a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f330c00b2e0 tx=0x7f330c005e80 comp rx=0 tx=0).stop 2023-10-10T17:42:25.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 msgr2=0x7f33100feee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100feee0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f330000d980 tx=0x7f330000dd40 comp rx=0 tx=0).stop 2023-10-10T17:42:25.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 shutdown_connections 2023-10-10T17:42:25.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f32fc0a1150 0x7f32fc0a3600 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3310100790 0x7f331019d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f33100ff480 0x7f33100feee0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 --2- 172.21.15.78:0/1987231445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f33100ffe30 0x7f331019ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 >> 172.21.15.78:0/1987231445 conn(0x7f33100f9ee0 msgr2=0x7f33101039c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:25.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 shutdown_connections 2023-10-10T17:42:25.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.699+0000 7f3317a82700 1 -- 172.21.15.78:0/1987231445 wait complete. 2023-10-10T17:42:25.712 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-10-10T17:42:25.713 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2023-10-10T17:42:24.243511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597936,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":648,"num_read_kb":389,"num_write":312,"num_write_kb":2983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":252,"ondisk_log_size":252,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":80844,"kb_used_data":11884,"kb_used_omap":0,"kb_used_meta":68864,"kb_avail":1124860980,"statfs":{"total":1151940427776,"available":1151857643520,"internally_reserved":0,"allocated":12169216,"data_stored":7693629,"data_compressed":319365,"data_compressed_allocated":430080,"data_compressed_original":2359296,"omap_allocated":0,"internal_metadata":70516736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":260,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":86,"num_read_kb":46,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003374"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788069+0000","last_change":"2023-10-10T17:40:41.604268+0000","last_active":"2023-10-10T17:41:37.788069+0000","last_peered":"2023-10-10T17:41:37.788069+0000","last_clean":"2023-10-10T17:41:37.788069+0000","last_became_active":"2023-10-10T17:40:41.603839+0000","last_became_peered":"2023-10-10T17:40:41.603839+0000","last_unstale":"2023-10-10T17:41:37.788069+0000","last_undegraded":"2023-10-10T17:41:37.788069+0000","last_fullsized":"2023-10-10T17:41:37.788069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:00:22.805728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599055+0000","last_change":"2023-10-10T17:40:41.603642+0000","last_active":"2023-10-10T17:41:37.599055+0000","last_peered":"2023-10-10T17:41:37.599055+0000","last_clean":"2023-10-10T17:41:37.599055+0000","last_became_active":"2023-10-10T17:40:41.603091+0000","last_became_peered":"2023-10-10T17:40:41.603091+0000","last_unstale":"2023-10-10T17:41:37.599055+0000","last_undegraded":"2023-10-10T17:41:37.599055+0000","last_fullsized":"2023-10-10T17:41:37.599055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:10:38.807138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598740+0000","last_change":"2023-10-10T17:40:41.610647+0000","last_active":"2023-10-10T17:41:37.598740+0000","last_peered":"2023-10-10T17:41:37.598740+0000","last_clean":"2023-10-10T17:41:37.598740+0000","last_became_active":"2023-10-10T17:40:41.610338+0000","last_became_peered":"2023-10-10T17:40:41.610338+0000","last_unstale":"2023-10-10T17:41:37.598740+0000","last_undegraded":"2023-10-10T17:41:37.598740+0000","last_fullsized":"2023-10-10T17:41:37.598740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:16:20.110335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602832+0000","last_change":"2023-10-10T17:40:41.607832+0000","last_active":"2023-10-10T17:41:37.602832+0000","last_peered":"2023-10-10T17:41:37.602832+0000","last_clean":"2023-10-10T17:41:37.602832+0000","last_became_active":"2023-10-10T17:40:41.607492+0000","last_became_peered":"2023-10-10T17:40:41.607492+0000","last_unstale":"2023-10-10T17:41:37.602832+0000","last_undegraded":"2023-10-10T17:41:37.602832+0000","last_fullsized":"2023-10-10T17:41:37.602832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:55:43.514129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600669+0000","last_change":"2023-10-10T17:40:41.601975+0000","last_active":"2023-10-10T17:41:37.600669+0000","last_peered":"2023-10-10T17:41:37.600669+0000","last_clean":"2023-10-10T17:41:37.600669+0000","last_became_active":"2023-10-10T17:40:41.601668+0000","last_became_peered":"2023-10-10T17:40:41.601668+0000","last_unstale":"2023-10-10T17:41:37.600669+0000","last_undegraded":"2023-10-10T17:41:37.600669+0000","last_fullsized":"2023-10-10T17:41:37.600669+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:25.433615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.604098+0000","last_change":"2023-10-10T17:41:37.604098+0000","last_active":"2023-10-10T17:41:37.604098+0000","last_peered":"2023-10-10T17:41:37.604098+0000","last_clean":"2023-10-10T17:41:37.604098+0000","last_became_active":"2023-10-10T17:41:37.603750+0000","last_became_peered":"2023-10-10T17:41:37.603750+0000","last_unstale":"2023-10-10T17:41:37.604098+0000","last_undegraded":"2023-10-10T17:41:37.604098+0000","last_fullsized":"2023-10-10T17:41:37.604098+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:52:52.491557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602564+0000","last_change":"2023-10-10T17:40:41.594628+0000","last_active":"2023-10-10T17:41:37.602564+0000","last_peered":"2023-10-10T17:41:37.602564+0000","last_clean":"2023-10-10T17:41:37.602564+0000","last_became_active":"2023-10-10T17:40:41.594325+0000","last_became_peered":"2023-10-10T17:40:41.594325+0000","last_unstale":"2023-10-10T17:41:37.602564+0000","last_undegraded":"2023-10-10T17:41:37.602564+0000","last_fullsized":"2023-10-10T17:41:37.602564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:09:20.447064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978940+0000","last_change":"2023-10-10T17:40:41.598032+0000","last_active":"2023-10-10T17:41:37.978940+0000","last_peered":"2023-10-10T17:41:37.978940+0000","last_clean":"2023-10-10T17:41:37.978940+0000","last_became_active":"2023-10-10T17:40:41.597671+0000","last_became_peered":"2023-10-10T17:40:41.597671+0000","last_unstale":"2023-10-10T17:41:37.978940+0000","last_undegraded":"2023-10-10T17:41:37.978940+0000","last_fullsized":"2023-10-10T17:41:37.978940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:56:22.583298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598085+0000","last_change":"2023-10-10T17:40:41.599770+0000","last_active":"2023-10-10T17:41:37.598085+0000","last_peered":"2023-10-10T17:41:37.598085+0000","last_clean":"2023-10-10T17:41:37.598085+0000","last_became_active":"2023-10-10T17:40:41.599449+0000","last_became_peered":"2023-10-10T17:40:41.599449+0000","last_unstale":"2023-10-10T17:41:37.598085+0000","last_undegraded":"2023-10-10T17:41:37.598085+0000","last_fullsized":"2023-10-10T17:41:37.598085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:25:18.482096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598837+0000","last_change":"2023-10-10T17:40:41.610655+0000","last_active":"2023-10-10T17:41:37.598837+0000","last_peered":"2023-10-10T17:41:37.598837+0000","last_clean":"2023-10-10T17:41:37.598837+0000","last_became_active":"2023-10-10T17:40:41.610133+0000","last_became_peered":"2023-10-10T17:40:41.610133+0000","last_unstale":"2023-10-10T17:41:37.598837+0000","last_undegraded":"2023-10-10T17:41:37.598837+0000","last_fullsized":"2023-10-10T17:41:37.598837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:32:12.198400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602157+0000","last_change":"2023-10-10T17:40:41.601065+0000","last_active":"2023-10-10T17:41:37.602157+0000","last_peered":"2023-10-10T17:41:37.602157+0000","last_clean":"2023-10-10T17:41:37.602157+0000","last_became_active":"2023-10-10T17:40:41.600530+0000","last_became_peered":"2023-10-10T17:40:41.600530+0000","last_unstale":"2023-10-10T17:41:37.602157+0000","last_undegraded":"2023-10-10T17:41:37.602157+0000","last_fullsized":"2023-10-10T17:41:37.602157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:48:30.086020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602647+0000","last_change":"2023-10-10T17:40:41.596067+0000","last_active":"2023-10-10T17:41:37.602647+0000","last_peered":"2023-10-10T17:41:37.602647+0000","last_clean":"2023-10-10T17:41:37.602647+0000","last_became_active":"2023-10-10T17:40:41.595703+0000","last_became_peered":"2023-10-10T17:40:41.595703+0000","last_unstale":"2023-10-10T17:41:37.602647+0000","last_undegraded":"2023-10-10T17:41:37.602647+0000","last_fullsized":"2023-10-10T17:41:37.602647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:29:13.451772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598863+0000","last_change":"2023-10-10T17:40:41.593296+0000","last_active":"2023-10-10T17:41:37.598863+0000","last_peered":"2023-10-10T17:41:37.598863+0000","last_clean":"2023-10-10T17:41:37.598863+0000","last_became_active":"2023-10-10T17:40:41.592965+0000","last_became_peered":"2023-10-10T17:40:41.592965+0000","last_unstale":"2023-10-10T17:41:37.598863+0000","last_undegraded":"2023-10-10T17:41:37.598863+0000","last_fullsized":"2023-10-10T17:41:37.598863+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:30:41.515067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598115+0000","last_change":"2023-10-10T17:40:41.604542+0000","last_active":"2023-10-10T17:41:37.598115+0000","last_peered":"2023-10-10T17:41:37.598115+0000","last_clean":"2023-10-10T17:41:37.598115+0000","last_became_active":"2023-10-10T17:40:41.604000+0000","last_became_peered":"2023-10-10T17:40:41.604000+0000","last_unstale":"2023-10-10T17:41:37.598115+0000","last_undegraded":"2023-10-10T17:41:37.598115+0000","last_fullsized":"2023-10-10T17:41:37.598115+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:42:32.658592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602108+0000","last_change":"2023-10-10T17:41:37.602108+0000","last_active":"2023-10-10T17:41:37.602108+0000","last_peered":"2023-10-10T17:41:37.602108+0000","last_clean":"2023-10-10T17:41:37.602108+0000","last_became_active":"2023-10-10T17:41:37.601778+0000","last_became_peered":"2023-10-10T17:41:37.601778+0000","last_unstale":"2023-10-10T17:41:37.602108+0000","last_undegraded":"2023-10-10T17:41:37.602108+0000","last_fullsized":"2023-10-10T17:41:37.602108+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:48:51.539525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595527+0000","last_change":"2023-10-10T17:40:41.606659+0000","last_active":"2023-10-10T17:41:37.595527+0000","last_peered":"2023-10-10T17:41:37.595527+0000","last_clean":"2023-10-10T17:41:37.595527+0000","last_became_active":"2023-10-10T17:40:41.606061+0000","last_became_peered":"2023-10-10T17:40:41.606061+0000","last_unstale":"2023-10-10T17:41:37.595527+0000","last_undegraded":"2023-10-10T17:41:37.595527+0000","last_fullsized":"2023-10-10T17:41:37.595527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:04:19.222774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788308+0000","last_change":"2023-10-10T17:40:41.607630+0000","last_active":"2023-10-10T17:41:37.788308+0000","last_peered":"2023-10-10T17:41:37.788308+0000","last_clean":"2023-10-10T17:41:37.788308+0000","last_became_active":"2023-10-10T17:40:41.607297+0000","last_became_peered":"2023-10-10T17:40:41.607297+0000","last_unstale":"2023-10-10T17:41:37.788308+0000","last_undegraded":"2023-10-10T17:41:37.788308+0000","last_fullsized":"2023-10-10T17:41:37.788308+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:15:06.512286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595355+0000","last_change":"2023-10-10T17:40:41.594082+0000","last_active":"2023-10-10T17:41:37.595355+0000","last_peered":"2023-10-10T17:41:37.595355+0000","last_clean":"2023-10-10T17:41:37.595355+0000","last_became_active":"2023-10-10T17:40:41.593787+0000","last_became_peered":"2023-10-10T17:40:41.593787+0000","last_unstale":"2023-10-10T17:41:37.595355+0000","last_undegraded":"2023-10-10T17:41:37.595355+0000","last_fullsized":"2023-10-10T17:41:37.595355+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:10:48.548705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596760+0000","last_change":"2023-10-10T17:40:41.608529+0000","last_active":"2023-10-10T17:41:37.596760+0000","last_peered":"2023-10-10T17:41:37.596760+0000","last_clean":"2023-10-10T17:41:37.596760+0000","last_became_active":"2023-10-10T17:40:41.608247+0000","last_became_peered":"2023-10-10T17:40:41.608247+0000","last_unstale":"2023-10-10T17:41:37.596760+0000","last_undegraded":"2023-10-10T17:41:37.596760+0000","last_fullsized":"2023-10-10T17:41:37.596760+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:20:05.900726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596528+0000","last_change":"2023-10-10T17:40:45.707455+0000","last_active":"2023-10-10T17:41:37.596528+0000","last_peered":"2023-10-10T17:41:37.596528+0000","last_clean":"2023-10-10T17:41:37.596528+0000","last_became_active":"2023-10-10T17:40:45.707024+0000","last_became_peered":"2023-10-10T17:40:45.707024+0000","last_unstale":"2023-10-10T17:41:37.596528+0000","last_undegraded":"2023-10-10T17:41:37.596528+0000","last_fullsized":"2023-10-10T17:41:37.596528+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:34:24.408486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595807+0000","last_change":"2023-10-10T17:40:41.165983+0000","last_active":"2023-10-10T17:41:37.595807+0000","last_peered":"2023-10-10T17:41:37.595807+0000","last_clean":"2023-10-10T17:41:37.595807+0000","last_became_active":"2023-10-10T17:40:41.165487+0000","last_became_peered":"2023-10-10T17:40:41.165487+0000","last_unstale":"2023-10-10T17:41:37.595807+0000","last_undegraded":"2023-10-10T17:41:37.595807+0000","last_fullsized":"2023-10-10T17:41:37.595807+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:08:45.809159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979220+0000","last_change":"2023-10-10T17:40:41.614258+0000","last_active":"2023-10-10T17:41:37.979220+0000","last_peered":"2023-10-10T17:41:37.979220+0000","last_clean":"2023-10-10T17:41:37.979220+0000","last_became_active":"2023-10-10T17:40:41.613951+0000","last_became_peered":"2023-10-10T17:40:41.613951+0000","last_unstale":"2023-10-10T17:41:37.979220+0000","last_undegraded":"2023-10-10T17:41:37.979220+0000","last_fullsized":"2023-10-10T17:41:37.979220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:43:34.983899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:45.708568+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:45.708108+0000","last_became_peered":"2023-10-10T17:40:45.708108+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:30:24.153573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980255+0000","last_change":"2023-10-10T17:40:41.167744+0000","last_active":"2023-10-10T17:41:37.980255+0000","last_peered":"2023-10-10T17:41:37.980255+0000","last_clean":"2023-10-10T17:41:37.980255+0000","last_became_active":"2023-10-10T17:40:41.167189+0000","last_became_peered":"2023-10-10T17:40:41.167189+0000","last_unstale":"2023-10-10T17:41:37.980255+0000","last_undegraded":"2023-10-10T17:41:37.980255+0000","last_fullsized":"2023-10-10T17:41:37.980255+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:21:47.667198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598064+0000","last_change":"2023-10-10T17:40:41.608514+0000","last_active":"2023-10-10T17:41:37.598064+0000","last_peered":"2023-10-10T17:41:37.598064+0000","last_clean":"2023-10-10T17:41:37.598064+0000","last_became_active":"2023-10-10T17:40:41.608208+0000","last_became_peered":"2023-10-10T17:40:41.608208+0000","last_unstale":"2023-10-10T17:41:37.598064+0000","last_undegraded":"2023-10-10T17:41:37.598064+0000","last_fullsized":"2023-10-10T17:41:37.598064+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:48:25.411862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.981315+0000","last_change":"2023-10-10T17:41:37.981315+0000","last_active":"2023-10-10T17:41:37.981315+0000","last_peered":"2023-10-10T17:41:37.981315+0000","last_clean":"2023-10-10T17:41:37.981315+0000","last_became_active":"2023-10-10T17:41:37.980691+0000","last_became_peered":"2023-10-10T17:41:37.980691+0000","last_unstale":"2023-10-10T17:41:37.981315+0000","last_undegraded":"2023-10-10T17:41:37.981315+0000","last_fullsized":"2023-10-10T17:41:37.981315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:36:12.102836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,10,5],"acting":[4,3,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601523+0000","last_change":"2023-10-10T17:40:41.169821+0000","last_active":"2023-10-10T17:41:37.601523+0000","last_peered":"2023-10-10T17:41:37.601523+0000","last_clean":"2023-10-10T17:41:37.601523+0000","last_became_active":"2023-10-10T17:40:41.169502+0000","last_became_peered":"2023-10-10T17:40:41.169502+0000","last_unstale":"2023-10-10T17:41:37.601523+0000","last_undegraded":"2023-10-10T17:41:37.601523+0000","last_fullsized":"2023-10-10T17:41:37.601523+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:32:01.106471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595719+0000","last_change":"2023-10-10T17:40:41.602720+0000","last_active":"2023-10-10T17:41:37.595719+0000","last_peered":"2023-10-10T17:41:37.595719+0000","last_clean":"2023-10-10T17:41:37.595719+0000","last_became_active":"2023-10-10T17:40:41.602187+0000","last_became_peered":"2023-10-10T17:40:41.602187+0000","last_unstale":"2023-10-10T17:41:37.595719+0000","last_undegraded":"2023-10-10T17:41:37.595719+0000","last_fullsized":"2023-10-10T17:41:37.595719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:18:35.228329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601501+0000","last_change":"2023-10-10T17:40:45.706946+0000","last_active":"2023-10-10T17:41:37.601501+0000","last_peered":"2023-10-10T17:41:37.601501+0000","last_clean":"2023-10-10T17:41:37.601501+0000","last_became_active":"2023-10-10T17:40:45.706495+0000","last_became_peered":"2023-10-10T17:40:45.706495+0000","last_unstale":"2023-10-10T17:41:37.601501+0000","last_undegraded":"2023-10-10T17:41:37.601501+0000","last_fullsized":"2023-10-10T17:41:37.601501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:11.140718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601201+0000","last_change":"2023-10-10T17:40:41.171855+0000","last_active":"2023-10-10T17:41:37.601201+0000","last_peered":"2023-10-10T17:41:37.601201+0000","last_clean":"2023-10-10T17:41:37.601201+0000","last_became_active":"2023-10-10T17:40:41.171515+0000","last_became_peered":"2023-10-10T17:40:41.171515+0000","last_unstale":"2023-10-10T17:41:37.601201+0000","last_undegraded":"2023-10-10T17:41:37.601201+0000","last_fullsized":"2023-10-10T17:41:37.601201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:52:15.556629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652087+0000","last_change":"2023-10-10T17:40:41.587083+0000","last_active":"2023-10-10T17:41:37.652087+0000","last_peered":"2023-10-10T17:41:37.652087+0000","last_clean":"2023-10-10T17:41:37.652087+0000","last_became_active":"2023-10-10T17:40:41.586758+0000","last_became_peered":"2023-10-10T17:40:41.586758+0000","last_unstale":"2023-10-10T17:41:37.652087+0000","last_undegraded":"2023-10-10T17:41:37.652087+0000","last_fullsized":"2023-10-10T17:41:37.652087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:31:30.273631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596265+0000","last_change":"2023-10-10T17:40:45.710492+0000","last_active":"2023-10-10T17:41:37.596265+0000","last_peered":"2023-10-10T17:41:37.596265+0000","last_clean":"2023-10-10T17:41:37.596265+0000","last_became_active":"2023-10-10T17:40:45.710069+0000","last_became_peered":"2023-10-10T17:40:45.710069+0000","last_unstale":"2023-10-10T17:41:37.596265+0000","last_undegraded":"2023-10-10T17:41:37.596265+0000","last_fullsized":"2023-10-10T17:41:37.596265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:58:28.507189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:19.708911+0000","last_change":"2023-10-10T17:40:41.167449+0000","last_active":"2023-10-10T17:42:19.708911+0000","last_peered":"2023-10-10T17:42:19.708911+0000","last_clean":"2023-10-10T17:42:19.708911+0000","last_became_active":"2023-10-10T17:40:41.167038+0000","last_became_peered":"2023-10-10T17:40:41.167038+0000","last_unstale":"2023-10-10T17:42:19.708911+0000","last_undegraded":"2023-10-10T17:42:19.708911+0000","last_fullsized":"2023-10-10T17:42:19.708911+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:48:38.144880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":574,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599067+0000","last_change":"2023-10-10T17:40:41.594327+0000","last_active":"2023-10-10T17:41:37.599067+0000","last_peered":"2023-10-10T17:41:37.599067+0000","last_clean":"2023-10-10T17:41:37.599067+0000","last_became_active":"2023-10-10T17:40:41.594008+0000","last_became_peered":"2023-10-10T17:40:41.594008+0000","last_unstale":"2023-10-10T17:41:37.599067+0000","last_undegraded":"2023-10-10T17:41:37.599067+0000","last_fullsized":"2023-10-10T17:41:37.599067+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:56:08.300633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.653403+0000","last_change":"2023-10-10T17:41:37.653403+0000","last_active":"2023-10-10T17:41:37.653403+0000","last_peered":"2023-10-10T17:41:37.653403+0000","last_clean":"2023-10-10T17:41:37.653403+0000","last_became_active":"2023-10-10T17:41:37.652946+0000","last_became_peered":"2023-10-10T17:41:37.652946+0000","last_unstale":"2023-10-10T17:41:37.653403+0000","last_undegraded":"2023-10-10T17:41:37.653403+0000","last_fullsized":"2023-10-10T17:41:37.653403+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:16:29.417131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,8,5],"acting":[2,6,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598235+0000","last_change":"2023-10-10T17:40:41.171926+0000","last_active":"2023-10-10T17:41:37.598235+0000","last_peered":"2023-10-10T17:41:37.598235+0000","last_clean":"2023-10-10T17:41:37.598235+0000","last_became_active":"2023-10-10T17:40:41.171627+0000","last_became_peered":"2023-10-10T17:40:41.171627+0000","last_unstale":"2023-10-10T17:41:37.598235+0000","last_undegraded":"2023-10-10T17:41:37.598235+0000","last_fullsized":"2023-10-10T17:41:37.598235+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:00:51.367484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651905+0000","last_change":"2023-10-10T17:40:41.593135+0000","last_active":"2023-10-10T17:41:37.651905+0000","last_peered":"2023-10-10T17:41:37.651905+0000","last_clean":"2023-10-10T17:41:37.651905+0000","last_became_active":"2023-10-10T17:40:41.592871+0000","last_became_peered":"2023-10-10T17:40:41.592871+0000","last_unstale":"2023-10-10T17:41:37.651905+0000","last_undegraded":"2023-10-10T17:41:37.651905+0000","last_fullsized":"2023-10-10T17:41:37.651905+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:04:32.143194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788755+0000","last_change":"2023-10-10T17:40:45.703557+0000","last_active":"2023-10-10T17:41:37.788755+0000","last_peered":"2023-10-10T17:41:37.788755+0000","last_clean":"2023-10-10T17:41:37.788755+0000","last_became_active":"2023-10-10T17:40:45.703085+0000","last_became_peered":"2023-10-10T17:40:45.703085+0000","last_unstale":"2023-10-10T17:41:37.788755+0000","last_undegraded":"2023-10-10T17:41:37.788755+0000","last_fullsized":"2023-10-10T17:41:37.788755+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:09:11.130227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598088+0000","last_change":"2023-10-10T17:40:41.162829+0000","last_active":"2023-10-10T17:41:37.598088+0000","last_peered":"2023-10-10T17:41:37.598088+0000","last_clean":"2023-10-10T17:41:37.598088+0000","last_became_active":"2023-10-10T17:40:41.162516+0000","last_became_peered":"2023-10-10T17:40:41.162516+0000","last_unstale":"2023-10-10T17:41:37.598088+0000","last_undegraded":"2023-10-10T17:41:37.598088+0000","last_fullsized":"2023-10-10T17:41:37.598088+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:07:29.590599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.789909+0000","last_change":"2023-10-10T17:41:37.789510+0000","last_active":"2023-10-10T17:41:37.789909+0000","last_peered":"2023-10-10T17:41:37.789909+0000","last_clean":"2023-10-10T17:41:37.789909+0000","last_became_active":"2023-10-10T17:41:37.789166+0000","last_became_peered":"2023-10-10T17:41:37.789166+0000","last_unstale":"2023-10-10T17:41:37.789909+0000","last_undegraded":"2023-10-10T17:41:37.789909+0000","last_fullsized":"2023-10-10T17:41:37.789909+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:33:55.993371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598938+0000","last_change":"2023-10-10T17:40:45.707139+0000","last_active":"2023-10-10T17:41:37.598938+0000","last_peered":"2023-10-10T17:41:37.598938+0000","last_clean":"2023-10-10T17:41:37.598938+0000","last_became_active":"2023-10-10T17:40:45.706708+0000","last_became_peered":"2023-10-10T17:40:45.706708+0000","last_unstale":"2023-10-10T17:41:37.598938+0000","last_undegraded":"2023-10-10T17:41:37.598938+0000","last_fullsized":"2023-10-10T17:41:37.598938+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:46:42.381782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651470+0000","last_change":"2023-10-10T17:40:41.170512+0000","last_active":"2023-10-10T17:41:37.651470+0000","last_peered":"2023-10-10T17:41:37.651470+0000","last_clean":"2023-10-10T17:41:37.651470+0000","last_became_active":"2023-10-10T17:40:41.170264+0000","last_became_peered":"2023-10-10T17:40:41.170264+0000","last_unstale":"2023-10-10T17:41:37.651470+0000","last_undegraded":"2023-10-10T17:41:37.651470+0000","last_fullsized":"2023-10-10T17:41:37.651470+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:21:55.138815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594921+0000","last_change":"2023-10-10T17:40:41.597647+0000","last_active":"2023-10-10T17:41:37.594921+0000","last_peered":"2023-10-10T17:41:37.594921+0000","last_clean":"2023-10-10T17:41:37.594921+0000","last_became_active":"2023-10-10T17:40:41.597245+0000","last_became_peered":"2023-10-10T17:40:41.597245+0000","last_unstale":"2023-10-10T17:41:37.594921+0000","last_undegraded":"2023-10-10T17:41:37.594921+0000","last_fullsized":"2023-10-10T17:41:37.594921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:40:02.702932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979667+0000","last_change":"2023-10-10T17:40:45.705089+0000","last_active":"2023-10-10T17:41:37.979667+0000","last_peered":"2023-10-10T17:41:37.979667+0000","last_clean":"2023-10-10T17:41:37.979667+0000","last_became_active":"2023-10-10T17:40:45.704475+0000","last_became_peered":"2023-10-10T17:40:45.704475+0000","last_unstale":"2023-10-10T17:41:37.979667+0000","last_undegraded":"2023-10-10T17:41:37.979667+0000","last_fullsized":"2023-10-10T17:41:37.979667+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:26.896709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":111,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:21.710912+0000","last_change":"2023-10-10T17:40:41.167409+0000","last_active":"2023-10-10T17:42:21.710912+0000","last_peered":"2023-10-10T17:42:21.710912+0000","last_clean":"2023-10-10T17:42:21.710912+0000","last_became_active":"2023-10-10T17:40:41.167125+0000","last_became_peered":"2023-10-10T17:40:41.167125+0000","last_unstale":"2023-10-10T17:42:21.710912+0000","last_undegraded":"2023-10-10T17:42:21.710912+0000","last_fullsized":"2023-10-10T17:42:21.710912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:12:36.905039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598010+0000","last_change":"2023-10-10T17:40:41.606216+0000","last_active":"2023-10-10T17:41:37.598010+0000","last_peered":"2023-10-10T17:41:37.598010+0000","last_clean":"2023-10-10T17:41:37.598010+0000","last_became_active":"2023-10-10T17:40:41.603258+0000","last_became_peered":"2023-10-10T17:40:41.603258+0000","last_unstale":"2023-10-10T17:41:37.598010+0000","last_undegraded":"2023-10-10T17:41:37.598010+0000","last_fullsized":"2023-10-10T17:41:37.598010+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:30:40.923843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.594917+0000","last_change":"2023-10-10T17:40:45.705623+0000","last_active":"2023-10-10T17:41:37.594917+0000","last_peered":"2023-10-10T17:41:37.594917+0000","last_clean":"2023-10-10T17:41:37.594917+0000","last_became_active":"2023-10-10T17:40:45.705242+0000","last_became_peered":"2023-10-10T17:40:45.705242+0000","last_unstale":"2023-10-10T17:41:37.594917+0000","last_undegraded":"2023-10-10T17:41:37.594917+0000","last_fullsized":"2023-10-10T17:41:37.594917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:43:18.391641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601242+0000","last_change":"2023-10-10T17:40:41.171552+0000","last_active":"2023-10-10T17:41:37.601242+0000","last_peered":"2023-10-10T17:41:37.601242+0000","last_clean":"2023-10-10T17:41:37.601242+0000","last_became_active":"2023-10-10T17:40:41.171207+0000","last_became_peered":"2023-10-10T17:40:41.171207+0000","last_unstale":"2023-10-10T17:41:37.601242+0000","last_undegraded":"2023-10-10T17:41:37.601242+0000","last_fullsized":"2023-10-10T17:41:37.601242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:15:35.436090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599824+0000","last_change":"2023-10-10T17:40:41.609655+0000","last_active":"2023-10-10T17:41:37.599824+0000","last_peered":"2023-10-10T17:41:37.599824+0000","last_clean":"2023-10-10T17:41:37.599824+0000","last_became_active":"2023-10-10T17:40:41.609318+0000","last_became_peered":"2023-10-10T17:40:41.609318+0000","last_unstale":"2023-10-10T17:41:37.599824+0000","last_undegraded":"2023-10-10T17:41:37.599824+0000","last_fullsized":"2023-10-10T17:41:37.599824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:06:22.359139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602379+0000","last_change":"2023-10-10T17:40:45.709513+0000","last_active":"2023-10-10T17:41:37.602379+0000","last_peered":"2023-10-10T17:41:37.602379+0000","last_clean":"2023-10-10T17:41:37.602379+0000","last_became_active":"2023-10-10T17:40:45.708963+0000","last_became_peered":"2023-10-10T17:40:45.708963+0000","last_unstale":"2023-10-10T17:41:37.602379+0000","last_undegraded":"2023-10-10T17:41:37.602379+0000","last_fullsized":"2023-10-10T17:41:37.602379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:15:02.687427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652007+0000","last_change":"2023-10-10T17:40:41.163234+0000","last_active":"2023-10-10T17:41:37.652007+0000","last_peered":"2023-10-10T17:41:37.652007+0000","last_clean":"2023-10-10T17:41:37.652007+0000","last_became_active":"2023-10-10T17:40:41.162916+0000","last_became_peered":"2023-10-10T17:40:41.162916+0000","last_unstale":"2023-10-10T17:41:37.652007+0000","last_undegraded":"2023-10-10T17:41:37.652007+0000","last_fullsized":"2023-10-10T17:41:37.652007+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:09:06.358768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598352+0000","last_change":"2023-10-10T17:40:41.602852+0000","last_active":"2023-10-10T17:41:37.598352+0000","last_peered":"2023-10-10T17:41:37.598352+0000","last_clean":"2023-10-10T17:41:37.598352+0000","last_became_active":"2023-10-10T17:40:41.602530+0000","last_became_peered":"2023-10-10T17:40:41.602530+0000","last_unstale":"2023-10-10T17:41:37.598352+0000","last_undegraded":"2023-10-10T17:41:37.598352+0000","last_fullsized":"2023-10-10T17:41:37.598352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:58:41.949886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595188+0000","last_change":"2023-10-10T17:40:45.713714+0000","last_active":"2023-10-10T17:41:37.595188+0000","last_peered":"2023-10-10T17:41:37.595188+0000","last_clean":"2023-10-10T17:41:37.595188+0000","last_became_active":"2023-10-10T17:40:45.712953+0000","last_became_peered":"2023-10-10T17:40:45.712953+0000","last_unstale":"2023-10-10T17:41:37.595188+0000","last_undegraded":"2023-10-10T17:41:37.595188+0000","last_fullsized":"2023-10-10T17:41:37.595188+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:33:15.222779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":204,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:22.711933+0000","last_change":"2023-10-10T17:40:41.168328+0000","last_active":"2023-10-10T17:42:22.711933+0000","last_peered":"2023-10-10T17:42:22.711933+0000","last_clean":"2023-10-10T17:42:22.711933+0000","last_became_active":"2023-10-10T17:40:41.168044+0000","last_became_peered":"2023-10-10T17:40:41.168044+0000","last_unstale":"2023-10-10T17:42:22.711933+0000","last_undegraded":"2023-10-10T17:42:22.711933+0000","last_fullsized":"2023-10-10T17:42:22.711933+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:52:24.701647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":166,"num_read_kb":87,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599535+0000","last_change":"2023-10-10T17:40:41.597795+0000","last_active":"2023-10-10T17:41:37.599535+0000","last_peered":"2023-10-10T17:41:37.599535+0000","last_clean":"2023-10-10T17:41:37.599535+0000","last_became_active":"2023-10-10T17:40:41.597537+0000","last_became_peered":"2023-10-10T17:40:41.597537+0000","last_unstale":"2023-10-10T17:41:37.599535+0000","last_undegraded":"2023-10-10T17:41:37.599535+0000","last_fullsized":"2023-10-10T17:41:37.599535+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:16:34.371564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652449+0000","last_change":"2023-10-10T17:40:45.709827+0000","last_active":"2023-10-10T17:41:37.652449+0000","last_peered":"2023-10-10T17:41:37.652449+0000","last_clean":"2023-10-10T17:41:37.652449+0000","last_became_active":"2023-10-10T17:40:45.709373+0000","last_became_peered":"2023-10-10T17:40:45.709373+0000","last_unstale":"2023-10-10T17:41:37.652449+0000","last_undegraded":"2023-10-10T17:41:37.652449+0000","last_fullsized":"2023-10-10T17:41:37.652449+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:35:42.826536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596527+0000","last_change":"2023-10-10T17:40:41.160804+0000","last_active":"2023-10-10T17:41:37.596527+0000","last_peered":"2023-10-10T17:41:37.596527+0000","last_clean":"2023-10-10T17:41:37.596527+0000","last_became_active":"2023-10-10T17:40:41.160493+0000","last_became_peered":"2023-10-10T17:40:41.160493+0000","last_unstale":"2023-10-10T17:41:37.596527+0000","last_undegraded":"2023-10-10T17:41:37.596527+0000","last_fullsized":"2023-10-10T17:41:37.596527+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:32:59.561663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598334+0000","last_change":"2023-10-10T17:40:41.601357+0000","last_active":"2023-10-10T17:41:37.598334+0000","last_peered":"2023-10-10T17:41:37.598334+0000","last_clean":"2023-10-10T17:41:37.598334+0000","last_became_active":"2023-10-10T17:40:41.601057+0000","last_became_peered":"2023-10-10T17:40:41.601057+0000","last_unstale":"2023-10-10T17:41:37.598334+0000","last_undegraded":"2023-10-10T17:41:37.598334+0000","last_fullsized":"2023-10-10T17:41:37.598334+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:58:07.286077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788393+0000","last_change":"2023-10-10T17:40:45.708067+0000","last_active":"2023-10-10T17:41:37.788393+0000","last_peered":"2023-10-10T17:41:37.788393+0000","last_clean":"2023-10-10T17:41:37.788393+0000","last_became_active":"2023-10-10T17:40:45.707686+0000","last_became_peered":"2023-10-10T17:40:45.707686+0000","last_unstale":"2023-10-10T17:41:37.788393+0000","last_undegraded":"2023-10-10T17:41:37.788393+0000","last_fullsized":"2023-10-10T17:41:37.788393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:08.956407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651829+0000","last_change":"2023-10-10T17:40:41.168006+0000","last_active":"2023-10-10T17:41:37.651829+0000","last_peered":"2023-10-10T17:41:37.651829+0000","last_clean":"2023-10-10T17:41:37.651829+0000","last_became_active":"2023-10-10T17:40:41.167777+0000","last_became_peered":"2023-10-10T17:40:41.167777+0000","last_unstale":"2023-10-10T17:41:37.651829+0000","last_undegraded":"2023-10-10T17:41:37.651829+0000","last_fullsized":"2023-10-10T17:41:37.651829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:30:32.283329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601103+0000","last_change":"2023-10-10T17:40:41.591215+0000","last_active":"2023-10-10T17:41:37.601103+0000","last_peered":"2023-10-10T17:41:37.601103+0000","last_clean":"2023-10-10T17:41:37.601103+0000","last_became_active":"2023-10-10T17:40:41.590929+0000","last_became_peered":"2023-10-10T17:40:41.590929+0000","last_unstale":"2023-10-10T17:41:37.601103+0000","last_undegraded":"2023-10-10T17:41:37.601103+0000","last_fullsized":"2023-10-10T17:41:37.601103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:11:13.144245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595485+0000","last_change":"2023-10-10T17:40:45.709038+0000","last_active":"2023-10-10T17:41:37.595485+0000","last_peered":"2023-10-10T17:41:37.595485+0000","last_clean":"2023-10-10T17:41:37.595485+0000","last_became_active":"2023-10-10T17:40:45.708683+0000","last_became_peered":"2023-10-10T17:40:45.708683+0000","last_unstale":"2023-10-10T17:41:37.595485+0000","last_undegraded":"2023-10-10T17:41:37.595485+0000","last_fullsized":"2023-10-10T17:41:37.595485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:50:01.174506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'4","reported_seq":116,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:22.722538+0000","last_change":"2023-10-10T17:40:41.163817+0000","last_active":"2023-10-10T17:42:22.722538+0000","last_peered":"2023-10-10T17:42:22.722538+0000","last_clean":"2023-10-10T17:42:22.722538+0000","last_became_active":"2023-10-10T17:40:41.163497+0000","last_became_peered":"2023-10-10T17:40:41.163497+0000","last_unstale":"2023-10-10T17:42:22.722538+0000","last_undegraded":"2023-10-10T17:42:22.722538+0000","last_fullsized":"2023-10-10T17:42:22.722538+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:54:45.382546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":800,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.607553+0000","last_change":"2023-10-10T17:41:37.607553+0000","last_active":"2023-10-10T17:41:37.607553+0000","last_peered":"2023-10-10T17:41:37.607553+0000","last_clean":"2023-10-10T17:41:37.607553+0000","last_became_active":"2023-10-10T17:41:37.606536+0000","last_became_peered":"2023-10-10T17:41:37.606536+0000","last_unstale":"2023-10-10T17:41:37.607553+0000","last_undegraded":"2023-10-10T17:41:37.607553+0000","last_fullsized":"2023-10-10T17:41:37.607553+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:58:09.182227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,10],"acting":[7,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600059+0000","last_change":"2023-10-10T17:40:45.714900+0000","last_active":"2023-10-10T17:41:37.600059+0000","last_peered":"2023-10-10T17:41:37.600059+0000","last_clean":"2023-10-10T17:41:37.600059+0000","last_became_active":"2023-10-10T17:40:45.714525+0000","last_became_peered":"2023-10-10T17:40:45.714525+0000","last_unstale":"2023-10-10T17:41:37.600059+0000","last_undegraded":"2023-10-10T17:41:37.600059+0000","last_fullsized":"2023-10-10T17:41:37.600059+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:01:36.283546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":66,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:38.795173+0000","last_change":"2023-10-10T17:41:38.795173+0000","last_active":"2023-10-10T17:41:38.795173+0000","last_peered":"2023-10-10T17:41:38.795173+0000","last_clean":"2023-10-10T17:41:38.795173+0000","last_became_active":"2023-10-10T17:41:37.603917+0000","last_became_peered":"2023-10-10T17:41:37.603917+0000","last_unstale":"2023-10-10T17:41:38.795173+0000","last_undegraded":"2023-10-10T17:41:38.795173+0000","last_fullsized":"2023-10-10T17:41:38.795173+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:32:38.418130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,4],"acting":[8,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651410+0000","last_change":"2023-10-10T17:40:41.586591+0000","last_active":"2023-10-10T17:41:37.651410+0000","last_peered":"2023-10-10T17:41:37.651410+0000","last_clean":"2023-10-10T17:41:37.651410+0000","last_became_active":"2023-10-10T17:40:41.586293+0000","last_became_peered":"2023-10-10T17:40:41.586293+0000","last_unstale":"2023-10-10T17:41:37.651410+0000","last_undegraded":"2023-10-10T17:41:37.651410+0000","last_fullsized":"2023-10-10T17:41:37.651410+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:39:26.663874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651375+0000","last_change":"2023-10-10T17:40:45.708348+0000","last_active":"2023-10-10T17:41:37.651375+0000","last_peered":"2023-10-10T17:41:37.651375+0000","last_clean":"2023-10-10T17:41:37.651375+0000","last_became_active":"2023-10-10T17:40:45.707899+0000","last_became_peered":"2023-10-10T17:40:45.707899+0000","last_unstale":"2023-10-10T17:41:37.651375+0000","last_undegraded":"2023-10-10T17:41:37.651375+0000","last_fullsized":"2023-10-10T17:41:37.651375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:29:39.345572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601729+0000","last_change":"2023-10-10T17:40:41.168209+0000","last_active":"2023-10-10T17:41:37.601729+0000","last_peered":"2023-10-10T17:41:37.601729+0000","last_clean":"2023-10-10T17:41:37.601729+0000","last_became_active":"2023-10-10T17:40:41.167907+0000","last_became_peered":"2023-10-10T17:40:41.167907+0000","last_unstale":"2023-10-10T17:41:37.601729+0000","last_undegraded":"2023-10-10T17:41:37.601729+0000","last_fullsized":"2023-10-10T17:41:37.601729+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:35:19.344734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.594848+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.594562+0000","last_became_peered":"2023-10-10T17:40:41.594562+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:15:21.805794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600519+0000","last_change":"2023-10-10T17:40:45.711043+0000","last_active":"2023-10-10T17:41:37.600519+0000","last_peered":"2023-10-10T17:41:37.600519+0000","last_clean":"2023-10-10T17:41:37.600519+0000","last_became_active":"2023-10-10T17:40:45.710617+0000","last_became_peered":"2023-10-10T17:40:45.710617+0000","last_unstale":"2023-10-10T17:41:37.600519+0000","last_undegraded":"2023-10-10T17:41:37.600519+0000","last_fullsized":"2023-10-10T17:41:37.600519+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:00:48.152401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595133+0000","last_change":"2023-10-10T17:40:41.163238+0000","last_active":"2023-10-10T17:41:37.595133+0000","last_peered":"2023-10-10T17:41:37.595133+0000","last_clean":"2023-10-10T17:41:37.595133+0000","last_became_active":"2023-10-10T17:40:41.162934+0000","last_became_peered":"2023-10-10T17:40:41.162934+0000","last_unstale":"2023-10-10T17:41:37.595133+0000","last_undegraded":"2023-10-10T17:41:37.595133+0000","last_fullsized":"2023-10-10T17:41:37.595133+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:23:10.440706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598371+0000","last_change":"2023-10-10T17:40:41.597026+0000","last_active":"2023-10-10T17:41:37.598371+0000","last_peered":"2023-10-10T17:41:37.598371+0000","last_clean":"2023-10-10T17:41:37.598371+0000","last_became_active":"2023-10-10T17:40:41.596747+0000","last_became_peered":"2023-10-10T17:40:41.596747+0000","last_unstale":"2023-10-10T17:41:37.598371+0000","last_undegraded":"2023-10-10T17:41:37.598371+0000","last_fullsized":"2023-10-10T17:41:37.598371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:03:52.010575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599056+0000","last_change":"2023-10-10T17:40:45.707218+0000","last_active":"2023-10-10T17:41:37.599056+0000","last_peered":"2023-10-10T17:41:37.599056+0000","last_clean":"2023-10-10T17:41:37.599056+0000","last_became_active":"2023-10-10T17:40:45.706850+0000","last_became_peered":"2023-10-10T17:40:45.706850+0000","last_unstale":"2023-10-10T17:41:37.599056+0000","last_undegraded":"2023-10-10T17:41:37.599056+0000","last_fullsized":"2023-10-10T17:41:37.599056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T05:36:44.235237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601441+0000","last_change":"2023-10-10T17:40:41.166713+0000","last_active":"2023-10-10T17:41:37.601441+0000","last_peered":"2023-10-10T17:41:37.601441+0000","last_clean":"2023-10-10T17:41:37.601441+0000","last_became_active":"2023-10-10T17:40:41.165951+0000","last_became_peered":"2023-10-10T17:40:41.165951+0000","last_unstale":"2023-10-10T17:41:37.601441+0000","last_undegraded":"2023-10-10T17:41:37.601441+0000","last_fullsized":"2023-10-10T17:41:37.601441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:56:00.129165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597783+0000","last_change":"2023-10-10T17:40:41.605860+0000","last_active":"2023-10-10T17:41:37.597783+0000","last_peered":"2023-10-10T17:41:37.597783+0000","last_clean":"2023-10-10T17:41:37.597783+0000","last_became_active":"2023-10-10T17:40:41.605556+0000","last_became_peered":"2023-10-10T17:40:41.605556+0000","last_unstale":"2023-10-10T17:41:37.597783+0000","last_undegraded":"2023-10-10T17:41:37.597783+0000","last_fullsized":"2023-10-10T17:41:37.597783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:45:07.438058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599757+0000","last_change":"2023-10-10T17:40:45.711844+0000","last_active":"2023-10-10T17:41:37.599757+0000","last_peered":"2023-10-10T17:41:37.599757+0000","last_clean":"2023-10-10T17:41:37.599757+0000","last_became_active":"2023-10-10T17:40:45.711413+0000","last_became_peered":"2023-10-10T17:40:45.711413+0000","last_unstale":"2023-10-10T17:41:37.599757+0000","last_undegraded":"2023-10-10T17:41:37.599757+0000","last_fullsized":"2023-10-10T17:41:37.599757+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:13:03.476627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.650650+0000","last_change":"2023-10-10T17:40:41.608625+0000","last_active":"2023-10-10T17:41:37.650650+0000","last_peered":"2023-10-10T17:41:37.650650+0000","last_clean":"2023-10-10T17:41:37.650650+0000","last_became_active":"2023-10-10T17:40:41.608332+0000","last_became_peered":"2023-10-10T17:40:41.608332+0000","last_unstale":"2023-10-10T17:41:37.650650+0000","last_undegraded":"2023-10-10T17:41:37.650650+0000","last_fullsized":"2023-10-10T17:41:37.650650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:07:55.183440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":113,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:42:21.720594+0000","last_change":"2023-10-10T17:40:41.169807+0000","last_active":"2023-10-10T17:42:21.720594+0000","last_peered":"2023-10-10T17:42:21.720594+0000","last_clean":"2023-10-10T17:42:21.720594+0000","last_became_active":"2023-10-10T17:40:41.169475+0000","last_became_peered":"2023-10-10T17:40:41.169475+0000","last_unstale":"2023-10-10T17:42:21.720594+0000","last_undegraded":"2023-10-10T17:42:21.720594+0000","last_fullsized":"2023-10-10T17:42:21.720594+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:50:22.887502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601878+0000","last_change":"2023-10-10T17:40:45.710544+0000","last_active":"2023-10-10T17:41:37.601878+0000","last_peered":"2023-10-10T17:41:37.601878+0000","last_clean":"2023-10-10T17:41:37.601878+0000","last_became_active":"2023-10-10T17:40:45.710084+0000","last_became_peered":"2023-10-10T17:40:45.710084+0000","last_unstale":"2023-10-10T17:41:37.601878+0000","last_undegraded":"2023-10-10T17:41:37.601878+0000","last_fullsized":"2023-10-10T17:41:37.601878+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:44:01.303666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979130+0000","last_change":"2023-10-10T17:40:41.599028+0000","last_active":"2023-10-10T17:41:37.979130+0000","last_peered":"2023-10-10T17:41:37.979130+0000","last_clean":"2023-10-10T17:41:37.979130+0000","last_became_active":"2023-10-10T17:40:41.598684+0000","last_became_peered":"2023-10-10T17:40:41.598684+0000","last_unstale":"2023-10-10T17:41:37.979130+0000","last_undegraded":"2023-10-10T17:41:37.979130+0000","last_fullsized":"2023-10-10T17:41:37.979130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:16:57.333515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602213+0000","last_change":"2023-10-10T17:40:41.165756+0000","last_active":"2023-10-10T17:41:37.602213+0000","last_peered":"2023-10-10T17:41:37.602213+0000","last_clean":"2023-10-10T17:41:37.602213+0000","last_became_active":"2023-10-10T17:40:41.165368+0000","last_became_peered":"2023-10-10T17:40:41.165368+0000","last_unstale":"2023-10-10T17:41:37.602213+0000","last_undegraded":"2023-10-10T17:41:37.602213+0000","last_fullsized":"2023-10-10T17:41:37.602213+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:16.089880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600889+0000","last_change":"2023-10-10T17:39:42.508828+0000","last_active":"2023-10-10T17:41:37.600889+0000","last_peered":"2023-10-10T17:41:37.600889+0000","last_clean":"2023-10-10T17:41:37.600889+0000","last_became_active":"2023-10-10T17:39:42.508475+0000","last_became_peered":"2023-10-10T17:39:42.508475+0000","last_unstale":"2023-10-10T17:41:37.600889+0000","last_undegraded":"2023-10-10T17:41:37.600889+0000","last_fullsized":"2023-10-10T17:41:37.600889+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:36:46.553699+0000","last_clean_scrub_stamp":"2023-10-10T17:36:46.553699+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:08:38.008210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652108+0000","last_change":"2023-10-10T17:40:45.714604+0000","last_active":"2023-10-10T17:41:37.652108+0000","last_peered":"2023-10-10T17:41:37.652108+0000","last_clean":"2023-10-10T17:41:37.652108+0000","last_became_active":"2023-10-10T17:40:45.714185+0000","last_became_peered":"2023-10-10T17:40:45.714185+0000","last_unstale":"2023-10-10T17:41:37.652108+0000","last_undegraded":"2023-10-10T17:41:37.652108+0000","last_fullsized":"2023-10-10T17:41:37.652108+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:55:26.031411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597853+0000","last_change":"2023-10-10T17:40:41.610002+0000","last_active":"2023-10-10T17:41:37.597853+0000","last_peered":"2023-10-10T17:41:37.597853+0000","last_clean":"2023-10-10T17:41:37.597853+0000","last_became_active":"2023-10-10T17:40:41.609708+0000","last_became_peered":"2023-10-10T17:40:41.609708+0000","last_unstale":"2023-10-10T17:41:37.597853+0000","last_undegraded":"2023-10-10T17:41:37.597853+0000","last_fullsized":"2023-10-10T17:41:37.597853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:20:34.611184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601055+0000","last_change":"2023-10-10T17:40:41.167715+0000","last_active":"2023-10-10T17:41:37.601055+0000","last_peered":"2023-10-10T17:41:37.601055+0000","last_clean":"2023-10-10T17:41:37.601055+0000","last_became_active":"2023-10-10T17:40:41.166385+0000","last_became_peered":"2023-10-10T17:40:41.166385+0000","last_unstale":"2023-10-10T17:41:37.601055+0000","last_undegraded":"2023-10-10T17:41:37.601055+0000","last_fullsized":"2023-10-10T17:41:37.601055+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:49:05.385383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788299+0000","last_change":"2023-10-10T17:40:45.707236+0000","last_active":"2023-10-10T17:41:37.788299+0000","last_peered":"2023-10-10T17:41:37.788299+0000","last_clean":"2023-10-10T17:41:37.788299+0000","last_became_active":"2023-10-10T17:40:45.706827+0000","last_became_peered":"2023-10-10T17:40:45.706827+0000","last_unstale":"2023-10-10T17:41:37.788299+0000","last_undegraded":"2023-10-10T17:41:37.788299+0000","last_fullsized":"2023-10-10T17:41:37.788299+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:46:58.930376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601396+0000","last_change":"2023-10-10T17:40:41.170523+0000","last_active":"2023-10-10T17:41:37.601396+0000","last_peered":"2023-10-10T17:41:37.601396+0000","last_clean":"2023-10-10T17:41:37.601396+0000","last_became_active":"2023-10-10T17:40:41.170243+0000","last_became_peered":"2023-10-10T17:40:41.170243+0000","last_unstale":"2023-10-10T17:41:37.601396+0000","last_undegraded":"2023-10-10T17:41:37.601396+0000","last_fullsized":"2023-10-10T17:41:37.601396+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:40.239129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599195+0000","last_change":"2023-10-10T17:40:41.610472+0000","last_active":"2023-10-10T17:41:37.599195+0000","last_peered":"2023-10-10T17:41:37.599195+0000","last_clean":"2023-10-10T17:41:37.599195+0000","last_became_active":"2023-10-10T17:40:41.610139+0000","last_became_peered":"2023-10-10T17:40:41.610139+0000","last_unstale":"2023-10-10T17:41:37.599195+0000","last_undegraded":"2023-10-10T17:41:37.599195+0000","last_fullsized":"2023-10-10T17:41:37.599195+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:22:52.414485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979558+0000","last_change":"2023-10-10T17:40:45.712179+0000","last_active":"2023-10-10T17:41:37.979558+0000","last_peered":"2023-10-10T17:41:37.979558+0000","last_clean":"2023-10-10T17:41:37.979558+0000","last_became_active":"2023-10-10T17:40:45.711708+0000","last_became_peered":"2023-10-10T17:40:45.711708+0000","last_unstale":"2023-10-10T17:41:37.979558+0000","last_undegraded":"2023-10-10T17:41:37.979558+0000","last_fullsized":"2023-10-10T17:41:37.979558+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:25:37.789886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595330+0000","last_change":"2023-10-10T17:40:41.164882+0000","last_active":"2023-10-10T17:41:37.595330+0000","last_peered":"2023-10-10T17:41:37.595330+0000","last_clean":"2023-10-10T17:41:37.595330+0000","last_became_active":"2023-10-10T17:40:41.164536+0000","last_became_peered":"2023-10-10T17:40:41.164536+0000","last_unstale":"2023-10-10T17:41:37.595330+0000","last_undegraded":"2023-10-10T17:41:37.595330+0000","last_fullsized":"2023-10-10T17:41:37.595330+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:20:38.161441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.980304+0000","last_change":"2023-10-10T17:40:41.595043+0000","last_active":"2023-10-10T17:41:37.980304+0000","last_peered":"2023-10-10T17:41:37.980304+0000","last_clean":"2023-10-10T17:41:37.980304+0000","last_became_active":"2023-10-10T17:40:41.594703+0000","last_became_peered":"2023-10-10T17:40:41.594703+0000","last_unstale":"2023-10-10T17:41:37.980304+0000","last_undegraded":"2023-10-10T17:41:37.980304+0000","last_fullsized":"2023-10-10T17:41:37.980304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:29:21.277302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596130+0000","last_change":"2023-10-10T17:40:45.705232+0000","last_active":"2023-10-10T17:41:37.596130+0000","last_peered":"2023-10-10T17:41:37.596130+0000","last_clean":"2023-10-10T17:41:37.596130+0000","last_became_active":"2023-10-10T17:40:45.704811+0000","last_became_peered":"2023-10-10T17:40:45.704811+0000","last_unstale":"2023-10-10T17:41:37.596130+0000","last_undegraded":"2023-10-10T17:41:37.596130+0000","last_fullsized":"2023-10-10T17:41:37.596130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:54:50.034871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602441+0000","last_change":"2023-10-10T17:40:41.165620+0000","last_active":"2023-10-10T17:41:37.602441+0000","last_peered":"2023-10-10T17:41:37.602441+0000","last_clean":"2023-10-10T17:41:37.602441+0000","last_became_active":"2023-10-10T17:40:41.165262+0000","last_became_peered":"2023-10-10T17:40:41.165262+0000","last_unstale":"2023-10-10T17:41:37.602441+0000","last_undegraded":"2023-10-10T17:41:37.602441+0000","last_fullsized":"2023-10-10T17:41:37.602441+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:42:19.126274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.599185+0000","last_change":"2023-10-10T17:40:41.603549+0000","last_active":"2023-10-10T17:41:37.599185+0000","last_peered":"2023-10-10T17:41:37.599185+0000","last_clean":"2023-10-10T17:41:37.599185+0000","last_became_active":"2023-10-10T17:40:41.602991+0000","last_became_peered":"2023-10-10T17:40:41.602991+0000","last_unstale":"2023-10-10T17:41:37.599185+0000","last_undegraded":"2023-10-10T17:41:37.599185+0000","last_fullsized":"2023-10-10T17:41:37.599185+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:43:30.283350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.606557+0000","last_change":"2023-10-10T17:41:37.606557+0000","last_active":"2023-10-10T17:41:37.606557+0000","last_peered":"2023-10-10T17:41:37.606557+0000","last_clean":"2023-10-10T17:41:37.606557+0000","last_became_active":"2023-10-10T17:41:37.606070+0000","last_became_peered":"2023-10-10T17:41:37.606070+0000","last_unstale":"2023-10-10T17:41:37.606557+0000","last_undegraded":"2023-10-10T17:41:37.606557+0000","last_fullsized":"2023-10-10T17:41:37.606557+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:09:50.492126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595549+0000","last_change":"2023-10-10T17:40:41.169859+0000","last_active":"2023-10-10T17:41:37.595549+0000","last_peered":"2023-10-10T17:41:37.595549+0000","last_clean":"2023-10-10T17:41:37.595549+0000","last_became_active":"2023-10-10T17:40:41.169507+0000","last_became_peered":"2023-10-10T17:40:41.169507+0000","last_unstale":"2023-10-10T17:41:37.595549+0000","last_undegraded":"2023-10-10T17:41:37.595549+0000","last_fullsized":"2023-10-10T17:41:37.595549+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:29:19.452819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600382+0000","last_change":"2023-10-10T17:40:41.593929+0000","last_active":"2023-10-10T17:41:37.600382+0000","last_peered":"2023-10-10T17:41:37.600382+0000","last_clean":"2023-10-10T17:41:37.600382+0000","last_became_active":"2023-10-10T17:40:41.593565+0000","last_became_peered":"2023-10-10T17:40:41.593565+0000","last_unstale":"2023-10-10T17:41:37.600382+0000","last_undegraded":"2023-10-10T17:41:37.600382+0000","last_fullsized":"2023-10-10T17:41:37.600382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:27:52.950921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.595626+0000","last_change":"2023-10-10T17:40:45.710759+0000","last_active":"2023-10-10T17:41:37.595626+0000","last_peered":"2023-10-10T17:41:37.595626+0000","last_clean":"2023-10-10T17:41:37.595626+0000","last_became_active":"2023-10-10T17:40:45.710118+0000","last_became_peered":"2023-10-10T17:40:45.710118+0000","last_unstale":"2023-10-10T17:41:37.595626+0000","last_undegraded":"2023-10-10T17:41:37.595626+0000","last_fullsized":"2023-10-10T17:41:37.595626+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:03:22.069829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979760+0000","last_change":"2023-10-10T17:40:41.165988+0000","last_active":"2023-10-10T17:41:37.979760+0000","last_peered":"2023-10-10T17:41:37.979760+0000","last_clean":"2023-10-10T17:41:37.979760+0000","last_became_active":"2023-10-10T17:40:41.165407+0000","last_became_peered":"2023-10-10T17:40:41.165407+0000","last_unstale":"2023-10-10T17:41:37.979760+0000","last_undegraded":"2023-10-10T17:41:37.979760+0000","last_fullsized":"2023-10-10T17:41:37.979760+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:16:55.568098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787946+0000","last_change":"2023-10-10T17:40:41.609834+0000","last_active":"2023-10-10T17:41:37.787946+0000","last_peered":"2023-10-10T17:41:37.787946+0000","last_clean":"2023-10-10T17:41:37.787946+0000","last_became_active":"2023-10-10T17:40:41.609509+0000","last_became_peered":"2023-10-10T17:40:41.609509+0000","last_unstale":"2023-10-10T17:41:37.787946+0000","last_undegraded":"2023-10-10T17:41:37.787946+0000","last_fullsized":"2023-10-10T17:41:37.787946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:26:49.739028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602247+0000","last_change":"2023-10-10T17:40:45.709003+0000","last_active":"2023-10-10T17:41:37.602247+0000","last_peered":"2023-10-10T17:41:37.602247+0000","last_clean":"2023-10-10T17:41:37.602247+0000","last_became_active":"2023-10-10T17:40:45.708610+0000","last_became_peered":"2023-10-10T17:40:45.708610+0000","last_unstale":"2023-10-10T17:41:37.602247+0000","last_undegraded":"2023-10-10T17:41:37.602247+0000","last_fullsized":"2023-10-10T17:41:37.602247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:04:55.725864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596673+0000","last_change":"2023-10-10T17:40:41.168104+0000","last_active":"2023-10-10T17:41:37.596673+0000","last_peered":"2023-10-10T17:41:37.596673+0000","last_clean":"2023-10-10T17:41:37.596673+0000","last_became_active":"2023-10-10T17:40:41.167802+0000","last_became_peered":"2023-10-10T17:40:41.167802+0000","last_unstale":"2023-10-10T17:41:37.596673+0000","last_undegraded":"2023-10-10T17:41:37.596673+0000","last_fullsized":"2023-10-10T17:41:37.596673+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:22:40.050459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979300+0000","last_change":"2023-10-10T17:40:41.606553+0000","last_active":"2023-10-10T17:41:37.979300+0000","last_peered":"2023-10-10T17:41:37.979300+0000","last_clean":"2023-10-10T17:41:37.979300+0000","last_became_active":"2023-10-10T17:40:41.606087+0000","last_became_peered":"2023-10-10T17:40:41.606087+0000","last_unstale":"2023-10-10T17:41:37.979300+0000","last_undegraded":"2023-10-10T17:41:37.979300+0000","last_fullsized":"2023-10-10T17:41:37.979300+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T21:44:40.056635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602105+0000","last_change":"2023-10-10T17:40:45.712028+0000","last_active":"2023-10-10T17:41:37.602105+0000","last_peered":"2023-10-10T17:41:37.602105+0000","last_clean":"2023-10-10T17:41:37.602105+0000","last_became_active":"2023-10-10T17:40:45.711534+0000","last_became_peered":"2023-10-10T17:40:45.711534+0000","last_unstale":"2023-10-10T17:41:37.602105+0000","last_undegraded":"2023-10-10T17:41:37.602105+0000","last_fullsized":"2023-10-10T17:41:37.602105+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:51:36.426627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600532+0000","last_change":"2023-10-10T17:40:41.164456+0000","last_active":"2023-10-10T17:41:37.600532+0000","last_peered":"2023-10-10T17:41:37.600532+0000","last_clean":"2023-10-10T17:41:37.600532+0000","last_became_active":"2023-10-10T17:40:41.164018+0000","last_became_peered":"2023-10-10T17:40:41.164018+0000","last_unstale":"2023-10-10T17:41:37.600532+0000","last_undegraded":"2023-10-10T17:41:37.600532+0000","last_fullsized":"2023-10-10T17:41:37.600532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:54:18.517460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.601647+0000","last_change":"2023-10-10T17:40:41.599724+0000","last_active":"2023-10-10T17:41:37.601647+0000","last_peered":"2023-10-10T17:41:37.601647+0000","last_clean":"2023-10-10T17:41:37.601647+0000","last_became_active":"2023-10-10T17:40:41.599356+0000","last_became_peered":"2023-10-10T17:40:41.599356+0000","last_unstale":"2023-10-10T17:41:37.601647+0000","last_undegraded":"2023-10-10T17:41:37.601647+0000","last_fullsized":"2023-10-10T17:41:37.601647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:18:57.775792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596111+0000","last_change":"2023-10-10T17:40:45.704911+0000","last_active":"2023-10-10T17:41:37.596111+0000","last_peered":"2023-10-10T17:41:37.596111+0000","last_clean":"2023-10-10T17:41:37.596111+0000","last_became_active":"2023-10-10T17:40:45.704299+0000","last_became_peered":"2023-10-10T17:40:45.704299+0000","last_unstale":"2023-10-10T17:41:37.596111+0000","last_undegraded":"2023-10-10T17:41:37.596111+0000","last_fullsized":"2023-10-10T17:41:37.596111+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:13:42.759241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.597150+0000","last_change":"2023-10-10T17:40:41.166091+0000","last_active":"2023-10-10T17:41:37.597150+0000","last_peered":"2023-10-10T17:41:37.597150+0000","last_clean":"2023-10-10T17:41:37.597150+0000","last_became_active":"2023-10-10T17:40:41.165610+0000","last_became_peered":"2023-10-10T17:40:41.165610+0000","last_unstale":"2023-10-10T17:41:37.597150+0000","last_undegraded":"2023-10-10T17:41:37.597150+0000","last_fullsized":"2023-10-10T17:41:37.597150+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:31:09.299708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602252+0000","last_change":"2023-10-10T17:40:41.600837+0000","last_active":"2023-10-10T17:41:37.602252+0000","last_peered":"2023-10-10T17:41:37.602252+0000","last_clean":"2023-10-10T17:41:37.602252+0000","last_became_active":"2023-10-10T17:40:41.600090+0000","last_became_peered":"2023-10-10T17:40:41.600090+0000","last_unstale":"2023-10-10T17:41:37.602252+0000","last_undegraded":"2023-10-10T17:41:37.602252+0000","last_fullsized":"2023-10-10T17:41:37.602252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:15:21.942555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603744+0000","last_change":"2023-10-10T17:41:37.603744+0000","last_active":"2023-10-10T17:41:37.603744+0000","last_peered":"2023-10-10T17:41:37.603744+0000","last_clean":"2023-10-10T17:41:37.603744+0000","last_became_active":"2023-10-10T17:41:37.603376+0000","last_became_peered":"2023-10-10T17:41:37.603376+0000","last_unstale":"2023-10-10T17:41:37.603744+0000","last_undegraded":"2023-10-10T17:41:37.603744+0000","last_fullsized":"2023-10-10T17:41:37.603744+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:59:51.305467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,1,4],"acting":[0,8,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.652254+0000","last_change":"2023-10-10T17:40:41.169820+0000","last_active":"2023-10-10T17:41:37.652254+0000","last_peered":"2023-10-10T17:41:37.652254+0000","last_clean":"2023-10-10T17:41:37.652254+0000","last_became_active":"2023-10-10T17:40:41.168767+0000","last_became_peered":"2023-10-10T17:40:41.168767+0000","last_unstale":"2023-10-10T17:41:37.652254+0000","last_undegraded":"2023-10-10T17:41:37.652254+0000","last_fullsized":"2023-10-10T17:41:37.652254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:22:55.616442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603091+0000","last_change":"2023-10-10T17:40:41.597482+0000","last_active":"2023-10-10T17:41:37.603091+0000","last_peered":"2023-10-10T17:41:37.603091+0000","last_clean":"2023-10-10T17:41:37.603091+0000","last_became_active":"2023-10-10T17:40:41.597166+0000","last_became_peered":"2023-10-10T17:40:41.597166+0000","last_unstale":"2023-10-10T17:41:37.603091+0000","last_undegraded":"2023-10-10T17:41:37.603091+0000","last_fullsized":"2023-10-10T17:41:37.603091+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T23:25:11.444261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651261+0000","last_change":"2023-10-10T17:40:45.707175+0000","last_active":"2023-10-10T17:41:37.651261+0000","last_peered":"2023-10-10T17:41:37.651261+0000","last_clean":"2023-10-10T17:41:37.651261+0000","last_became_active":"2023-10-10T17:40:45.706448+0000","last_became_peered":"2023-10-10T17:40:45.706448+0000","last_unstale":"2023-10-10T17:41:37.651261+0000","last_undegraded":"2023-10-10T17:41:37.651261+0000","last_fullsized":"2023-10-10T17:41:37.651261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:44.679598+0000","last_clean_scrub_stamp":"2023-10-10T17:40:44.679598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:19:33.970113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.600136+0000","last_change":"2023-10-10T17:40:41.169806+0000","last_active":"2023-10-10T17:41:37.600136+0000","last_peered":"2023-10-10T17:41:37.600136+0000","last_clean":"2023-10-10T17:41:37.600136+0000","last_became_active":"2023-10-10T17:40:41.169496+0000","last_became_peered":"2023-10-10T17:40:41.169496+0000","last_unstale":"2023-10-10T17:41:37.600136+0000","last_undegraded":"2023-10-10T17:41:37.600136+0000","last_fullsized":"2023-10-10T17:41:37.600136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:40.140250+0000","last_clean_scrub_stamp":"2023-10-10T17:40:40.140250+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T18:28:16.024298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.603755+0000","last_change":"2023-10-10T17:40:41.598366+0000","last_active":"2023-10-10T17:41:37.603755+0000","last_peered":"2023-10-10T17:41:37.603755+0000","last_clean":"2023-10-10T17:41:37.603755+0000","last_became_active":"2023-10-10T17:40:41.598036+0000","last_became_peered":"2023-10-10T17:40:41.598036+0000","last_unstale":"2023-10-10T17:41:37.603755+0000","last_undegraded":"2023-10-10T17:41:37.603755+0000","last_fullsized":"2023-10-10T17:41:37.603755+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T19:59:22.632882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.787912+0000","last_change":"2023-10-10T17:40:41.609923+0000","last_active":"2023-10-10T17:41:37.787912+0000","last_peered":"2023-10-10T17:41:37.787912+0000","last_clean":"2023-10-10T17:41:37.787912+0000","last_became_active":"2023-10-10T17:40:41.609575+0000","last_became_peered":"2023-10-10T17:40:41.609575+0000","last_unstale":"2023-10-10T17:41:37.787912+0000","last_undegraded":"2023-10-10T17:41:37.787912+0000","last_fullsized":"2023-10-10T17:41:37.787912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:14:10.560266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596871+0000","last_change":"2023-10-10T17:40:41.609012+0000","last_active":"2023-10-10T17:41:37.596871+0000","last_peered":"2023-10-10T17:41:37.596871+0000","last_clean":"2023-10-10T17:41:37.596871+0000","last_became_active":"2023-10-10T17:40:41.608738+0000","last_became_peered":"2023-10-10T17:40:41.608738+0000","last_unstale":"2023-10-10T17:41:37.596871+0000","last_undegraded":"2023-10-10T17:41:37.596871+0000","last_fullsized":"2023-10-10T17:41:37.596871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:11:39.107240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596125+0000","last_change":"2023-10-10T17:40:41.608801+0000","last_active":"2023-10-10T17:41:37.596125+0000","last_peered":"2023-10-10T17:41:37.596125+0000","last_clean":"2023-10-10T17:41:37.596125+0000","last_became_active":"2023-10-10T17:40:41.608510+0000","last_became_peered":"2023-10-10T17:40:41.608510+0000","last_unstale":"2023-10-10T17:41:37.596125+0000","last_undegraded":"2023-10-10T17:41:37.596125+0000","last_fullsized":"2023-10-10T17:41:37.596125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T00:16:53.945359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598418+0000","last_change":"2023-10-10T17:40:41.603516+0000","last_active":"2023-10-10T17:41:37.598418+0000","last_peered":"2023-10-10T17:41:37.598418+0000","last_clean":"2023-10-10T17:41:37.598418+0000","last_became_active":"2023-10-10T17:40:41.602948+0000","last_became_peered":"2023-10-10T17:40:41.602948+0000","last_unstale":"2023-10-10T17:41:37.598418+0000","last_undegraded":"2023-10-10T17:41:37.598418+0000","last_fullsized":"2023-10-10T17:41:37.598418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:18:04.694328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.602112+0000","last_change":"2023-10-10T17:41:37.602112+0000","last_active":"2023-10-10T17:41:37.602112+0000","last_peered":"2023-10-10T17:41:37.602112+0000","last_clean":"2023-10-10T17:41:37.602112+0000","last_became_active":"2023-10-10T17:41:37.601781+0000","last_became_peered":"2023-10-10T17:41:37.601781+0000","last_unstale":"2023-10-10T17:41:37.602112+0000","last_undegraded":"2023-10-10T17:41:37.602112+0000","last_fullsized":"2023-10-10T17:41:37.602112+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T20:34:00.565768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788702+0000","last_change":"2023-10-10T17:40:41.597595+0000","last_active":"2023-10-10T17:41:37.788702+0000","last_peered":"2023-10-10T17:41:37.788702+0000","last_clean":"2023-10-10T17:41:37.788702+0000","last_became_active":"2023-10-10T17:40:41.597301+0000","last_became_peered":"2023-10-10T17:40:41.597301+0000","last_unstale":"2023-10-10T17:41:37.788702+0000","last_undegraded":"2023-10-10T17:41:37.788702+0000","last_fullsized":"2023-10-10T17:41:37.788702+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T22:51:11.601251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596929+0000","last_change":"2023-10-10T17:40:41.611841+0000","last_active":"2023-10-10T17:41:37.596929+0000","last_peered":"2023-10-10T17:41:37.596929+0000","last_clean":"2023-10-10T17:41:37.596929+0000","last_became_active":"2023-10-10T17:40:41.611438+0000","last_became_peered":"2023-10-10T17:40:41.611438+0000","last_unstale":"2023-10-10T17:41:37.596929+0000","last_undegraded":"2023-10-10T17:41:37.596929+0000","last_fullsized":"2023-10-10T17:41:37.596929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T04:29:05.260738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.979112+0000","last_change":"2023-10-10T17:40:41.607707+0000","last_active":"2023-10-10T17:41:37.979112+0000","last_peered":"2023-10-10T17:41:37.979112+0000","last_clean":"2023-10-10T17:41:37.979112+0000","last_became_active":"2023-10-10T17:40:41.607412+0000","last_became_peered":"2023-10-10T17:40:41.607412+0000","last_unstale":"2023-10-10T17:41:37.979112+0000","last_undegraded":"2023-10-10T17:41:37.979112+0000","last_fullsized":"2023-10-10T17:41:37.979112+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:09:49.015272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.651033+0000","last_change":"2023-10-10T17:40:41.593674+0000","last_active":"2023-10-10T17:41:37.651033+0000","last_peered":"2023-10-10T17:41:37.651033+0000","last_clean":"2023-10-10T17:41:37.651033+0000","last_became_active":"2023-10-10T17:40:41.593339+0000","last_became_peered":"2023-10-10T17:40:41.593339+0000","last_unstale":"2023-10-10T17:41:37.651033+0000","last_undegraded":"2023-10-10T17:41:37.651033+0000","last_fullsized":"2023-10-10T17:41:37.651033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:39:37.705884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.978977+0000","last_change":"2023-10-10T17:40:41.610075+0000","last_active":"2023-10-10T17:41:37.978977+0000","last_peered":"2023-10-10T17:41:37.978977+0000","last_clean":"2023-10-10T17:41:37.978977+0000","last_became_active":"2023-10-10T17:40:41.609669+0000","last_became_peered":"2023-10-10T17:40:41.609669+0000","last_unstale":"2023-10-10T17:41:37.978977+0000","last_undegraded":"2023-10-10T17:41:37.978977+0000","last_fullsized":"2023-10-10T17:41:37.978977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T02:39:42.281285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.596551+0000","last_change":"2023-10-10T17:40:41.596780+0000","last_active":"2023-10-10T17:41:37.596551+0000","last_peered":"2023-10-10T17:41:37.596551+0000","last_clean":"2023-10-10T17:41:37.596551+0000","last_became_active":"2023-10-10T17:40:41.596477+0000","last_became_peered":"2023-10-10T17:40:41.596477+0000","last_unstale":"2023-10-10T17:41:37.596551+0000","last_undegraded":"2023-10-10T17:41:37.596551+0000","last_fullsized":"2023-10-10T17:41:37.596551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-11T17:51:14.200732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.598899+0000","last_change":"2023-10-10T17:40:41.606226+0000","last_active":"2023-10-10T17:41:37.598899+0000","last_peered":"2023-10-10T17:41:37.598899+0000","last_clean":"2023-10-10T17:41:37.598899+0000","last_became_active":"2023-10-10T17:40:41.605843+0000","last_became_peered":"2023-10-10T17:40:41.605843+0000","last_unstale":"2023-10-10T17:41:37.598899+0000","last_undegraded":"2023-10-10T17:41:37.598899+0000","last_fullsized":"2023-10-10T17:41:37.598899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T01:26:44.165622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-10T17:41:37.788525+0000","last_change":"2023-10-10T17:40:41.596122+0000","last_active":"2023-10-10T17:41:37.788525+0000","last_peered":"2023-10-10T17:41:37.788525+0000","last_clean":"2023-10-10T17:41:37.788525+0000","last_became_active":"2023-10-10T17:40:41.595818+0000","last_became_peered":"2023-10-10T17:40:41.595818+0000","last_unstale":"2023-10-10T17:41:37.788525+0000","last_undegraded":"2023-10-10T17:41:37.788525+0000","last_fullsized":"2023-10-10T17:41:37.788525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-10T17:40:41.144942+0000","last_clean_scrub_stamp":"2023-10-10T17:40:41.144942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-12T03:24:32.658064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":250871,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7568,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":512,"num_read_kb":270,"num_write":131,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":22704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6156,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738996,"statfs":{"total":95995035648,"available":95988731904,"internally_reserved":0,"allocated":987136,"data_stored":492386,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.82199999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79200000000000004,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.56399999999999995}]},{"osd":1,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.75800000000000001,"15min":0.77700000000000002},"min":{"1min":0.46899999999999997,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.90200000000000002,"5min":4.2000000000000002,"15min":4.2000000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.68600000000000005,"15min":0.69599999999999995},"min":{"1min":0.45700000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86099999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.60099999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48899999999999999,"15min":0.48999999999999999},"min":{"1min":0.32800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.65400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.49299999999999999,"15min":0.48999999999999999},"min":{"1min":0.34200000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.66000000000000003,"5min":1.026,"15min":1.026},"last":0.50600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.80600000000000005,"15min":0.82599999999999996},"min":{"1min":0.48299999999999998,"5min":0.375,"15min":0.375},"max":{"1min":0.95799999999999996,"5min":4.1319999999999997,"15min":4.1319999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.70999999999999996,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.97499999999999998,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.80300000000000005}]},{"osd":5,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.33600000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78000000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.628,"15min":0.628},"min":{"1min":0.40400000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81599999999999995,"5min":1.129,"15min":1.129},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66800000000000004},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.85999999999999999,"5min":1.107,"15min":1.107},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.66700000000000004}]},{"osd":7,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.80900000000000005,"15min":0.81999999999999995},"min":{"1min":0.51300000000000001,"5min":0.439,"15min":0.439},"max":{"1min":1.01,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.81200000000000006,"15min":0.82799999999999996},"min":{"1min":0.437,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0309999999999999,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.85299999999999998}]},{"osd":8,"last update":"Tue Oct 10 17:41:41 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.75800000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.72699999999999998,"15min":0.72899999999999998},"min":{"1min":0.47599999999999998,"5min":0.44,"15min":0.44},"max":{"1min":0.93600000000000005,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.752,"15min":0.753},"min":{"1min":0.52100000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.05,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.83799999999999997}]},{"osd":10,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.97599999999999998,"15min":1.01},"min":{"1min":0.60699999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.117,"5min":5.0919999999999996,"15min":5.0919999999999996},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.84599999999999997,"15min":0.85899999999999999},"min":{"1min":0.61399999999999999,"5min":0.502,"15min":0.502},"max":{"1min":1.0720000000000001,"5min":2.5510000000000002,"15min":2.5510000000000002},"last":0.82699999999999996}]}]},{"osd":10,"up_from":73,"seq":313532612643,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6432,"kb_used_data":1048,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738720,"statfs":{"total":95995035648,"available":95988449280,"internally_reserved":0,"allocated":1073152,"data_stored":1087425,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.435}]},{"osd":1,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.44500000000000001,"15min":0.441},"min":{"1min":0.32700000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45600000000000002,"15min":0.45200000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54200000000000004}]},{"osd":2,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.47399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78200000000000003,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.59899999999999998,"15min":0.58899999999999997},"min":{"1min":0.51500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.80200000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68600000000000005}]},{"osd":3,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57599999999999996,"15min":0.58399999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.76700000000000002,"5min":1.071,"15min":1.071},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.437,"5min":0.374,"15min":0.374},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.501}]},{"osd":4,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.35299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53200000000000003,"15min":0.52500000000000002},"min":{"1min":0.41199999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.82199999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.63}]},{"osd":5,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.65900000000000003,"15min":0.65400000000000003},"min":{"1min":0.52900000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.81599999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.67200000000000004,"15min":0.66300000000000003},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.73499999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.56699999999999995,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.90100000000000002,"5min":1.05,"15min":1.05},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.64400000000000002,"15min":0.64300000000000002},"min":{"1min":0.41999999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.71599999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:40:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.70899999999999996}]},{"osd":8,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70399999999999996,"15min":0.69999999999999996},"min":{"1min":0.60299999999999998,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.85499999999999998,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.753},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.63500000000000001,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.91900000000000004,"5min":1.157,"15min":1.157},"last":0.76300000000000001}]},{"osd":9,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88800000000000001,"5min":1.004,"15min":1.004},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.755,"15min":0.76000000000000001},"min":{"1min":0.53100000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92800000000000005,"5min":1.2,"15min":1.2},"last":0.80800000000000005}]},{"osd":11,"last update":"Tue Oct 10 17:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.76300000000000001,"15min":0.76100000000000001},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77100000000000002,"15min":0.77000000000000002},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.94299999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.78300000000000003}]}]},{"osd":9,"up_from":68,"seq":292057776169,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6364,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":1003520,"data_stored":492622,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.42399999999999999,"15min":0.436},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.49099999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.373},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.48199999999999998,"15min":0.502},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.54000000000000004,"5min":1.016,"15min":1.016},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.60799999999999998,"15min":0.61499999999999999},"min":{"1min":0.433,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.85799999999999998,"5min":1.103,"15min":1.103},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.627,"15min":0.63900000000000001},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.755,"5min":1.03,"15min":1.03},"last":0.52100000000000002}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.59899999999999998,"15min":0.61299999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.69199999999999995,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.60299999999999998,"15min":0.61499999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.70199999999999996,"5min":1.151,"15min":1.151},"last":0.5}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.54000000000000004,"15min":0.56299999999999994},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.62,"5min":0.748,"15min":0.748},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.54600000000000004,"15min":0.56899999999999995},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64300000000000002,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":0.38700000000000001}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.73799999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.68999999999999995,"15min":0.70999999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.84099999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.69599999999999995,"15min":0.71999999999999997},"min":{"1min":0.439,"5min":0.436,"15min":0.436},"max":{"1min":0.77000000000000002,"5min":1.3169999999999999,"15min":1.3169999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.69899999999999995,"15min":0.72099999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.71299999999999997,"5min":1.417,"15min":1.417},"last":0.48099999999999998}]},{"osd":6,"last update":"Tue Oct 10 17:40:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1160000000000001,"5min":1.1160000000000001,"15min":1.1160000000000001},"last":0.52600000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.76700000000000002,"15min":0.79100000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.90400000000000003,"5min":1.2589999999999999,"15min":1.2589999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.76300000000000001,"15min":0.78700000000000003},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.80900000000000005,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67000000000000004}]},{"osd":8,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.753,"15min":0.77500000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77600000000000002,"5min":1.4450000000000001,"15min":1.4450000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.76900000000000002,"15min":0.78900000000000003},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.79400000000000004,"5min":1.377,"15min":1.377},"last":0.67800000000000005}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69799999999999995,"15min":0.70999999999999996},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.83399999999999996,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.71699999999999997,"15min":0.72299999999999998},"min":{"1min":0.52900000000000003,"5min":0.497,"15min":0.497},"max":{"1min":0.876,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.64500000000000002}]},{"osd":11,"last update":"Tue Oct 10 17:41:50 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.65000000000000002}]}]},{"osd":8,"up_from":60,"seq":257698037805,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6492,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738660,"statfs":{"total":95995035648,"available":95988387840,"internally_reserved":0,"allocated":1003520,"data_stored":492542,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.53500000000000003,"15min":0.52300000000000002},"min":{"1min":0.436,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.64400000000000002,"15min":0.58399999999999996},"min":{"1min":0.44500000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.61899999999999999}]},{"osd":1,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.58199999999999996,"15min":0.56100000000000005},"min":{"1min":0.40300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84599999999999997,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.624,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.86099999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.42799999999999999,"15min":0.40000000000000002},"min":{"1min":0.29399999999999998,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64100000000000001,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.47599999999999998,"15min":0.45200000000000001},"min":{"1min":0.27000000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52200000000000002}]},{"osd":3,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.57999999999999996},"min":{"1min":0.439,"5min":0.37,"15min":0.37},"max":{"1min":0.94799999999999995,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.012,"5min":0.65500000000000003,"15min":0.58999999999999997},"min":{"1min":0.52100000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.3959999999999999,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.68799999999999994}]},{"osd":4,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.60999999999999999,"15min":0.58799999999999997},"min":{"1min":0.35499999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.85099999999999998,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.63100000000000001,"15min":0.60299999999999998},"min":{"1min":0.441,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72999999999999998}]},{"osd":5,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.53900000000000003,"15min":0.50900000000000001},"min":{"1min":0.33600000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.56499999999999995,"15min":0.53100000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.93799999999999994,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.69899999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.48999999999999999,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.93799999999999994,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.70899999999999996,"15min":0.69399999999999995},"min":{"1min":0.56299999999999994,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.90400000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.77200000000000002}]},{"osd":7,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.71599999999999997,"15min":0.68700000000000006},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.999,"5min":1.022,"15min":1.022},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.69899999999999995,"15min":0.67300000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91900000000000004,"5min":1.101,"15min":1.101},"last":0.74299999999999999}]},{"osd":9,"last update":"Tue Oct 10 17:42:20 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.79500000000000004,"15min":0.72499999999999998},"min":{"1min":0.58499999999999996,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.012,"5min":6.3630000000000004,"15min":6.3630000000000004},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.76400000000000001,"15min":0.73999999999999999},"min":{"1min":0.58499999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.036,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.83499999999999996}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.751,"15min":0.73199999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.87},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.73699999999999999,"15min":0.71499999999999997},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.88600000000000001}]},{"osd":11,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.76000000000000001}]}]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6596,"kb_used_data":1020,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738556,"statfs":{"total":95995035648,"available":95988281344,"internally_reserved":0,"allocated":1044480,"data_stored":1083855,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.53500000000000003},"min":{"1min":0.376,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.72099999999999997,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.374}]},{"osd":1,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40899999999999997,"15min":0.39300000000000002},"min":{"1min":0.28699999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.318,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72099999999999997,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57999999999999996,"15min":0.58099999999999996},"min":{"1min":0.436,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.75600000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.78000000000000003,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70099999999999996}]},{"osd":3,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57599999999999996,"15min":0.56699999999999995},"min":{"1min":0.39100000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.59799999999999998,"15min":0.58899999999999997},"min":{"1min":0.48399999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.755,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.46200000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51300000000000001,"15min":0.50700000000000001},"min":{"1min":0.312,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.495},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48099999999999998,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.432}]},{"osd":5,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.73499999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.64200000000000002,"15min":0.63200000000000001},"min":{"1min":0.442,"5min":0.441,"15min":0.441},"max":{"1min":0.85499999999999998,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64000000000000001}]},{"osd":6,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.66400000000000003,"15min":0.65700000000000003},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.63500000000000001,"15min":0.63300000000000001},"min":{"1min":0.40799999999999997,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.374}]},{"osd":8,"last update":"Tue Oct 10 17:42:03 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.70699999999999996,"15min":0.69099999999999995},"min":{"1min":0.623,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.68200000000000005,"15min":0.67200000000000004},"min":{"1min":0.58799999999999997,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.85799999999999998,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.71399999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.72499999999999998,"15min":0.71499999999999997},"min":{"1min":0.625,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.754,"15min":0.74399999999999999},"min":{"1min":0.61399999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"last":0.77300000000000002}]},{"osd":10,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.751,"15min":0.745},"min":{"1min":0.55600000000000005,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.98299999999999998,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.496,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92300000000000004,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.48199999999999998}]},{"osd":11,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66800000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6476,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738676,"statfs":{"total":95995035648,"available":95988404224,"internally_reserved":0,"allocated":987136,"data_stored":491894,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.36799999999999999,"15min":0.35599999999999998},"min":{"1min":0.28299999999999997,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.375,"15min":0.375},"min":{"1min":0.27700000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.41299999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.78200000000000003,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.54500000000000004,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.38500000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"min":{"1min":0.39800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83099999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.54100000000000004}]},{"osd":3,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.29699999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.68999999999999995,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.442,"15min":0.42599999999999999},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.33600000000000002}]},{"osd":4,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40600000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.441},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57399999999999995,"15min":0.56000000000000005},"min":{"1min":0.441,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82799999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.50800000000000001}]},{"osd":5,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59199999999999997,"15min":0.57599999999999996},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.59599999999999997},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.61399999999999999}]},{"osd":7,"last update":"Tue Oct 10 17:41:37 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59599999999999997,"15min":0.57799999999999996},"min":{"1min":0.46400000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.58299999999999996},"min":{"1min":0.40899999999999997,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89500000000000002,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.627}]},{"osd":8,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.63800000000000001,"15min":0.64000000000000001},"min":{"1min":0.46899999999999997,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.85099999999999998,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.65800000000000003,"15min":0.66400000000000003},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2090000000000001,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.60399999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.61299999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76500000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63600000000000001,"15min":0.64300000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.42899999999999999}]},{"osd":10,"last update":"Tue Oct 10 17:41:41 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.73199999999999998,"15min":0.74299999999999999},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.501},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.73899999999999999,"15min":0.753},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.875,"5min":1.042,"15min":1.042},"last":0.59099999999999997}]},{"osd":11,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7380,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737772,"statfs":{"total":95995035648,"available":95987478528,"internally_reserved":0,"allocated":995328,"data_stored":495598,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.51600000000000001,"15min":0.48599999999999999},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.60599999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57999999999999996,"15min":0.67700000000000005},"min":{"1min":0.39900000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.80400000000000005,"5min":5.859,"15min":5.859},"last":0.44400000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.58099999999999996,"15min":0.51900000000000002},"min":{"1min":0.46200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.88300000000000001,"5min":1,"15min":1},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.623,"15min":0.60899999999999999},"min":{"1min":0.47299999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86899999999999999,"5min":2.8479999999999999,"15min":2.8479999999999999},"last":0.84499999999999997}]},{"osd":3,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58599999999999997,"15min":0.56399999999999995},"min":{"1min":0.45500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.72699999999999998,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59699999999999998,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80500000000000005,"5min":0.875,"15min":0.875},"last":0.627}]},{"osd":4,"last update":"Tue Oct 10 17:41:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.45800000000000002,"15min":0.40400000000000003},"min":{"1min":0.28100000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.46999999999999997,"15min":0.41499999999999998},"min":{"1min":0.38700000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.501}]},{"osd":5,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59499999999999997,"15min":0.55700000000000005},"min":{"1min":0.438,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.60799999999999998,"15min":0.55600000000000005},"min":{"1min":0.42799999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.70199999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:42:16 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60299999999999998,"15min":0.59299999999999997},"min":{"1min":0.41799999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.63900000000000001,"15min":0.61299999999999999},"min":{"1min":0.45200000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88800000000000001,"5min":1.034,"15min":1.034},"last":0.68700000000000006}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.59999999999999998,"15min":0.55700000000000005},"min":{"1min":0.42199999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.60799999999999998,"15min":0.58399999999999996},"min":{"1min":0.495,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1,"5min":1,"15min":1},"last":0.76700000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67300000000000004,"15min":0.65100000000000002},"min":{"1min":0.54500000000000004,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.67700000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.99099999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.85799999999999998}]},{"osd":9,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.76300000000000001,"15min":0.749},"min":{"1min":0.60999999999999999,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.91900000000000004,"5min":1.0589999999999999,"15min":1.0589999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.73399999999999999,"15min":0.71399999999999997},"min":{"1min":0.53900000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.01,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.871}]},{"osd":10,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.75800000000000001,"15min":0.748},"min":{"1min":0.58499999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.77500000000000002,"15min":0.76300000000000001},"min":{"1min":0.57599999999999996,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.006,"5min":1.008,"15min":1.008},"last":0.754}]},{"osd":11,"last update":"Tue Oct 10 17:41:44 2023","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.89000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738447,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7396,"kb_used_data":988,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737756,"statfs":{"total":95995035648,"available":95987462144,"internally_reserved":0,"allocated":1011712,"data_stored":493282,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 10 17:41:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.54500000000000004,"15min":0.53500000000000003},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55900000000000005,"15min":0.53700000000000003},"min":{"1min":0.44500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.59299999999999997,"15min":0.54300000000000004},"min":{"1min":0.42299999999999999,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.59099999999999997,"15min":0.53600000000000003},"min":{"1min":0.41299999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.274,"5min":1.274,"15min":1.274},"last":0.49199999999999999}]},{"osd":3,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":7.274,"5min":7.274,"15min":7.274},"last":0.57799999999999996}]},{"osd":4,"last update":"Tue Oct 10 17:41:36 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.55600000000000005,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.55800000000000005,"15min":0.52300000000000002},"min":{"1min":0.38700000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.5,"5min":1.5,"15min":1.5},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54500000000000004,"15min":0.51500000000000001},"min":{"1min":0.433,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.53100000000000003},"min":{"1min":0.38500000000000001,"5min":0.379,"15min":0.379},"max":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.48299999999999998,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.129,"5min":1.129,"15min":1.129},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.32100000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.75800000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63100000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":0.65200000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:41:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.69099999999999995,"15min":0.70399999999999996},"min":{"1min":0.44500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.622}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64000000000000001,"15min":0.64200000000000002},"min":{"1min":0.36099999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79900000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68000000000000005}]},{"osd":10,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71399999999999997,"15min":0.71599999999999997},"min":{"1min":0.503,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.75},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72999999999999998,"15min":0.72799999999999998},"min":{"1min":0.55300000000000005,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.427,"5min":1.427,"15min":1.427},"last":0.59799999999999998}]},{"osd":11,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.67200000000000004,"15min":0.66700000000000004},"min":{"1min":0.53000000000000003,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.70599999999999996,"15min":0.70199999999999996},"min":{"1min":0.51800000000000002,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.86599999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7456,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737696,"statfs":{"total":95995035648,"available":95987400704,"internally_reserved":0,"allocated":1007616,"data_stored":496169,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:41:49 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.51600000000000001,"15min":0.498},"min":{"1min":0.378,"5min":0.252,"15min":0.252},"max":{"1min":0.88700000000000001,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.49199999999999999,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.68300000000000005,"5min":1.177,"15min":1.177},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:49 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.60099999999999998,"15min":0.53100000000000003},"min":{"1min":0.50900000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.81399999999999995,"5min":5.484,"15min":5.484},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.56699999999999995,"15min":0.53400000000000003},"min":{"1min":0.46500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.81399999999999995,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.56999999999999995,"15min":0.52500000000000002},"min":{"1min":0.39600000000000002,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.84299999999999997,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.58599999999999997,"15min":0.54400000000000004},"min":{"1min":0.51300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84799999999999998,"5min":1.099,"15min":1.099},"last":0.56699999999999995}]},{"osd":4,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.64300000000000002}]},{"osd":5,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.48599999999999999,"15min":0.42299999999999999},"min":{"1min":0.39900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.435},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.52000000000000002,"15min":0.46800000000000003},"min":{"1min":0.38300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.41499999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.97899999999999998,"5min":1.129,"15min":1.129},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60999999999999999,"15min":0.59199999999999997},"min":{"1min":0.42299999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92300000000000004,"5min":1.075,"15min":1.075},"last":0.504}]},{"osd":7,"last update":"Tue Oct 10 17:41:45 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.755},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.84499999999999997}]},{"osd":8,"last update":"Tue Oct 10 17:42:02 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.59399999999999997,"15min":0.57799999999999996},"min":{"1min":0.45600000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.622,"15min":0.60599999999999998},"min":{"1min":0.48699999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.627}]},{"osd":9,"last update":"Tue Oct 10 17:41:20 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70299999999999996,"15min":0.70699999999999996},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.93600000000000005,"5min":1.087,"15min":1.087},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51100000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.996,"5min":1.226,"15min":1.226},"last":0.79100000000000004}]},{"osd":10,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.66100000000000003,"15min":0.64400000000000002},"min":{"1min":0.56599999999999995,"5min":0.442,"15min":0.442},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.70599999999999996,"15min":0.68300000000000005},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.83099999999999996}]},{"osd":11,"last update":"Tue Oct 10 17:41:43 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.879},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.80900000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149761,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6740,"kb_used_data":972,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738412,"statfs":{"total":95995035648,"available":95988133888,"internally_reserved":0,"allocated":995328,"data_stored":491757,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.35499999999999998,"15min":0.34799999999999998},"min":{"1min":0.24199999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.621,"5min":0.752,"15min":0.752},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.36599999999999999,"15min":0.36199999999999999},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56499999999999995,"15min":0.53900000000000003},"min":{"1min":0.39200000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.55800000000000005,"15min":0.52500000000000002},"min":{"1min":0.40300000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":1.359,"5min":1.359,"15min":1.359},"last":0.60799999999999998}]},{"osd":2,"last update":"Tue Oct 10 17:42:15 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56599999999999995,"15min":0.54000000000000004},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73099999999999998,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57099999999999995,"15min":0.54100000000000004},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80000000000000004,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.66600000000000004}]},{"osd":4,"last update":"Tue Oct 10 17:41:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59099999999999997,"15min":0.56799999999999995},"min":{"1min":0.41099999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.84099999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.627},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61299999999999999,"15min":0.59299999999999997},"min":{"1min":0.46000000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.746,"5min":1.214,"15min":1.214},"last":0.753}]},{"osd":5,"last update":"Tue Oct 10 17:42:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58699999999999997,"15min":0.59799999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.78900000000000003,"5min":1.2430000000000001,"15min":1.2430000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58899999999999997,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76100000000000001,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.67600000000000005}]},{"osd":6,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.5,"15min":0.50800000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.621,"5min":1.3149999999999999,"15min":1.3149999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.45800000000000002,"15min":0.47799999999999998},"min":{"1min":0.308,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.46999999999999997}]},{"osd":7,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.61399999999999999,"15min":0.61099999999999999},"min":{"1min":0.42699999999999999,"5min":0.375,"15min":0.375},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66700000000000004,"15min":0.66600000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.80200000000000005}]},{"osd":8,"last update":"Tue Oct 10 17:39:58 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53700000000000003}]},{"osd":9,"last update":"Tue Oct 10 17:42:19 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.624,"15min":0.65400000000000003},"min":{"1min":0.377,"5min":0.317,"15min":0.317},"max":{"1min":0.69899999999999995,"5min":1.3500000000000001,"15min":1.3500000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.64400000000000002,"15min":0.68200000000000005},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68999999999999995,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.54200000000000004}]},{"osd":10,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.847,"5min":1.847,"15min":1.847},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.52300000000000002,"5min":0.499,"15min":0.499},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.76300000000000001}]},{"osd":11,"last update":"Tue Oct 10 17:41:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.4930000000000001,"5min":1.4930000000000001,"15min":1.4930000000000001},"last":0.70799999999999996}]}]},{"osd":4,"up_from":34,"seq":146028888127,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6620,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738532,"statfs":{"total":95995035648,"available":95988256768,"internally_reserved":0,"allocated":1003520,"data_stored":492772,"data_compressed":17535,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:40:29 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.39100000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.432,"5min":0.372,"15min":0.372},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.39400000000000002,"15min":0.36599999999999999},"min":{"1min":0.31,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.65200000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.40799999999999997,"15min":0.372},"min":{"1min":0.28899999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.59999999999999998,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.63400000000000001,"15min":0.57499999999999996},"min":{"1min":0.47999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.85099999999999998}]},{"osd":3,"last update":"Tue Oct 10 17:41:30 2023","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.60899999999999999,"15min":0.54300000000000004},"min":{"1min":0.42599999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":6.1289999999999996,"5min":6.1289999999999996,"15min":6.1289999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53900000000000003,"15min":0.53700000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.75900000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57099999999999995}]},{"osd":5,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.626,"15min":0.58599999999999997},"min":{"1min":0.45500000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.83099999999999996,"5min":1.589,"15min":1.589},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.56100000000000005,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81100000000000005,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.69499999999999995}]},{"osd":6,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.60699999999999998,"15min":0.58799999999999997},"min":{"1min":0.48299999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85699999999999998,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66500000000000004,"15min":0.64300000000000002},"min":{"1min":0.499,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95399999999999996,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.57299999999999995,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.61799999999999999,"15min":0.56799999999999995},"min":{"1min":0.46899999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.76900000000000002}]},{"osd":8,"last update":"Tue Oct 10 17:42:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66700000000000004,"15min":0.65300000000000002},"min":{"1min":0.46899999999999997,"5min":0.432,"15min":0.432},"max":{"1min":0.93400000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76100000000000001,"15min":0.71799999999999997},"min":{"1min":0.60699999999999998,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.006,"5min":6.1639999999999997,"15min":6.1639999999999997},"last":0.83899999999999997}]},{"osd":9,"last update":"Tue Oct 10 17:42:21 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.73199999999999998,"15min":0.71499999999999997},"min":{"1min":0.627,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.95899999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.78100000000000003,"15min":0.76600000000000001},"min":{"1min":0.61699999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.98999999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.93799999999999994}]},{"osd":10,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68200000000000005,"15min":0.67400000000000004},"min":{"1min":0.52300000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.80400000000000005,"15min":0.79600000000000004},"min":{"1min":0.56100000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.038,"5min":1.169,"15min":1.169},"last":0.89900000000000002}]},{"osd":11,"last update":"Tue Oct 10 17:41:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.88600000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6736,"kb_used_data":1032,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738416,"statfs":{"total":95995035648,"available":95988137984,"internally_reserved":0,"allocated":1056768,"data_stored":1083327,"data_compressed":53850,"data_compressed_allocated":69632,"data_compressed_original":638976,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.57599999999999996,"15min":0.52200000000000002},"min":{"1min":0.33500000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.79600000000000004,"5min":7.1230000000000002,"15min":7.1230000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.503,"5min":0.52600000000000002,"15min":0.51900000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.69999999999999996,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.34300000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.68700000000000006,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.61899999999999999,"15min":0.55400000000000005},"min":{"1min":0.497,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.71599999999999997,"5min":7.0570000000000004,"15min":7.0570000000000004},"last":0.61699999999999999}]},{"osd":2,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.40899999999999997,"15min":0.40100000000000002},"min":{"1min":0.318,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.55700000000000005,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46100000000000002,"15min":0.42499999999999999},"min":{"1min":0.32000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.57399999999999995}]},{"osd":3,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56899999999999995,"15min":0.53800000000000003},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.9950000000000001,"5min":2.9950000000000001,"15min":2.9950000000000001},"last":0.438},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.56999999999999995,"15min":0.55700000000000005},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.83799999999999997,"5min":2.4209999999999998,"15min":2.4209999999999998},"last":0.60099999999999998}]},{"osd":4,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65100000000000002,"15min":0.59199999999999997},"min":{"1min":0.438,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76300000000000001,"5min":7.3479999999999999,"15min":7.3479999999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.41499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.73499999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.70499999999999996}]},{"osd":6,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.63200000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.1240000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.62,"15min":0.623},"min":{"1min":0.432,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.63300000000000001}]},{"osd":7,"last update":"Tue Oct 10 17:41:39 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.63700000000000001,"15min":0.61699999999999999},"min":{"1min":0.504,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82599999999999996,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.624},"min":{"1min":0.54300000000000004,"5min":0.443,"15min":0.443},"max":{"1min":0.80200000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66000000000000003}]},{"osd":8,"last update":"Tue Oct 10 17:42:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.61199999999999999,"15min":0.55600000000000005},"min":{"1min":0.47399999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80200000000000005,"5min":6.7229999999999999,"15min":6.7229999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81899999999999995,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.64700000000000002}]},{"osd":9,"last update":"Tue Oct 10 17:41:23 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.70899999999999996,"15min":0.71299999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90300000000000002,"5min":1.034,"15min":1.034},"last":0.751},{"interface":"front","average":{"1min":0.79700000000000004,"5min":0.71299999999999997,"15min":0.69899999999999995},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":3.3149999999999999,"5min":3.3149999999999999,"15min":3.3149999999999999},"last":0.76500000000000001}]},{"osd":10,"last update":"Tue Oct 10 17:41:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71699999999999997,"15min":0.70899999999999996},"min":{"1min":0.56200000000000006,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.88100000000000001,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.70799999999999996,"15min":0.70299999999999996},"min":{"1min":0.65200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34648,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36315,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":4984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":5584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-10T17:42:25.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:42:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.872+0000 7f79fceb6700 1 -- 172.21.15.78:0/4114629178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100370 msgr2=0x7f79f81007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.872+0000 7f79fceb6700 1 --2- 172.21.15.78:0/4114629178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100370 0x7f79f81007c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009990 tx=0x7f79ec009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- 172.21.15.78:0/4114629178 shutdown_connections 2023-10-10T17:42:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 --2- 172.21.15.78:0/4114629178 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f80ff180 0x7f79f80ff590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 --2- 172.21.15.78:0/4114629178 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79f8100d00 0x7f79f81075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 --2- 172.21.15.78:0/4114629178 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100370 0x7f79f81007c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- 172.21.15.78:0/4114629178 >> 172.21.15.78:0/4114629178 conn(0x7f79f80fa760 msgr2=0x7f79f80fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- 172.21.15.78:0/4114629178 shutdown_connections 2023-10-10T17:42:25.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- 172.21.15.78:0/4114629178 wait complete. 2023-10-10T17:42:25.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 Processor -- start 2023-10-10T17:42:25.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- start start 2023-10-10T17:42:25.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79f80ff180 0x7f79f8194e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f8100370 0x7f79f8195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100d00 0x7f79f819c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f819c9b0 con 0x7f79f8100d00 2023-10-10T17:42:25.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f819caf0 con 0x7f79f80ff180 2023-10-10T17:42:25.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.873+0000 7f79fceb6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f819cc80 con 0x7f79f8100370 2023-10-10T17:42:25.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100d00 0x7f79f819c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100d00 0x7f79f819c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44016/0 (socket says 172.21.15.78:44016) 2023-10-10T17:42:25.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 -- 172.21.15.78:0/1478565819 learned_addr learned my addr 172.21.15.78:0/1478565819 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:25.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f659c700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79f80ff180 0x7f79f8194e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f5d9b700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f8100370 0x7f79f8195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 -- 172.21.15.78:0/1478565819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f8100370 msgr2=0x7f79f8195370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f8100370 0x7f79f8195370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 -- 172.21.15.78:0/1478565819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79f80ff180 msgr2=0x7f79f8194e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:25.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79f80ff180 0x7f79f8194e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:25.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 -- 172.21.15.78:0/1478565819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec0095f0 con 0x7f79f8100d00 2023-10-10T17:42:25.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f5d9b700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f8100370 0x7f79f8195370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:25.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f659c700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79f80ff180 0x7f79f8194e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:25.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79f6d9d700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100d00 0x7f79f819c3e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f79f0007ee0 tx=0x7f79f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:25.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79e77fe700 1 -- 172.21.15.78:0/1478565819 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0017070 con 0x7f79f8100d00 2023-10-10T17:42:25.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79e77fe700 1 -- 172.21.15.78:0/1478565819 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f79f000f040 con 0x7f79f8100d00 2023-10-10T17:42:25.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f810aae0 con 0x7f79f8100d00 2023-10-10T17:42:25.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79e77fe700 1 -- 172.21.15.78:0/1478565819 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f0005740 con 0x7f79f8100d00 2023-10-10T17:42:25.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.874+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f810a950 con 0x7f79f8100d00 2023-10-10T17:42:25.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.875+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f8106710 con 0x7f79f8100d00 2023-10-10T17:42:25.888 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:25 smithi078 ceph-mon[99494]: pgmap v315: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-10-10T17:42:25.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.876+0000 7f79e77fe700 1 -- 172.21.15.78:0/1478565819 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f79f00058a0 con 0x7f79f8100d00 2023-10-10T17:42:25.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.877+0000 7f79e77fe700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f79e00a1150 0x7f79e00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:25.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.877+0000 7f79f659c700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f79e00a1150 0x7f79e00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:25.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.877+0000 7f79e77fe700 1 -- 172.21.15.78:0/1478565819 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f79f00cc4b0 con 0x7f79f8100d00 2023-10-10T17:42:25.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.877+0000 7f79f659c700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f79e00a1150 0x7f79e00a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f79e8005950 tx=0x7f79e80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:25.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:25.881+0000 7f79e77fe700 1 -- 172.21.15.78:0/1478565819 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f79f00958f0 con 0x7f79f8100d00 2023-10-10T17:42:25.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:25 smithi190 ceph-mon[101220]: pgmap v315: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-10-10T17:42:25.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:25 smithi157 ceph-mon[100896]: pgmap v315: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-10-10T17:42:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.069+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79f8061b70 con 0x7f79f8100d00 2023-10-10T17:42:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.071+0000 7f79e77fe700 1 -- 172.21.15.78:0/1478565819 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7f79f0098f10 con 0x7f79f8100d00 2023-10-10T17:42:26.073 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:26.073 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:42:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f79e00a1150 msgr2=0x7f79e00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f79e00a1150 0x7f79e00a3600 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f79e8005950 tx=0x7f79e80058e0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100d00 msgr2=0x7f79f819c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100d00 0x7f79f819c3e0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f79f0007ee0 tx=0x7f79f000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 shutdown_connections 2023-10-10T17:42:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f79e00a1150 0x7f79e00a3600 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f79f8100d00 0x7f79f819c3e0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f79f80ff180 0x7f79f8194e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 --2- 172.21.15.78:0/1478565819 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f79f8100370 0x7f79f8195370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 >> 172.21.15.78:0/1478565819 conn(0x7f79f80fa760 msgr2=0x7f79f8103850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.073+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 shutdown_connections 2023-10-10T17:42:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.074+0000 7f79fceb6700 1 -- 172.21.15.78:0/1478565819 wait complete. 2023-10-10T17:42:26.088 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:42:26.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:42:26.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 -- 172.21.15.78:0/3730024322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a81003b0 msgr2=0x7fd2a8100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/3730024322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a81003b0 0x7fd2a8100800 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd298009990 tx=0x7fd298009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 -- 172.21.15.78:0/3730024322 shutdown_connections 2023-10-10T17:42:26.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/3730024322 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a8100d40 0x7fd2a8107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/3730024322 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2a80ff1c0 0x7fd2a80ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/3730024322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a81003b0 0x7fd2a8100800 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 -- 172.21.15.78:0/3730024322 >> 172.21.15.78:0/3730024322 conn(0x7fd2a80fa7c0 msgr2=0x7fd2a80fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:26.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 -- 172.21.15.78:0/3730024322 shutdown_connections 2023-10-10T17:42:26.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.210+0000 7fd2ae01c700 1 -- 172.21.15.78:0/3730024322 wait complete. 2023-10-10T17:42:26.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 Processor -- start 2023-10-10T17:42:26.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 -- start start 2023-10-10T17:42:26.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a80ff1c0 0x7fd2a8194f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a81003b0 0x7fd2a8195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2a8100d40 0x7fd2a819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a819cab0 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a819cbf0 con 0x7fd2a8100d40 2023-10-10T17:42:26.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2ae01c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a819cd80 con 0x7fd2a81003b0 2023-10-10T17:42:26.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2a77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a80ff1c0 0x7fd2a8194f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2a77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a80ff1c0 0x7fd2a8194f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44026/0 (socket says 172.21.15.78:44026) 2023-10-10T17:42:26.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.211+0000 7fd2a77fe700 1 -- 172.21.15.78:0/710566081 learned_addr learned my addr 172.21.15.78:0/710566081 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:26.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a7fff700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2a8100d40 0x7fd2a819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a6ffd700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a81003b0 0x7fd2a8195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a77fe700 1 -- 172.21.15.78:0/710566081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a81003b0 msgr2=0x7fd2a8195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a77fe700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a81003b0 0x7fd2a8195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a77fe700 1 -- 172.21.15.78:0/710566081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2a8100d40 msgr2=0x7fd2a819c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a77fe700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2a8100d40 0x7fd2a819c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a77fe700 1 -- 172.21.15.78:0/710566081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980095f0 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a77fe700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a80ff1c0 0x7fd2a8194f30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0098f0 tx=0x7fd29c009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:26.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a4ff9700 1 -- 172.21.15.78:0/710566081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c012070 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a4ff9700 1 -- 172.21.15.78:0/710566081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd29c0044e0 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a810ab30 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2a4ff9700 1 -- 172.21.15.78:0/710566081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c007d10 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.212+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a810a9a0 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.213+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a80682a0 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.214+0000 7fd2a4ff9700 1 -- 172.21.15.78:0/710566081 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd29c016030 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.215+0000 7fd2a4ff9700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2900a1150 0x7fd2900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.215+0000 7fd2a6ffd700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2900a1150 0x7fd2900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.215+0000 7fd2a4ff9700 1 -- 172.21.15.78:0/710566081 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd29c0cdf00 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.215+0000 7fd2a6ffd700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2900a1150 0x7fd2900a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd2980095c0 tx=0x7fd298009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:26.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.219+0000 7fd2a4ff9700 1 -- 172.21.15.78:0/710566081 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd29c0972e0 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.406+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd2a8106160 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.406+0000 7fd2a4ff9700 1 -- 172.21.15.78:0/710566081 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd29c09a900 con 0x7fd2a80ff1c0 2023-10-10T17:42:26.408 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:42:26.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2900a1150 msgr2=0x7fd2900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2900a1150 0x7fd2900a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd2980095c0 tx=0x7fd298009510 comp rx=0 tx=0).stop 2023-10-10T17:42:26.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a80ff1c0 msgr2=0x7fd2a8194f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a80ff1c0 0x7fd2a8194f30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd29c0098f0 tx=0x7fd29c009cb0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 shutdown_connections 2023-10-10T17:42:26.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fd2900a1150 0x7fd2900a3600 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd2a8100d40 0x7fd2a819c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd2a80ff1c0 0x7fd2a8194f30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 --2- 172.21.15.78:0/710566081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd2a81003b0 0x7fd2a8195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 >> 172.21.15.78:0/710566081 conn(0x7fd2a80fa7c0 msgr2=0x7fd2a80fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:26.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 shutdown_connections 2023-10-10T17:42:26.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.409+0000 7fd2ae01c700 1 -- 172.21.15.78:0/710566081 wait complete. 2023-10-10T17:42:26.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:42:26.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 -- 172.21.15.78:0/3719552937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c0ff1e0 msgr2=0x7f8a5c0ff5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 --2- 172.21.15.78:0/3719552937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c0ff1e0 0x7f8a5c0ff5f0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009990 tx=0x7f8a4c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 -- 172.21.15.78:0/3719552937 shutdown_connections 2023-10-10T17:42:26.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 --2- 172.21.15.78:0/3719552937 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c100d60 0x7f8a5c107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 --2- 172.21.15.78:0/3719552937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c0ff1e0 0x7f8a5c0ff5f0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 --2- 172.21.15.78:0/3719552937 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a5c1003d0 0x7f8a5c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 -- 172.21.15.78:0/3719552937 >> 172.21.15.78:0/3719552937 conn(0x7f8a5c0fa800 msgr2=0x7f8a5c0fcc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:26.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 -- 172.21.15.78:0/3719552937 shutdown_connections 2023-10-10T17:42:26.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.546+0000 7f8a63bda700 1 -- 172.21.15.78:0/3719552937 wait complete. 2023-10-10T17:42:26.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 Processor -- start 2023-10-10T17:42:26.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 -- start start 2023-10-10T17:42:26.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a5c0ff1e0 0x7f8a5c196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c1003d0 0x7f8a5c1965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c100d60 0x7f8a5c19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c19dbe0 con 0x7f8a5c100d60 2023-10-10T17:42:26.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c19dd20 con 0x7f8a5c0ff1e0 2023-10-10T17:42:26.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a63bda700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c19deb0 con 0x7f8a5c1003d0 2023-10-10T17:42:26.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c100d60 0x7f8a5c19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c100d60 0x7f8a5c19d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44044/0 (socket says 172.21.15.78:44044) 2023-10-10T17:42:26.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 -- 172.21.15.78:0/202142374 learned_addr learned my addr 172.21.15.78:0/202142374 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:26.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a61976700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a5c0ff1e0 0x7f8a5c196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a61175700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c1003d0 0x7f8a5c1965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 -- 172.21.15.78:0/202142374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c1003d0 msgr2=0x7f8a5c1965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c1003d0 0x7f8a5c1965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 -- 172.21.15.78:0/202142374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a5c0ff1e0 msgr2=0x7f8a5c196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a5c0ff1e0 0x7f8a5c196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a62177700 1 -- 172.21.15.78:0/202142374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0095f0 con 0x7f8a5c100d60 2023-10-10T17:42:26.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.547+0000 7f8a61175700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c1003d0 0x7f8a5c1965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:26.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.548+0000 7f8a62177700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c100d60 0x7f8a5c19d610 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8a54007ee0 tx=0x7f8a5400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.548+0000 7f8a52ffd700 1 -- 172.21.15.78:0/202142374 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54017070 con 0x7f8a5c100d60 2023-10-10T17:42:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.548+0000 7f8a52ffd700 1 -- 172.21.15.78:0/202142374 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a5400f040 con 0x7f8a5c100d60 2023-10-10T17:42:26.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.548+0000 7f8a52ffd700 1 -- 172.21.15.78:0/202142374 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54005740 con 0x7f8a5c100d60 2023-10-10T17:42:26.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.548+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c10ab40 con 0x7f8a5c100d60 2023-10-10T17:42:26.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.548+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c10a9b0 con 0x7f8a5c100d60 2023-10-10T17:42:26.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.549+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c1072e0 con 0x7f8a5c100d60 2023-10-10T17:42:26.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.550+0000 7f8a52ffd700 1 -- 172.21.15.78:0/202142374 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8a5400dc60 con 0x7f8a5c100d60 2023-10-10T17:42:26.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.550+0000 7f8a52ffd700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8a480a1150 0x7f8a480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.550+0000 7f8a61976700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8a480a1150 0x7f8a480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.550+0000 7f8a52ffd700 1 -- 172.21.15.78:0/202142374 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8a540cca70 con 0x7f8a5c100d60 2023-10-10T17:42:26.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.551+0000 7f8a61976700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8a480a1150 0x7f8a480a3600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00b5c0 tx=0x7f8a4c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:26.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.554+0000 7f8a52ffd700 1 -- 172.21.15.78:0/202142374 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8a54095eb0 con 0x7f8a5c100d60 2023-10-10T17:42:26.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.745+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a5c061700 con 0x7f8a5c100d60 2023-10-10T17:42:26.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.745+0000 7f8a52ffd700 1 -- 172.21.15.78:0/202142374 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8a540994d0 con 0x7f8a5c100d60 2023-10-10T17:42:26.747 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8a480a1150 msgr2=0x7f8a480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8a480a1150 0x7f8a480a3600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c00b5c0 tx=0x7f8a4c005960 comp rx=0 tx=0).stop 2023-10-10T17:42:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c100d60 msgr2=0x7f8a5c19d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c100d60 0x7f8a5c19d610 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f8a54007ee0 tx=0x7f8a5400d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 shutdown_connections 2023-10-10T17:42:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f8a480a1150 0x7f8a480a3600 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a5c100d60 0x7f8a5c19d610 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8a5c0ff1e0 0x7f8a5c196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 --2- 172.21.15.78:0/202142374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8a5c1003d0 0x7f8a5c1965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.747+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 >> 172.21.15.78:0/202142374 conn(0x7f8a5c0fa800 msgr2=0x7f8a5c0fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.748+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 shutdown_connections 2023-10-10T17:42:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.748+0000 7f8a63bda700 1 -- 172.21.15.78:0/202142374 wait complete. 2023-10-10T17:42:26.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:26 smithi078 ceph-mon[99494]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:26.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:26 smithi078 ceph-mon[99494]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:26.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:26 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1478565819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:26.760 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:26 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/710566081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:26.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:42:26.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2176740013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a080ff430 msgr2=0x7f1a080ff8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2176740013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a080ff430 0x7f1a080ff8a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f19fc009990 tx=0x7f19fc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2176740013 shutdown_connections 2023-10-10T17:42:26.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2176740013 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a081007e0 0x7f1a08100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2176740013 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a080ffde0 0x7f1a08107690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2176740013 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a080ff430 0x7f1a080ff8a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2176740013 >> 172.21.15.78:0/2176740013 conn(0x7f1a080f9fb0 msgr2=0x7f1a080fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:26.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2176740013 shutdown_connections 2023-10-10T17:42:26.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2176740013 wait complete. 2023-10-10T17:42:26.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.885+0000 7f1a0d57b700 1 Processor -- start 2023-10-10T17:42:26.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a0d57b700 1 -- start start 2023-10-10T17:42:26.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a0d57b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a080ff430 0x7f1a080feeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a0d57b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a080ffde0 0x7f1a0819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a0d57b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a081007e0 0x7f1a0819d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a0d57b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0819dab0 con 0x7f1a081007e0 2023-10-10T17:42:26.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a0d57b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0819dc20 con 0x7f1a080ffde0 2023-10-10T17:42:26.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a0d57b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0819dde0 con 0x7f1a080ff430 2023-10-10T17:42:26.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a081007e0 0x7f1a0819d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a081007e0 0x7f1a0819d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44058/0 (socket says 172.21.15.78:44058) 2023-10-10T17:42:26.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 -- 172.21.15.78:0/2067741017 learned_addr learned my addr 172.21.15.78:0/2067741017 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:26.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a06ffd700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a080ff430 0x7f1a080feeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a067fc700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a080ffde0 0x7f1a0819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 -- 172.21.15.78:0/2067741017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a080ff430 msgr2=0x7f1a080feeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a080ff430 0x7f1a080feeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 -- 172.21.15.78:0/2067741017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a080ffde0 msgr2=0x7f1a0819ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:26.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a080ffde0 0x7f1a0819ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:26.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 -- 172.21.15.78:0/2067741017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0095f0 con 0x7f1a081007e0 2023-10-10T17:42:26.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f1a077fe700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a081007e0 0x7f1a0819d2e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000ef30 tx=0x7f1a0000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:26.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f19effff700 1 -- 172.21.15.78:0/2067741017 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00018070 con 0x7f1a081007e0 2023-10-10T17:42:26.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f19effff700 1 -- 172.21.15.78:0/2067741017 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1a0000f040 con 0x7f1a081007e0 2023-10-10T17:42:26.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.886+0000 7f19effff700 1 -- 172.21.15.78:0/2067741017 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0000cdb0 con 0x7f1a081007e0 2023-10-10T17:42:26.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.887+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a0810ab80 con 0x7f1a081007e0 2023-10-10T17:42:26.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.887+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a0810a9f0 con 0x7f1a081007e0 2023-10-10T17:42:26.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.887+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a081066f0 con 0x7f1a081007e0 2023-10-10T17:42:26.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.888+0000 7f19effff700 1 -- 172.21.15.78:0/2067741017 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1a00003a00 con 0x7f1a081007e0 2023-10-10T17:42:26.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.889+0000 7f19effff700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19f00a11d0 0x7f19f00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:26.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.889+0000 7f1a06ffd700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19f00a11d0 0x7f19f00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:26.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.889+0000 7f19effff700 1 -- 172.21.15.78:0/2067741017 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1a000cc7d0 con 0x7f1a081007e0 2023-10-10T17:42:26.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.890+0000 7f1a06ffd700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19f00a11d0 0x7f19f00a3680 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1a08068c10 tx=0x7f19f800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:26.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:26.893+0000 7f19effff700 1 -- 172.21.15.78:0/2067741017 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1a00095ca0 con 0x7f1a081007e0 2023-10-10T17:42:26.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:26 smithi190 ceph-mon[101220]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:26.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:26 smithi190 ceph-mon[101220]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:26.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:26 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1478565819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:26.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:26 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/710566081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:26 smithi157 ceph-mon[100896]: from='client.15486 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:26 smithi157 ceph-mon[100896]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:26 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1478565819' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:26.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:26 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/710566081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.082+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1a08061700 con 0x7f1a081007e0 2023-10-10T17:42:27.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.082+0000 7f19effff700 1 -- 172.21.15.78:0/2067741017 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1a000992c0 con 0x7f1a081007e0 2023-10-10T17:42:27.085 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:42:27.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19f00a11d0 msgr2=0x7f19f00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19f00a11d0 0x7f19f00a3680 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1a08068c10 tx=0x7f19f800a3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a081007e0 msgr2=0x7f1a0819d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a081007e0 0x7f1a0819d2e0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000ef30 tx=0x7f1a0000c3a0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 shutdown_connections 2023-10-10T17:42:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f19f00a11d0 0x7f19f00a3680 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1a080ff430 0x7f1a080feeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1a081007e0 0x7f1a0819d2e0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 --2- 172.21.15.78:0/2067741017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1a080ffde0 0x7f1a0819ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 >> 172.21.15.78:0/2067741017 conn(0x7f1a080f9fb0 msgr2=0x7f1a08103830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:27.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 shutdown_connections 2023-10-10T17:42:27.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.084+0000 7f1a0d57b700 1 -- 172.21.15.78:0/2067741017 wait complete. 2023-10-10T17:42:27.099 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:42:27.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 -- 172.21.15.78:0/3473552056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 msgr2=0x7f9278100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 --2- 172.21.15.78:0/3473552056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278100800 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9264009990 tx=0x7f9264009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 -- 172.21.15.78:0/3473552056 shutdown_connections 2023-10-10T17:42:27.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 --2- 172.21.15.78:0/3473552056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780ff1c0 0x7f92780ff5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 --2- 172.21.15.78:0/3473552056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9278100d40 0x7f9278107600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 --2- 172.21.15.78:0/3473552056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278100800 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 -- 172.21.15.78:0/3473552056 >> 172.21.15.78:0/3473552056 conn(0x7f92780fa7c0 msgr2=0x7f92780fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:27.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 -- 172.21.15.78:0/3473552056 shutdown_connections 2023-10-10T17:42:27.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.222+0000 7f927f10a700 1 -- 172.21.15.78:0/3473552056 wait complete. 2023-10-10T17:42:27.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 Processor -- start 2023-10-10T17:42:27.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 -- start start 2023-10-10T17:42:27.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780ff1c0 0x7f92781960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278196610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9278100d40 0x7f927819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927819dc50 con 0x7f92781003b0 2023-10-10T17:42:27.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927819dd90 con 0x7f9278100d40 2023-10-10T17:42:27.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927f10a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927819df20 con 0x7f92780ff1c0 2023-10-10T17:42:27.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278196610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278196610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44070/0 (socket says 172.21.15.78:44070) 2023-10-10T17:42:27.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 -- 172.21.15.78:0/4083371807 learned_addr learned my addr 172.21.15.78:0/4083371807 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:27.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927cea6700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780ff1c0 0x7f92781960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927d6a7700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9278100d40 0x7f927819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 -- 172.21.15.78:0/4083371807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780ff1c0 msgr2=0x7f92781960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780ff1c0 0x7f92781960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 -- 172.21.15.78:0/4083371807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9278100d40 msgr2=0x7f927819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9278100d40 0x7f927819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f9277fff700 1 -- 172.21.15.78:0/4083371807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92640095f0 con 0x7f92781003b0 2023-10-10T17:42:27.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.223+0000 7f927cea6700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780ff1c0 0x7f92781960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:27.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.224+0000 7f9277fff700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278196610 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f926400b2e0 tx=0x7f9264004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:27.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.224+0000 7f9275ffb700 1 -- 172.21.15.78:0/4083371807 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926401d070 con 0x7f92781003b0 2023-10-10T17:42:27.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.224+0000 7f9275ffb700 1 -- 172.21.15.78:0/4083371807 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9264005030 con 0x7f92781003b0 2023-10-10T17:42:27.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.224+0000 7f9275ffb700 1 -- 172.21.15.78:0/4083371807 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264004c50 con 0x7f92781003b0 2023-10-10T17:42:27.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.224+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927810ab70 con 0x7f92781003b0 2023-10-10T17:42:27.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.224+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927810a9e0 con 0x7f92781003b0 2023-10-10T17:42:27.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.225+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9278107340 con 0x7f92781003b0 2023-10-10T17:42:27.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.226+0000 7f9275ffb700 1 -- 172.21.15.78:0/4083371807 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f92640051a0 con 0x7f92781003b0 2023-10-10T17:42:27.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.226+0000 7f9275ffb700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f92680a11d0 0x7f92680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.227+0000 7f927cea6700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f92680a11d0 0x7f92680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.227+0000 7f9275ffb700 1 -- 172.21.15.78:0/4083371807 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f92640cdff0 con 0x7f92781003b0 2023-10-10T17:42:27.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.227+0000 7f927cea6700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f92680a11d0 0x7f92680a3680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f926c005950 tx=0x7f926c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:27.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.230+0000 7f9275ffb700 1 -- 172.21.15.78:0/4083371807 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9264097460 con 0x7f92781003b0 2023-10-10T17:42:27.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.415+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9278061700 con 0x7f92781003b0 2023-10-10T17:42:27.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.415+0000 7f9275ffb700 1 -- 172.21.15.78:0/4083371807 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7f92640163a0 con 0x7f92781003b0 2023-10-10T17:42:27.418 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:42:27.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f92680a11d0 msgr2=0x7f92680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f92680a11d0 0x7f92680a3680 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f926c005950 tx=0x7f926c0058e0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 msgr2=0x7f9278196610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278196610 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f926400b2e0 tx=0x7f9264004730 comp rx=0 tx=0).stop 2023-10-10T17:42:27.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 shutdown_connections 2023-10-10T17:42:27.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f92680a11d0 0x7f92680a3680 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92780ff1c0 0x7f92781960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9278100d40 0x7f927819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 --2- 172.21.15.78:0/4083371807 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92781003b0 0x7f9278196610 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 >> 172.21.15.78:0/4083371807 conn(0x7f92780fa7c0 msgr2=0x7f92780fcab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:27.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 shutdown_connections 2023-10-10T17:42:27.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.418+0000 7f927f10a700 1 -- 172.21.15.78:0/4083371807 wait complete. 2023-10-10T17:42:27.431 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:42:27.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 -- 172.21.15.78:0/203444025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc100480 msgr2=0x7fcabc1008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 --2- 172.21.15.78:0/203444025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc100480 0x7fcabc1008f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009990 tx=0x7fcaa8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 -- 172.21.15.78:0/203444025 shutdown_connections 2023-10-10T17:42:27.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 --2- 172.21.15.78:0/203444025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcabc100e30 0x7fcabc1076d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 --2- 172.21.15.78:0/203444025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcabc0ff770 0x7fcabc0ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 --2- 172.21.15.78:0/203444025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc100480 0x7fcabc1008f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 -- 172.21.15.78:0/203444025 >> 172.21.15.78:0/203444025 conn(0x7fcabc0f9fb0 msgr2=0x7fcabc0fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:27.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 -- 172.21.15.78:0/203444025 shutdown_connections 2023-10-10T17:42:27.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.551+0000 7fcac309f700 1 -- 172.21.15.78:0/203444025 wait complete. 2023-10-10T17:42:27.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 Processor -- start 2023-10-10T17:42:27.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 -- start start 2023-10-10T17:42:27.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc0ff770 0x7fcabc194e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcabc100480 0x7fcabc195390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcabc100e30 0x7fcabc19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc19c9d0 con 0x7fcabc0ff770 2023-10-10T17:42:27.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc19cb10 con 0x7fcabc100e30 2023-10-10T17:42:27.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.552+0000 7fcac309f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcabc19cca0 con 0x7fcabc100480 2023-10-10T17:42:27.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc0ff770 0x7fcabc194e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc0ff770 0x7fcabc194e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44082/0 (socket says 172.21.15.78:44082) 2023-10-10T17:42:27.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 -- 172.21.15.78:0/2798545017 learned_addr learned my addr 172.21.15.78:0/2798545017 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:27.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcabbfff700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcabc100480 0x7fcabc195390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac163c700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcabc100e30 0x7fcabc19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 -- 172.21.15.78:0/2798545017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcabc100480 msgr2=0x7fcabc195390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcabc100480 0x7fcabc195390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 -- 172.21.15.78:0/2798545017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcabc100e30 msgr2=0x7fcabc19c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcabc100e30 0x7fcabc19c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 -- 172.21.15.78:0/2798545017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa80095f0 con 0x7fcabc0ff770 2023-10-10T17:42:27.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac163c700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcabc100e30 0x7fcabc19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:27.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac0e3b700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc0ff770 0x7fcabc194e50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcab000e970 tx=0x7fcab000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:27.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcab9ffb700 1 -- 172.21.15.78:0/2798545017 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0009790 con 0x7fcabc0ff770 2023-10-10T17:42:27.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcab9ffb700 1 -- 172.21.15.78:0/2798545017 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcab000eec0 con 0x7fcabc0ff770 2023-10-10T17:42:27.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcabc10ab50 con 0x7fcabc0ff770 2023-10-10T17:42:27.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcab9ffb700 1 -- 172.21.15.78:0/2798545017 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab0003cf0 con 0x7fcabc0ff770 2023-10-10T17:42:27.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.553+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcabc10a9c0 con 0x7fcabc0ff770 2023-10-10T17:42:27.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.554+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcabc067eb0 con 0x7fcabc0ff770 2023-10-10T17:42:27.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.555+0000 7fcab9ffb700 1 -- 172.21.15.78:0/2798545017 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcab000ccd0 con 0x7fcabc0ff770 2023-10-10T17:42:27.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.556+0000 7fcab9ffb700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcaac0a11d0 0x7fcaac0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.556+0000 7fcab9ffb700 1 -- 172.21.15.78:0/2798545017 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcab0014070 con 0x7fcabc0ff770 2023-10-10T17:42:27.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.556+0000 7fcabbfff700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcaac0a11d0 0x7fcaac0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.556+0000 7fcabbfff700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcaac0a11d0 0x7fcaac0a3680 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800b2e0 tx=0x7fcaa8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:27.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.560+0000 7fcab9ffb700 1 -- 172.21.15.78:0/2798545017 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fcab00962b0 con 0x7fcabc0ff770 2023-10-10T17:42:27.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.794+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcabc19dc10 con 0x7fcabc0ff770 2023-10-10T17:42:27.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.794+0000 7fcab9ffb700 1 -- 172.21.15.78:0/2798545017 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7fcab00998d0 con 0x7fcabc0ff770 2023-10-10T17:42:27.797 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:27.797 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:42:27.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcaac0a11d0 msgr2=0x7fcaac0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcaac0a11d0 0x7fcaac0a3680 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcaa800b2e0 tx=0x7fcaa8005fb0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc0ff770 msgr2=0x7fcabc194e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc0ff770 0x7fcabc194e50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fcab000e970 tx=0x7fcab000ec80 comp rx=0 tx=0).stop 2023-10-10T17:42:27.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 shutdown_connections 2023-10-10T17:42:27.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fcaac0a11d0 0x7fcaac0a3680 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcabc100e30 0x7fcabc19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcabc0ff770 0x7fcabc194e50 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 --2- 172.21.15.78:0/2798545017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcabc100480 0x7fcabc195390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.796+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 >> 172.21.15.78:0/2798545017 conn(0x7fcabc0f9fb0 msgr2=0x7fcabc0fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:27.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.797+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 shutdown_connections 2023-10-10T17:42:27.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.797+0000 7fcac309f700 1 -- 172.21.15.78:0/2798545017 wait complete. 2023-10-10T17:42:27.801 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:42:27.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:27 smithi078 ceph-mon[99494]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:27.809 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:27 smithi078 ceph-mon[99494]: pgmap v316: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-10-10T17:42:27.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:27 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/202142374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:27 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2067741017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.810 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:27 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4083371807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:42:27.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:27 smithi190 ceph-mon[101220]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:27.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:27 smithi190 ceph-mon[101220]: pgmap v316: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-10-10T17:42:27.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:27 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/202142374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:27 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2067741017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:27 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4083371807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 -- 172.21.15.78:0/4032901370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 msgr2=0x7f35601086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 --2- 172.21.15.78:0/4032901370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f35601086d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f355800b0a0 tx=0x7f355800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 -- 172.21.15.78:0/4032901370 shutdown_connections 2023-10-10T17:42:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 --2- 172.21.15.78:0/4032901370 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3560100290 0x7f35601006a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 --2- 172.21.15.78:0/4032901370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f35601086d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 --2- 172.21.15.78:0/4032901370 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3560101480 0x7f35601018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 -- 172.21.15.78:0/4032901370 >> 172.21.15.78:0/4032901370 conn(0x7f35600fb870 msgr2=0x7f35600fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:27.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 -- 172.21.15.78:0/4032901370 shutdown_connections 2023-10-10T17:42:27.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.930+0000 7f3565b14700 1 -- 172.21.15.78:0/4032901370 wait complete. 2023-10-10T17:42:27.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 Processor -- start 2023-10-10T17:42:27.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 -- start start 2023-10-10T17:42:27.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3560100290 0x7f3560195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3560101480 0x7f35601964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f356019d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356019db10 con 0x7f3560101e10 2023-10-10T17:42:27.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356019dc50 con 0x7f3560100290 2023-10-10T17:42:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f3565b14700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356019dde0 con 0x7f3560101480 2023-10-10T17:42:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f355ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f356019d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f355ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f356019d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44100/0 (socket says 172.21.15.78:44100) 2023-10-10T17:42:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.931+0000 7f355ffff700 1 -- 172.21.15.78:0/31779176 learned_addr learned my addr 172.21.15.78:0/31779176 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355f7fe700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3560100290 0x7f3560195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355effd700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3560101480 0x7f35601964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355ffff700 1 -- 172.21.15.78:0/31779176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3560101480 msgr2=0x7f35601964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355ffff700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3560101480 0x7f35601964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355ffff700 1 -- 172.21.15.78:0/31779176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3560100290 msgr2=0x7f3560195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355ffff700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3560100290 0x7f3560195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355ffff700 1 -- 172.21.15.78:0/31779176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3558009d00 con 0x7f3560101e10 2023-10-10T17:42:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355effd700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3560101480 0x7f35601964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355ffff700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f356019d540 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f355800b830 tx=0x7f3558004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355cff9700 1 -- 172.21.15.78:0/31779176 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355800e040 con 0x7f3560101e10 2023-10-10T17:42:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355cff9700 1 -- 172.21.15.78:0/31779176 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f35580094c0 con 0x7f3560101e10 2023-10-10T17:42:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356010bc20 con 0x7f3560101e10 2023-10-10T17:42:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f355cff9700 1 -- 172.21.15.78:0/31779176 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355801bc70 con 0x7f3560101e10 2023-10-10T17:42:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.932+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356010ba90 con 0x7f3560101e10 2023-10-10T17:42:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.933+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35600671d0 con 0x7f3560101e10 2023-10-10T17:42:27.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.934+0000 7f355cff9700 1 -- 172.21.15.78:0/31779176 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f355801d890 con 0x7f3560101e10 2023-10-10T17:42:27.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.935+0000 7f355cff9700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f35480a1150 0x7f35480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:27.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.935+0000 7f355f7fe700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f35480a1150 0x7f35480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:27.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.935+0000 7f355cff9700 1 -- 172.21.15.78:0/31779176 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f35580cefd0 con 0x7f3560101e10 2023-10-10T17:42:27.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.935+0000 7f355f7fe700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f35480a1150 0x7f35480a3600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3550005fd0 tx=0x7f3550005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:27.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:27.939+0000 7f355cff9700 1 -- 172.21.15.78:0/31779176 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f35580984a0 con 0x7f3560101e10 2023-10-10T17:42:27.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:27 smithi157 ceph-mon[100896]: from='client.15498 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-10T17:42:27.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:27 smithi157 ceph-mon[100896]: pgmap v316: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-10-10T17:42:27.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:27 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/202142374' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:27 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2067741017' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:42:27.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:27 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4083371807' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:42:28.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.128+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3560107060 con 0x7f3560101e10 2023-10-10T17:42:28.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.130+0000 7f355cff9700 1 -- 172.21.15.78:0/31779176 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24418 (secure 0 0 0) 0x7f355809bac0 con 0x7f3560101e10 2023-10-10T17:42:28.132 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:28.132 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":91,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:41:37.591297+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:42:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f35480a1150 msgr2=0x7f35480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f35480a1150 0x7f35480a3600 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3550005fd0 tx=0x7f3550005e50 comp rx=0 tx=0).stop 2023-10-10T17:42:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 msgr2=0x7f356019d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f356019d540 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f355800b830 tx=0x7f3558004d30 comp rx=0 tx=0).stop 2023-10-10T17:42:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 shutdown_connections 2023-10-10T17:42:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f35480a1150 0x7f35480a3600 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3560101e10 0x7f356019d540 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3560100290 0x7f3560195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 --2- 172.21.15.78:0/31779176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3560101480 0x7f35601964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 >> 172.21.15.78:0/31779176 conn(0x7f35600fb870 msgr2=0x7f35600fda40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 shutdown_connections 2023-10-10T17:42:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.132+0000 7f3565b14700 1 -- 172.21.15.78:0/31779176 wait complete. 2023-10-10T17:42:28.148 DEBUG:tasks.fwd_scrub:Yielding 2023-10-10T17:42:28.148 INFO:teuthology.run_tasks:Running task exec... 2023-10-10T17:42:28.174 INFO:teuthology.task.exec:Executing custom commands... 2023-10-10T17:42:28.174 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:42:28.175 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-10-10T17:42:28.175 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2023-10-10T17:42:28.176 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:42:28.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:42:28.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 -- 172.21.15.78:0/3512809111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c105fa0 msgr2=0x7ff52c108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3512809111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c105fa0 0x7ff52c108380 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7ff52400b0a0 tx=0x7ff52400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 -- 172.21.15.78:0/3512809111 shutdown_connections 2023-10-10T17:42:28.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3512809111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c105fa0 0x7ff52c108380 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3512809111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff52c100d60 0x7ff52c103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3512809111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c103680 0x7ff52c105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 -- 172.21.15.78:0/3512809111 >> 172.21.15.78:0/3512809111 conn(0x7ff52c0fa720 msgr2=0x7ff52c0fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 -- 172.21.15.78:0/3512809111 shutdown_connections 2023-10-10T17:42:28.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.299+0000 7ff5342ee700 1 -- 172.21.15.78:0/3512809111 wait complete. 2023-10-10T17:42:28.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 Processor -- start 2023-10-10T17:42:28.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 -- start start 2023-10-10T17:42:28.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff52c100d60 0x7ff52c196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c103680 0x7ff52c196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c105fa0 0x7ff52c19d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c19dcb0 con 0x7ff52c103680 2023-10-10T17:42:28.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c19ddf0 con 0x7ff52c100d60 2023-10-10T17:42:28.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff5342ee700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c19df80 con 0x7ff52c105fa0 2023-10-10T17:42:28.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff531889700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c103680 0x7ff52c196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff531889700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c103680 0x7ff52c196670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44112/0 (socket says 172.21.15.78:44112) 2023-10-10T17:42:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff531889700 1 -- 172.21.15.78:0/3849844203 learned_addr learned my addr 172.21.15.78:0/3849844203 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff53208a700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff52c100d60 0x7ff52c196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.300+0000 7ff53288b700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c105fa0 0x7ff52c19d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff531889700 1 -- 172.21.15.78:0/3849844203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c105fa0 msgr2=0x7ff52c19d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff531889700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c105fa0 0x7ff52c19d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff531889700 1 -- 172.21.15.78:0/3849844203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff52c100d60 msgr2=0x7ff52c196130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff531889700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff52c100d60 0x7ff52c196130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff531889700 1 -- 172.21.15.78:0/3849844203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff524009d00 con 0x7ff52c103680 2023-10-10T17:42:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff53288b700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c105fa0 0x7ff52c19d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff531889700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c103680 0x7ff52c196670 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00e9a0 tx=0x7ff51c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c009790 con 0x7ff52c103680 2023-10-10T17:42:28.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff51c00cbd0 con 0x7ff52c103680 2023-10-10T17:42:28.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c10bbf0 con 0x7ff52c103680 2023-10-10T17:42:28.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c018680 con 0x7ff52c103680 2023-10-10T17:42:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.301+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c10ba60 con 0x7ff52c103680 2023-10-10T17:42:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.302+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c067c10 con 0x7ff52c103680 2023-10-10T17:42:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.303+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff51c00cd40 con 0x7ff52c103680 2023-10-10T17:42:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.304+0000 7ff5237fe700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5180a1150 0x7ff5180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.304+0000 7ff53208a700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5180a1150 0x7ff5180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.304+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff51c014070 con 0x7ff52c103680 2023-10-10T17:42:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.304+0000 7ff53208a700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5180a1150 0x7ff5180a3600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff528005950 tx=0x7ff52800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.307+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff51c096590 con 0x7ff52c103680 2023-10-10T17:42:28.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 -- 172.21.15.78:0/721492440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 msgr2=0x7ff89c106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 --2- 172.21.15.78:0/721492440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c106d70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff89400b0a0 tx=0x7ff89400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 -- 172.21.15.78:0/721492440 shutdown_connections 2023-10-10T17:42:28.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 --2- 172.21.15.78:0/721492440 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89c0fe930 0x7ff89c0fed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 --2- 172.21.15.78:0/721492440 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c106d70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 --2- 172.21.15.78:0/721492440 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff89c0ffb20 0x7ff89c0fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 -- 172.21.15.78:0/721492440 >> 172.21.15.78:0/721492440 conn(0x7ff89c0f9f30 msgr2=0x7ff89c0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 -- 172.21.15.78:0/721492440 shutdown_connections 2023-10-10T17:42:28.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.308+0000 7ff8a4373700 1 -- 172.21.15.78:0/721492440 wait complete. 2023-10-10T17:42:28.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 Processor -- start 2023-10-10T17:42:28.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 -- start start 2023-10-10T17:42:28.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89c0fe930 0x7ff89c1956c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff89c0ffb20 0x7ff89c193d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c194240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c1949e0 con 0x7ff89c1004b0 2023-10-10T17:42:28.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c194b50 con 0x7ff89c0ffb20 2023-10-10T17:42:28.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a4373700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89c1a0190 con 0x7ff89c0fe930 2023-10-10T17:42:28.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c194240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c194240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44128/0 (socket says 172.21.15.78:44128) 2023-10-10T17:42:28.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.309+0000 7ff8a2910700 1 -- 172.21.15.78:0/3465660516 learned_addr learned my addr 172.21.15.78:0/3465660516 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a190e700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff89c0ffb20 0x7ff89c193d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a210f700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89c0fe930 0x7ff89c1956c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a2910700 1 -- 172.21.15.78:0/3465660516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89c0fe930 msgr2=0x7ff89c1956c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89c0fe930 0x7ff89c1956c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a2910700 1 -- 172.21.15.78:0/3465660516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff89c0ffb20 msgr2=0x7ff89c193d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff89c0ffb20 0x7ff89c193d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a2910700 1 -- 172.21.15.78:0/3465660516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff894009d00 con 0x7ff89c1004b0 2023-10-10T17:42:28.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a190e700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff89c0ffb20 0x7ff89c193d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:28.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c194240 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff89400d010 tx=0x7ff89400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8937fe700 1 -- 172.21.15.78:0/3465660516 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89400e040 con 0x7ff89c1004b0 2023-10-10T17:42:28.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89c109a80 con 0x7ff89c1004b0 2023-10-10T17:42:28.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8937fe700 1 -- 172.21.15.78:0/3465660516 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff894003e10 con 0x7ff89c1004b0 2023-10-10T17:42:28.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8937fe700 1 -- 172.21.15.78:0/3465660516 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8940053c0 con 0x7ff89c1004b0 2023-10-10T17:42:28.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.310+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89c1098f0 con 0x7ff89c1004b0 2023-10-10T17:42:28.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.312+0000 7ff8937fe700 1 -- 172.21.15.78:0/3465660516 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff89401da40 con 0x7ff89c1004b0 2023-10-10T17:42:28.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.313+0000 7ff8937fe700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff8880a1150 0x7ff8880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.313+0000 7ff8a210f700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff8880a1150 0x7ff8880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.313+0000 7ff8937fe700 1 -- 172.21.15.78:0/3465660516 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff8940cdcf0 con 0x7ff89c1004b0 2023-10-10T17:42:28.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.313+0000 7ff8a210f700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff8880a1150 0x7ff8880a3600 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff898005950 tx=0x7ff8980058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.313+0000 7ff8b04ffb80 1 Processor -- start 2023-10-10T17:42:28.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8b04ffb80 1 -- start start 2023-10-10T17:42:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa934090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa9345d0 0x563afa934bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa935600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa8390e0 con 0x563afa935190 2023-10-10T17:42:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa88b920 con 0x563afa933c80 2023-10-10T17:42:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa884db0 con 0x563afa9345d0 2023-10-10T17:42:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa935600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa935600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44136/0 (socket says 172.21.15.78:44136) 2023-10-10T17:42:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 -- 172.21.15.78:0/1648734789 learned_addr learned my addr 172.21.15.78:0/1648734789 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a190e700 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa9345d0 0x563afa934bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a210f700 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa934090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 -- 172.21.15.78:0/1648734789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa9345d0 msgr2=0x563afa934bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa9345d0 0x563afa934bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 -- 172.21.15.78:0/1648734789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 msgr2=0x563afa934090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa934090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 -- 172.21.15.78:0/1648734789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563afa7066b0 con 0x563afa935190 2023-10-10T17:42:28.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff8a2910700 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa935600 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff8940a2d10 tx=0x7ff89401dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1250ace493ccc5f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.314+0000 7ff890dac700 1 -- 172.21.15.78:0/1648734789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89409eeb0 con 0x563afa935190 2023-10-10T17:42:28.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff890dac700 1 -- 172.21.15.78:0/1648734789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8940a3240 con 0x563afa935190 2023-10-10T17:42:28.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff890dac700 1 -- 172.21.15.78:0/1648734789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8940a2db0 con 0x563afa935190 2023-10-10T17:42:28.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1648734789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 msgr2=0x563afa935600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa935600 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff8940a2d10 tx=0x7ff89401dde0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1648734789 shutdown_connections 2023-10-10T17:42:28.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa935600 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa934090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1648734789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa9345d0 0x563afa934bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1648734789 >> 172.21.15.78:0/1648734789 conn(0x563afa9293e0 msgr2=0x563afa92e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1648734789 shutdown_connections 2023-10-10T17:42:28.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.315+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1648734789 wait complete. 2023-10-10T17:42:28.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 -- start start 2023-10-10T17:42:28.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 Processor -- start 2023-10-10T17:42:28.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa93d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa9345d0 0x563afa93d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa935190 0x563afa944820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa826960 con 0x563afa9345d0 2023-10-10T17:42:28.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa826600 con 0x563afa933c80 2023-10-10T17:42:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa7089a0 con 0x563afa935190 2023-10-10T17:42:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8b04ffb80 10 client.0 _ll_register_callbacks cb 0x563afa844800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa9345d0 0x563afa93d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa9345d0 0x563afa93d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44140/0 (socket says 172.21.15.78:44140) 2023-10-10T17:42:28.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 -- 172.21.15.78:0/1885556396 learned_addr learned my addr 172.21.15.78:0/1885556396 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a210f700 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa93d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a2910700 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa935190 0x563afa944820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 -- 172.21.15.78:0/1885556396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa935190 msgr2=0x563afa944820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa935190 0x563afa944820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 -- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 msgr2=0x563afa93d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa93d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.316+0000 7ff8a190e700 1 -- 172.21.15.78:0/1885556396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89c1098f0 con 0x563afa9345d0 2023-10-10T17:42:28.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8a190e700 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa9345d0 0x563afa93d7e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff88c012840 tx=0x7ff88c012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5936fdbeedfb7a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff885ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:28.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff885ffb700 1 -- 172.21.15.78:0/1885556396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c017070 con 0x563afa9345d0 2023-10-10T17:42:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff885ffb700 1 -- 172.21.15.78:0/1885556396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff88c011840 con 0x563afa9345d0 2023-10-10T17:42:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563afa926ba0 con 0x563afa9345d0 2023-10-10T17:42:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff885ffb700 1 -- 172.21.15.78:0/1885556396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff88c011a00 con 0x563afa9345d0 2023-10-10T17:42:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 10 client.15558 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:28.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563afa84b210 con 0x563afa9345d0 2023-10-10T17:42:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff885ffb700 1 -- 172.21.15.78:0/1885556396 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563afa926ba0 con 0x563afa9345d0 2023-10-10T17:42:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff885ffb700 1 -- 172.21.15.78:0/1885556396 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff88c01c420 con 0x563afa9345d0 2023-10-10T17:42:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 10 client.15558 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 10 client.15558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:28.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 20 client.15558 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:28.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa9490e0 0x563afa94b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 4 client.15558 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:28.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.317+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563afa847120 con 0x563afa9490e0 2023-10-10T17:42:28.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.318+0000 7ff8a210f700 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa9490e0 0x563afa94b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.318+0000 7ff885ffb700 10 client.15558 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.318+0000 7ff8a210f700 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa9490e0 0x563afa94b4f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8a75a6e03aa3efd server_cookie=807364d4adf6323a in_seq=0 out_seq=0 2023-10-10T17:42:28.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff885ffb700 1 -- 172.21.15.78:0/1885556396 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563afa847120 con 0x563afa9490e0 2023-10-10T17:42:28.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff885ffb700 10 client.15558 handle_command_reply: tid=0 2023-10-10T17:42:28.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 client.15558 shutdown 2023-10-10T17:42:28.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa9345d0 msgr2=0x563afa93d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa9345d0 0x563afa93d7e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7ff88c012840 tx=0x7ff88c012c00 comp rx=0 tx=0).stop 2023-10-10T17:42:28.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 shutdown_connections 2023-10-10T17:42:28.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa935190 0x563afa944820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa9490e0 0x563afa94b4f0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa933c80 0x563afa93d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/1885556396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa9345d0 0x563afa93d7e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff885ffb700 0 client.15558 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 >> 172.21.15.78:0/1885556396 conn(0x563afa9293e0 msgr2=0x563afa92de60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 shutdown_connections 2023-10-10T17:42:28.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/1885556396 wait complete. 2023-10-10T17:42:28.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.319+0000 7ff8b04ffb80 20 client.15558 trim_cache size 0 max 16384 2023-10-10T17:42:28.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 Processor -- start 2023-10-10T17:42:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 -- start start 2023-10-10T17:42:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9380a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa9385e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa9447e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa9301c0 con 0x563afa935190 2023-10-10T17:42:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa9349a0 con 0x563afa9345d0 2023-10-10T17:42:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.336+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa93b270 con 0x563afa933c80 2023-10-10T17:42:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa9447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa9447e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44150/0 (socket says 172.21.15.78:44150) 2023-10-10T17:42:28.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 -- 172.21.15.78:0/3875836735 learned_addr learned my addr 172.21.15.78:0/3875836735 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a210f700 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9380a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a190e700 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa9385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 -- 172.21.15.78:0/3875836735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 msgr2=0x563afa9380a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9380a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 -- 172.21.15.78:0/3875836735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 msgr2=0x563afa9385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa9385e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 -- 172.21.15.78:0/3875836735 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563afa84af60 con 0x563afa935190 2023-10-10T17:42:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa9447e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff8940cb530 tx=0x7ff8940972f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa16afe8c6aef19 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8915ad700 1 -- 172.21.15.78:0/3875836735 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894099220 con 0x563afa935190 2023-10-10T17:42:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8915ad700 1 -- 172.21.15.78:0/3875836735 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff89409b0e0 con 0x563afa935190 2023-10-10T17:42:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.337+0000 7ff8915ad700 1 -- 172.21.15.78:0/3875836735 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff894024be0 con 0x563afa935190 2023-10-10T17:42:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3875836735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 msgr2=0x563afa9447e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa9447e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7ff8940cb530 tx=0x7ff8940972f0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3875836735 shutdown_connections 2023-10-10T17:42:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9380a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa9447e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3875836735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa9385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3875836735 >> 172.21.15.78:0/3875836735 conn(0x563afa9293e0 msgr2=0x563afa92e590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3875836735 shutdown_connections 2023-10-10T17:42:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3875836735 wait complete. 2023-10-10T17:42:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 -- start start 2023-10-10T17:42:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.338+0000 7ff8b04ffb80 1 Processor -- start 2023-10-10T17:42:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9446f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa941190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa941710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa99cf90 con 0x563afa935190 2023-10-10T17:42:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa99d0d0 con 0x563afa9345d0 2023-10-10T17:42:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563afa99d210 con 0x563afa933c80 2023-10-10T17:42:28.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 10 client.0 _ll_register_callbacks cb 0x7ff88c009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:28.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa941710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa941710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44156/0 (socket says 172.21.15.78:44156) 2023-10-10T17:42:28.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 -- 172.21.15.78:0/3097264145 learned_addr learned my addr 172.21.15.78:0/3097264145 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a190e700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa941190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a210f700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 -- 172.21.15.78:0/3097264145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 msgr2=0x563afa9446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9446f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 -- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 msgr2=0x563afa941190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa941190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 -- 172.21.15.78:0/3097264145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563afa7066b0 con 0x563afa935190 2023-10-10T17:42:28.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a190e700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa941190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:28.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8a2910700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa941710 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff894021000 tx=0x7ff89400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1a5354584428a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff885ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:28.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff885ffb700 1 -- 172.21.15.78:0/3097264145 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8940a3020 con 0x563afa935190 2023-10-10T17:42:28.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff885ffb700 1 -- 172.21.15.78:0/3097264145 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff894024be0 con 0x563afa935190 2023-10-10T17:42:28.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8940a3240 con 0x563afa935190 2023-10-10T17:42:28.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.339+0000 7ff885ffb700 1 -- 172.21.15.78:0/3097264145 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff894024de0 con 0x563afa935190 2023-10-10T17:42:28.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 10 client.15564 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:28.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563afa69bce0 con 0x563afa935190 2023-10-10T17:42:28.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff885ffb700 1 -- 172.21.15.78:0/3097264145 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff8940be080 con 0x563afa935190 2023-10-10T17:42:28.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff885ffb700 1 -- 172.21.15.78:0/3097264145 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff894099de0 con 0x563afa935190 2023-10-10T17:42:28.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 10 client.15564 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 10 client.15564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 20 client.15564 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa99e710 0x563afa9a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 4 client.15564 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:42:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563afa8536e0 con 0x563afa99e710 2023-10-10T17:42:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.340+0000 7ff8a210f700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa99e710 0x563afa9a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff885ffb700 10 client.15564 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8a210f700 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa99e710 0x563afa9a0b20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaca48216166448 server_cookie=e1087922e273646f in_seq=0 out_seq=0 2023-10-10T17:42:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff885ffb700 1 -- 172.21.15.78:0/3097264145 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563afa8536e0 con 0x563afa99e710 2023-10-10T17:42:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff885ffb700 10 client.15564 handle_command_reply: tid=0 2023-10-10T17:42:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 client.15564 shutdown 2023-10-10T17:42:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 msgr2=0x563afa941710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa941710 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7ff894021000 tx=0x7ff89400bf10 comp rx=0 tx=0).stop 2023-10-10T17:42:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 shutdown_connections 2023-10-10T17:42:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563afa933c80 0x563afa9446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563afa99e710 0x563afa9a0b20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563afa935190 0x563afa941710 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 --2- 172.21.15.78:0/3097264145 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563afa9345d0 0x563afa941190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.341+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 >> 172.21.15.78:0/3097264145 conn(0x563afa9293e0 msgr2=0x563afa93d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff885ffb700 0 client.15564 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 shutdown_connections 2023-10-10T17:42:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8b04ffb80 1 -- 172.21.15.78:0/3097264145 wait complete. 2023-10-10T17:42:28.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8b04ffb80 20 client.15564 trim_cache size 0 max 16384 2023-10-10T17:42:28.372 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:42:28.372 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:42:28.373 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "be922620-ce54-4381-aec5-9b7ca373784b", 2023-10-10T17:42:28.373 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:42:28.373 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:42:28.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff8880a1150 msgr2=0x7ff8880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff8880a1150 0x7ff8880a3600 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff898005950 tx=0x7ff8980058e0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 msgr2=0x7ff89c194240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c194240 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff89400d010 tx=0x7ff89400bd30 comp rx=0 tx=0).stop 2023-10-10T17:42:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 shutdown_connections 2023-10-10T17:42:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff8880a1150 0x7ff8880a3600 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff89c0fe930 0x7ff89c1956c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff89c1004b0 0x7ff89c194240 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 --2- 172.21.15.78:0/3465660516 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff89c0ffb20 0x7ff89c193d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 >> 172.21.15.78:0/3465660516 conn(0x7ff89c0f9f30 msgr2=0x7ff89c0fc130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 shutdown_connections 2023-10-10T17:42:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.342+0000 7ff8a4373700 1 -- 172.21.15.78:0/3465660516 wait complete. 2023-10-10T17:42:28.377 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:42:28.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 -- 172.21.15.78:0/1655413612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 msgr2=0x7f0e341083e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/1655413612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e341083e0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00b0a0 tx=0x7f0e2c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 -- 172.21.15.78:0/1655413612 shutdown_connections 2023-10-10T17:42:28.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/1655413612 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e341083e0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/1655413612 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e34100d40 0x7f0e34103120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/1655413612 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e34103660 0x7f0e34105a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 -- 172.21.15.78:0/1655413612 >> 172.21.15.78:0/1655413612 conn(0x7f0e340fa7c0 msgr2=0x7f0e340fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 -- 172.21.15.78:0/1655413612 shutdown_connections 2023-10-10T17:42:28.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 -- 172.21.15.78:0/1655413612 wait complete. 2023-10-10T17:42:28.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 Processor -- start 2023-10-10T17:42:28.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.501+0000 7f0e3a8db700 1 -- start start 2023-10-10T17:42:28.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e34100d40 0x7f0e34196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e34103660 0x7f0e341966a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e3419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3419dce0 con 0x7f0e34105f80 2023-10-10T17:42:28.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3419de20 con 0x7f0e34103660 2023-10-10T17:42:28.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3419dfb0 con 0x7f0e34100d40 2023-10-10T17:42:28.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e3419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e3419d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44170/0 (socket says 172.21.15.78:44170) 2023-10-10T17:42:28.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 -- 172.21.15.78:0/847156016 learned_addr learned my addr 172.21.15.78:0/847156016 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e33fff700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e34100d40 0x7f0e34196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e337fe700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e34103660 0x7f0e341966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 -- 172.21.15.78:0/847156016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e34100d40 msgr2=0x7f0e34196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e34100d40 0x7f0e34196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 -- 172.21.15.78:0/847156016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e34103660 msgr2=0x7f0e341966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e34103660 0x7f0e341966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 -- 172.21.15.78:0/847156016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c009d00 con 0x7f0e34105f80 2023-10-10T17:42:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e38e78700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e3419d710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00b780 tx=0x7f0e2c007990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e317fa700 1 -- 172.21.15.78:0/847156016 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c00e040 con 0x7f0e34105f80 2023-10-10T17:42:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3410bc20 con 0x7f0e34105f80 2023-10-10T17:42:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e317fa700 1 -- 172.21.15.78:0/847156016 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e2c00be40 con 0x7f0e34105f80 2023-10-10T17:42:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e317fa700 1 -- 172.21.15.78:0/847156016 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c0052a0 con 0x7f0e34105f80 2023-10-10T17:42:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.502+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3410ba90 con 0x7f0e34105f80 2023-10-10T17:42:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e317fa700 1 -- 172.21.15.78:0/847156016 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0e2c01d470 con 0x7f0e34105f80 2023-10-10T17:42:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 Processor -- start 2023-10-10T17:42:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 -- start start 2023-10-10T17:42:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e317fa700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e1c0a1150 0x7f0e1c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31f0a0 0x5604ba31f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba3203a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba327180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba220060 con 0x5604ba31f0a0 2023-10-10T17:42:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba2728a0 con 0x5604ba3208e0 2023-10-10T17:42:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba26bd30 con 0x5604ba31ff30 2023-10-10T17:42:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e33fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31f0a0 0x5604ba31f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e317fa700 1 -- 172.21.15.78:0/847156016 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e2c0ce0d0 con 0x7f0e34105f80 2023-10-10T17:42:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e33fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31f0a0 0x5604ba31f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44174/0 (socket says 172.21.15.78:44174) 2023-10-10T17:42:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e33fff700 1 -- 172.21.15.78:0/3684109914 learned_addr learned my addr 172.21.15.78:0/3684109914 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e337fe700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e1c0a1150 0x7f0e1c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.506+0000 7f0e337fe700 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba3203a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e38e78700 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba327180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e33fff700 1 -- 172.21.15.78:0/3684109914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 msgr2=0x5604ba3203a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e33fff700 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba3203a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e33fff700 1 -- 172.21.15.78:0/3684109914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 msgr2=0x5604ba327180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e33fff700 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba327180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e33fff700 1 -- 172.21.15.78:0/3684109914 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604ba0ed630 con 0x5604ba31f0a0 2023-10-10T17:42:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e33fff700 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31f0a0 0x5604ba31f4b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009d80 tx=0x7f0e24011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef93ca70e1541f8c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e337fe700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e1c0a1150 0x7f0e1c0a3600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009eb0 tx=0x7f0e280047c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e1a7fc700 1 -- 172.21.15.78:0/3684109914 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24004230 con 0x5604ba31f0a0 2023-10-10T17:42:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e1a7fc700 1 -- 172.21.15.78:0/3684109914 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e24004390 con 0x5604ba31f0a0 2023-10-10T17:42:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.507+0000 7f0e1a7fc700 1 -- 172.21.15.78:0/3684109914 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e24019880 con 0x5604ba31f0a0 2023-10-10T17:42:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3684109914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31f0a0 msgr2=0x5604ba31f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31f0a0 0x5604ba31f4b0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009d80 tx=0x7f0e24011720 comp rx=0 tx=0).stop 2023-10-10T17:42:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3684109914 shutdown_connections 2023-10-10T17:42:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba327180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31f0a0 0x5604ba31f4b0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3684109914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba3203a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3684109914 >> 172.21.15.78:0/3684109914 conn(0x5604ba310360 msgr2=0x5604ba315670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3684109914 shutdown_connections 2023-10-10T17:42:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3684109914 wait complete. 2023-10-10T17:42:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 -- start start 2023-10-10T17:42:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 Processor -- start 2023-10-10T17:42:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba328940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.508+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba328e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba3208e0 0x5604ba32fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba20d8e0 con 0x5604ba3208e0 2023-10-10T17:42:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba20d580 con 0x5604ba31f0a0 2023-10-10T17:42:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba0ef920 con 0x5604ba31ff30 2023-10-10T17:42:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 10 client.0 _ll_register_callbacks cb 0x5604ba22b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba3208e0 0x5604ba32fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba3208e0 0x5604ba32fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44184/0 (socket says 172.21.15.78:44184) 2023-10-10T17:42:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 -- 172.21.15.78:0/2164579571 learned_addr learned my addr 172.21.15.78:0/2164579571 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e33fff700 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba328940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e337fe700 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba328e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 -- 172.21.15.78:0/2164579571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 msgr2=0x5604ba328e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba328e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 -- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 msgr2=0x5604ba328940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba328940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 -- 172.21.15.78:0/2164579571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e240099e0 con 0x5604ba3208e0 2023-10-10T17:42:28.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e38e78700 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba3208e0 0x5604ba32fec0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0cb860 tx=0x7f0e2c09ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b29892e9b51477 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e137fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:28.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e137fe700 1 -- 172.21.15.78:0/2164579571 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2c015300 con 0x5604ba3208e0 2023-10-10T17:42:28.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e137fe700 1 -- 172.21.15.78:0/2164579571 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e2c02d2f0 con 0x5604ba3208e0 2023-10-10T17:42:28.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604ba30db20 con 0x5604ba3208e0 2023-10-10T17:42:28.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e137fe700 1 -- 172.21.15.78:0/2164579571 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e2c015120 con 0x5604ba3208e0 2023-10-10T17:42:28.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 10 client.15576 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:28.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.509+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5604ba232190 con 0x5604ba3208e0 2023-10-10T17:42:28.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e137fe700 1 -- 172.21.15.78:0/2164579571 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5604ba30db20 con 0x5604ba3208e0 2023-10-10T17:42:28.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e137fe700 1 -- 172.21.15.78:0/2164579571 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0e2c0cbd20 con 0x5604ba3208e0 2023-10-10T17:42:28.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e46a67b80 10 client.15576 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:28.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e46a67b80 10 client.15576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:28.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e46a67b80 20 client.15576 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:28.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba334440 0x5604ba336850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e46a67b80 4 client.15576 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:28.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604ba22e0a0 con 0x5604ba334440 2023-10-10T17:42:28.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e33fff700 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba334440 0x5604ba336850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e137fe700 10 client.15576 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.510+0000 7f0e33fff700 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba334440 0x5604ba336850 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f1b3e488ebfce13 server_cookie=f736d7be064466a3 in_seq=0 out_seq=0 2023-10-10T17:42:28.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e137fe700 1 -- 172.21.15.78:0/2164579571 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5604ba22e0a0 con 0x5604ba334440 2023-10-10T17:42:28.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e137fe700 10 client.15576 handle_command_reply: tid=0 2023-10-10T17:42:28.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 client.15576 shutdown 2023-10-10T17:42:28.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba3208e0 msgr2=0x5604ba32fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba3208e0 0x5604ba32fec0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c0cb860 tx=0x7f0e2c09ad40 comp rx=0 tx=0).stop 2023-10-10T17:42:28.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 shutdown_connections 2023-10-10T17:42:28.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba334440 0x5604ba336850 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba3208e0 0x5604ba32fec0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba328940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/2164579571 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31ff30 0x5604ba328e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 >> 172.21.15.78:0/2164579571 conn(0x5604ba310360 msgr2=0x5604ba314de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e137fe700 0 client.15576 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 shutdown_connections 2023-10-10T17:42:28.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.511+0000 7f0e46a67b80 1 -- 172.21.15.78:0/2164579571 wait complete. 2023-10-10T17:42:28.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.512+0000 7f0e46a67b80 20 client.15576 trim_cache size 0 max 16384 2023-10-10T17:42:28.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 Processor -- start 2023-10-10T17:42:28.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 -- start start 2023-10-10T17:42:28.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31f0a0 0x5604ba327750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba326c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba327df0 con 0x5604ba31ff30 2023-10-10T17:42:28.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba334260 con 0x5604ba3208e0 2023-10-10T17:42:28.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba3284b0 con 0x5604ba31f0a0 2023-10-10T17:42:28.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44190/0 (socket says 172.21.15.78:44190) 2023-10-10T17:42:28.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.528+0000 7f0e337fe700 1 -- 172.21.15.78:0/4021841809 learned_addr learned my addr 172.21.15.78:0/4021841809 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e33fff700 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31f0a0 0x5604ba327750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e38e78700 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba326c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e337fe700 1 -- 172.21.15.78:0/4021841809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31f0a0 msgr2=0x5604ba327750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e337fe700 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31f0a0 0x5604ba327750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e337fe700 1 -- 172.21.15.78:0/4021841809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 msgr2=0x5604ba326c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e337fe700 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba326c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e337fe700 1 -- 172.21.15.78:0/4021841809 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604ba231ee0 con 0x5604ba31ff30 2023-10-10T17:42:28.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e337fe700 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32fe80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0e28011e30 tx=0x7f0e28012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa85477f7e281be server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e1affd700 1 -- 172.21.15.78:0/4021841809 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28025070 con 0x5604ba31ff30 2023-10-10T17:42:28.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e1affd700 1 -- 172.21.15.78:0/4021841809 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e28010bd0 con 0x5604ba31ff30 2023-10-10T17:42:28.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.529+0000 7f0e1affd700 1 -- 172.21.15.78:0/4021841809 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2801b640 con 0x5604ba31ff30 2023-10-10T17:42:28.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 -- 172.21.15.78:0/4021841809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 msgr2=0x5604ba32fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32fe80 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f0e28011e30 tx=0x7f0e28012040 comp rx=0 tx=0).stop 2023-10-10T17:42:28.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 -- 172.21.15.78:0/4021841809 shutdown_connections 2023-10-10T17:42:28.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba31f0a0 0x5604ba327750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba3208e0 0x5604ba326c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/4021841809 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32fe80 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 -- 172.21.15.78:0/4021841809 >> 172.21.15.78:0/4021841809 conn(0x5604ba310360 msgr2=0x5604ba33b170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 -- 172.21.15.78:0/4021841809 shutdown_connections 2023-10-10T17:42:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 -- 172.21.15.78:0/4021841809 wait complete. 2023-10-10T17:42:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 -- start start 2023-10-10T17:42:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.530+0000 7f0e46a67b80 1 Processor -- start 2023-10-10T17:42:28.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba32d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba3208e0 0x5604ba32eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba32f3f0 con 0x5604ba31ff30 2023-10-10T17:42:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba32f560 con 0x5604ba31f0a0 2023-10-10T17:42:28.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604ba32f720 con 0x5604ba3208e0 2023-10-10T17:42:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 10 client.0 _ll_register_callbacks cb 0x7f0e2c097540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44196/0 (socket says 172.21.15.78:44196) 2023-10-10T17:42:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 -- 172.21.15.78:0/3902439782 learned_addr learned my addr 172.21.15.78:0/3902439782 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e33fff700 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba32d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e38e78700 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba3208e0 0x5604ba32eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 -- 172.21.15.78:0/3902439782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba3208e0 msgr2=0x5604ba32eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba3208e0 0x5604ba32eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 -- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 msgr2=0x5604ba32d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:28.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba32d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 -- 172.21.15.78:0/3902439782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e28018070 con 0x5604ba31ff30 2023-10-10T17:42:28.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e337fe700 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32dea0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0e2801fac0 tx=0x7f0e28010f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=912e7e464210f458 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e137fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:28.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e137fe700 1 -- 172.21.15.78:0/3902439782 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28025050 con 0x5604ba31ff30 2023-10-10T17:42:28.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e137fe700 1 -- 172.21.15.78:0/3902439782 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0e2801ee40 con 0x5604ba31ff30 2023-10-10T17:42:28.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.531+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e24004390 con 0x5604ba31ff30 2023-10-10T17:42:28.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e137fe700 1 -- 172.21.15.78:0/3902439782 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e2802f070 con 0x5604ba31ff30 2023-10-10T17:42:28.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 10 client.15582 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:28.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5604ba082c60 con 0x5604ba31ff30 2023-10-10T17:42:28.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e137fe700 1 -- 172.21.15.78:0/3902439782 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0e24004390 con 0x5604ba31ff30 2023-10-10T17:42:28.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e137fe700 1 -- 172.21.15.78:0/3902439782 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0e28034820 con 0x5604ba31ff30 2023-10-10T17:42:28.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 10 client.15582 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:28.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 10 client.15582 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:28.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 20 client.15582 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:28.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba388e30 0x5604ba38b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 4 client.15582 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:42:28.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5604ba23a660 con 0x5604ba388e30 2023-10-10T17:42:28.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.532+0000 7f0e33fff700 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba388e30 0x5604ba38b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e137fe700 10 client.15582 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e33fff700 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba388e30 0x5604ba38b240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7af27aad9ee046a8 server_cookie=ae009ec39d6eaeaf in_seq=0 out_seq=0 2023-10-10T17:42:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e137fe700 1 -- 172.21.15.78:0/3902439782 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5604ba23a660 con 0x5604ba388e30 2023-10-10T17:42:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e137fe700 10 client.15582 handle_command_reply: tid=0 2023-10-10T17:42:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 client.15582 shutdown 2023-10-10T17:42:28.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 msgr2=0x5604ba32dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32dea0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f0e2801fac0 tx=0x7f0e28010f70 comp rx=0 tx=0).stop 2023-10-10T17:42:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 shutdown_connections 2023-10-10T17:42:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5604ba3208e0 0x5604ba32eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5604ba388e30 0x5604ba38b240 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5604ba31f0a0 0x5604ba32d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e137fe700 0 client.15582 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 --2- 172.21.15.78:0/3902439782 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5604ba31ff30 0x5604ba32dea0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 >> 172.21.15.78:0/3902439782 conn(0x5604ba310360 msgr2=0x5604ba326c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 shutdown_connections 2023-10-10T17:42:28.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 1 -- 172.21.15.78:0/3902439782 wait complete. 2023-10-10T17:42:28.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.533+0000 7f0e46a67b80 20 client.15582 trim_cache size 0 max 16384 2023-10-10T17:42:28.560 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:42:28.560 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:42:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:42:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "be922620-ce54-4381-aec5-9b7ca373784b": { 2023-10-10T17:42:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:42:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "be922620-ce54-4381-aec5-9b7ca373784b", 2023-10-10T17:42:28.561 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:42:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:42:28.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:42:28.562 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:42:28.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e1c0a1150 msgr2=0x7f0e1c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e1c0a1150 0x7f0e1c0a3600 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e28009eb0 tx=0x7f0e280047c0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 msgr2=0x7f0e3419d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e3419d710 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c00b780 tx=0x7f0e2c007990 comp rx=0 tx=0).stop 2023-10-10T17:42:28.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 shutdown_connections 2023-10-10T17:42:28.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f0e1c0a1150 0x7f0e1c0a3600 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e34100d40 0x7f0e34196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e34105f80 0x7f0e3419d710 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 --2- 172.21.15.78:0/847156016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e34103660 0x7f0e341966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 >> 172.21.15.78:0/847156016 conn(0x7f0e340fa7c0 msgr2=0x7f0e341019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 shutdown_connections 2023-10-10T17:42:28.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.534+0000 7f0e3a8db700 1 -- 172.21.15.78:0/847156016 wait complete. 2023-10-10T17:42:28.566 INFO:tasks.cephfs.filesystem:scrub status for tag:be922620-ce54-4381-aec5-9b7ca373784b - {'path': '/', 'tag': 'be922620-ce54-4381-aec5-9b7ca373784b', 'options': 'recursive,force'} 2023-10-10T17:42:28.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.573+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff52c19eeb0 con 0x7ff52c103680 2023-10-10T17:42:28.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.616+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 17) v1 ==== 135732+0+0 (secure 0 0 0) 0x7ff51c096960 con 0x7ff52c103680 2023-10-10T17:42:28.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.617+0000 7ff5237fe700 1 -- 172.21.15.78:0/3849844203 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v17) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff51c08ada0 con 0x7ff52c103680 2023-10-10T17:42:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5180a1150 msgr2=0x7ff5180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5180a1150 0x7ff5180a3600 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff528005950 tx=0x7ff52800b410 comp rx=0 tx=0).stop 2023-10-10T17:42:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c103680 msgr2=0x7ff52c196670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c103680 0x7ff52c196670 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff51c00e9a0 tx=0x7ff51c00ecb0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 shutdown_connections 2023-10-10T17:42:28.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7ff5180a1150 0x7ff5180a3600 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff52c105fa0 0x7ff52c19d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff52c100d60 0x7ff52c196130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 --2- 172.21.15.78:0/3849844203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff52c103680 0x7ff52c196670 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 >> 172.21.15.78:0/3849844203 conn(0x7ff52c0fa720 msgr2=0x7ff52c1021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 shutdown_connections 2023-10-10T17:42:28.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.619+0000 7ff5342ee700 1 -- 172.21.15.78:0/3849844203 wait complete. 2023-10-10T17:42:28.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-10-10T17:42:28.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2798545017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:28.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/31779176' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:28.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3849844203' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-10T17:42:28.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 -- 172.21.15.78:0/3235443328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700105fa0 msgr2=0x7fc700108380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 --2- 172.21.15.78:0/3235443328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700105fa0 0x7fc700108380 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fc6f800b0a0 tx=0x7fc6f800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 -- 172.21.15.78:0/3235443328 shutdown_connections 2023-10-10T17:42:28.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 --2- 172.21.15.78:0/3235443328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700105fa0 0x7fc700108380 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 --2- 172.21.15.78:0/3235443328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc700100d60 0x7fc700103140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 --2- 172.21.15.78:0/3235443328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc700103680 0x7fc700105a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 -- 172.21.15.78:0/3235443328 >> 172.21.15.78:0/3235443328 conn(0x7fc7000fa720 msgr2=0x7fc7000fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:28.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 -- 172.21.15.78:0/3235443328 shutdown_connections 2023-10-10T17:42:28.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.786+0000 7fc706d99700 1 -- 172.21.15.78:0/3235443328 wait complete. 2023-10-10T17:42:28.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 Processor -- start 2023-10-10T17:42:28.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 -- start start 2023-10-10T17:42:28.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc700100d60 0x7fc7001960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700103680 0x7fc700196630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc700105fa0 0x7fc70019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70019dc70 con 0x7fc700103680 2023-10-10T17:42:28.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70019ddb0 con 0x7fc700105fa0 2023-10-10T17:42:28.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc706d99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70019df40 con 0x7fc700100d60 2023-10-10T17:42:28.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc6fffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700103680 0x7fc700196630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc6fffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700103680 0x7fc700196630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44240/0 (socket says 172.21.15.78:44240) 2023-10-10T17:42:28.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc6fffff700 1 -- 172.21.15.78:0/3835405637 learned_addr learned my addr 172.21.15.78:0/3835405637 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:28.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc704b35700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc700100d60 0x7fc7001960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.787+0000 7fc705336700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc700105fa0 0x7fc70019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:28.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fffff700 1 -- 172.21.15.78:0/3835405637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc700100d60 msgr2=0x7fc7001960f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fffff700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc700100d60 0x7fc7001960f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fffff700 1 -- 172.21.15.78:0/3835405637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc700105fa0 msgr2=0x7fc70019d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:28.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fffff700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc700105fa0 0x7fc70019d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:28.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fffff700 1 -- 172.21.15.78:0/3835405637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f8009d00 con 0x7fc700103680 2023-10-10T17:42:28.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc704b35700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc700100d60 0x7fc7001960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:28.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fffff700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700103680 0x7fc700196630 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00e9a0 tx=0x7fc6ec00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:28.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fdffb700 1 -- 172.21.15.78:0/3835405637 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec009790 con 0x7fc700103680 2023-10-10T17:42:28.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fdffb700 1 -- 172.21.15.78:0/3835405637 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6ec010430 con 0x7fc700103680 2023-10-10T17:42:28.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70010bc00 con 0x7fc700103680 2023-10-10T17:42:28.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc6fdffb700 1 -- 172.21.15.78:0/3835405637 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec018680 con 0x7fc700103680 2023-10-10T17:42:28.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.788+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70010ba70 con 0x7fc700103680 2023-10-10T17:42:28.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.789+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7000fc290 con 0x7fc700103680 2023-10-10T17:42:28.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.791+0000 7fc6fdffb700 1 -- 172.21.15.78:0/3835405637 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135732+0+0 (secure 0 0 0) 0x7fc6ec0105a0 con 0x7fc700103680 2023-10-10T17:42:28.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.791+0000 7fc6fdffb700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 0x7fc6f00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:28.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.791+0000 7fc704b35700 1 -- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 msgr2=0x7fc6f00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:28.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.791+0000 7fc6fdffb700 1 -- 172.21.15.78:0/3835405637 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc6ec014070 con 0x7fc700103680 2023-10-10T17:42:28.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.791+0000 7fc704b35700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 0x7fc6f00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:42:28.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.796+0000 7fc6fdffb700 1 -- 172.21.15.78:0/3835405637 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc6ec051730 con 0x7fc700103680 2023-10-10T17:42:28.868 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: ignoring --setuser ceph since I am not root 2023-10-10T17:42:28.868 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: ignoring --setgroup ceph since I am not root 2023-10-10T17:42:28.869 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-mgr[102372]: -- 172.21.15.157:0/2583613366 <== mon.2 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b829c27520 con 0x55b829bfd000 2023-10-10T17:42:28.869 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:28.785+0000 7f3969574200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:42:28.869 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2798545017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:28.870 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/31779176' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:28.870 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3849844203' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-10T17:42:28.870 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2798545017' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:28.871 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/31779176' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:42:28.871 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3849844203' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-10-10T17:42:28.871 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: ignoring --setuser ceph since I am not root 2023-10-10T17:42:28.872 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: ignoring --setgroup ceph since I am not root 2023-10-10T17:42:28.872 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:28.788+0000 7f27cf744200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:42:28.957 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: ignoring --setuser ceph since I am not root 2023-10-10T17:42:28.958 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: ignoring --setgroup ceph since I am not root 2023-10-10T17:42:28.958 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:28.780+0000 7f47c595f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-10T17:42:28.958 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:28.865+0000 7f47c595f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:42:28.959 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:28 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:28.956+0000 7f47c595f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:42:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.992+0000 7fc704b35700 1 -- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 msgr2=0x7fc6f00a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.992+0000 7fc704b35700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 0x7fc6f00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-10T17:42:28.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.994+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fc700061930 con 0x7fc700103680 2023-10-10T17:42:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:28.998+0000 7fc6fdffb700 1 -- 172.21.15.78:0/3835405637 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc6ec099d10 con 0x7fc700103680 2023-10-10T17:42:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 msgr2=0x7fc6f00a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:42:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 0x7fc6f00a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700103680 msgr2=0x7fc700196630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700103680 0x7fc700196630 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00e9a0 tx=0x7fc6ec00ecb0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 shutdown_connections 2023-10-10T17:42:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7fc6f00a11a0 0x7fc6f00a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc700100d60 0x7fc7001960f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc700105fa0 0x7fc70019d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 --2- 172.21.15.78:0/3835405637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc700103680 0x7fc700196630 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 >> 172.21.15.78:0/3835405637 conn(0x7fc7000fa720 msgr2=0x7fc7001019c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 shutdown_connections 2023-10-10T17:42:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.001+0000 7fc706d99700 1 -- 172.21.15.78:0/3835405637 wait complete. 2023-10-10T17:42:29.016 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-10-10T17:42:29.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 -- 172.21.15.78:0/245930224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 msgr2=0x7f72c0100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/245930224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0100780 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009990 tx=0x7f72b0009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 -- 172.21.15.78:0/245930224 shutdown_connections 2023-10-10T17:42:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/245930224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c0100cc0 0x7f72c0107580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/245930224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c00ff140 0x7f72c00ff550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/245930224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0100780 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 -- 172.21.15.78:0/245930224 >> 172.21.15.78:0/245930224 conn(0x7f72c00fa720 msgr2=0x7f72c00fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:29.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 -- 172.21.15.78:0/245930224 shutdown_connections 2023-10-10T17:42:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 -- 172.21.15.78:0/245930224 wait complete. 2023-10-10T17:42:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.139+0000 7f72c5f7c700 1 Processor -- start 2023-10-10T17:42:29.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72c5f7c700 1 -- start start 2023-10-10T17:42:29.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72c5f7c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c00ff140 0x7f72c0196020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72c5f7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0196560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72c5f7c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c0100cc0 0x7f72c019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72c5f7c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c019dba0 con 0x7f72c0100330 2023-10-10T17:42:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72c5f7c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c019dce0 con 0x7f72c0100cc0 2023-10-10T17:42:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72c5f7c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c019de70 con 0x7f72c00ff140 2023-10-10T17:42:29.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0196560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:29.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0196560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44304/0 (socket says 172.21.15.78:44304) 2023-10-10T17:42:29.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 -- 172.21.15.78:0/3980961043 learned_addr learned my addr 172.21.15.78:0/3980961043 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:29.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72bf7fe700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c00ff140 0x7f72c0196020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:29.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72bffff700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c0100cc0 0x7f72c019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:29.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 -- 172.21.15.78:0/3980961043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c00ff140 msgr2=0x7f72c0196020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c00ff140 0x7f72c0196020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 -- 172.21.15.78:0/3980961043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c0100cc0 msgr2=0x7f72c019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c0100cc0 0x7f72c019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 -- 172.21.15.78:0/3980961043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b00095f0 con 0x7f72c0100330 2023-10-10T17:42:29.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.140+0000 7f72beffd700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0196560 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f72b0004760 tx=0x7f72b0004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:29.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.141+0000 7f72bcff9700 1 -- 172.21.15.78:0/3980961043 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b001d070 con 0x7f72c0100330 2023-10-10T17:42:29.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.141+0000 7f72bcff9700 1 -- 172.21.15.78:0/3980961043 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f72b0004830 con 0x7f72c0100330 2023-10-10T17:42:29.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.141+0000 7f72bcff9700 1 -- 172.21.15.78:0/3980961043 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72b0005300 con 0x7f72c0100330 2023-10-10T17:42:29.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.141+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72c010ab00 con 0x7f72c0100330 2023-10-10T17:42:29.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.141+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72c010a970 con 0x7f72c0100330 2023-10-10T17:42:29.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.142+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c0107270 con 0x7f72c0100330 2023-10-10T17:42:29.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.143+0000 7f72bcff9700 1 -- 172.21.15.78:0/3980961043 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135732+0+0 (secure 0 0 0) 0x7f72b00054a0 con 0x7f72c0100330 2023-10-10T17:42:29.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.144+0000 7f72bcff9700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 0x7f72a80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.144+0000 7f72bf7fe700 1 -- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 msgr2=0x7f72a80a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:29.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.144+0000 7f72bcff9700 1 -- 172.21.15.78:0/3980961043 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f72b00ce270 con 0x7f72c0100330 2023-10-10T17:42:29.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.144+0000 7f72bf7fe700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 0x7f72a80a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:42:29.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.147+0000 7f72bcff9700 1 -- 172.21.15.78:0/3980961043 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f72b0097740 con 0x7f72c0100330 2023-10-10T17:42:29.164 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:28.868+0000 7f27cf744200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:42:29.164 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:28.957+0000 7f27cf744200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:42:29.164 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:29.049+0000 7f27cf744200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:42:29.221 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:28.867+0000 7f3969574200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-10T17:42:29.221 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:28.956+0000 7f3969574200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-10T17:42:29.222 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:29.050+0000 7f3969574200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:42:29.234 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:29.055+0000 7f47c595f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-10T17:42:29.235 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:29.233+0000 7f47c595f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:42:29.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.345+0000 7f72bf7fe700 1 -- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 msgr2=0x7f72a80a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:29.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.345+0000 7f72bf7fe700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 0x7f72a80a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-10T17:42:29.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.345+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f72c002cfe0 con 0x7f72c0100330 2023-10-10T17:42:29.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.350+0000 7f72bcff9700 1 -- 172.21.15.78:0/3980961043 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f72b009ad60 con 0x7f72c0100330 2023-10-10T17:42:29.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 msgr2=0x7f72a80a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:42:29.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 0x7f72a80a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 msgr2=0x7f72c0196560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0196560 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f72b0004760 tx=0x7f72b0004790 comp rx=0 tx=0).stop 2023-10-10T17:42:29.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 shutdown_connections 2023-10-10T17:42:29.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f72a80a11a0 0x7f72a80a3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f72c00ff140 0x7f72c0196020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f72c0100cc0 0x7f72c019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 --2- 172.21.15.78:0/3980961043 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72c0100330 0x7f72c0196560 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 >> 172.21.15.78:0/3980961043 conn(0x7f72c00fa720 msgr2=0x7f72c00fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:29.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 shutdown_connections 2023-10-10T17:42:29.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.352+0000 7f72c5f7c700 1 -- 172.21.15.78:0/3980961043 wait complete. 2023-10-10T17:42:29.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-10-10T17:42:29.477 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:29.219+0000 7f3969574200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:42:29.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.490+0000 7f6885220700 1 -- 172.21.15.78:0/3151842257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880100cb0 msgr2=0x7f6880103090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.490+0000 7f6885220700 1 --2- 172.21.15.78:0/3151842257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880100cb0 0x7f6880103090 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6870009990 tx=0x7f6870009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.490+0000 7f6885220700 1 -- 172.21.15.78:0/3151842257 shutdown_connections 2023-10-10T17:42:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.490+0000 7f6885220700 1 --2- 172.21.15.78:0/3151842257 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6880105ef0 0x7f6880108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.490+0000 7f6885220700 1 --2- 172.21.15.78:0/3151842257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880100cb0 0x7f6880103090 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.491+0000 7f6885220700 1 --2- 172.21.15.78:0/3151842257 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f68801035d0 0x7f68801059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.491+0000 7f6885220700 1 -- 172.21.15.78:0/3151842257 >> 172.21.15.78:0/3151842257 conn(0x7f68800fa750 msgr2=0x7f68800fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.491+0000 7f6885220700 1 -- 172.21.15.78:0/3151842257 shutdown_connections 2023-10-10T17:42:29.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.491+0000 7f6885220700 1 -- 172.21.15.78:0/3151842257 wait complete. 2023-10-10T17:42:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.491+0000 7f6885220700 1 Processor -- start 2023-10-10T17:42:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f6885220700 1 -- start start 2023-10-10T17:42:29.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f6885220700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6880100cb0 0x7f6880100620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f6885220700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68801035d0 0x7f68800fec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f6885220700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880105ef0 0x7f68800ff1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f6885220700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68800ff940 con 0x7f6880105ef0 2023-10-10T17:42:29.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f6885220700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68800ffab0 con 0x7f6880100cb0 2023-10-10T17:42:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f6885220700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68801a0030 con 0x7f68801035d0 2023-10-10T17:42:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880105ef0 0x7f68800ff1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880105ef0 0x7f68800ff1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44372/0 (socket says 172.21.15.78:44372) 2023-10-10T17:42:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687ed9d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6880100cb0 0x7f6880100620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:29.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 -- 172.21.15.78:0/3810335832 learned_addr learned my addr 172.21.15.78:0/3810335832 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687e59c700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68801035d0 0x7f68800fec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68801035d0 msgr2=0x7f68800fec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f68801035d0 0x7f68800fec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6880100cb0 msgr2=0x7f6880100620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:29.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6880100cb0 0x7f6880100620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687f59e700 1 -- 172.21.15.78:0/3810335832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68700095f0 con 0x7f6880105ef0 2023-10-10T17:42:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.492+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6880100cb0 0x7f6880100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.493+0000 7f687f59e700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880105ef0 0x7f68800ff1a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6878007ee0 tx=0x7f687800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.493+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687800dd20 con 0x7f6880105ef0 2023-10-10T17:42:29.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.493+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f687800f040 con 0x7f6880105ef0 2023-10-10T17:42:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.493+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f688010bbe0 con 0x7f6880105ef0 2023-10-10T17:42:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.493+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68780048a0 con 0x7f6880105ef0 2023-10-10T17:42:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.493+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f688010ba50 con 0x7f6880105ef0 2023-10-10T17:42:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.494+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68800fc2c0 con 0x7f6880105ef0 2023-10-10T17:42:29.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.495+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135732+0+0 (secure 0 0 0) 0x7f6878003a00 con 0x7f6880105ef0 2023-10-10T17:42:29.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.495+0000 7f6867fff700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 0x7f68680a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:29.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.495+0000 7f687ed9d700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 msgr2=0x7f68680a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:29.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.496+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f68780cc490 con 0x7f6880105ef0 2023-10-10T17:42:29.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.496+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 0x7f68680a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-10T17:42:29.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.500+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f687804e730 con 0x7f6880105ef0 2023-10-10T17:42:29.526 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:29.219+0000 7f27cf744200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-10T17:42:29.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.697+0000 7f687ed9d700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 msgr2=0x7f68680a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:29.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.697+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 0x7f68680a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-10T17:42:29.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:29.739+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 --> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6880061980 con 0x7f68680a11a0 2023-10-10T17:42:29.842 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:29.525+0000 7f27cf744200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:42:29.842 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:29.613+0000 7f27cf744200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:42:29.842 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:29.746+0000 7f27cf744200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:42:29.843 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: pgmap v317: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-10-10T17:42:29.843 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:42:29.843 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:42:29.843 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:42:29.844 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:42:29.844 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3849844203' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-10-10T17:42:29.844 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-10-10T17:42:29.844 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3835405637' entity='client.admin' 2023-10-10T17:42:29.844 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3980961043' entity='client.admin' 2023-10-10T17:42:29.850 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:29.529+0000 7f3969574200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:42:29.851 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:29.620+0000 7f3969574200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:42:29.851 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:29.754+0000 7f3969574200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:42:29.851 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: pgmap v317: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-10-10T17:42:29.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:42:29.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:42:29.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:42:29.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:42:29.852 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3849844203' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-10-10T17:42:29.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-10-10T17:42:29.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3835405637' entity='client.admin' 2023-10-10T17:42:29.853 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3980961043' entity='client.admin' 2023-10-10T17:42:29.877 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:29.567+0000 7f47c595f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-10T17:42:29.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:29.659+0000 7f47c595f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-10T17:42:29.878 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:29.793+0000 7f47c595f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-10T17:42:29.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: pgmap v317: 129 pgs: 129 active+clean; 584 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-10-10T17:42:29.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:42:29.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:42:29.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:42:29.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:42:29.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3849844203' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-10-10T17:42:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: mgrmap e17: x(active, since 7m), standbys: z, y 2023-10-10T17:42:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3835405637' entity='client.admin' 2023-10-10T17:42:29.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3980961043' entity='client.admin' 2023-10-10T17:42:30.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:30.098+0000 7f687ed9d700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 msgr2=0x7f68680a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:30.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:30.098+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 0x7f68680a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-10T17:42:30.227 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:29.849+0000 7f3969574200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:42:30.307 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:29.887+0000 7f47c595f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:42:30.674 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:29.840+0000 7f27cf744200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-10T17:42:30.674 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:30 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:30.259+0000 7f27cf744200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:42:30.727 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:30 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:30.272+0000 7f3969574200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:42:30.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:30.899+0000 7f687ed9d700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 msgr2=0x7f68680a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:30.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:30.899+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 0x7f68680a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-10T17:42:31.127 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:30.305+0000 7f47c595f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-10T17:42:31.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:30.865+0000 7f47c595f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:42:31.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:31.034+0000 7f47c595f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:42:31.164 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:30 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:30.820+0000 7f27cf744200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:42:31.164 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:30 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:30.994+0000 7f27cf744200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:42:31.227 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:30 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:30.850+0000 7f3969574200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-10T17:42:31.228 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:31 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:31.023+0000 7f3969574200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-10T17:42:32.018 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:31 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:31.755+0000 7f27cf744200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:42:32.018 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:31 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:31.846+0000 7f27cf744200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:42:32.055 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:31.792+0000 7f47c595f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:42:32.055 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:31.883+0000 7f47c595f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:42:32.056 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:31 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:31.791+0000 7f3969574200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:42:32.056 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:31 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:31.883+0000 7f3969574200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-10T17:42:32.294 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:32.017+0000 7f27cf744200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:42:32.332 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:32.053+0000 7f47c595f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:42:32.333 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:32.054+0000 7f3969574200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-10T17:42:32.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:32.500+0000 7f687ed9d700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 msgr2=0x7f68680a3650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/321866266 2023-10-10T17:42:32.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:32.500+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 0x7f68680a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-10T17:42:32.627 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:32.331+0000 7f47c595f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:42:32.628 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:32.554+0000 7f47c595f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:42:32.664 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:32.293+0000 7f27cf744200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:42:32.664 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:32.522+0000 7f27cf744200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:42:32.708 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:32.332+0000 7f3969574200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-10T17:42:32.708 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:32.559+0000 7f3969574200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-10T17:42:32.931 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:32.669+0000 7f27cf744200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:42:32.966 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:32.702+0000 7f47c595f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:42:32.970 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:32.707+0000 7f3969574200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-10T17:42:33.227 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:32.968+0000 7f3969574200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:42:33.377 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:32.964+0000 7f47c595f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:42:33.414 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:32.930+0000 7f27cf744200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-10T17:42:33.914 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:33.639+0000 7f27cf744200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:42:33.915 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:33.770+0000 7f27cf744200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:42:33.977 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:33 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:33.685+0000 7f3969574200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:42:33.978 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:33 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:33.817+0000 7f3969574200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:42:33.992 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:33.668+0000 7f47c595f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-10T17:42:33.992 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:33.799+0000 7f47c595f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-10T17:42:34.289 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:33.961+0000 7f27cf744200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:42:34.290 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:34 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:34.075+0000 7f27cf744200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:42:34.327 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:33.991+0000 7f47c595f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:42:34.327 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:34.106+0000 7f47c595f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:42:34.327 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:34.325+0000 7f47c595f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:42:34.338 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:34 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:34.008+0000 7f3969574200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-10T17:42:34.338 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:34 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:34.121+0000 7f3969574200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-10T17:42:34.664 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:34 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:34.289+0000 7f27cf744200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:42:34.727 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:34 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:34.337+0000 7f3969574200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-10T17:42:35.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:35.209+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 18) v1 ==== 135400+0+0 (secure 0 0 0) 0x7f6878003cb0 con 0x7f6880105ef0 2023-10-10T17:42:35.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:35.210+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 msgr2=0x7f68680a3650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:42:35.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:35.210+0000 7f6867fff700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/321866266,v1:172.21.15.78:6801/321866266] conn(0x7f68680a11a0 0x7f68680a3650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:35.414 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z[103200]: 2023-10-10T17:42:35.144+0000 7f27cf744200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:42:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: Standby manager daemon z restarted 2023-10-10T17:42:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: Standby manager daemon z started 2023-10-10T17:42:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-10T17:42:35.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:42:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-10T17:42:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:42:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: Active manager daemon x restarted 2023-10-10T17:42:35.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:35 smithi190 ceph-mon[101220]: Activating manager daemon x 2023-10-10T17:42:35.475 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: Standby manager daemon z restarted 2023-10-10T17:42:35.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: Standby manager daemon z started 2023-10-10T17:42:35.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-10T17:42:35.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:42:35.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-10T17:42:35.476 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:42:35.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: Active manager daemon x restarted 2023-10-10T17:42:35.477 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-mon[99494]: Activating manager daemon x 2023-10-10T17:42:35.477 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.186+0000 7f47c595f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:42:35.477 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4721db7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.478 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4721db7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.478 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4721db7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.478 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4721db7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.478 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4721db7700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.478 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4724dbd700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.479 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4724dbd700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.479 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4724dbd700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.479 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4724dbd700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.479 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:42:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:42:35.400+0000 7f4724dbd700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:42:35.480 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y[102347]: 2023-10-10T17:42:35.197+0000 7f3969574200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-10T17:42:35.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: Standby manager daemon z restarted 2023-10-10T17:42:35.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: Standby manager daemon z started 2023-10-10T17:42:35.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-10-10T17:42:35.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:42:35.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-10-10T17:42:35.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.190:0/861519140' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:42:35.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: Active manager daemon x restarted 2023-10-10T17:42:35.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:35 smithi157 ceph-mon[100896]: Activating manager daemon x 2023-10-10T17:42:36.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.230+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 19) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f68780c9cd0 con 0x7f6880105ef0 2023-10-10T17:42:36.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.230+0000 7f6867fff700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6868069200 0x7f68680695d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.231+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 --> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f6880061980 con 0x7f6868069200 2023-10-10T17:42:36.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.234+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6868069200 0x7f68680695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.236+0000 7f687ed9d700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6868069200 0x7f68680695d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f687000b2e0 tx=0x7f6870005fb0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:36.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.294+0000 7f6867fff700 1 -- 172.21.15.78:0/3810335832 <== mgr.15588 v2:172.21.15.78:6800/2386743294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6880061980 con 0x7f6868069200 2023-10-10T17:42:36.297 INFO:teuthology.orchestra.run.smithi078.stdout:Schedule set for path / 2023-10-10T17:42:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6868069200 msgr2=0x7f68680695d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6868069200 0x7f68680695d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f687000b2e0 tx=0x7f6870005fb0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880105ef0 msgr2=0x7f68800ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 --2- 172.21.15.78:0/3810335832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6880105ef0 0x7f68800ff1a0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6878007ee0 tx=0x7f687800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f687f59e700 1 -- 172.21.15.78:0/3810335832 reap_dead start 2023-10-10T17:42:36.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 shutdown_connections 2023-10-10T17:42:36.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 >> 172.21.15.78:0/3810335832 conn(0x7f68800fa750 msgr2=0x7f6880101910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 shutdown_connections 2023-10-10T17:42:36.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.297+0000 7f6885220700 1 -- 172.21.15.78:0/3810335832 wait complete. 2023-10-10T17:42:36.312 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-10-10T17:42:36.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 -- 172.21.15.78:0/3196000248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c1006f0 msgr2=0x7fe60c100b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 --2- 172.21.15.78:0/3196000248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c1006f0 0x7fe60c100b00 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc009990 tx=0x7fe5fc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 -- 172.21.15.78:0/3196000248 shutdown_connections 2023-10-10T17:42:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 --2- 172.21.15.78:0/3196000248 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe60c0ffcf0 0x7fe60c1075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 --2- 172.21.15.78:0/3196000248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c1006f0 0x7fe60c100b00 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 --2- 172.21.15.78:0/3196000248 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe60c0ff340 0x7fe60c0ff7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 -- 172.21.15.78:0/3196000248 >> 172.21.15.78:0/3196000248 conn(0x7fe60c0f9ea0 msgr2=0x7fe60c0fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:36.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 -- 172.21.15.78:0/3196000248 shutdown_connections 2023-10-10T17:42:36.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.440+0000 7fe611846700 1 -- 172.21.15.78:0/3196000248 wait complete. 2023-10-10T17:42:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 Processor -- start 2023-10-10T17:42:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 -- start start 2023-10-10T17:42:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe60c0ff340 0x7fe60c0feda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffcf0 0x7fe60c19d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe60c1006f0 0x7fe60c19f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c19fd10 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c19fe80 con 0x7fe60c1006f0 2023-10-10T17:42:36.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe611846700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c1a0040 con 0x7fe60c0ff340 2023-10-10T17:42:36.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffcf0 0x7fe60c19d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffcf0 0x7fe60c19d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44652/0 (socket says 172.21.15.78:44652) 2023-10-10T17:42:36.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 -- 172.21.15.78:0/3929966451 learned_addr learned my addr 172.21.15.78:0/3929966451 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60b7fe700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe60c1006f0 0x7fe60c19f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60affd700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe60c0ff340 0x7fe60c0feda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 -- 172.21.15.78:0/3929966451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe60c0ff340 msgr2=0x7fe60c0feda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe60c0ff340 0x7fe60c0feda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 -- 172.21.15.78:0/3929966451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe60c1006f0 msgr2=0x7fe60c19f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe60c1006f0 0x7fe60c19f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60a7fc700 1 -- 172.21.15.78:0/3929966451 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc0095f0 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.441+0000 7fe60b7fe700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe60c1006f0 0x7fe60c19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.442+0000 7fe60a7fc700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffcf0 0x7fe60c19d110 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe60000d950 tx=0x7fe60000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:36.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.442+0000 7fe610844700 1 -- 172.21.15.78:0/3929966451 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60000f890 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.442+0000 7fe610844700 1 -- 172.21.15.78:0/3929966451 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe600010460 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.442+0000 7fe610844700 1 -- 172.21.15.78:0/3929966451 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60000e610 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.442+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c10aad0 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.443+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c10a940 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.444+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60c106090 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.445+0000 7fe610844700 1 -- 172.21.15.78:0/3929966451 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135527+0+0 (secure 0 0 0) 0x7fe6000108d0 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.445+0000 7fe610844700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe5f40a0fd0 0x7fe5f40a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.445+0000 7fe60affd700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe5f40a0fd0 0x7fe5f40a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.445+0000 7fe610844700 1 -- 172.21.15.78:0/3929966451 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fe6000cc690 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.446+0000 7fe60affd700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe5f40a0fd0 0x7fe5f40a3480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00b5c0 tx=0x7fe5fc005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:36.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.453+0000 7fe610844700 1 -- 172.21.15.78:0/3929966451 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe600095cc0 con 0x7fe60c0ffcf0 2023-10-10T17:42:36.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: osdmap e92: 12 total, 12 up, 12 in 2023-10-10T17:42:36.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: mgrmap e18: x(active, starting, since 0.0120696s), standbys: y, z 2023-10-10T17:42:36.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-10T17:42:36.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:42:36.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:42:36.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:42:36.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-10T17:42:36.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-10T17:42:36.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-10T17:42:36.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:42:36.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-10T17:42:36.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-10T17:42:36.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: Standby manager daemon y restarted 2023-10-10T17:42:36.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: Standby manager daemon y started 2023-10-10T17:42:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:42:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-10T17:42:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:42:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:42:36.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:42:36.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:42:36.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:42:36.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:42:36.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:42:36.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-10T17:42:36.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-10T17:42:36.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-10T17:42:36.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:42:36.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:42:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:42:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:42:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:42:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:42:36.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:42:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:42:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:42:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:42:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:42:36.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:42:36.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-10T17:42:36.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-10T17:42:36.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-10T17:42:36.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: Manager daemon x is now available 2023-10-10T17:42:36.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:36.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:36.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-10T17:42:36.488 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-10T17:42:36.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: osdmap e92: 12 total, 12 up, 12 in 2023-10-10T17:42:36.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: mgrmap e18: x(active, starting, since 0.0120696s), standbys: y, z 2023-10-10T17:42:36.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-10T17:42:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:42:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:42:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:42:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-10T17:42:36.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-10T17:42:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-10T17:42:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:42:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-10T17:42:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-10T17:42:36.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: Standby manager daemon y restarted 2023-10-10T17:42:36.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: Standby manager daemon y started 2023-10-10T17:42:36.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:42:36.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-10T17:42:36.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:42:36.631 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:42:36.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:42:36.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:42:36.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:42:36.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:42:36.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:42:36.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-10T17:42:36.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-10T17:42:36.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-10T17:42:36.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:42:36.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:42:36.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:42:36.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:42:36.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:42:36.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:42:36.634 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:42:36.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:42:36.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:42:36.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:42:36.635 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:42:36.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:42:36.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-10T17:42:36.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-10T17:42:36.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-10T17:42:36.636 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: Manager daemon x is now available 2023-10-10T17:42:36.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:36.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:36.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-10T17:42:36.637 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-10T17:42:36.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: osdmap e92: 12 total, 12 up, 12 in 2023-10-10T17:42:36.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: mgrmap e18: x(active, starting, since 0.0120696s), standbys: y, z 2023-10-10T17:42:36.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-10-10T17:42:36.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-10T17:42:36.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-10T17:42:36.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-10T17:42:36.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-10-10T17:42:36.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-10-10T17:42:36.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-10-10T17:42:36.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-10-10T17:42:36.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-10-10T17:42:36.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-10-10T17:42:36.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: Standby manager daemon y restarted 2023-10-10T17:42:36.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: Standby manager daemon y started 2023-10-10T17:42:36.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:42:36.669 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-10-10T17:42:36.670 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:42:36.670 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:42:36.670 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.? 172.21.15.157:0/3386599632' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-10-10T17:42:36.671 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:42:36.671 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:42:36.671 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:42:36.671 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:42:36.671 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-10T17:42:36.672 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-10-10T17:42:36.672 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-10-10T17:42:36.672 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-10T17:42:36.672 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-10T17:42:36.673 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-10T17:42:36.673 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-10T17:42:36.673 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-10T17:42:36.673 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-10T17:42:36.673 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-10T17:42:36.673 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-10T17:42:36.674 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-10-10T17:42:36.674 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-10-10T17:42:36.674 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-10-10T17:42:36.674 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-10-10T17:42:36.674 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-10T17:42:36.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-10T17:42:36.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-10T17:42:36.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: Manager daemon x is now available 2023-10-10T17:42:36.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:36.675 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:36.676 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-10T17:42:36.676 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-10T17:42:36.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.676+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 --> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7fe60c061c60 con 0x7fe5f40a0fd0 2023-10-10T17:42:36.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.727+0000 7fe610844700 1 -- 172.21.15.78:0/3929966451 <== mgr.15588 v2:172.21.15.78:6800/2386743294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fe60c061c60 con 0x7fe5f40a0fd0 2023-10-10T17:42:36.729 INFO:teuthology.orchestra.run.smithi078.stdout:Retention added to path / 2023-10-10T17:42:36.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe5f40a0fd0 msgr2=0x7fe5f40a3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe5f40a0fd0 0x7fe5f40a3480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe5fc00b5c0 tx=0x7fe5fc005fd0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffcf0 msgr2=0x7fe60c19d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffcf0 0x7fe60c19d110 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fe60000d950 tx=0x7fe60000dc60 comp rx=0 tx=0).stop 2023-10-10T17:42:36.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 shutdown_connections 2023-10-10T17:42:36.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe60c0ff340 0x7fe60c0feda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe5f40a0fd0 0x7fe5f40a3480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe60c1006f0 0x7fe60c19f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 --2- 172.21.15.78:0/3929966451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe60c0ffcf0 0x7fe60c19d110 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 >> 172.21.15.78:0/3929966451 conn(0x7fe60c0f9ea0 msgr2=0x7fe60c0fc140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:36.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 shutdown_connections 2023-10-10T17:42:36.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.729+0000 7fe611846700 1 -- 172.21.15.78:0/3929966451 wait complete. 2023-10-10T17:42:36.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-10-10T17:42:36.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 -- 172.21.15.78:0/3136059325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100360 msgr2=0x7f8c0c1007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/3136059325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100360 0x7f8c0c1007b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009990 tx=0x7f8c00009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 -- 172.21.15.78:0/3136059325 shutdown_connections 2023-10-10T17:42:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/3136059325 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c0c0ff170 0x7f8c0c0ff580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/3136059325 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c0c100cf0 0x7f8c0c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/3136059325 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100360 0x7f8c0c1007b0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 -- 172.21.15.78:0/3136059325 >> 172.21.15.78:0/3136059325 conn(0x7f8c0c0fa7b0 msgr2=0x7f8c0c0fcbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 -- 172.21.15.78:0/3136059325 shutdown_connections 2023-10-10T17:42:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.873+0000 7f8c12ba1700 1 -- 172.21.15.78:0/3136059325 wait complete. 2023-10-10T17:42:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 Processor -- start 2023-10-10T17:42:36.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 -- start start 2023-10-10T17:42:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c0c0ff170 0x7f8c0c194f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c0c100360 0x7f8c0c195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100cf0 0x7f8c0c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c19cab0 con 0x7f8c0c100cf0 2023-10-10T17:42:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c19cbf0 con 0x7f8c0c0ff170 2023-10-10T17:42:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c12ba1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c0c19cd80 con 0x7f8c0c100360 2023-10-10T17:42:36.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100cf0 0x7f8c0c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100cf0 0x7f8c0c19c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44668/0 (socket says 172.21.15.78:44668) 2023-10-10T17:42:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 -- 172.21.15.78:0/2481914477 learned_addr learned my addr 172.21.15.78:0/2481914477 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1093d700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c0c0ff170 0x7f8c0c194f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c0bfff700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c0c100360 0x7f8c0c195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 -- 172.21.15.78:0/2481914477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c0c100360 msgr2=0x7f8c0c195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c0c100360 0x7f8c0c195470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 -- 172.21.15.78:0/2481914477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c0c0ff170 msgr2=0x7f8c0c194f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:36.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c0c0ff170 0x7f8c0c194f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:36.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.874+0000 7f8c1113e700 1 -- 172.21.15.78:0/2481914477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c000095f0 con 0x7f8c0c100cf0 2023-10-10T17:42:36.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.875+0000 7f8c1113e700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100cf0 0x7f8c0c19c4e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00e940 tx=0x7f8bfc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:36.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.875+0000 7f8c09ffb700 1 -- 172.21.15.78:0/2481914477 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc0097b0 con 0x7f8c0c100cf0 2023-10-10T17:42:36.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.875+0000 7f8c09ffb700 1 -- 172.21.15.78:0/2481914477 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8bfc00ee90 con 0x7f8c0c100cf0 2023-10-10T17:42:36.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.875+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c0c10ab50 con 0x7f8c0c100cf0 2023-10-10T17:42:36.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.875+0000 7f8c09ffb700 1 -- 172.21.15.78:0/2481914477 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bfc005520 con 0x7f8c0c100cf0 2023-10-10T17:42:36.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.875+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c0c10a9c0 con 0x7f8c0c100cf0 2023-10-10T17:42:36.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.876+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c0c106730 con 0x7f8c0c100cf0 2023-10-10T17:42:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.877+0000 7f8c09ffb700 1 -- 172.21.15.78:0/2481914477 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135527+0+0 (secure 0 0 0) 0x7f8bfc004750 con 0x7f8c0c100cf0 2023-10-10T17:42:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.877+0000 7f8c09ffb700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8bf40a0fd0 0x7f8bf40a3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.877+0000 7f8c1093d700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8bf40a0fd0 0x7f8bf40a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.877+0000 7f8c09ffb700 1 -- 172.21.15.78:0/2481914477 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8bfc014070 con 0x7f8c0c100cf0 2023-10-10T17:42:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.878+0000 7f8c1093d700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8bf40a0fd0 0x7f8bf40a3480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c1001c0 tx=0x7f8c0400b040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:36.882+0000 7f8c09ffb700 1 -- 172.21.15.78:0/2481914477 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f8bfc0960b0 con 0x7f8c0c100cf0 2023-10-10T17:42:37.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.114+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 --> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f8c0c061c30 con 0x7f8bf40a0fd0 2023-10-10T17:42:37.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.123+0000 7f8c09ffb700 1 -- 172.21.15.78:0/2481914477 <== mgr.15588 v2:172.21.15.78:6800/2386743294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f8c0c061c30 con 0x7f8bf40a0fd0 2023-10-10T17:42:37.125 INFO:teuthology.orchestra.run.smithi078.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-10-10T00:00:00", "created": "2023-10-10T17:42:36", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-10-10T17:42:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8bf40a0fd0 msgr2=0x7f8bf40a3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8bf40a0fd0 0x7f8bf40a3480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c1001c0 tx=0x7f8c0400b040 comp rx=0 tx=0).stop 2023-10-10T17:42:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100cf0 msgr2=0x7f8c0c19c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:37.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100cf0 0x7f8c0c19c4e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc00e940 tx=0x7f8bfc00ec50 comp rx=0 tx=0).stop 2023-10-10T17:42:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 shutdown_connections 2023-10-10T17:42:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8bf40a0fd0 0x7f8bf40a3480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c0c100cf0 0x7f8c0c19c4e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c0c0ff170 0x7f8c0c194f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 --2- 172.21.15.78:0/2481914477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c0c100360 0x7f8c0c195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 >> 172.21.15.78:0/2481914477 conn(0x7f8c0c0fa7b0 msgr2=0x7f8c0c103870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:37.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 shutdown_connections 2023-10-10T17:42:37.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.125+0000 7f8c12ba1700 1 -- 172.21.15.78:0/2481914477 wait complete. 2023-10-10T17:42:37.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-10-10T17:42:37.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.281+0000 7f4ca3eee700 1 -- 172.21.15.78:0/3203414957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 msgr2=0x7f4c9c105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:37.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.281+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/3203414957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c105a10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c009990 tx=0x7f4c8c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.281+0000 7f4ca3eee700 1 -- 172.21.15.78:0/3203414957 shutdown_connections 2023-10-10T17:42:37.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.281+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/3203414957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c9c105f50 0x7f4c9c108350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.281+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/3203414957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c9c100d10 0x7f4c9c1030f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.281+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/3203414957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c105a10 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.281+0000 7f4ca3eee700 1 -- 172.21.15.78:0/3203414957 >> 172.21.15.78:0/3203414957 conn(0x7f4c9c0fa750 msgr2=0x7f4c9c0fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:37.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 -- 172.21.15.78:0/3203414957 shutdown_connections 2023-10-10T17:42:37.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 -- 172.21.15.78:0/3203414957 wait complete. 2023-10-10T17:42:37.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 Processor -- start 2023-10-10T17:42:37.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 -- start start 2023-10-10T17:42:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c9c100d10 0x7f4c9c195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c9c105f50 0x7f4c9c19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c19db10 con 0x7f4c9c103630 2023-10-10T17:42:37.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c19dc50 con 0x7f4c9c105f50 2023-10-10T17:42:37.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca3eee700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c19dde0 con 0x7f4c9c100d10 2023-10-10T17:42:37.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca1489700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:37.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca1489700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c1964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44680/0 (socket says 172.21.15.78:44680) 2023-10-10T17:42:37.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca1489700 1 -- 172.21.15.78:0/2006188447 learned_addr learned my addr 172.21.15.78:0/2006188447 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:37.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca248b700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c9c105f50 0x7f4c9c19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:37.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.282+0000 7f4ca1c8a700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c9c100d10 0x7f4c9c195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:37.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca1489700 1 -- 172.21.15.78:0/2006188447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c9c100d10 msgr2=0x7f4c9c195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:37.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca1489700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c9c100d10 0x7f4c9c195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca1489700 1 -- 172.21.15.78:0/2006188447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c9c105f50 msgr2=0x7f4c9c19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:37.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca1489700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c9c105f50 0x7f4c9c19d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca1489700 1 -- 172.21.15.78:0/2006188447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c0095f0 con 0x7f4c9c103630 2023-10-10T17:42:37.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca1489700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c1964d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c000c00 tx=0x7f4c8c00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:37.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4c92ffd700 1 -- 172.21.15.78:0/2006188447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c01d070 con 0x7f4c9c103630 2023-10-10T17:42:37.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4c92ffd700 1 -- 172.21.15.78:0/2006188447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c8c00fb50 con 0x7f4c9c103630 2023-10-10T17:42:37.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9c10bbf0 con 0x7f4c9c103630 2023-10-10T17:42:37.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4c92ffd700 1 -- 172.21.15.78:0/2006188447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c0173f0 con 0x7f4c9c103630 2023-10-10T17:42:37.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.283+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9c10ba60 con 0x7f4c9c103630 2023-10-10T17:42:37.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.284+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9c0fc2c0 con 0x7f4c9c103630 2023-10-10T17:42:37.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.285+0000 7f4c92ffd700 1 -- 172.21.15.78:0/2006188447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7f4c8c00fcc0 con 0x7f4c9c103630 2023-10-10T17:42:37.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.286+0000 7f4c92ffd700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4c880a1350 0x7f4c880a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:37.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.286+0000 7f4ca1c8a700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4c880a1350 0x7f4c880a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:37.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.286+0000 7f4c92ffd700 1 -- 172.21.15.78:0/2006188447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4c8c0ce820 con 0x7f4c9c103630 2023-10-10T17:42:37.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.286+0000 7f4ca1c8a700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4c880a1350 0x7f4c880a3800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4c98005950 tx=0x7f4c9800b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:37.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.288+0000 7f4c92ffd700 1 -- 172.21.15.78:0/2006188447 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4c8c097860 con 0x7f4c9c103630 2023-10-10T17:42:37.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.523+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 --> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4c9c061c10 con 0x7f4c880a1350 2023-10-10T17:42:37.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.530+0000 7f4c92ffd700 1 -- 172.21.15.78:0/2006188447 <== mgr.15588 v2:172.21.15.78:6800/2386743294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f4c9c061c10 con 0x7f4c880a1350 2023-10-10T17:42:37.532 INFO:teuthology.orchestra.run.smithi078.stdout:/ 1M 6M3h 2023-10-10T17:42:37.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4c880a1350 msgr2=0x7f4c880a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:37.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4c880a1350 0x7f4c880a3800 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4c98005950 tx=0x7f4c9800b410 comp rx=0 tx=0).stop 2023-10-10T17:42:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 msgr2=0x7f4c9c1964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c1964d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c000c00 tx=0x7f4c8c00f710 comp rx=0 tx=0).stop 2023-10-10T17:42:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 shutdown_connections 2023-10-10T17:42:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4c9c100d10 0x7f4c9c195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4c880a1350 0x7f4c880a3800 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4c9c105f50 0x7f4c9c19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 --2- 172.21.15.78:0/2006188447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4c9c103630 0x7f4c9c1964d0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 >> 172.21.15.78:0/2006188447 conn(0x7f4c9c0fa750 msgr2=0x7f4c9c101970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 shutdown_connections 2023-10-10T17:42:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:37.533+0000 7f4ca3eee700 1 -- 172.21.15.78:0/2006188447 wait complete. 2023-10-10T17:42:37.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:37 smithi078 ceph-mon[99494]: [10/Oct/2023:17:42:35] ENGINE Bus STARTING 2023-10-10T17:42:37.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:37 smithi078 ceph-mon[99494]: [10/Oct/2023:17:42:36] ENGINE Serving on https://172.21.15.78:7150 2023-10-10T17:42:37.545 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:37 smithi078 ceph-mon[99494]: [10/Oct/2023:17:42:36] ENGINE Serving on http://172.21.15.78:8765 2023-10-10T17:42:37.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:37 smithi078 ceph-mon[99494]: [10/Oct/2023:17:42:36] ENGINE Bus STARTED 2023-10-10T17:42:37.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:37 smithi078 ceph-mon[99494]: mgrmap e19: x(active, since 1.03376s), standbys: y, z 2023-10-10T17:42:37.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:37 smithi078 ceph-mon[99494]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:37.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:37 smithi078 ceph-mon[99494]: pgmap v3: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:37.547 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-10-10T17:42:37.575 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-10-10T17:42:37.618 INFO:teuthology.run_tasks:Running task background_exec... 2023-10-10T17:42:37.632 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:42:37.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2023-10-10T17:42:37.635 INFO:teuthology.run_tasks:Running task workunit... 2023-10-10T17:42:37.711 INFO:tasks.workunit:Pulling workunits from ref 48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf 2023-10-10T17:42:37.712 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-10T17:42:37.712 INFO:tasks.workunit:timeout=3h 2023-10-10T17:42:37.712 INFO:tasks.workunit:cleanup=True 2023-10-10T17:42:37.713 DEBUG:teuthology.orchestra.run.smithi078:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:37.714 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:37 smithi190 ceph-mon[101220]: [10/Oct/2023:17:42:35] ENGINE Bus STARTING 2023-10-10T17:42:37.714 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:37 smithi190 ceph-mon[101220]: [10/Oct/2023:17:42:36] ENGINE Serving on https://172.21.15.78:7150 2023-10-10T17:42:37.715 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:37 smithi190 ceph-mon[101220]: [10/Oct/2023:17:42:36] ENGINE Serving on http://172.21.15.78:8765 2023-10-10T17:42:37.715 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:37 smithi190 ceph-mon[101220]: [10/Oct/2023:17:42:36] ENGINE Bus STARTED 2023-10-10T17:42:37.715 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:37 smithi190 ceph-mon[101220]: mgrmap e19: x(active, since 1.03376s), standbys: y, z 2023-10-10T17:42:37.715 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:37 smithi190 ceph-mon[101220]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:37.715 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:37 smithi190 ceph-mon[101220]: pgmap v3: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:37.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:37 smithi157 ceph-mon[100896]: [10/Oct/2023:17:42:35] ENGINE Bus STARTING 2023-10-10T17:42:37.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:37 smithi157 ceph-mon[100896]: [10/Oct/2023:17:42:36] ENGINE Serving on https://172.21.15.78:7150 2023-10-10T17:42:37.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:37 smithi157 ceph-mon[100896]: [10/Oct/2023:17:42:36] ENGINE Serving on http://172.21.15.78:8765 2023-10-10T17:42:37.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:37 smithi157 ceph-mon[100896]: [10/Oct/2023:17:42:36] ENGINE Bus STARTED 2023-10-10T17:42:37.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:37 smithi157 ceph-mon[100896]: mgrmap e19: x(active, since 1.03376s), standbys: y, z 2023-10-10T17:42:37.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:37 smithi157 ceph-mon[100896]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:37.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:37 smithi157 ceph-mon[100896]: pgmap v3: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:37.730 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-10T17:42:37.731 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 7ch/124d Inode: 1 Links: 2 2023-10-10T17:42:37.731 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-10T17:42:37.731 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:cephfs_t:s0 2023-10-10T17:42:37.731 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-10T17:42:37.732 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-10-10 17:40:41.577773566 +0000 2023-10-10T17:42:37.732 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-10-10 17:40:52.869401175 +0000 2023-10-10T17:42:37.732 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: 2023-10-10 17:40:41.577773566 +0000 2023-10-10T17:42:37.732 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-10T17:42:37.733 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-10T17:42:37.827 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/chrisphoffman/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf 2023-10-10T17:42:37.844 INFO:tasks.workunit.client.0.smithi078.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-10T17:42:38.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:38 smithi078 ceph-mon[99494]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:38 smithi078 ceph-mon[99494]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:38 smithi078 ceph-mon[99494]: pgmap v4: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:38.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:38 smithi078 ceph-mon[99494]: mgrmap e20: x(active, since 2s), standbys: y, z 2023-10-10T17:42:38.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:38 smithi078 ceph-mon[99494]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:38 smithi190 ceph-mon[101220]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:38 smithi190 ceph-mon[101220]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:38 smithi190 ceph-mon[101220]: pgmap v4: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:38.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:38 smithi190 ceph-mon[101220]: mgrmap e20: x(active, since 2s), standbys: y, z 2023-10-10T17:42:38.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:38 smithi190 ceph-mon[101220]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:38 smithi157 ceph-mon[100896]: from='client.15666 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:38 smithi157 ceph-mon[100896]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:38.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:38 smithi157 ceph-mon[100896]: pgmap v4: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:38.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:38 smithi157 ceph-mon[100896]: mgrmap e20: x(active, since 2s), standbys: y, z 2023-10-10T17:42:38.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:38 smithi157 ceph-mon[100896]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:42:39.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:39 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:39 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:39 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:39 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:39 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:39 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:39 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.667 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:39 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:39 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:39 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:39 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.668 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:39 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:39 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:39 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:39 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:39 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:39 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:39.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:39 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: pgmap v5: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:40.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi078 to 196.2M 2023-10-10T17:42:40.883 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Unable to set osd_memory_target on smithi078 to 205745254: error parsing value: Value '205745254' is below minimum 939524096 2023-10-10T17:42:40.884 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi190 to 196.2M 2023-10-10T17:42:40.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Unable to set osd_memory_target on smithi190 to 205773926: error parsing value: Value '205773926' is below minimum 939524096 2023-10-10T17:42:40.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi157 to 196.2M 2023-10-10T17:42:40.885 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Unable to set osd_memory_target on smithi157 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-10T17:42:40.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:42:40.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:42:40.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:42:40.886 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Updating smithi078:/etc/ceph/ceph.conf 2023-10-10T17:42:40.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Updating smithi157:/etc/ceph/ceph.conf 2023-10-10T17:42:40.887 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:40 smithi078 ceph-mon[99494]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:42:40.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: pgmap v5: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi078 to 196.2M 2023-10-10T17:42:40.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Unable to set osd_memory_target on smithi078 to 205745254: error parsing value: Value '205745254' is below minimum 939524096 2023-10-10T17:42:40.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi190 to 196.2M 2023-10-10T17:42:40.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Unable to set osd_memory_target on smithi190 to 205773926: error parsing value: Value '205773926' is below minimum 939524096 2023-10-10T17:42:40.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi157 to 196.2M 2023-10-10T17:42:40.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Unable to set osd_memory_target on smithi157 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-10T17:42:40.920 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:42:40.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:42:40.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:42:40.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Updating smithi078:/etc/ceph/ceph.conf 2023-10-10T17:42:40.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Updating smithi157:/etc/ceph/ceph.conf 2023-10-10T17:42:40.921 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:40 smithi190 ceph-mon[101220]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:42:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: pgmap v5: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:40.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.983 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.983 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi078 to 196.2M 2023-10-10T17:42:40.983 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.984 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Unable to set osd_memory_target on smithi078 to 205745254: error parsing value: Value '205745254' is below minimum 939524096 2023-10-10T17:42:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi190 to 196.2M 2023-10-10T17:42:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:42:40.985 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Unable to set osd_memory_target on smithi190 to 205773926: error parsing value: Value '205773926' is below minimum 939524096 2023-10-10T17:42:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi157 to 196.2M 2023-10-10T17:42:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Unable to set osd_memory_target on smithi157 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-10T17:42:40.986 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:42:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:42:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:42:40.987 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Updating smithi078:/etc/ceph/ceph.conf 2023-10-10T17:42:40.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Updating smithi157:/etc/ceph/ceph.conf 2023-10-10T17:42:40.988 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:40 smithi157 ceph-mon[100896]: Updating smithi190:/etc/ceph/ceph.conf 2023-10-10T17:42:41.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:42:41.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:42:41.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:42:41.881 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:41.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:42:41.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:42:41.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.882 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:41 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:42:41.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:42:41.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:42:41.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:42:41.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:41.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:42:41.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:42:41.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:41 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:42:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.conf 2023-10-10T17:42:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi157:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2023-10-10T17:42:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Updating smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-10T17:42:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:42:41.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:42:41.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:42:41.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:42:41.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:42:41.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:42:41.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:42:41.982 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:41 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:42:42.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:42 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:42:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:42 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:42:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:42 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:42:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:42 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:42:42.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:42 smithi078 ceph-mon[99494]: pgmap v6: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:42.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:42 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:42:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:42 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:42:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:42 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:42:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:42 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:42:42.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:42 smithi190 ceph-mon[101220]: pgmap v6: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:42 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:42:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:42 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:42:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:42 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:42:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:42 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:42:42.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:42 smithi157 ceph-mon[100896]: pgmap v6: 129 pgs: 129 active+clean; 970 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:42:44.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:44 smithi078 ceph-mon[99494]: pgmap v7: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 51 op/s 2023-10-10T17:42:44.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:44 smithi190 ceph-mon[101220]: pgmap v7: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 51 op/s 2023-10-10T17:42:44.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:44 smithi157 ceph-mon[100896]: pgmap v7: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 132 KiB/s wr, 51 op/s 2023-10-10T17:42:46.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:46 smithi078 ceph-mon[99494]: pgmap v8: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 102 KiB/s wr, 39 op/s 2023-10-10T17:42:46.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:46 smithi190 ceph-mon[101220]: pgmap v8: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 102 KiB/s wr, 39 op/s 2023-10-10T17:42:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:46 smithi157 ceph-mon[100896]: pgmap v8: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 102 KiB/s wr, 39 op/s 2023-10-10T17:42:48.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:48 smithi190 ceph-mon[101220]: pgmap v9: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 84 KiB/s wr, 35 op/s 2023-10-10T17:42:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:48 smithi157 ceph-mon[100896]: pgmap v9: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 84 KiB/s wr, 35 op/s 2023-10-10T17:42:49.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:48 smithi078 ceph-mon[99494]: pgmap v9: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 84 KiB/s wr, 35 op/s 2023-10-10T17:42:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.797+0000 7f671687a700 1 -- 172.21.15.78:0/2708512141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 msgr2=0x7f671010bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.797+0000 7f671687a700 1 --2- 172.21.15.78:0/2708512141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f671010bfd0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f670800b0d0 tx=0x7f670800b3e0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.797+0000 7f671687a700 1 -- 172.21.15.78:0/2708512141 shutdown_connections 2023-10-10T17:42:50.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.797+0000 7f671687a700 1 --2- 172.21.15.78:0/2708512141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f671010bfd0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.797+0000 7f671687a700 1 --2- 172.21.15.78:0/2708512141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67100ff3c0 0x7f67100ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.797+0000 7f671687a700 1 --2- 172.21.15.78:0/2708512141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67100ffd60 0x7f6710103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.797+0000 7f671687a700 1 -- 172.21.15.78:0/2708512141 >> 172.21.15.78:0/2708512141 conn(0x7f67100763a0 msgr2=0x7f67100767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.798+0000 7f671687a700 1 -- 172.21.15.78:0/2708512141 shutdown_connections 2023-10-10T17:42:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.798+0000 7f671687a700 1 -- 172.21.15.78:0/2708512141 wait complete. 2023-10-10T17:42:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.798+0000 7f671687a700 1 Processor -- start 2023-10-10T17:42:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.798+0000 7f671687a700 1 -- start start 2023-10-10T17:42:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67100ff3c0 0x7f67101972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67100ffd60 0x7f6710197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f67101a2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6710198130 con 0x7f6710104400 2023-10-10T17:42:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67101a28a0 con 0x7f67100ffd60 2023-10-10T17:42:50.804 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67101a2a10 con 0x7f67100ff3c0 2023-10-10T17:42:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f67101a2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f67101a2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44700/0 (socket says 172.21.15.78:44700) 2023-10-10T17:42:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 -- 172.21.15.78:0/1985609957 learned_addr learned my addr 172.21.15.78:0/1985609957 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f670ffff700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67100ff3c0 0x7f67101972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 -- 172.21.15.78:0/1985609957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67100ff3c0 msgr2=0x7f67101972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67100ff3c0 0x7f67101972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 -- 172.21.15.78:0/1985609957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67100ffd60 msgr2=0x7f6710197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f670f7fe700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67100ffd60 0x7f6710197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67100ffd60 0x7f6710197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 -- 172.21.15.78:0/1985609957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009d00 con 0x7f6710104400 2023-10-10T17:42:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f670ffff700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67100ff3c0 0x7f67101972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f670f7fe700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67100ffd60 0x7f6710197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f6714e17700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f67101a2360 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6708007870 tx=0x7f6708007950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f670d7fa700 1 -- 172.21.15.78:0/1985609957 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800e040 con 0x7f6710104400 2023-10-10T17:42:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f670d7fa700 1 -- 172.21.15.78:0/1985609957 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6708003e10 con 0x7f6710104400 2023-10-10T17:42:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f670d7fa700 1 -- 172.21.15.78:0/1985609957 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6708012af0 con 0x7f6710104400 2023-10-10T17:42:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671010f1d0 con 0x7f6710104400 2023-10-10T17:42:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.799+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671010f040 con 0x7f6710104400 2023-10-10T17:42:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.802+0000 7f670d7fa700 1 -- 172.21.15.78:0/1985609957 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7f67080056c0 con 0x7f6710104400 2023-10-10T17:42:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.802+0000 7f670d7fa700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f66f80a1350 0x7f66f80a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.803+0000 7f670ffff700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f66f80a1350 0x7f66f80a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.803+0000 7f670d7fa700 1 -- 172.21.15.78:0/1985609957 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f67080ce930 con 0x7f6710104400 2023-10-10T17:42:50.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.803+0000 7f670ffff700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f66f80a1350 0x7f66f80a3800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6700005fd0 tx=0x7f6700005dc0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:50.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.803+0000 7f6722a06b80 1 Processor -- start 2023-10-10T17:42:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.803+0000 7f6722a06b80 1 -- start start 2023-10-10T17:42:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662276560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662276b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f6722a06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66217c000 con 0x55c66227c720 2023-10-10T17:42:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f6722a06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6621ce840 con 0x55c66227cb90 2023-10-10T17:42:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f6722a06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6621c7cd0 con 0x55c6620d26b0 2023-10-10T17:42:50.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662276560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662276560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44706/0 (socket says 172.21.15.78:44706) 2023-10-10T17:42:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 -- 172.21.15.78:0/3366902156 learned_addr learned my addr 172.21.15.78:0/3366902156 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f6714e17700 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662276b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670ffff700 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 -- 172.21.15.78:0/3366902156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 msgr2=0x55c66227c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 -- 172.21.15.78:0/3366902156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 msgr2=0x55c662276b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662276b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 -- 172.21.15.78:0/3366902156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6620495d0 con 0x55c66227c720 2023-10-10T17:42:50.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670ffff700 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.804+0000 7f670f7fe700 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662276560 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6704009bd0 tx=0x7f6704009ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476f3b3091723d64 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.805+0000 7f66f67fc700 1 -- 172.21.15.78:0/3366902156 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400eaa0 con 0x55c66227c720 2023-10-10T17:42:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.805+0000 7f66f67fc700 1 -- 172.21.15.78:0/3366902156 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6704004cf0 con 0x55c66227c720 2023-10-10T17:42:50.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.805+0000 7f66f67fc700 1 -- 172.21.15.78:0/3366902156 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704005620 con 0x55c66227c720 2023-10-10T17:42:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.805+0000 7f6722a06b80 1 -- 172.21.15.78:0/3366902156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 msgr2=0x55c662276560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.805+0000 7f6722a06b80 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662276560 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6704009bd0 tx=0x7f6704009ee0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- 172.21.15.78:0/3366902156 shutdown_connections 2023-10-10T17:42:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662276b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 --2- 172.21.15.78:0/3366902156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662276560 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- 172.21.15.78:0/3366902156 >> 172.21.15.78:0/3366902156 conn(0x55c66226bf50 msgr2=0x55c662271270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- 172.21.15.78:0/3366902156 shutdown_connections 2023-10-10T17:42:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- 172.21.15.78:0/3366902156 wait complete. 2023-10-10T17:42:50.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- start start 2023-10-10T17:42:50.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 Processor -- start 2023-10-10T17:42:50.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662285a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662285fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c66227fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c662169880 con 0x55c66227c720 2023-10-10T17:42:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c662169520 con 0x55c66227cb90 2023-10-10T17:42:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.806+0000 7f6722a06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66204b8c0 con 0x55c6620d26b0 2023-10-10T17:42:50.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f6722a06b80 10 client.0 _ll_register_callbacks cb 0x55c662187720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662285fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662285fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44712/0 (socket says 172.21.15.78:44712) 2023-10-10T17:42:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 -- 172.21.15.78:0/250645828 learned_addr learned my addr 172.21.15.78:0/250645828 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f6714e17700 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c66227fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670ffff700 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662285a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 -- 172.21.15.78:0/250645828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 msgr2=0x55c662285a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662285a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 -- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 msgr2=0x55c66227fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c66227fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 -- 172.21.15.78:0/250645828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6704009800 con 0x55c66227c720 2023-10-10T17:42:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f670f7fe700 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662285fc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6704000c00 tx=0x7f6704005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02268b0a587ea17 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f66ef7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f66ef7fe700 1 -- 172.21.15.78:0/250645828 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704021070 con 0x55c66227c720 2023-10-10T17:42:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f66ef7fe700 1 -- 172.21.15.78:0/250645828 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f670400e8a0 con 0x55c66227c720 2023-10-10T17:42:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c662269710 con 0x55c66227c720 2023-10-10T17:42:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f66ef7fe700 1 -- 172.21.15.78:0/250645828 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6704005000 con 0x55c66227c720 2023-10-10T17:42:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f6722a06b80 10 client.15690 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.807+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c66218e130 con 0x55c66227c720 2023-10-10T17:42:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f66ef7fe700 1 -- 172.21.15.78:0/250645828 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c662269710 con 0x55c66227c720 2023-10-10T17:42:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f66ef7fe700 1 -- 172.21.15.78:0/250645828 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f670400c070 con 0x55c66227c720 2023-10-10T17:42:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f6722a06b80 10 client.15690 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f6722a06b80 10 client.15690 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f6722a06b80 20 client.15690 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f6722a06b80 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c66228fec0 0x55c6622922d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f6722a06b80 4 client.15690 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c66218a040 con 0x55c66228fec0 2023-10-10T17:42:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.808+0000 7f670ffff700 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c66228fec0 0x55c6622922d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f66ef7fe700 10 client.15690 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f670ffff700 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c66228fec0 0x55c6622922d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a71721f3a3122378 server_cookie=8d91653d52d81909 in_seq=0 out_seq=0 2023-10-10T17:42:50.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f66ef7fe700 1 -- 172.21.15.78:0/250645828 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c66218a040 con 0x55c66228fec0 2023-10-10T17:42:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f66ef7fe700 10 client.15690 handle_command_reply: tid=0 2023-10-10T17:42:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 client.15690 shutdown 2023-10-10T17:42:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 msgr2=0x55c662285fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662285fc0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f6704000c00 tx=0x7f6704005490 comp rx=0 tx=0).stop 2023-10-10T17:42:50.842 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 shutdown_connections 2023-10-10T17:42:50.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662285a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c66228fec0 0x55c6622922d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c66227fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 --2- 172.21.15.78:0/250645828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662285fc0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.843 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f66ef7fe700 0 client.15690 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:50.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 >> 172.21.15.78:0/250645828 conn(0x55c66226bf50 msgr2=0x55c6622707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:50.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.809+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 shutdown_connections 2023-10-10T17:42:50.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.810+0000 7f6722a06b80 1 -- 172.21.15.78:0/250645828 wait complete. 2023-10-10T17:42:50.844 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.810+0000 7f6722a06b80 20 client.15690 trim_cache size 0 max 16384 2023-10-10T17:42:50.849 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 Processor -- start 2023-10-10T17:42:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 -- start start 2023-10-10T17:42:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662270f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662285650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.850 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66227aa20 con 0x55c66227c720 2023-10-10T17:42:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66228c8a0 con 0x55c66227cb90 2023-10-10T17:42:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6722a06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66227c1e0 con 0x55c6620d26b0 2023-10-10T17:42:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662270f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662270f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44720/0 (socket says 172.21.15.78:44720) 2023-10-10T17:42:50.851 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f670f7fe700 1 -- 172.21.15.78:0/1862245342 learned_addr learned my addr 172.21.15.78:0/1862245342 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.824+0000 7f6714e17700 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662285650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f670f7fe700 1 -- 172.21.15.78:0/1862245342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 msgr2=0x55c66227fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f670f7fe700 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f670f7fe700 1 -- 172.21.15.78:0/1862245342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 msgr2=0x55c662285650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.852 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f670f7fe700 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662285650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f670f7fe700 1 -- 172.21.15.78:0/1862245342 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c66218de80 con 0x55c66227c720 2023-10-10T17:42:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f670f7fe700 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662270f60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6704004cc0 tx=0x7f670400e700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605f2db48fe83ec6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f66f6ffd700 1 -- 172.21.15.78:0/1862245342 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704021070 con 0x55c66227c720 2023-10-10T17:42:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f66f6ffd700 1 -- 172.21.15.78:0/1862245342 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6704009460 con 0x55c66227c720 2023-10-10T17:42:50.853 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.825+0000 7f66f6ffd700 1 -- 172.21.15.78:0/1862245342 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704004490 con 0x55c66227c720 2023-10-10T17:42:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 -- 172.21.15.78:0/1862245342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 msgr2=0x55c662270f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662270f60 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6704004cc0 tx=0x7f670400e700 comp rx=0 tx=0).stop 2023-10-10T17:42:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 -- 172.21.15.78:0/1862245342 shutdown_connections 2023-10-10T17:42:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c66227fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.854 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c662285650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 --2- 172.21.15.78:0/1862245342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662270f60 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 -- 172.21.15.78:0/1862245342 >> 172.21.15.78:0/1862245342 conn(0x55c66226bf50 msgr2=0x55c662272a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 -- 172.21.15.78:0/1862245342 shutdown_connections 2023-10-10T17:42:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 -- 172.21.15.78:0/1862245342 wait complete. 2023-10-10T17:42:50.855 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.826+0000 7f6722a06b80 1 -- start start 2023-10-10T17:42:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 1 Processor -- start 2023-10-10T17:42:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662283630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662283b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c6622840b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6622e2db0 con 0x55c66227c720 2023-10-10T17:42:50.856 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6622e2f20 con 0x55c66227cb90 2023-10-10T17:42:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6622e30e0 con 0x55c6620d26b0 2023-10-10T17:42:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6722a06b80 10 client.0 _ll_register_callbacks cb 0x7f67080cbb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662283b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.857 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662283b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44726/0 (socket says 172.21.15.78:44726) 2023-10-10T17:42:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 -- 172.21.15.78:0/2964774391 learned_addr learned my addr 172.21.15.78:0/2964774391 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f6714e17700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c6622840b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670ffff700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662283630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 -- 172.21.15.78:0/2964774391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 msgr2=0x55c662283630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.858 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662283630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 -- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 msgr2=0x55c6622840b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c6622840b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670f7fe700 1 -- 172.21.15.78:0/2964774391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c66218e130 con 0x55c66227c720 2023-10-10T17:42:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.827+0000 7f670ffff700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662283630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:50.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f670f7fe700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662283b70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6704009f20 tx=0x7f6704011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da795b7f41027e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f66ef7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f66ef7fe700 1 -- 172.21.15.78:0/2964774391 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6704021040 con 0x55c66227c720 2023-10-10T17:42:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f66ef7fe700 1 -- 172.21.15.78:0/2964774391 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f6704000e40 con 0x55c66227c720 2023-10-10T17:42:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6704004cf0 con 0x55c66227c720 2023-10-10T17:42:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f66ef7fe700 1 -- 172.21.15.78:0/2964774391 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6704004fd0 con 0x55c66227c720 2023-10-10T17:42:50.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 10 client.15696 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c661fdec00 con 0x55c66227c720 2023-10-10T17:42:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f66ef7fe700 1 -- 172.21.15.78:0/2964774391 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6704004cf0 con 0x55c66227c720 2023-10-10T17:42:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f66ef7fe700 1 -- 172.21.15.78:0/2964774391 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6704009c00 con 0x55c66227c720 2023-10-10T17:42:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 10 client.15696 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:50.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 10 client.15696 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 20 client.15696 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6622e4370 0x55c6622e6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 4 client.15696 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:42:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.828+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c662196600 con 0x55c6622e4370 2023-10-10T17:42:50.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.829+0000 7f670ffff700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6622e4370 0x55c6622e6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.829+0000 7f66ef7fe700 10 client.15696 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.829+0000 7f670ffff700 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6622e4370 0x55c6622e6750 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba6bdc9d95152a70 server_cookie=84141c66f85b3dcc in_seq=0 out_seq=0 2023-10-10T17:42:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f66ef7fe700 1 -- 172.21.15.78:0/2964774391 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c662196600 con 0x55c6622e4370 2023-10-10T17:42:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f66ef7fe700 10 client.15696 handle_command_reply: tid=0 2023-10-10T17:42:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f6722a06b80 1 client.15696 shutdown 2023-10-10T17:42:50.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 msgr2=0x55c662283b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f6722a06b80 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662283b70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6704009f20 tx=0x7f6704011720 comp rx=0 tx=0).stop 2023-10-10T17:42:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 shutdown_connections 2023-10-10T17:42:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f6722a06b80 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6620d26b0 0x55c662283630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.843+0000 7f6722a06b80 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6622e4370 0x55c6622e6750 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f6722a06b80 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c66227cb90 0x55c6622840b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f66ef7fe700 0 client.15696 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f6722a06b80 1 --2- 172.21.15.78:0/2964774391 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c66227c720 0x55c662283b70 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 >> 172.21.15.78:0/2964774391 conn(0x55c66226bf50 msgr2=0x55c66227ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 shutdown_connections 2023-10-10T17:42:50.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f6722a06b80 1 -- 172.21.15.78:0/2964774391 wait complete. 2023-10-10T17:42:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f6722a06b80 20 client.15696 trim_cache size 0 max 16384 2023-10-10T17:42:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:42:50.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:42:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:42:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:42:50.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f66f80a1350 msgr2=0x7f66f80a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f66f80a1350 0x7f66f80a3800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6700005fd0 tx=0x7f6700005dc0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 msgr2=0x7f67101a2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f67101a2360 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6708007870 tx=0x7f6708007950 comp rx=0 tx=0).stop 2023-10-10T17:42:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 shutdown_connections 2023-10-10T17:42:50.868 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f67100ff3c0 0x7f67101972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f66f80a1350 0x7f66f80a3800 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6710104400 0x7f67101a2360 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 --2- 172.21.15.78:0/1985609957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67100ffd60 0x7f6710197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 >> 172.21.15.78:0/1985609957 conn(0x7f67100763a0 msgr2=0x7f67100fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:50.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 shutdown_connections 2023-10-10T17:42:50.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:42:50.844+0000 7f671687a700 1 -- 172.21.15.78:0/1985609957 wait complete. 2023-10-10T17:42:50.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:50 smithi190 ceph-mon[101220]: pgmap v10: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-10T17:42:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:50 smithi157 ceph-mon[100896]: pgmap v10: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-10T17:42:51.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:50 smithi078 ceph-mon[99494]: pgmap v10: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-10T17:42:52.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:52 smithi190 ceph-mon[101220]: pgmap v11: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-10T17:42:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:52 smithi157 ceph-mon[100896]: pgmap v11: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-10T17:42:53.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:52 smithi078 ceph-mon[99494]: pgmap v11: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 77 KiB/s wr, 32 op/s 2023-10-10T17:42:54.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:54 smithi190 ceph-mon[101220]: pgmap v12: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 36 op/s 2023-10-10T17:42:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:54 smithi157 ceph-mon[100896]: pgmap v12: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 36 op/s 2023-10-10T17:42:55.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:54 smithi078 ceph-mon[99494]: pgmap v12: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 77 KiB/s wr, 36 op/s 2023-10-10T17:42:56.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:56 smithi190 ceph-mon[101220]: pgmap v13: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:42:56.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:56 smithi157 ceph-mon[100896]: pgmap v13: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:42:57.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:56 smithi078 ceph-mon[99494]: pgmap v13: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:42:58.570 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:42:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 -- 172.21.15.78:0/1719554849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc1055d0 msgr2=0x7f37cc10d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 --2- 172.21.15.78:0/1719554849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc1055d0 0x7f37cc10d1e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f37c400b0d0 tx=0x7f37c400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 -- 172.21.15.78:0/1719554849 shutdown_connections 2023-10-10T17:42:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 --2- 172.21.15.78:0/1719554849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37cc0feaf0 0x7f37cc0feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 --2- 172.21.15.78:0/1719554849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc1055d0 0x7f37cc10d1e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 --2- 172.21.15.78:0/1719554849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37cc0ff490 0x7f37cc104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 -- 172.21.15.78:0/1719554849 >> 172.21.15.78:0/1719554849 conn(0x7f37cc075920 msgr2=0x7f37cc075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 -- 172.21.15.78:0/1719554849 shutdown_connections 2023-10-10T17:42:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.751+0000 7f37d35ea700 1 -- 172.21.15.78:0/1719554849 wait complete. 2023-10-10T17:42:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 Processor -- start 2023-10-10T17:42:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 -- start start 2023-10-10T17:42:58.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37cc0feaf0 0x7f37cc196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc0ff490 0x7f37cc196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37cc1055d0 0x7f37cc19aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc19b4f0 con 0x7f37cc0ff490 2023-10-10T17:42:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc19b660 con 0x7f37cc1055d0 2023-10-10T17:42:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d35ea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37cc19b820 con 0x7f37cc0feaf0 2023-10-10T17:42:58.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d0b85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc0ff490 0x7f37cc196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d0b85700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc0ff490 0x7f37cc196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44740/0 (socket says 172.21.15.78:44740) 2023-10-10T17:42:58.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.752+0000 7f37d0b85700 1 -- 172.21.15.78:0/483816051 learned_addr learned my addr 172.21.15.78:0/483816051 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:58.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d1b87700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37cc1055d0 0x7f37cc19aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d1386700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37cc0feaf0 0x7f37cc196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d0b85700 1 -- 172.21.15.78:0/483816051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37cc0feaf0 msgr2=0x7f37cc196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d0b85700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37cc0feaf0 0x7f37cc196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d0b85700 1 -- 172.21.15.78:0/483816051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37cc1055d0 msgr2=0x7f37cc19aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d0b85700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37cc1055d0 0x7f37cc19aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d0b85700 1 -- 172.21.15.78:0/483816051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c4009d00 con 0x7f37cc0ff490 2023-10-10T17:42:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d1b87700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37cc1055d0 0x7f37cc19aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d0b85700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc0ff490 0x7f37cc196a50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00e940 tx=0x7f37bc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37c27fc700 1 -- 172.21.15.78:0/483816051 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc009730 con 0x7f37cc0ff490 2023-10-10T17:42:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37cc1103f0 con 0x7f37cc0ff490 2023-10-10T17:42:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37c27fc700 1 -- 172.21.15.78:0/483816051 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37bc00cbd0 con 0x7f37cc0ff490 2023-10-10T17:42:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc110260 con 0x7f37cc0ff490 2023-10-10T17:42:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.753+0000 7f37c27fc700 1 -- 172.21.15.78:0/483816051 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc018680 con 0x7f37cc0ff490 2023-10-10T17:42:58.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.757+0000 7f37c27fc700 1 -- 172.21.15.78:0/483816051 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7f37bc00cd40 con 0x7f37cc0ff490 2023-10-10T17:42:58.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.758+0000 7f37df776b80 1 Processor -- start 2023-10-10T17:42:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.758+0000 7f37c27fc700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37b80a5760 0x7f37b80a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.758+0000 7f37c27fc700 1 -- 172.21.15.78:0/483816051 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f37bc014070 con 0x7f37cc0ff490 2023-10-10T17:42:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37df776b80 1 -- start start 2023-10-10T17:42:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37d1b87700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37b80a5760 0x7f37b80a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37df776b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a30740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37df776b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 0x55e0f5a34b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37df776b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a35510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37df776b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f592d060 con 0x55e0f5a30c80 2023-10-10T17:42:58.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37df776b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f597f8a0 con 0x55e0f5a350c0 2023-10-10T17:42:58.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37df776b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f5978d30 con 0x55e0f5a30370 2023-10-10T17:42:58.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37d1b87700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a35510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37d1b87700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a35510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:46468/0 (socket says 172.21.15.78:46468) 2023-10-10T17:42:58.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37d1b87700 1 -- 172.21.15.78:0/1244102640 learned_addr learned my addr 172.21.15.78:0/1244102640 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:58.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.760+0000 7f37d1b87700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37b80a5760 0x7f37b80a7c10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009fd0 tx=0x7f37c4008c10 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37d1b87700 1 -- 172.21.15.78:0/1244102640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 msgr2=0x55e0f5a30740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:42:58.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37d1b87700 1 --2- 172.21.15.78:0/1244102640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a30740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37d1b87700 1 -- 172.21.15.78:0/1244102640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 msgr2=0x55e0f5a34b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:58.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37d1b87700 1 --2- 172.21.15.78:0/1244102640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 0x55e0f5a34b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37d1b87700 1 -- 172.21.15.78:0/1244102640 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0f57fa630 con 0x55e0f5a350c0 2023-10-10T17:42:58.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37d1b87700 1 --2- 172.21.15.78:0/1244102640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a35510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37c4008f80 tx=0x7f37c4024000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77338792d3472c02 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37b77fe700 1 -- 172.21.15.78:0/1244102640 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c400e040 con 0x55e0f5a350c0 2023-10-10T17:42:58.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37b77fe700 1 -- 172.21.15.78:0/1244102640 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37c400be30 con 0x55e0f5a350c0 2023-10-10T17:42:58.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.761+0000 7f37b77fe700 1 -- 172.21.15.78:0/1244102640 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c401ee40 con 0x55e0f5a350c0 2023-10-10T17:42:58.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 -- 172.21.15.78:0/1244102640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 msgr2=0x55e0f5a35510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 --2- 172.21.15.78:0/1244102640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a35510 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f37c4008f80 tx=0x7f37c4024000 comp rx=0 tx=0).stop 2023-10-10T17:42:58.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 -- 172.21.15.78:0/1244102640 shutdown_connections 2023-10-10T17:42:58.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 --2- 172.21.15.78:0/1244102640 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a30740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 --2- 172.21.15.78:0/1244102640 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a35510 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 --2- 172.21.15.78:0/1244102640 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 0x55e0f5a34b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 -- 172.21.15.78:0/1244102640 >> 172.21.15.78:0/1244102640 conn(0x55e0f5a1d360 msgr2=0x55e0f5a22b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:58.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 -- 172.21.15.78:0/1244102640 shutdown_connections 2023-10-10T17:42:58.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 -- 172.21.15.78:0/1244102640 wait complete. 2023-10-10T17:42:58.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 -- start start 2023-10-10T17:42:58.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.762+0000 7f37df776b80 1 Processor -- start 2023-10-10T17:42:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37df776b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a3c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37df776b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a39a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37df776b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a3a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37df776b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f591a8e0 con 0x55e0f5a30370 2023-10-10T17:42:58.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37df776b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f591a580 con 0x55e0f5a350c0 2023-10-10T17:42:58.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37df776b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f57fc920 con 0x55e0f5a30c80 2023-10-10T17:42:58.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37df776b80 10 client.0 _ll_register_callbacks cb 0x55e0f5938780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:58.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37d1386700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a3c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37d1386700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a3c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44748/0 (socket says 172.21.15.78:44748) 2023-10-10T17:42:58.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37d1386700 1 -- 172.21.15.78:0/1631621220 learned_addr learned my addr 172.21.15.78:0/1631621220 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:58.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37d0b85700 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a39a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.763+0000 7f37d1b87700 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a3a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37d1386700 1 -- 172.21.15.78:0/1631621220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 msgr2=0x55e0f5a39a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37d1386700 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a39a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37d1386700 1 -- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 msgr2=0x55e0f5a3a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37d1386700 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a3a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37d1386700 1 -- 172.21.15.78:0/1631621220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37cc10da20 con 0x55e0f5a30370 2023-10-10T17:42:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37d1386700 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a3c7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f37c800de20 tx=0x7f37c800f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6962696fd4c884 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37b4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1631621220 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8011440 con 0x55e0f5a30370 2023-10-10T17:42:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1631621220 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f37c80115a0 con 0x55e0f5a30370 2023-10-10T17:42:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0f5a1ab20 con 0x55e0f5a30370 2023-10-10T17:42:58.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1631621220 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37c80117a0 con 0x55e0f5a30370 2023-10-10T17:42:58.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37df776b80 10 client.15705 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:58.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e0f593f190 con 0x55e0f5a30370 2023-10-10T17:42:58.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.764+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1631621220 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e0f5a1ab20 con 0x55e0f5a30370 2023-10-10T17:42:58.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1631621220 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f37c801b9a0 con 0x55e0f5a30370 2023-10-10T17:42:58.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37df776b80 10 client.15705 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:58.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37df776b80 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:58.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37df776b80 20 client.15705 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:58.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37df776b80 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5a63860 0x55e0f5a65c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37df776b80 4 client.15705 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:58.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0f593b0a0 con 0x55e0f5a63860 2023-10-10T17:42:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37d0b85700 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5a63860 0x55e0f5a65c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37b4ff9700 10 client.15705 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.765+0000 7f37d0b85700 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5a63860 0x55e0f5a65c70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22502d8df27d39e5 server_cookie=9c11c0adb2bd27ee in_seq=0 out_seq=0 2023-10-10T17:42:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1631621220 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e0f593b0a0 con 0x55e0f5a63860 2023-10-10T17:42:58.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37b4ff9700 10 client.15705 handle_command_reply: tid=0 2023-10-10T17:42:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 client.15705 shutdown 2023-10-10T17:42:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 msgr2=0x55e0f5a3c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a3c7a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f37c800de20 tx=0x7f37c800f640 comp rx=0 tx=0).stop 2023-10-10T17:42:58.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 shutdown_connections 2023-10-10T17:42:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5a63860 0x55e0f5a65c70 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a3a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a3c7a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 --2- 172.21.15.78:0/1631621220 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a39a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37b4ff9700 0 client.15705 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 >> 172.21.15.78:0/1631621220 conn(0x55e0f5a1d360 msgr2=0x55e0f5a21de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.766+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 shutdown_connections 2023-10-10T17:42:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.767+0000 7f37df776b80 1 -- 172.21.15.78:0/1631621220 wait complete. 2023-10-10T17:42:58.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.767+0000 7f37df776b80 20 client.15705 trim_cache size 0 max 16384 2023-10-10T17:42:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.783+0000 7f37df776b80 1 Processor -- start 2023-10-10T17:42:58.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.783+0000 7f37df776b80 1 -- start start 2023-10-10T17:42:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37df776b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a329b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37df776b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 0x55e0f5a32ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37df776b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a22560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37df776b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f5a33430 con 0x55e0f5a30c80 2023-10-10T17:42:58.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37df776b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f5a33c60 con 0x55e0f5a350c0 2023-10-10T17:42:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37df776b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f5a37fc0 con 0x55e0f5a30370 2023-10-10T17:42:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a329b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a329b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:36286/0 (socket says 172.21.15.78:36286) 2023-10-10T17:42:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 -- 172.21.15.78:0/2324221494 learned_addr learned my addr 172.21.15.78:0/2324221494 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:58.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1b87700 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a22560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d0b85700 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 0x55e0f5a32ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 -- 172.21.15.78:0/2324221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 msgr2=0x55e0f5a22560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:58.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a22560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 -- 172.21.15.78:0/2324221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 msgr2=0x55e0f5a32ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:58.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 0x55e0f5a32ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.784+0000 7f37d1386700 1 -- 172.21.15.78:0/2324221494 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e0f593eee0 con 0x55e0f5a30370 2023-10-10T17:42:58.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.785+0000 7f37d1386700 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a329b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f37c8005950 tx=0x7f37c80057a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36bfe84507552e67 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.785+0000 7f37b7fff700 1 -- 172.21.15.78:0/2324221494 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c800f990 con 0x55e0f5a30370 2023-10-10T17:42:58.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.785+0000 7f37b7fff700 1 -- 172.21.15.78:0/2324221494 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37c800d510 con 0x55e0f5a30370 2023-10-10T17:42:58.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.785+0000 7f37b7fff700 1 -- 172.21.15.78:0/2324221494 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8010bd0 con 0x55e0f5a30370 2023-10-10T17:42:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 -- 172.21.15.78:0/2324221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 msgr2=0x55e0f5a329b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a329b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f37c8005950 tx=0x7f37c80057a0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 -- 172.21.15.78:0/2324221494 shutdown_connections 2023-10-10T17:42:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30370 0x55e0f5a329b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a22560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 --2- 172.21.15.78:0/2324221494 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30c80 0x55e0f5a32ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 -- 172.21.15.78:0/2324221494 >> 172.21.15.78:0/2324221494 conn(0x55e0f5a1d360 msgr2=0x55e0f5a34120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:58.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 -- 172.21.15.78:0/2324221494 shutdown_connections 2023-10-10T17:42:58.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 -- 172.21.15.78:0/2324221494 wait complete. 2023-10-10T17:42:58.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 -- start start 2023-10-10T17:42:58.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.786+0000 7f37df776b80 1 Processor -- start 2023-10-10T17:42:58.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37df776b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a2b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37df776b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a2bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37df776b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a29e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37df776b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f5a2a530 con 0x55e0f5a30370 2023-10-10T17:42:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37df776b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f5a2a6a0 con 0x55e0f5a350c0 2023-10-10T17:42:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37df776b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0f5a2a860 con 0x55e0f5a30c80 2023-10-10T17:42:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37df776b80 10 client.0 _ll_register_callbacks cb 0x7f37c800d870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:58.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a2b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a2b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44762/0 (socket says 172.21.15.78:44762) 2023-10-10T17:42:58.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 -- 172.21.15.78:0/1153744681 learned_addr learned my addr 172.21.15.78:0/1153744681 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:58.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1b87700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a29e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d0b85700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a2bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 -- 172.21.15.78:0/1153744681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 msgr2=0x55e0f5a2bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:58.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a2bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 -- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 msgr2=0x55e0f5a29e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:58.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a29e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1386700 1 -- 172.21.15.78:0/1153744681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e0f593f190 con 0x55e0f5a30370 2023-10-10T17:42:58.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.787+0000 7f37d1b87700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a29e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:58.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37d1386700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a2b980 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f37c80103f0 tx=0x7f37c800f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d759fc0bcc006cdb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37b4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:58.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1153744681 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8004450 con 0x55e0f5a30370 2023-10-10T17:42:58.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1153744681 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f37c8000e40 con 0x55e0f5a30370 2023-10-10T17:42:58.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37c400be30 con 0x55e0f5a30370 2023-10-10T17:42:58.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1153744681 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37c800f990 con 0x55e0f5a30370 2023-10-10T17:42:58.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 10 client.15708 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:58.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e0f578fc60 con 0x55e0f5a30370 2023-10-10T17:42:58.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1153744681 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37c400be30 con 0x55e0f5a30370 2023-10-10T17:42:58.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1153744681 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f37c8010bf0 con 0x55e0f5a30370 2023-10-10T17:42:58.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 10 client.15708 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:58.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:58.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 20 client.15708 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:58.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5ab8740 0x55e0f5abab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 4 client.15708 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:42:58.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.788+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e0f5947660 con 0x55e0f5ab8740 2023-10-10T17:42:58.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.789+0000 7f37d0b85700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5ab8740 0x55e0f5abab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.789+0000 7f37b4ff9700 10 client.15708 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:58.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.789+0000 7f37d0b85700 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5ab8740 0x55e0f5abab50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1d66bec8138f1a7 server_cookie=cce24987050cb8ac in_seq=0 out_seq=0 2023-10-10T17:42:58.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37b4ff9700 1 -- 172.21.15.78:0/1153744681 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e0f5947660 con 0x55e0f5ab8740 2023-10-10T17:42:58.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37b4ff9700 10 client.15708 handle_command_reply: tid=0 2023-10-10T17:42:58.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 client.15708 shutdown 2023-10-10T17:42:58.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 msgr2=0x55e0f5a2b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a2b980 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f37c80103f0 tx=0x7f37c800f640 comp rx=0 tx=0).stop 2023-10-10T17:42:58.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 shutdown_connections 2023-10-10T17:42:58.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e0f5ab8740 0x55e0f5abab50 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e0f5a350c0 0x55e0f5a29e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e0f5a30370 0x55e0f5a2b980 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 --2- 172.21.15.78:0/1153744681 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e0f5a30c80 0x55e0f5a2bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 >> 172.21.15.78:0/1153744681 conn(0x55e0f5a1d360 msgr2=0x55e0f5a26960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:58.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37b4ff9700 0 client.15708 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:58.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 shutdown_connections 2023-10-10T17:42:58.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 1 -- 172.21.15.78:0/1153744681 wait complete. 2023-10-10T17:42:58.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.790+0000 7f37df776b80 20 client.15708 trim_cache size 0 max 16384 2023-10-10T17:42:58.812 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:42:58.812 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:42:58.813 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:42:58.813 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:42:58.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37b80a5760 msgr2=0x7f37b80a7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37b80a5760 0x7f37b80a7c10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009fd0 tx=0x7f37c4008c10 comp rx=0 tx=0).stop 2023-10-10T17:42:58.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc0ff490 msgr2=0x7f37cc196a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc0ff490 0x7f37cc196a50 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f37bc00e940 tx=0x7f37bc00ed00 comp rx=0 tx=0).stop 2023-10-10T17:42:58.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 shutdown_connections 2023-10-10T17:42:58.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f37cc0feaf0 0x7f37cc196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37b80a5760 0x7f37b80a7c10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37cc1055d0 0x7f37cc19aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 --2- 172.21.15.78:0/483816051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37cc0ff490 0x7f37cc196a50 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 >> 172.21.15.78:0/483816051 conn(0x7f37cc075920 msgr2=0x7f37cc0fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:58.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 shutdown_connections 2023-10-10T17:42:58.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.791+0000 7f37d35ea700 1 -- 172.21.15.78:0/483816051 wait complete. 2023-10-10T17:42:58.817 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:42:58.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:42:58.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:58 smithi078 ceph-mon[99494]: pgmap v14: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 9 op/s 2023-10-10T17:42:58.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:58 smithi190 ceph-mon[101220]: pgmap v14: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 9 op/s 2023-10-10T17:42:58.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 -- 172.21.15.78:0/2726070103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7700ffdc0 msgr2=0x7fe770103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 --2- 172.21.15.78:0/2726070103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7700ffdc0 0x7fe770103d80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe7600099c0 tx=0x7fe760009cd0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 -- 172.21.15.78:0/2726070103 shutdown_connections 2023-10-10T17:42:58.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 --2- 172.21.15.78:0/2726070103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe770104460 0x7fe77010c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 --2- 172.21.15.78:0/2726070103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7700ff420 0x7fe7700ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 --2- 172.21.15.78:0/2726070103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7700ffdc0 0x7fe770103d80 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 -- 172.21.15.78:0/2726070103 >> 172.21.15.78:0/2726070103 conn(0x7fe770076310 msgr2=0x7fe770076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 -- 172.21.15.78:0/2726070103 shutdown_connections 2023-10-10T17:42:58.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.955+0000 7fe777e17700 1 -- 172.21.15.78:0/2726070103 wait complete. 2023-10-10T17:42:58.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 Processor -- start 2023-10-10T17:42:58.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 -- start start 2023-10-10T17:42:58.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7700ff420 0x7fe770197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7700ffdc0 0x7fe770197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe770104460 0x7fe7701a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7701980d0 con 0x7fe770104460 2023-10-10T17:42:58.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7701a2810 con 0x7fe7700ff420 2023-10-10T17:42:58.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe777e17700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7701a29d0 con 0x7fe7700ffdc0 2023-10-10T17:42:58.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe770104460 0x7fe7701a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe770104460 0x7fe7701a22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44780/0 (socket says 172.21.15.78:44780) 2023-10-10T17:42:58.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 -- 172.21.15.78:0/1986611159 learned_addr learned my addr 172.21.15.78:0/1986611159 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:58.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe775bb3700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7700ff420 0x7fe770197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7753b2700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7700ffdc0 0x7fe770197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 -- 172.21.15.78:0/1986611159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7700ffdc0 msgr2=0x7fe770197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7700ffdc0 0x7fe770197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 -- 172.21.15.78:0/1986611159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7700ff420 msgr2=0x7fe770197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7700ff420 0x7fe770197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.956+0000 7fe7763b4700 1 -- 172.21.15.78:0/1986611159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7600095f0 con 0x7fe770104460 2023-10-10T17:42:58.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.957+0000 7fe7763b4700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe770104460 0x7fe7701a22d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe7680094f0 tx=0x7fe76800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.957+0000 7fe766ffd700 1 -- 172.21.15.78:0/1986611159 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76800ddd0 con 0x7fe770104460 2023-10-10T17:42:58.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.957+0000 7fe766ffd700 1 -- 172.21.15.78:0/1986611159 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe76800f040 con 0x7fe770104460 2023-10-10T17:42:58.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.957+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe77010f1e0 con 0x7fe770104460 2023-10-10T17:42:58.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.957+0000 7fe766ffd700 1 -- 172.21.15.78:0/1986611159 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768015400 con 0x7fe770104460 2023-10-10T17:42:58.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.957+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe77010f050 con 0x7fe770104460 2023-10-10T17:42:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.958+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe77010f370 con 0x7fe770104460 2023-10-10T17:42:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.959+0000 7fe766ffd700 1 -- 172.21.15.78:0/1986611159 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7fe768004c40 con 0x7fe770104460 2023-10-10T17:42:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.960+0000 7fe766ffd700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe75c0a13d0 0x7fe75c0a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:58.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.960+0000 7fe775bb3700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe75c0a13d0 0x7fe75c0a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.960+0000 7fe766ffd700 1 -- 172.21.15.78:0/1986611159 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fe7680cca70 con 0x7fe770104460 2023-10-10T17:42:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.960+0000 7fe775bb3700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe75c0a13d0 0x7fe75c0a3880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009c60 tx=0x7fe76c009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:58.966+0000 7fe766ffd700 1 -- 172.21.15.78:0/1986611159 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fe76804e730 con 0x7fe770104460 2023-10-10T17:42:58.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:58 smithi157 ceph-mon[100896]: pgmap v14: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 682 B/s wr, 9 op/s 2023-10-10T17:42:59.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.209+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe770067a60 con 0x7fe770104460 2023-10-10T17:42:59.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.210+0000 7fe766ffd700 1 -- 172.21.15.78:0/1986611159 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7fe768095a20 con 0x7fe770104460 2023-10-10T17:42:59.211 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:42:59.211 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:42:59.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe75c0a13d0 msgr2=0x7fe75c0a3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe75c0a13d0 0x7fe75c0a3880 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe76c009c60 tx=0x7fe76c009260 comp rx=0 tx=0).stop 2023-10-10T17:42:59.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe770104460 msgr2=0x7fe7701a22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe770104460 0x7fe7701a22d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fe7680094f0 tx=0x7fe76800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 shutdown_connections 2023-10-10T17:42:59.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe75c0a13d0 0x7fe75c0a3880 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe770104460 0x7fe7701a22d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe7700ff420 0x7fe770197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 --2- 172.21.15.78:0/1986611159 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe7700ffdc0 0x7fe770197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.212+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 >> 172.21.15.78:0/1986611159 conn(0x7fe770076310 msgr2=0x7fe7700fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.213+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 shutdown_connections 2023-10-10T17:42:59.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.213+0000 7fe777e17700 1 -- 172.21.15.78:0/1986611159 wait complete. 2023-10-10T17:42:59.216 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:42:59.229 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:42:59.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1516967738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f480ffdf0 msgr2=0x7f7f48103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1516967738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f480ffdf0 0x7f7f48103db0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0099c0 tx=0x7f7f3c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1516967738 shutdown_connections 2023-10-10T17:42:59.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1516967738 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f480ff450 0x7f7f480ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1516967738 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f48104490 0x7f7f4810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1516967738 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f480ffdf0 0x7f7f48103db0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1516967738 >> 172.21.15.78:0/1516967738 conn(0x7f7f48077080 msgr2=0x7f7f48077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1516967738 shutdown_connections 2023-10-10T17:42:59.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.362+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1516967738 wait complete. 2023-10-10T17:42:59.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 Processor -- start 2023-10-10T17:42:59.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 -- start start 2023-10-10T17:42:59.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f480ff450 0x7f7f481963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f480ffdf0 0x7f7f48196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48104490 0x7f7f4819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4819b3a0 con 0x7f7f48104490 2023-10-10T17:42:59.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4819b510 con 0x7f7f480ffdf0 2023-10-10T17:42:59.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.363+0000 7f7f4d1f0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4819b6d0 con 0x7f7f480ff450 2023-10-10T17:42:59.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48104490 0x7f7f4819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48104490 0x7f7f4819ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44792/0 (socket says 172.21.15.78:44792) 2023-10-10T17:42:59.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 -- 172.21.15.78:0/1305866685 learned_addr learned my addr 172.21.15.78:0/1305866685 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f480ffdf0 0x7f7f48196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f480ff450 0x7f7f481963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 -- 172.21.15.78:0/1305866685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f480ff450 msgr2=0x7f7f481963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f480ff450 0x7f7f481963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 -- 172.21.15.78:0/1305866685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f480ffdf0 msgr2=0x7f7f48196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f480ffdf0 0x7f7f48196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 -- 172.21.15.78:0/1305866685 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3c0095f0 con 0x7f7f48104490 2023-10-10T17:42:59.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f480ff450 0x7f7f481963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:59.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f480ffdf0 0x7f7f48196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:59.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4759e700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48104490 0x7f7f4819ad60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000ef30 tx=0x7f7f4000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f2ffff700 1 -- 172.21.15.78:0/1305866685 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40018070 con 0x7f7f48104490 2023-10-10T17:42:59.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f48110370 con 0x7f7f48104490 2023-10-10T17:42:59.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f2ffff700 1 -- 172.21.15.78:0/1305866685 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f4000f040 con 0x7f7f48104490 2023-10-10T17:42:59.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.364+0000 7f7f2ffff700 1 -- 172.21.15.78:0/1305866685 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4000cdb0 con 0x7f7f48104490 2023-10-10T17:42:59.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.365+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f481101e0 con 0x7f7f48104490 2023-10-10T17:42:59.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.368+0000 7f7f2ffff700 1 -- 172.21.15.78:0/1305866685 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7f7f40003a00 con 0x7f7f48104490 2023-10-10T17:42:59.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.368+0000 7f7f2ffff700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7f300a1350 0x7f7f300a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.368+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7f300a1350 0x7f7f300a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.368+0000 7f7f2ffff700 1 -- 172.21.15.78:0/1305866685 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f7f400cc980 con 0x7f7f48104490 2023-10-10T17:42:59.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.369+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7f300a1350 0x7f7f300a3800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009cb0 tx=0x7f7f38009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 Processor -- start 2023-10-10T17:42:59.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 -- start start 2023-10-10T17:42:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa6895970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa6895eb0 0x55faa6896320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa679b060 con 0x55faa68a85d0 2023-10-10T17:42:59.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa67ed8a0 con 0x55faa6895eb0 2023-10-10T17:42:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.371+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa67e6d30 con 0x55faa68997c0 2023-10-10T17:42:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68aa9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44802/0 (socket says 172.21.15.78:44802) 2023-10-10T17:42:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4659c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa6895eb0 0x55faa6896320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 -- 172.21.15.78:0/3687583676 learned_addr learned my addr 172.21.15.78:0/3687583676 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa6895970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 -- 172.21.15.78:0/3687583676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 msgr2=0x55faa6895970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa6895970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 -- 172.21.15.78:0/3687583676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa6895eb0 msgr2=0x55faa6896320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa6895eb0 0x55faa6896320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 -- 172.21.15.78:0/3687583676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faa6668630 con 0x55faa68a85d0 2023-10-10T17:42:59.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f4759e700 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68aa9b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7f40095b80 tx=0x7f7f40095a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f7ee51e6a14a5c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f2d7fa700 1 -- 172.21.15.78:0/3687583676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f40098fe0 con 0x55faa68a85d0 2023-10-10T17:42:59.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f2d7fa700 1 -- 172.21.15.78:0/3687583676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f40004ad0 con 0x55faa68a85d0 2023-10-10T17:42:59.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.372+0000 7f7f2d7fa700 1 -- 172.21.15.78:0/3687583676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f400ca230 con 0x55faa68a85d0 2023-10-10T17:42:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 -- 172.21.15.78:0/3687583676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 msgr2=0x55faa68aa9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68aa9b0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7f40095b80 tx=0x7f7f40095a40 comp rx=0 tx=0).stop 2023-10-10T17:42:59.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 -- 172.21.15.78:0/3687583676 shutdown_connections 2023-10-10T17:42:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa6895970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68aa9b0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/3687583676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa6895eb0 0x55faa6896320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 -- 172.21.15.78:0/3687583676 >> 172.21.15.78:0/3687583676 conn(0x55faa688b360 msgr2=0x55faa6890b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.373+0000 7f7f5937cb80 1 -- 172.21.15.78:0/3687583676 shutdown_connections 2023-10-10T17:42:59.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 -- 172.21.15.78:0/3687583676 wait complete. 2023-10-10T17:42:59.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 -- start start 2023-10-10T17:42:59.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 Processor -- start 2023-10-10T17:42:59.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa6895eb0 0x55faa68a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa68a2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa68a7850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa67888e0 con 0x55faa6895eb0 2023-10-10T17:42:59.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa6788580 con 0x55faa68a85d0 2023-10-10T17:42:59.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa666a920 con 0x55faa68997c0 2023-10-10T17:42:59.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.374+0000 7f7f5937cb80 10 client.0 _ll_register_callbacks cb 0x55faa67a6780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:59.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa6895eb0 0x55faa68a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa6895eb0 0x55faa68a1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44804/0 (socket says 172.21.15.78:44804) 2023-10-10T17:42:59.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 -- 172.21.15.78:0/2916946484 learned_addr learned my addr 172.21.15.78:0/2916946484 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f4759e700 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa68a7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 -- 172.21.15.78:0/2916946484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 msgr2=0x55faa68a2070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:42:59.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa68a2070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 -- 172.21.15.78:0/2916946484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 msgr2=0x55faa68a7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa68a7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 -- 172.21.15.78:0/2916946484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f481101e0 con 0x55faa6895eb0 2023-10-10T17:42:59.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa6895eb0 0x55faa68a1b30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7f38003930 tx=0x7f7f380047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e9f01a0b7ce49ca server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f2aff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:59.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f2aff5700 1 -- 172.21.15.78:0/2916946484 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f38012650 con 0x55faa6895eb0 2023-10-10T17:42:59.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f2aff5700 1 -- 172.21.15.78:0/2916946484 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f7f38018070 con 0x55faa6895eb0 2023-10-10T17:42:59.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55faa6888b20 con 0x55faa6895eb0 2023-10-10T17:42:59.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.375+0000 7f7f2aff5700 1 -- 172.21.15.78:0/2916946484 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f38004570 con 0x55faa6895eb0 2023-10-10T17:42:59.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 10 client.15726 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:59.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55faa67ad190 con 0x55faa6895eb0 2023-10-10T17:42:59.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f2aff5700 1 -- 172.21.15.78:0/2916946484 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55faa6888b20 con 0x55faa6895eb0 2023-10-10T17:42:59.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f2aff5700 1 -- 172.21.15.78:0/2916946484 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7f380217c0 con 0x55faa6895eb0 2023-10-10T17:42:59.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 10 client.15726 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:59.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 10 client.15726 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:42:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 20 client.15726 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa68c0930 0x55faa68c2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 4 client.15726 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55faa67a90a0 con 0x55faa68c0930 2023-10-10T17:42:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.376+0000 7f7f4659c700 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa68c0930 0x55faa68c2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.377+0000 7f7f4659c700 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa68c0930 0x55faa68c2d40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4bd5c26dce499089 server_cookie=2db31fdf5f174748 in_seq=0 out_seq=0 2023-10-10T17:42:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.377+0000 7f7f2aff5700 10 client.15726 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.377+0000 7f7f2aff5700 1 -- 172.21.15.78:0/2916946484 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55faa67a90a0 con 0x55faa68c0930 2023-10-10T17:42:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.377+0000 7f7f2aff5700 10 client.15726 handle_command_reply: tid=0 2023-10-10T17:42:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.377+0000 7f7f5937cb80 1 client.15726 shutdown 2023-10-10T17:42:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.377+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa6895eb0 msgr2=0x55faa68a1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.377+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa6895eb0 0x55faa68a1b30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7f38003930 tx=0x7f7f380047c0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 shutdown_connections 2023-10-10T17:42:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa68a7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa6895eb0 0x55faa68a1b30 secure :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f7f38003930 tx=0x7f7f380047c0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa68c0930 0x55faa68c2d40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/2916946484 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa68a2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f2aff5700 0 client.15726 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 >> 172.21.15.78:0/2916946484 conn(0x55faa688b360 msgr2=0x55faa688fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 shutdown_connections 2023-10-10T17:42:59.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 1 -- 172.21.15.78:0/2916946484 wait complete. 2023-10-10T17:42:59.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.378+0000 7f7f5937cb80 20 client.15726 trim_cache size 0 max 16384 2023-10-10T17:42:59.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 Processor -- start 2023-10-10T17:42:59.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 -- start start 2023-10-10T17:42:59.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68997c0 0x55faa68a77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68955b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa689cf80 0x55faa6890560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa68967a0 con 0x55faa68a85d0 2023-10-10T17:42:59.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa6894a90 con 0x55faa68997c0 2023-10-10T17:42:59.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.402+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa6888980 con 0x55faa689cf80 2023-10-10T17:42:59.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68955b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68955b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44812/0 (socket says 172.21.15.78:44812) 2023-10-10T17:42:59.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 -- 172.21.15.78:0/1195785018 learned_addr learned my addr 172.21.15.78:0/1195785018 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68997c0 0x55faa68a77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4759e700 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa689cf80 0x55faa6890560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 -- 172.21.15.78:0/1195785018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa689cf80 msgr2=0x55faa6890560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa689cf80 0x55faa6890560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 -- 172.21.15.78:0/1195785018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68997c0 msgr2=0x55faa68a77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68997c0 0x55faa68a77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 -- 172.21.15.78:0/1195785018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faa688cb00 con 0x55faa68a85d0 2023-10-10T17:42:59.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68997c0 0x55faa68a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:59.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68955b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c006010 tx=0x7f7f3c00bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57caac4596bebd3d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.403+0000 7f7f2dffb700 1 -- 172.21.15.78:0/1195785018 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c027cc0 con 0x55faa68a85d0 2023-10-10T17:42:59.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f2dffb700 1 -- 172.21.15.78:0/1195785018 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f3c01e040 con 0x55faa68a85d0 2023-10-10T17:42:59.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f2dffb700 1 -- 172.21.15.78:0/1195785018 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c003cb0 con 0x55faa68a85d0 2023-10-10T17:42:59.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1195785018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 msgr2=0x55faa68955b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68955b0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c006010 tx=0x7f7f3c00bdb0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1195785018 shutdown_connections 2023-10-10T17:42:59.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa689cf80 0x55faa6890560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68997c0 0x55faa68a77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1195785018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa68a85d0 0x55faa68955b0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1195785018 >> 172.21.15.78:0/1195785018 conn(0x55faa688b360 msgr2=0x55faa6894ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1195785018 shutdown_connections 2023-10-10T17:42:59.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.404+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1195785018 wait complete. 2023-10-10T17:42:59.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 -- start start 2023-10-10T17:42:59.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 Processor -- start 2023-10-10T17:42:59.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa68a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa689cf80 0x55faa68a5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa6913810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa68a5fc0 con 0x55faa689cf80 2023-10-10T17:42:59.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa6913d80 con 0x55faa68a85d0 2023-10-10T17:42:59.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa6913f40 con 0x55faa68997c0 2023-10-10T17:42:59.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f5937cb80 10 client.0 _ll_register_callbacks cb 0x7f7f380043d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:59.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f4659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa689cf80 0x55faa68a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f4659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa689cf80 0x55faa68a5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44820/0 (socket says 172.21.15.78:44820) 2023-10-10T17:42:59.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f4659c700 1 -- 172.21.15.78:0/1102059774 learned_addr learned my addr 172.21.15.78:0/1102059774 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.405+0000 7f7f4759e700 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa6913810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa68a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f4659c700 1 -- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 msgr2=0x55faa68a52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa68a52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f4659c700 1 -- 172.21.15.78:0/1102059774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 msgr2=0x55faa6913810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa6913810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f4659c700 1 -- 172.21.15.78:0/1102059774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f480692c0 con 0x55faa689cf80 2023-10-10T17:42:59.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f4659c700 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa689cf80 0x55faa68a5830 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0036c0 tx=0x7f7f3c00b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf184152a699a38 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f2aff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:59.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f2aff5700 1 -- 172.21.15.78:0/1102059774 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c003ab0 con 0x55faa689cf80 2023-10-10T17:42:59.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f2aff5700 1 -- 172.21.15.78:0/1102059774 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f7f3c0270f0 con 0x55faa689cf80 2023-10-10T17:42:59.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7f40004ad0 con 0x55faa689cf80 2023-10-10T17:42:59.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f2aff5700 1 -- 172.21.15.78:0/1102059774 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7f3c0272b0 con 0x55faa689cf80 2023-10-10T17:42:59.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f5937cb80 10 client.15732 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:59.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.406+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55faa6614de0 con 0x55faa689cf80 2023-10-10T17:42:59.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f2aff5700 1 -- 172.21.15.78:0/1102059774 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7f40004ad0 con 0x55faa689cf80 2023-10-10T17:42:59.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f2aff5700 1 -- 172.21.15.78:0/1102059774 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7f3c00f5f0 con 0x55faa689cf80 2023-10-10T17:42:59.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f5937cb80 10 client.15732 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:59.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f5937cb80 10 client.15732 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:42:59.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f5937cb80 20 client.15732 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:59.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa69152d0 0x55faa69176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f5937cb80 4 client.15732 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:42:59.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55faa67b5660 con 0x55faa69152d0 2023-10-10T17:42:59.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa69152d0 0x55faa69176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f2aff5700 10 client.15732 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:59.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.407+0000 7f7f46d9d700 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa69152d0 0x55faa69176b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7e35a46f6ea8606f server_cookie=724446ff781301cf in_seq=0 out_seq=0 2023-10-10T17:42:59.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f2aff5700 1 -- 172.21.15.78:0/1102059774 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55faa67b5660 con 0x55faa69152d0 2023-10-10T17:42:59.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f2aff5700 10 client.15732 handle_command_reply: tid=0 2023-10-10T17:42:59.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 client.15732 shutdown 2023-10-10T17:42:59.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa689cf80 msgr2=0x55faa68a5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa689cf80 0x55faa68a5830 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c0036c0 tx=0x7f7f3c00b6b0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 shutdown_connections 2023-10-10T17:42:59.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55faa69152d0 0x55faa69176b0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55faa68997c0 0x55faa68a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55faa68a85d0 0x55faa6913810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 --2- 172.21.15.78:0/1102059774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55faa689cf80 0x55faa68a5830 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 >> 172.21.15.78:0/1102059774 conn(0x55faa688b360 msgr2=0x55faa6895730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f2aff5700 0 client.15732 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:42:59.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 shutdown_connections 2023-10-10T17:42:59.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 1 -- 172.21.15.78:0/1102059774 wait complete. 2023-10-10T17:42:59.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.408+0000 7f7f5937cb80 20 client.15732 trim_cache size 0 max 16384 2023-10-10T17:42:59.424 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:42:59.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7f300a1350 msgr2=0x7f7f300a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7f300a1350 0x7f7f300a3800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7f38009cb0 tx=0x7f7f38009260 comp rx=0 tx=0).stop 2023-10-10T17:42:59.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48104490 msgr2=0x7f7f4819ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48104490 0x7f7f4819ad60 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f7f4000ef30 tx=0x7f7f4000c3a0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 shutdown_connections 2023-10-10T17:42:59.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7f480ff450 0x7f7f481963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7f300a1350 0x7f7f300a3800 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7f48104490 0x7f7f4819ad60 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 --2- 172.21.15.78:0/1305866685 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7f480ffdf0 0x7f7f48196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 >> 172.21.15.78:0/1305866685 conn(0x7f7f48077080 msgr2=0x7f7f480fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 shutdown_connections 2023-10-10T17:42:59.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.409+0000 7f7f4d1f0700 1 -- 172.21.15.78:0/1305866685 wait complete. 2023-10-10T17:42:59.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:42:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 -- 172.21.15.78:0/2446775695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe540ff450 msgr2=0x7fbe540ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2446775695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe540ff450 0x7fbe540ff820 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fbe440099c0 tx=0x7fbe44009cd0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 -- 172.21.15.78:0/2446775695 shutdown_connections 2023-10-10T17:42:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2446775695 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe54104490 0x7fbe5410c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2446775695 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe540ff450 0x7fbe540ff820 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2446775695 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe540ffdf0 0x7fbe54103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 -- 172.21.15.78:0/2446775695 >> 172.21.15.78:0/2446775695 conn(0x7fbe540770b0 msgr2=0x7fbe540774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 -- 172.21.15.78:0/2446775695 shutdown_connections 2023-10-10T17:42:59.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.554+0000 7fbe5997b700 1 -- 172.21.15.78:0/2446775695 wait complete. 2023-10-10T17:42:59.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 Processor -- start 2023-10-10T17:42:59.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 -- start start 2023-10-10T17:42:59.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe540ff450 0x7fbe54195310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe540ffdf0 0x7fbe54195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54104490 0x7fbe54199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5419a2f0 con 0x7fbe54104490 2023-10-10T17:42:59.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5419a460 con 0x7fbe540ff450 2023-10-10T17:42:59.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe5997b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5419a620 con 0x7fbe540ffdf0 2023-10-10T17:42:59.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54104490 0x7fbe54199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54104490 0x7fbe54199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44834/0 (socket says 172.21.15.78:44834) 2023-10-10T17:42:59.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe537fe700 1 -- 172.21.15.78:0/2250117919 learned_addr learned my addr 172.21.15.78:0/2250117919 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe527fc700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe540ffdf0 0x7fbe54195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.555+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe540ff450 0x7fbe54195310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe537fe700 1 -- 172.21.15.78:0/2250117919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe540ffdf0 msgr2=0x7fbe54195850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe537fe700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe540ffdf0 0x7fbe54195850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe537fe700 1 -- 172.21.15.78:0/2250117919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe540ff450 msgr2=0x7fbe54195310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe537fe700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe540ff450 0x7fbe54195310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe537fe700 1 -- 172.21.15.78:0/2250117919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe440095f0 con 0x7fbe54104490 2023-10-10T17:42:59.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe537fe700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54104490 0x7fbe54199cb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c007ee0 tx=0x7fbe4c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe58979700 1 -- 172.21.15.78:0/2250117919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c017070 con 0x7fbe54104490 2023-10-10T17:42:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5410f2a0 con 0x7fbe54104490 2023-10-10T17:42:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe58979700 1 -- 172.21.15.78:0/2250117919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe4c00f040 con 0x7fbe54104490 2023-10-10T17:42:59.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe58979700 1 -- 172.21.15.78:0/2250117919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c005580 con 0x7fbe54104490 2023-10-10T17:42:59.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.556+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5410f110 con 0x7fbe54104490 2023-10-10T17:42:59.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.559+0000 7fbe58979700 1 -- 172.21.15.78:0/2250117919 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7fbe4c00db20 con 0x7fbe54104490 2023-10-10T17:42:59.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.560+0000 7fbe58979700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fbe3c0a1360 0x7fbe3c0a3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.560+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fbe3c0a1360 0x7fbe3c0a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.560+0000 7fbe58979700 1 -- 172.21.15.78:0/2250117919 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fbe4c0cc8a0 con 0x7fbe54104490 2023-10-10T17:42:59.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.560+0000 7fbe65b07b80 1 Processor -- start 2023-10-10T17:42:59.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe65b07b80 1 -- start start 2023-10-10T17:42:59.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fbe3c0a1360 0x7fbe3c0a3810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009990 tx=0x7fbe44005fb0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df8c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df90b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df91510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262de89060 con 0x56262df910c0 2023-10-10T17:42:59.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262dedb8a0 con 0x56262df8cc80 2023-10-10T17:42:59.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262ded4d30 con 0x56262df8c370 2023-10-10T17:42:59.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df91510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe52ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df8c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe52ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df8c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:36368/0 (socket says 172.21.15.78:36368) 2023-10-10T17:42:59.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe52ffd700 1 -- 172.21.15.78:0/2742921551 learned_addr learned my addr 172.21.15.78:0/2742921551 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df91510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44842/0 (socket says 172.21.15.78:44842) 2023-10-10T17:42:59.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.561+0000 7fbe527fc700 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df90b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe537fe700 1 -- 172.21.15.78:0/2742921551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 msgr2=0x56262df8c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe537fe700 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df8c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe537fe700 1 -- 172.21.15.78:0/2742921551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 msgr2=0x56262df90b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe537fe700 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df90b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe537fe700 1 -- 172.21.15.78:0/2742921551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56262dd56630 con 0x56262df910c0 2023-10-10T17:42:59.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df8c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:59.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe527fc700 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df90b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:59.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe537fe700 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df91510 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0c9c40 tx=0x7fbe4c004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec2c3db2e886ec1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe39ffb700 1 -- 172.21.15.78:0/2742921551 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c09d540 con 0x56262df910c0 2023-10-10T17:42:59.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe39ffb700 1 -- 172.21.15.78:0/2742921551 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe4c0a1960 con 0x56262df910c0 2023-10-10T17:42:59.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.562+0000 7fbe39ffb700 1 -- 172.21.15.78:0/2742921551 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c09d370 con 0x56262df910c0 2023-10-10T17:42:59.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2742921551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 msgr2=0x56262df91510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df91510 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0c9c40 tx=0x7fbe4c004b50 comp rx=0 tx=0).stop 2023-10-10T17:42:59.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2742921551 shutdown_connections 2023-10-10T17:42:59.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df8c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df91510 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2742921551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df90b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2742921551 >> 172.21.15.78:0/2742921551 conn(0x56262df79360 msgr2=0x56262df7eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2742921551 shutdown_connections 2023-10-10T17:42:59.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2742921551 wait complete. 2023-10-10T17:42:59.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.563+0000 7fbe65b07b80 1 -- start start 2023-10-10T17:42:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 1 Processor -- start 2023-10-10T17:42:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df95bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df96130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df96670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262de768e0 con 0x56262df910c0 2023-10-10T17:42:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262de76580 con 0x56262df8cc80 2023-10-10T17:42:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262dd58920 con 0x56262df8c370 2023-10-10T17:42:59.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe65b07b80 10 client.0 _ll_register_callbacks cb 0x56262de94780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df96670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df96670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44846/0 (socket says 172.21.15.78:44846) 2023-10-10T17:42:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 -- 172.21.15.78:0/1797229184 learned_addr learned my addr 172.21.15.78:0/1797229184 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 -- 172.21.15.78:0/1797229184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 msgr2=0x56262df95bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df95bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 -- 172.21.15.78:0/1797229184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 msgr2=0x56262df96130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:42:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df96130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.564+0000 7fbe537fe700 1 -- 172.21.15.78:0/1797229184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5410f110 con 0x56262df910c0 2023-10-10T17:42:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe537fe700 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df96670 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0ca300 tx=0x7fbe4c098f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4472e2e49351d05c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe32ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe32ffd700 1 -- 172.21.15.78:0/1797229184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c09d370 con 0x56262df910c0 2023-10-10T17:42:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe32ffd700 1 -- 172.21.15.78:0/1797229184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fbe4c097970 con 0x56262df910c0 2023-10-10T17:42:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56262df76b20 con 0x56262df910c0 2023-10-10T17:42:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe32ffd700 1 -- 172.21.15.78:0/1797229184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe4c0c9ef0 con 0x56262df910c0 2023-10-10T17:42:59.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 10 client.15744 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:59.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56262de9b190 con 0x56262df910c0 2023-10-10T17:42:59.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe32ffd700 1 -- 172.21.15.78:0/1797229184 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56262df76b20 con 0x56262df910c0 2023-10-10T17:42:59.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe32ffd700 1 -- 172.21.15.78:0/1797229184 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbe4c098a00 con 0x56262df910c0 2023-10-10T17:42:59.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 10 client.15744 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:59.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 10 client.15744 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:42:59.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 20 client.15744 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:59.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262dfae640 0x56262dfb0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 4 client.15744 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:59.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.565+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56262de970a0 con 0x56262dfae640 2023-10-10T17:42:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.566+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262dfae640 0x56262dfb0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.566+0000 7fbe32ffd700 10 client.15744 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.566+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262dfae640 0x56262dfb0a20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=96f2a6cacd807e0f server_cookie=12bad8b186e5ccac in_seq=0 out_seq=0 2023-10-10T17:42:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe32ffd700 1 -- 172.21.15.78:0/1797229184 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56262de970a0 con 0x56262dfae640 2023-10-10T17:42:59.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe32ffd700 10 client.15744 handle_command_reply: tid=0 2023-10-10T17:42:59.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 client.15744 shutdown 2023-10-10T17:42:59.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 msgr2=0x56262df96670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df96670 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c0ca300 tx=0x7fbe4c098f70 comp rx=0 tx=0).stop 2023-10-10T17:42:59.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 shutdown_connections 2023-10-10T17:42:59.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262dfae640 0x56262dfb0a20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df95bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df96670 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/1797229184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df96130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 >> 172.21.15.78:0/1797229184 conn(0x56262df79360 msgr2=0x56262df92570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe32ffd700 0 client.15744 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:59.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 shutdown_connections 2023-10-10T17:42:59.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 1 -- 172.21.15.78:0/1797229184 wait complete. 2023-10-10T17:42:59.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.567+0000 7fbe65b07b80 20 client.15744 trim_cache size 0 max 16384 2023-10-10T17:42:59.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 Processor -- start 2023-10-10T17:42:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 -- start start 2023-10-10T17:42:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df87d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df831e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df8f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262df8fd10 con 0x56262df910c0 2023-10-10T17:42:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262df8d110 con 0x56262df8cc80 2023-10-10T17:42:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.592+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262df88240 con 0x56262df8c370 2023-10-10T17:42:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df87d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df8f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df87d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:36380/0 (socket says 172.21.15.78:36380) 2023-10-10T17:42:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df8f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44854/0 (socket says 172.21.15.78:44854) 2023-10-10T17:42:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 -- 172.21.15.78:0/2841073863 learned_addr learned my addr 172.21.15.78:0/2841073863 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe527fc700 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df831e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 -- 172.21.15.78:0/2841073863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 msgr2=0x56262df831e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df831e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 -- 172.21.15.78:0/2841073863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 msgr2=0x56262df8f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df8f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 -- 172.21.15.78:0/2841073863 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56262dfb55a0 con 0x56262df8c370 2023-10-10T17:42:59.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df87d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400b580 tx=0x7fbe4401f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48d0655b301c2e97 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe3a7fc700 1 -- 172.21.15.78:0/2841073863 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4402c040 con 0x56262df8c370 2023-10-10T17:42:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe3a7fc700 1 -- 172.21.15.78:0/2841073863 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe4401e040 con 0x56262df8c370 2023-10-10T17:42:59.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.593+0000 7fbe3a7fc700 1 -- 172.21.15.78:0/2841073863 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4400f770 con 0x56262df8c370 2023-10-10T17:42:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2841073863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 msgr2=0x56262df87d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df87d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbe4400b580 tx=0x7fbe4401f000 comp rx=0 tx=0).stop 2023-10-10T17:42:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2841073863 shutdown_connections 2023-10-10T17:42:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262df87d00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df8f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/2841073863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262df831e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2841073863 >> 172.21.15.78:0/2841073863 conn(0x56262df79360 msgr2=0x56262df963b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2841073863 shutdown_connections 2023-10-10T17:42:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.594+0000 7fbe65b07b80 1 -- 172.21.15.78:0/2841073863 wait complete. 2023-10-10T17:42:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 -- start start 2023-10-10T17:42:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 Processor -- start 2023-10-10T17:42:59.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262dffa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262dffaea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df86c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262df87200 con 0x56262df910c0 2023-10-10T17:42:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262df87370 con 0x56262df8cc80 2023-10-10T17:42:59.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.595+0000 7fbe65b07b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262df87530 con 0x56262df8c370 2023-10-10T17:42:59.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe65b07b80 10 client.0 _ll_register_callbacks cb 0x7fbe48009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:59.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df86c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe527fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262dffaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe527fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262dffaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:46580/0 (socket says 172.21.15.78:46580) 2023-10-10T17:42:59.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe527fc700 1 -- 172.21.15.78:0/784232531 learned_addr learned my addr 172.21.15.78:0/784232531 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262dffa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe537fe700 1 -- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 msgr2=0x56262dffa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe537fe700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262dffa960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe537fe700 1 -- 172.21.15.78:0/784232531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 msgr2=0x56262dffaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe537fe700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262dffaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe537fe700 1 -- 172.21.15.78:0/784232531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe44003b10 con 0x56262df910c0 2023-10-10T17:42:59.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262dffa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:42:59.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe527fc700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262dffaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:59.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.596+0000 7fbe537fe700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df86c00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c095f80 tx=0x7fbe4c099a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fa7466afcf1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe32ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:42:59.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe32ffd700 1 -- 172.21.15.78:0/784232531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4c09d370 con 0x56262df910c0 2023-10-10T17:42:59.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe32ffd700 1 -- 172.21.15.78:0/784232531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fbe4c097970 con 0x56262df910c0 2023-10-10T17:42:59.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe4c0a1960 con 0x56262df910c0 2023-10-10T17:42:59.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe32ffd700 1 -- 172.21.15.78:0/784232531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe4c09d540 con 0x56262df910c0 2023-10-10T17:42:59.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 10 client.15750 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:59.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56262dd02de0 con 0x56262df910c0 2023-10-10T17:42:59.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe32ffd700 1 -- 172.21.15.78:0/784232531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbe4c0a1960 con 0x56262df910c0 2023-10-10T17:42:59.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe32ffd700 1 -- 172.21.15.78:0/784232531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fbe4c095300 con 0x56262df910c0 2023-10-10T17:42:59.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 10 client.15750 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:59.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 10 client.15750 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:42:59.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 20 client.15750 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:59.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262e002370 0x56262e004750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 4 client.15750 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:42:59.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.597+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56262dea3660 con 0x56262e002370 2023-10-10T17:42:59.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.598+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262e002370 0x56262e004750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.598+0000 7fbe32ffd700 10 client.15750 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:59.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.598+0000 7fbe52ffd700 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262e002370 0x56262e004750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4940301d052e111e server_cookie=dfd5d1dc9e3636f9 in_seq=0 out_seq=0 2023-10-10T17:42:59.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.598+0000 7fbe32ffd700 1 -- 172.21.15.78:0/784232531 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56262dea3660 con 0x56262e002370 2023-10-10T17:42:59.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.598+0000 7fbe32ffd700 10 client.15750 handle_command_reply: tid=0 2023-10-10T17:42:59.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.598+0000 7fbe65b07b80 1 client.15750 shutdown 2023-10-10T17:42:59.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 msgr2=0x56262df86c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df86c00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c095f80 tx=0x7fbe4c099a80 comp rx=0 tx=0).stop 2023-10-10T17:42:59.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 shutdown_connections 2023-10-10T17:42:59.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56262e002370 0x56262e004750 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56262df8c370 0x56262dffa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56262df910c0 0x56262df86c00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 --2- 172.21.15.78:0/784232531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56262df8cc80 0x56262dffaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 >> 172.21.15.78:0/784232531 conn(0x56262df79360 msgr2=0x56262df831e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe32ffd700 0 client.15750 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:42:59.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 shutdown_connections 2023-10-10T17:42:59.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 1 -- 172.21.15.78:0/784232531 wait complete. 2023-10-10T17:42:59.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe65b07b80 20 client.15750 trim_cache size 0 max 16384 2023-10-10T17:42:59.615 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:42:59.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fbe3c0a1360 msgr2=0x7fbe3c0a3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fbe3c0a1360 0x7fbe3c0a3810 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbe44009990 tx=0x7fbe44005fb0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54104490 msgr2=0x7fbe54199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.599+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54104490 0x7fbe54199cb0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbe4c007ee0 tx=0x7fbe4c00d3b0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 shutdown_connections 2023-10-10T17:42:59.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fbe3c0a1360 0x7fbe3c0a3810 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe54104490 0x7fbe54199cb0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fbe540ff450 0x7fbe54195310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 --2- 172.21.15.78:0/2250117919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fbe540ffdf0 0x7fbe54195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 >> 172.21.15.78:0/2250117919 conn(0x7fbe540770b0 msgr2=0x7fbe540fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 shutdown_connections 2023-10-10T17:42:59.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.600+0000 7fbe5997b700 1 -- 172.21.15.78:0/2250117919 wait complete. 2023-10-10T17:42:59.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:42:59.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 -- 172.21.15.78:0/1880386531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80ffec0 msgr2=0x7f8ed8103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/1880386531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80ffec0 0x7f8ed8103e80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40099c0 tx=0x7f8ec4009cd0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 -- 172.21.15.78:0/1880386531 shutdown_connections 2023-10-10T17:42:59.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/1880386531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed80ff520 0x7f8ed80ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/1880386531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ed8104560 0x7f8ed810c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/1880386531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed80ffec0 0x7f8ed8103e80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 -- 172.21.15.78:0/1880386531 >> 172.21.15.78:0/1880386531 conn(0x7f8ed80760d0 msgr2=0x7f8ed80764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 -- 172.21.15.78:0/1880386531 shutdown_connections 2023-10-10T17:42:59.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.753+0000 7f8edf1d7700 1 -- 172.21.15.78:0/1880386531 wait complete. 2023-10-10T17:42:59.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 Processor -- start 2023-10-10T17:42:59.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 -- start start 2023-10-10T17:42:59.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed80ff520 0x7f8ed81963f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ed80ffec0 0x7f8ed8196930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed8104560 0x7f8ed819ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed819b3d0 con 0x7f8ed8104560 2023-10-10T17:42:59.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed819b540 con 0x7f8ed80ffec0 2023-10-10T17:42:59.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edf1d7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed819b700 con 0x7f8ed80ff520 2023-10-10T17:42:59.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.754+0000 7f8edd774700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed8104560 0x7f8ed819ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed8104560 0x7f8ed819ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44874/0 (socket says 172.21.15.78:44874) 2023-10-10T17:42:59.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 -- 172.21.15.78:0/2558337801 learned_addr learned my addr 172.21.15.78:0/2558337801 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ed80ffec0 0x7f8ed8196930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edcf73700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed80ff520 0x7f8ed81963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 -- 172.21.15.78:0/2558337801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed80ff520 msgr2=0x7f8ed81963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed80ff520 0x7f8ed81963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 -- 172.21.15.78:0/2558337801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ed80ffec0 msgr2=0x7f8ed8196930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ed80ffec0 0x7f8ed8196930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 -- 172.21.15.78:0/2558337801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec40095f0 con 0x7f8ed8104560 2023-10-10T17:42:59.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edd774700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed8104560 0x7f8ed819ad90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000ef30 tx=0x7f8ed000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8ed5ffb700 1 -- 172.21.15.78:0/2558337801 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0018070 con 0x7f8ed8104560 2023-10-10T17:42:59.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed81104a0 con 0x7f8ed8104560 2023-10-10T17:42:59.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8ed5ffb700 1 -- 172.21.15.78:0/2558337801 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ed000f040 con 0x7f8ed8104560 2023-10-10T17:42:59.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8ed5ffb700 1 -- 172.21.15.78:0/2558337801 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed000cd00 con 0x7f8ed8104560 2023-10-10T17:42:59.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.755+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8110310 con 0x7f8ed8104560 2023-10-10T17:42:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.759+0000 7f8ed5ffb700 1 -- 172.21.15.78:0/2558337801 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7f8ed0003a00 con 0x7f8ed8104560 2023-10-10T17:42:59.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.759+0000 7f8eeb363b80 1 Processor -- start 2023-10-10T17:42:59.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.759+0000 7f8ed5ffb700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ec80a1350 0x7f8ec80a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ec80a1350 0x7f8ec80a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8ed5ffb700 1 -- 172.21.15.78:0/2558337801 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8ed0004ad0 con 0x7f8ed8104560 2023-10-10T17:42:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8eeb363b80 1 -- start start 2023-10-10T17:42:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2e370 0x55b24cc2e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc32b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc33510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cb2b060 con 0x55b24cc330c0 2023-10-10T17:42:59.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cb7d8a0 con 0x55b24cc2ec80 2023-10-10T17:42:59.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cb76d30 con 0x55b24cc2e370 2023-10-10T17:42:59.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ec80a1350 0x7f8ec80a3800 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40098e0 tx=0x7f8ec4000bc0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2e370 0x55b24cc2e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2e370 0x55b24cc2e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:36408/0 (socket says 172.21.15.78:36408) 2023-10-10T17:42:59.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8edcf73700 1 -- 172.21.15.78:0/1581008506 learned_addr learned my addr 172.21.15.78:0/1581008506 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8edd774700 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc33510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.760+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc32b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8edd774700 1 -- 172.21.15.78:0/1581008506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2e370 msgr2=0x55b24cc2e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8edd774700 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2e370 0x55b24cc2e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8edd774700 1 -- 172.21.15.78:0/1581008506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 msgr2=0x55b24cc32b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8edd774700 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc32b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8edd774700 1 -- 172.21.15.78:0/1581008506 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b24c9f8630 con 0x55b24cc330c0 2023-10-10T17:42:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc32b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:59.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8edd774700 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc33510 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00c9a00 tx=0x7f8ed00092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728fce6cb434ee9e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/1581008506 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0098c30 con 0x55b24cc330c0 2023-10-10T17:42:59.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/1581008506 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ed000ce60 con 0x55b24cc330c0 2023-10-10T17:42:59.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.761+0000 7f8ec2ffd700 1 -- 172.21.15.78:0/1581008506 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0099790 con 0x55b24cc330c0 2023-10-10T17:42:59.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1581008506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 msgr2=0x55b24cc33510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc33510 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f8ed00c9a00 tx=0x7f8ed00092e0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1581008506 shutdown_connections 2023-10-10T17:42:59.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2e370 0x55b24cc2e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc33510 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1581008506 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc32b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1581008506 >> 172.21.15.78:0/1581008506 conn(0x55b24cc1b360 msgr2=0x55b24cc20b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1581008506 shutdown_connections 2023-10-10T17:42:59.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.762+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1581008506 wait complete. 2023-10-10T17:42:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 -- start start 2023-10-10T17:42:59.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 Processor -- start 2023-10-10T17:42:59.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc37b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc38190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cb188e0 con 0x55b24cc330c0 2023-10-10T17:42:59.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cb18580 con 0x55b24cc2e370 2023-10-10T17:42:59.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24c9fa920 con 0x55b24cc2ec80 2023-10-10T17:42:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.763+0000 7f8eeb363b80 10 client.0 _ll_register_callbacks cb 0x55b24cb36780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:46606/0 (socket says 172.21.15.78:46606) 2023-10-10T17:42:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 -- 172.21.15.78:0/2223904415 learned_addr learned my addr 172.21.15.78:0/2223904415 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edd774700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc38190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc37b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 -- 172.21.15.78:0/2223904415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 msgr2=0x55b24cc37b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc37b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 -- 172.21.15.78:0/2223904415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 msgr2=0x55b24cc38190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc38190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 -- 172.21.15.78:0/2223904415 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8110310 con 0x55b24cc2e370 2023-10-10T17:42:59.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc37b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:59.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edd774700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc38190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:59.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8edcf73700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3a840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0128d0 tx=0x7f8ecc012c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2c239340f6df02d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8eb7fff700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:42:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2223904415 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc017070 con 0x55b24cc2e370 2023-10-10T17:42:59.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b24cc18b20 con 0x55b24cc2e370 2023-10-10T17:42:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.764+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2223904415 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8ecc004050 con 0x55b24cc2e370 2023-10-10T17:42:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2223904415 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ecc004210 con 0x55b24cc2e370 2023-10-10T17:42:59.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 10 client.25253 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b24cb3d190 con 0x55b24cc2e370 2023-10-10T17:42:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2223904415 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b24cc18b20 con 0x55b24cc2e370 2023-10-10T17:42:59.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2223904415 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8ecc0037a0 con 0x55b24cc2e370 2023-10-10T17:42:59.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 10 client.25253 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:59.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 10 client.25253 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:59.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 20 client.25253 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:59.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc48080 0x55b24cc4a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 4 client.25253 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:42:59.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b24cb390a0 con 0x55b24cc48080 2023-10-10T17:42:59.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.765+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc48080 0x55b24cc4a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.766+0000 7f8eb7fff700 10 client.25253 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:59.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.766+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc48080 0x55b24cc4a490 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98b7ca2831959167 server_cookie=7e703883795d6a73 in_seq=0 out_seq=0 2023-10-10T17:42:59.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.766+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2223904415 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b24cb390a0 con 0x55b24cc48080 2023-10-10T17:42:59.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.766+0000 7f8eb7fff700 10 client.25253 handle_command_reply: tid=0 2023-10-10T17:42:59.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 client.25253 shutdown 2023-10-10T17:42:59.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 msgr2=0x55b24cc3a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3a840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc0128d0 tx=0x7f8ecc012c90 comp rx=0 tx=0).stop 2023-10-10T17:42:59.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 shutdown_connections 2023-10-10T17:42:59.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc48080 0x55b24cc4a490 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc38190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eb7fff700 0 client.25253 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:59.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3a840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2223904415 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc37b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 >> 172.21.15.78:0/2223904415 conn(0x55b24cc1b360 msgr2=0x55b24cc34570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 shutdown_connections 2023-10-10T17:42:59.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2223904415 wait complete. 2023-10-10T17:42:59.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.767+0000 7f8eeb363b80 20 client.25253 trim_cache size 0 max 16384 2023-10-10T17:42:59.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.780+0000 7f8eeb363b80 1 Processor -- start 2023-10-10T17:42:59.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.780+0000 7f8eeb363b80 1 -- start start 2023-10-10T17:42:59.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc2e370 0x55b24cc3f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc3a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc330c0 0x55b24cc20600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cc2f3d0 con 0x55b24cc2e370 2023-10-10T17:42:59.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cc18660 con 0x55b24cc2ec80 2023-10-10T17:42:59.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cc1aee0 con 0x55b24cc330c0 2023-10-10T17:42:59.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc2e370 0x55b24cc3f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc2e370 0x55b24cc3f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44892/0 (socket says 172.21.15.78:44892) 2023-10-10T17:42:59.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 -- 172.21.15.78:0/1543838113 learned_addr learned my addr 172.21.15.78:0/1543838113 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc3a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edd774700 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc330c0 0x55b24cc20600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 -- 172.21.15.78:0/1543838113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc330c0 msgr2=0x55b24cc20600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc330c0 0x55b24cc20600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 -- 172.21.15.78:0/1543838113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 msgr2=0x55b24cc3a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc3a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 -- 172.21.15.78:0/1543838113 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b24cc1cb00 con 0x55b24cc2e370 2023-10-10T17:42:59.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.781+0000 7f8edcf73700 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc2e370 0x55b24cc3f150 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009750 tx=0x7f8ecc003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f8287698720384 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8ec37fe700 1 -- 172.21.15.78:0/1543838113 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc017040 con 0x55b24cc2e370 2023-10-10T17:42:59.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8ec37fe700 1 -- 172.21.15.78:0/1543838113 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ecc0055d0 con 0x55b24cc2e370 2023-10-10T17:42:59.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8ec37fe700 1 -- 172.21.15.78:0/1543838113 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc00fec0 con 0x55b24cc2e370 2023-10-10T17:42:59.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1543838113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc2e370 msgr2=0x55b24cc3f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc2e370 0x55b24cc3f150 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009750 tx=0x7f8ecc003680 comp rx=0 tx=0).stop 2023-10-10T17:42:59.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1543838113 shutdown_connections 2023-10-10T17:42:59.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc330c0 0x55b24cc20600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc2e370 0x55b24cc3f150 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/1543838113 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2ec80 0x55b24cc3a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.782+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1543838113 >> 172.21.15.78:0/1543838113 conn(0x55b24cc1b360 msgr2=0x55b24cc31100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.783+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1543838113 shutdown_connections 2023-10-10T17:42:59.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.783+0000 7f8eeb363b80 1 -- 172.21.15.78:0/1543838113 wait complete. 2023-10-10T17:42:59.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.783+0000 7f8eeb363b80 1 -- start start 2023-10-10T17:42:59.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.783+0000 7f8eeb363b80 1 Processor -- start 2023-10-10T17:42:59.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc3e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eeb363b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc9af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cc3f1e0 con 0x55b24cc330c0 2023-10-10T17:42:59.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cc9b480 con 0x55b24cc2e370 2023-10-10T17:42:59.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eeb363b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24cc9b5c0 con 0x55b24cc2ec80 2023-10-10T17:42:59.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eeb363b80 10 client.0 _ll_register_callbacks cb 0x7f8ecc009ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:42:59.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edd774700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc9af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:46620/0 (socket says 172.21.15.78:46620) 2023-10-10T17:42:59.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edd774700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc9af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44902/0 (socket says 172.21.15.78:44902) 2023-10-10T17:42:59.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 -- 172.21.15.78:0/2746903182 learned_addr learned my addr 172.21.15.78:0/2746903182 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:42:59.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc3e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 -- 172.21.15.78:0/2746903182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 msgr2=0x55b24cc3e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc3e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 -- 172.21.15.78:0/2746903182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 msgr2=0x55b24cc9af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:42:59.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc9af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 -- 172.21.15.78:0/2746903182 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b24cb3d190 con 0x55b24cc2e370 2023-10-10T17:42:59.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edd774700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc9af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:42:59.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc3e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:42:59.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8edcf73700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3e380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc003b00 tx=0x7f8ecc00f720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49113bef70a83347 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:42:59.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.784+0000 7f8eb7fff700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:42:59.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2746903182 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc017040 con 0x55b24cc2e370 2023-10-10T17:42:59.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ed000ce60 con 0x55b24cc2e370 2023-10-10T17:42:59.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2746903182 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8ecc00fb00 con 0x55b24cc2e370 2023-10-10T17:42:59.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2746903182 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ecc00fcc0 con 0x55b24cc2e370 2023-10-10T17:42:59.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 10 client.25259 fetch_fsmap learned FSMap version 24 2023-10-10T17:42:59.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b24c9a4de0 con 0x55b24cc2e370 2023-10-10T17:42:59.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2746903182 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ed000ce60 con 0x55b24cc2e370 2023-10-10T17:42:59.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2746903182 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8ecc003d20 con 0x55b24cc2e370 2023-10-10T17:42:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 10 client.25259 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:42:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 10 client.25259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:42:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 20 client.25259 populate_metadata read hostname 'smithi078' 2023-10-10T17:42:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc9c850 0x55b24cc9ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:42:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 4 client.25259 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:42:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.785+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b24cb45660 con 0x55b24cc9c850 2023-10-10T17:42:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc9c850 0x55b24cc9ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:42:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8eb7fff700 10 client.25259 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8ed7fff700 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc9c850 0x55b24cc9ec60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=457ee4378577ac6d server_cookie=9a6d8b2431e05000 in_seq=0 out_seq=0 2023-10-10T17:42:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8eb7fff700 1 -- 172.21.15.78:0/2746903182 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b24cb45660 con 0x55b24cc9c850 2023-10-10T17:42:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8eb7fff700 10 client.25259 handle_command_reply: tid=0 2023-10-10T17:42:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8eeb363b80 1 client.25259 shutdown 2023-10-10T17:42:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 msgr2=0x55b24cc3e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.786+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3e380 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc003b00 tx=0x7f8ecc00f720 comp rx=0 tx=0).stop 2023-10-10T17:42:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 shutdown_connections 2023-10-10T17:42:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b24cc9c850 0x55b24cc9ec60 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b24cc330c0 0x55b24cc9af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b24cc2e370 0x55b24cc3e380 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 --2- 172.21.15.78:0/2746903182 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b24cc2ec80 0x55b24cc3e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 >> 172.21.15.78:0/2746903182 conn(0x55b24cc1b360 msgr2=0x55b24cc20600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eb7fff700 0 client.25259 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:42:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 shutdown_connections 2023-10-10T17:42:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 1 -- 172.21.15.78:0/2746903182 wait complete. 2023-10-10T17:42:59.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8eeb363b80 20 client.25259 trim_cache size 0 max 16384 2023-10-10T17:42:59.831 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:42:59.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ec80a1350 msgr2=0x7f8ec80a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ec80a1350 0x7f8ec80a3800 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40098e0 tx=0x7f8ec4000bc0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed8104560 msgr2=0x7f8ed819ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:42:59.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.787+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed8104560 0x7f8ed819ad90 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000ef30 tx=0x7f8ed000c3a0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 shutdown_connections 2023-10-10T17:42:59.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8ed80ff520 0x7f8ed81963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ec80a1350 0x7f8ec80a3800 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ed8104560 0x7f8ed819ad90 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 --2- 172.21.15.78:0/2558337801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8ed80ffec0 0x7f8ed8196930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:42:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 >> 172.21.15.78:0/2558337801 conn(0x7f8ed80760d0 msgr2=0x7f8ed80fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:42:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 shutdown_connections 2023-10-10T17:42:59.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:42:59.788+0000 7f8edf1d7700 1 -- 172.21.15.78:0/2558337801 wait complete. 2023-10-10T17:42:59.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:42:59 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1986611159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:59.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:42:59 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1986611159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:42:59.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:42:59 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1986611159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:43:00.837 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:43:00.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:43:00.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:00 smithi190 ceph-mon[101220]: pgmap v15: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:43:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.973+0000 7fcb3266f700 1 -- 172.21.15.78:0/2691462524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c1063f0 msgr2=0x7fcb2c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.973+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2691462524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c1063f0 0x7fcb2c1067c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009990 tx=0x7fcb20009ca0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.973+0000 7fcb3266f700 1 -- 172.21.15.78:0/2691462524 shutdown_connections 2023-10-10T17:43:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.973+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2691462524 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb2c101400 0x7fcb2c10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.973+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2691462524 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c1063f0 0x7fcb2c1067c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.974+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2691462524 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcb2c100a50 0x7fcb2c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.974+0000 7fcb3266f700 1 -- 172.21.15.78:0/2691462524 >> 172.21.15.78:0/2691462524 conn(0x7fcb2c077590 msgr2=0x7fcb2c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:00.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.974+0000 7fcb3266f700 1 -- 172.21.15.78:0/2691462524 shutdown_connections 2023-10-10T17:43:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.974+0000 7fcb3266f700 1 -- 172.21.15.78:0/2691462524 wait complete. 2023-10-10T17:43:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.974+0000 7fcb3266f700 1 Processor -- start 2023-10-10T17:43:00.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.974+0000 7fcb3266f700 1 -- start start 2023-10-10T17:43:00.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:00 smithi157 ceph-mon[100896]: pgmap v15: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:43:00.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb3266f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcb2c100a50 0x7fcb2c196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:00.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb3266f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c101400 0x7fcb2c196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:00.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb3266f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb2c1063f0 0x7fcb2c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:00.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb3266f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c19b3f0 con 0x7fcb2c101400 2023-10-10T17:43:00.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb3266f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c19b560 con 0x7fcb2c100a50 2023-10-10T17:43:00.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb3266f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2c19b720 con 0x7fcb2c1063f0 2023-10-10T17:43:00.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c101400 0x7fcb2c196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:00.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c101400 0x7fcb2c196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44928/0 (socket says 172.21.15.78:44928) 2023-10-10T17:43:00.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/2257269195 learned_addr learned my addr 172.21.15.78:0/2257269195 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:00.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcb2c100a50 0x7fcb2c196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:00.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb2c1063f0 0x7fcb2c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:00.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/2257269195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb2c1063f0 msgr2=0x7fcb2c19adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:00.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb2c1063f0 0x7fcb2c19adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/2257269195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcb2c100a50 msgr2=0x7fcb2c196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:00.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcb2c100a50 0x7fcb2c196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/2257269195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200095f0 con 0x7fcb2c101400 2023-10-10T17:43:00.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.975+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c101400 0x7fcb2c196950 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00c920 tx=0x7fcb1c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:00.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.976+0000 7fcb297fa700 1 -- 172.21.15.78:0/2257269195 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c007b00 con 0x7fcb2c101400 2023-10-10T17:43:00.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.976+0000 7fcb297fa700 1 -- 172.21.15.78:0/2257269195 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb1c00ce70 con 0x7fcb2c101400 2023-10-10T17:43:00.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.976+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2c110380 con 0x7fcb2c101400 2023-10-10T17:43:00.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.976+0000 7fcb297fa700 1 -- 172.21.15.78:0/2257269195 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c0054e0 con 0x7fcb2c101400 2023-10-10T17:43:00.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.976+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2c1101f0 con 0x7fcb2c101400 2023-10-10T17:43:00.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.978+0000 7fcb297fa700 1 -- 172.21.15.78:0/2257269195 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7fcb1c004750 con 0x7fcb2c101400 2023-10-10T17:43:00.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.978+0000 7fcb297fa700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcb140a1350 0x7fcb140a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:00.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.978+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcb140a1350 0x7fcb140a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:00.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.979+0000 7fcb297fa700 1 -- 172.21.15.78:0/2257269195 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fcb1c0ccce0 con 0x7fcb2c101400 2023-10-10T17:43:00.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.979+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcb140a1350 0x7fcb140a3800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcb20003800 tx=0x7fcb20005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:00.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 Processor -- start 2023-10-10T17:43:00.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 -- start start 2023-10-10T17:43:00.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac04fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:00.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac054f0 0x55df3ac093f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:00.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac09d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:00.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ab0a0e0 con 0x55df3ac09930 2023-10-10T17:43:00.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ab5c920 con 0x55df3ac054f0 2023-10-10T17:43:00.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ab55db0 con 0x55df3ac04be0 2023-10-10T17:43:00.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb30c0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac09d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:00.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb30c0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac09d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44936/0 (socket says 172.21.15.78:44936) 2023-10-10T17:43:00.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.980+0000 7fcb30c0c700 1 -- 172.21.15.78:0/1202261802 learned_addr learned my addr 172.21.15.78:0/1202261802 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:00.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac054f0 0x55df3ac093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:00.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac04fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:00.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb30c0c700 1 -- 172.21.15.78:0/1202261802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 msgr2=0x55df3ac04fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:00.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac04fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb30c0c700 1 -- 172.21.15.78:0/1202261802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac054f0 msgr2=0x55df3ac093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:00.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac054f0 0x55df3ac093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb30c0c700 1 -- 172.21.15.78:0/1202261802 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df3a9d76b0 con 0x55df3ac09930 2023-10-10T17:43:00.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac054f0 0x55df3ac093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:00.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac09d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcb240084f0 tx=0x7fcb24008800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36ef4ff85754bc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:00.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb127fc700 1 -- 172.21.15.78:0/1202261802 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24003c00 con 0x55df3ac09930 2023-10-10T17:43:00.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb127fc700 1 -- 172.21.15.78:0/1202261802 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb24010040 con 0x55df3ac09930 2023-10-10T17:43:00.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.981+0000 7fcb127fc700 1 -- 172.21.15.78:0/1202261802 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb24012760 con 0x55df3ac09930 2023-10-10T17:43:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1202261802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 msgr2=0x55df3ac09d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac09d80 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcb240084f0 tx=0x7fcb24008800 comp rx=0 tx=0).stop 2023-10-10T17:43:00.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1202261802 shutdown_connections 2023-10-10T17:43:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac04fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac09d80 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1202261802 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac054f0 0x55df3ac093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:00.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1202261802 >> 172.21.15.78:0/1202261802 conn(0x55df3abfa3e0 msgr2=0x55df3abffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1202261802 shutdown_connections 2023-10-10T17:43:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1202261802 wait complete. 2023-10-10T17:43:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.982+0000 7fcb3e7fbb80 1 -- start start 2023-10-10T17:43:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 1 Processor -- start 2023-10-10T17:43:00.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac11050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac054f0 0x55df3ac0e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac0e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3aaf7960 con 0x55df3ac054f0 2023-10-10T17:43:01.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3aaf7600 con 0x55df3ac09930 2023-10-10T17:43:01.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3a9d99a0 con 0x55df3ac04be0 2023-10-10T17:43:01.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb3e7fbb80 10 client.0 _ll_register_callbacks cb 0x55df3ab15800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:01.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb2b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac054f0 0x55df3ac0e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb2b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac054f0 0x55df3ac0e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44942/0 (socket says 172.21.15.78:44942) 2023-10-10T17:43:01.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/1566287198 learned_addr learned my addr 172.21.15.78:0/1566287198 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac11050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.983+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac0e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/1566287198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 msgr2=0x55df3ac11050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac11050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 msgr2=0x55df3ac0e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac0e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/1566287198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb24007d10 con 0x55df3ac054f0 2023-10-10T17:43:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac054f0 0x55df3ac0e340 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c099650 tx=0x7fcb1c0999d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c336ea0a07a61d9f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/1566287198 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c0a1da0 con 0x55df3ac054f0 2023-10-10T17:43:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/1566287198 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fcb1c099bb0 con 0x55df3ac054f0 2023-10-10T17:43:01.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55df3abf7ba0 con 0x55df3ac054f0 2023-10-10T17:43:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/1566287198 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb1c099d70 con 0x55df3ac054f0 2023-10-10T17:43:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb3e7fbb80 10 client.15786 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55df3ab1c210 con 0x55df3ac054f0 2023-10-10T17:43:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.984+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/1566287198 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55df3abf7ba0 con 0x55df3ac054f0 2023-10-10T17:43:01.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/1566287198 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcb1c0a1a10 con 0x55df3ac054f0 2023-10-10T17:43:01.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb3e7fbb80 10 client.15786 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:01.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb3e7fbb80 10 client.15786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:01.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb3e7fbb80 20 client.15786 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:01.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac1e7a0 0x55df3ac20bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb3e7fbb80 4 client.15786 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:01.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df3ab18120 con 0x55df3ac1e7a0 2023-10-10T17:43:01.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac1e7a0 0x55df3ac20bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac1e7a0 0x55df3ac20bb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3c3f030da8f98d0 server_cookie=217af2d097cb8769 in_seq=0 out_seq=0 2023-10-10T17:43:01.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.985+0000 7fcb0b7fe700 10 client.15786 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/1566287198 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55df3ab18120 con 0x55df3ac1e7a0 2023-10-10T17:43:01.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb0b7fe700 10 client.15786 handle_command_reply: tid=0 2023-10-10T17:43:01.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 client.15786 shutdown 2023-10-10T17:43:01.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac054f0 msgr2=0x55df3ac0e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac054f0 0x55df3ac0e340 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c099650 tx=0x7fcb1c0999d0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 shutdown_connections 2023-10-10T17:43:01.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac11050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac1e7a0 0x55df3ac20bb0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac0e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb0b7fe700 0 client.15786 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/1566287198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac054f0 0x55df3ac0e340 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.986+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 >> 172.21.15.78:0/1566287198 conn(0x55df3abfa3e0 msgr2=0x55df3ac0ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.987+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 shutdown_connections 2023-10-10T17:43:01.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.987+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/1566287198 wait complete. 2023-10-10T17:43:01.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:00.987+0000 7fcb3e7fbb80 20 client.15786 trim_cache size 0 max 16384 2023-10-10T17:43:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 Processor -- start 2023-10-10T17:43:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 -- start start 2023-10-10T17:43:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac04be0 0x55df3abffcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac054f0 0x55df3ac15a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac0db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3abf7ac0 con 0x55df3ac09930 2023-10-10T17:43:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ac05c70 con 0x55df3ac04be0 2023-10-10T17:43:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ac0c920 con 0x55df3ac054f0 2023-10-10T17:43:01.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb30c0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac0db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb2b7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac054f0 0x55df3ac15a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb2b7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac054f0 0x55df3ac15a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:36480/0 (socket says 172.21.15.78:36480) 2023-10-10T17:43:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb30c0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac0db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44952/0 (socket says 172.21.15.78:44952) 2023-10-10T17:43:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/3447803363 learned_addr learned my addr 172.21.15.78:0/3447803363 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.016+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac04be0 0x55df3abffcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/3447803363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac04be0 msgr2=0x55df3abffcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac04be0 0x55df3abffcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/3447803363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 msgr2=0x55df3ac0db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac0db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb2b7fe700 1 -- 172.21.15.78:0/3447803363 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df3ac0bfe0 con 0x55df3ac054f0 2023-10-10T17:43:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac0db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac04be0 0x55df3abffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac054f0 0x55df3ac15a50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c096370 tx=0x7fcb1c0b2260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=515019267e0752a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb12ffd700 1 -- 172.21.15.78:0/3447803363 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c0c9fc0 con 0x55df3ac054f0 2023-10-10T17:43:01.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb12ffd700 1 -- 172.21.15.78:0/3447803363 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb1c0b92b0 con 0x55df3ac054f0 2023-10-10T17:43:01.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.017+0000 7fcb12ffd700 1 -- 172.21.15.78:0/3447803363 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c098e40 con 0x55df3ac054f0 2023-10-10T17:43:01.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/3447803363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac054f0 msgr2=0x55df3ac15a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac054f0 0x55df3ac15a50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c096370 tx=0x7fcb1c0b2260 comp rx=0 tx=0).stop 2023-10-10T17:43:01.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/3447803363 shutdown_connections 2023-10-10T17:43:01.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac09930 0x55df3ac0db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac04be0 0x55df3abffcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/3447803363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac054f0 0x55df3ac15a50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/3447803363 >> 172.21.15.78:0/3447803363 conn(0x55df3abfa3e0 msgr2=0x55df3ac07930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/3447803363 shutdown_connections 2023-10-10T17:43:01.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/3447803363 wait complete. 2023-10-10T17:43:01.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.018+0000 7fcb3e7fbb80 1 -- start start 2023-10-10T17:43:01.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 1 Processor -- start 2023-10-10T17:43:01.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac14220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac70660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac14760 0x55df3ac14bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ac70ba0 con 0x55df3ac14760 2023-10-10T17:43:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ac83e30 con 0x55df3ac09930 2023-10-10T17:43:01.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df3ac83f70 con 0x55df3ac04be0 2023-10-10T17:43:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb3e7fbb80 10 client.0 _ll_register_callbacks cb 0x7fcb1c099490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb30c0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac14760 0x55df3ac14bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb2bfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac14220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb30c0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac14760 0x55df3ac14bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44958/0 (socket says 172.21.15.78:44958) 2023-10-10T17:43:01.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb30c0c700 1 -- 172.21.15.78:0/4074786315 learned_addr learned my addr 172.21.15.78:0/4074786315 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac70660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.019+0000 7fcb30c0c700 1 -- 172.21.15.78:0/4074786315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 msgr2=0x55df3ac14220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac14220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb30c0c700 1 -- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 msgr2=0x55df3ac70660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac70660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb30c0c700 1 -- 172.21.15.78:0/4074786315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df3a9d76b0 con 0x55df3ac14760 2023-10-10T17:43:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb2b7fe700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac70660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac14220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb30c0c700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac14760 0x55df3ac14bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcb240084f0 tx=0x7fcb24007c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e3bd8bc7d5e533 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb0b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:01.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/4074786315 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb240128c0 con 0x55df3ac14760 2023-10-10T17:43:01.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.020+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/4074786315 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fcb24012a20 con 0x55df3ac14760 2023-10-10T17:43:01.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcb24010040 con 0x55df3ac14760 2023-10-10T17:43:01.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/4074786315 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb24012c20 con 0x55df3ac14760 2023-10-10T17:43:01.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 10 client.15792 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:01.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55df3ab1c210 con 0x55df3ac14760 2023-10-10T17:43:01.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/4074786315 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb24010040 con 0x55df3ac14760 2023-10-10T17:43:01.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/4074786315 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fcb24026030 con 0x55df3ac14760 2023-10-10T17:43:01.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 10 client.15792 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:01.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 10 client.15792 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:01.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 20 client.15792 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:01.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac855c0 0x55df3ac879d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 4 client.15792 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:43:01.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55df3ab246e0 con 0x55df3ac855c0 2023-10-10T17:43:01.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.021+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac855c0 0x55df3ac879d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.022+0000 7fcb0b7fe700 10 client.15792 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.022+0000 7fcb2bfff700 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac855c0 0x55df3ac879d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71bf911f68a4ce87 server_cookie=65848a60ab2111c5 in_seq=0 out_seq=0 2023-10-10T17:43:01.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.022+0000 7fcb0b7fe700 1 -- 172.21.15.78:0/4074786315 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55df3ab246e0 con 0x55df3ac855c0 2023-10-10T17:43:01.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.022+0000 7fcb0b7fe700 10 client.15792 handle_command_reply: tid=0 2023-10-10T17:43:01.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 client.15792 shutdown 2023-10-10T17:43:01.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac14760 msgr2=0x55df3ac14bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac14760 0x55df3ac14bd0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fcb240084f0 tx=0x7fcb24007c40 comp rx=0 tx=0).stop 2023-10-10T17:43:01.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 shutdown_connections 2023-10-10T17:43:01.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55df3ac04be0 0x55df3ac14220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55df3ac855c0 0x55df3ac879d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55df3ac14760 0x55df3ac14bd0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 --2- 172.21.15.78:0/4074786315 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55df3ac09930 0x55df3ac70660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb0b7fe700 0 client.15792 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 >> 172.21.15.78:0/4074786315 conn(0x55df3abfa3e0 msgr2=0x55df3ac08870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 shutdown_connections 2023-10-10T17:43:01.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 1 -- 172.21.15.78:0/4074786315 wait complete. 2023-10-10T17:43:01.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.023+0000 7fcb3e7fbb80 20 client.15792 trim_cache size 0 max 16384 2023-10-10T17:43:01.040 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:43:01.040 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:43:01.040 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "4674d947-848d-4afb-9f1f-59ffba13c498", 2023-10-10T17:43:01.041 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:43:01.041 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:43:01.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcb140a1350 msgr2=0x7fcb140a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcb140a1350 0x7fcb140a3800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcb20003800 tx=0x7fcb20005fd0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c101400 msgr2=0x7fcb2c196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c101400 0x7fcb2c196950 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c00c920 tx=0x7fcb1c00cc30 comp rx=0 tx=0).stop 2023-10-10T17:43:01.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 shutdown_connections 2023-10-10T17:43:01.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcb2c1063f0 0x7fcb2c19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcb140a1350 0x7fcb140a3800 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcb2c100a50 0x7fcb2c196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 --2- 172.21.15.78:0/2257269195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb2c101400 0x7fcb2c196950 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 >> 172.21.15.78:0/2257269195 conn(0x7fcb2c077590 msgr2=0x7fcb2c0ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 shutdown_connections 2023-10-10T17:43:01.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.024+0000 7fcb3266f700 1 -- 172.21.15.78:0/2257269195 wait complete. 2023-10-10T17:43:01.044 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:00 smithi078 ceph-mon[99494]: pgmap v15: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:43:01.045 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:43:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:43:00.739+0000 7f4702a39700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:43:01.045 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:43:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:43:00.739+0000 7f4702a39700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:43:01.045 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:43:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:43:00.739+0000 7f4702a39700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:43:01.045 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:43:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:43:00.739+0000 7f4702a39700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:43:01.045 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:43:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:43:00.739+0000 7f4702a39700 -1 client.0 error registering admin socket command: (17) File exists 2023-10-10T17:43:01.046 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:43:01.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.171+0000 7f996dd3d700 1 -- 172.21.15.78:0/294028630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 msgr2=0x7f996810ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.171+0000 7f996dd3d700 1 --2- 172.21.15.78:0/294028630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f996810ccd0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f996000b0a0 tx=0x7f996000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 -- 172.21.15.78:0/294028630 shutdown_connections 2023-10-10T17:43:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 --2- 172.21.15.78:0/294028630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f996810ccd0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 --2- 172.21.15.78:0/294028630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9968101390 0x7f9968101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 --2- 172.21.15.78:0/294028630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9968101ca0 0x7f996810a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 -- 172.21.15.78:0/294028630 >> 172.21.15.78:0/294028630 conn(0x7f99680770b0 msgr2=0x7f99680774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 -- 172.21.15.78:0/294028630 shutdown_connections 2023-10-10T17:43:01.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 -- 172.21.15.78:0/294028630 wait complete. 2023-10-10T17:43:01.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.172+0000 7f996dd3d700 1 Processor -- start 2023-10-10T17:43:01.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f996dd3d700 1 -- start start 2023-10-10T17:43:01.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f996dd3d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9968101390 0x7f9968194270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f996dd3d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9968101ca0 0x7f99681947b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f996dd3d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f9968198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f996dd3d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968199250 con 0x7f996810a8b0 2023-10-10T17:43:01.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f996dd3d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99681993c0 con 0x7f9968101ca0 2023-10-10T17:43:01.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f996dd3d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968199580 con 0x7f9968101390 2023-10-10T17:43:01.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f9967fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f9968198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f9967fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f9968198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44972/0 (socket says 172.21.15.78:44972) 2023-10-10T17:43:01.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f9967fff700 1 -- 172.21.15.78:0/369543882 learned_addr learned my addr 172.21.15.78:0/369543882 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f99677fe700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9968101390 0x7f9968194270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f9966ffd700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9968101ca0 0x7f99681947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f9967fff700 1 -- 172.21.15.78:0/369543882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9968101390 msgr2=0x7f9968194270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.173+0000 7f9967fff700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9968101390 0x7f9968194270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f9967fff700 1 -- 172.21.15.78:0/369543882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9968101ca0 msgr2=0x7f99681947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f9967fff700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9968101ca0 0x7f99681947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f9967fff700 1 -- 172.21.15.78:0/369543882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9960009d00 con 0x7f996810a8b0 2023-10-10T17:43:01.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f9967fff700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f9968198c10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9960006070 tx=0x7f9960004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f9964ff9700 1 -- 172.21.15.78:0/369543882 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960026070 con 0x7f996810a8b0 2023-10-10T17:43:01.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f996810e150 con 0x7f996810a8b0 2023-10-10T17:43:01.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f9964ff9700 1 -- 172.21.15.78:0/369543882 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99600077d0 con 0x7f996810a8b0 2023-10-10T17:43:01.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f9964ff9700 1 -- 172.21.15.78:0/369543882 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9960004420 con 0x7f996810a8b0 2023-10-10T17:43:01.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.174+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996810dfc0 con 0x7f996810a8b0 2023-10-10T17:43:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.177+0000 7f9964ff9700 1 -- 172.21.15.78:0/369543882 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 20) v1 ==== 135891+0+0 (secure 0 0 0) 0x7f9960007940 con 0x7f996810a8b0 2023-10-10T17:43:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.178+0000 7f9964ff9700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99500a1350 0x7f99500a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.178+0000 7f9964ff9700 1 -- 172.21.15.78:0/369543882 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f99600d2020 con 0x7f996810a8b0 2023-10-10T17:43:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.178+0000 7f99677fe700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99500a1350 0x7f99500a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f99677fe700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99500a1350 0x7f99500a3800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9958005950 tx=0x7f995800b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 Processor -- start 2023-10-10T17:43:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 -- start start 2023-10-10T17:43:01.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b14f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b19370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b19d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288a1a060 con 0x562288b14b60 2023-10-10T17:43:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288a6c8a0 con 0x562288b15470 2023-10-10T17:43:01.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.179+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288a65d30 con 0x562288b198b0 2023-10-10T17:43:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b14f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b14f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44976/0 (socket says 172.21.15.78:44976) 2023-10-10T17:43:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 -- 172.21.15.78:0/2417372103 learned_addr learned my addr 172.21.15.78:0/2417372103 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f9966ffd700 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b19370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f9967fff700 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b19d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 -- 172.21.15.78:0/2417372103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 msgr2=0x562288b19d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b19d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 -- 172.21.15.78:0/2417372103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 msgr2=0x562288b19370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b19370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 -- 172.21.15.78:0/2417372103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622888e7630 con 0x562288b14b60 2023-10-10T17:43:01.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f9966ffd700 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b19370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f99677fe700 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b14f30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9958010fa0 tx=0x7f9958013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8c9d05d5033a83 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f994dffb700 1 -- 172.21.15.78:0/2417372103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958011db0 con 0x562288b14b60 2023-10-10T17:43:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f994dffb700 1 -- 172.21.15.78:0/2417372103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9958015070 con 0x562288b14b60 2023-10-10T17:43:01.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.180+0000 7f994dffb700 1 -- 172.21.15.78:0/2417372103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995801c610 con 0x562288b14b60 2023-10-10T17:43:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 -- 172.21.15.78:0/2417372103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 msgr2=0x562288b14f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b14f30 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f9958010fa0 tx=0x7f9958013000 comp rx=0 tx=0).stop 2023-10-10T17:43:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 -- 172.21.15.78:0/2417372103 shutdown_connections 2023-10-10T17:43:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b19d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b14f30 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/2417372103 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b19370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 -- 172.21.15.78:0/2417372103 >> 172.21.15.78:0/2417372103 conn(0x562288b0a360 msgr2=0x562288b0fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 -- 172.21.15.78:0/2417372103 shutdown_connections 2023-10-10T17:43:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.181+0000 7f9979ec9b80 1 -- 172.21.15.78:0/2417372103 wait complete. 2023-10-10T17:43:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.182+0000 7f9979ec9b80 1 -- start start 2023-10-10T17:43:01.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.182+0000 7f9979ec9b80 1 Processor -- start 2023-10-10T17:43:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b20fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b1e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 0x562288b1e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288a078e0 con 0x562288b15470 2023-10-10T17:43:01.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288a07580 con 0x562288b198b0 2023-10-10T17:43:01.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622888e9920 con 0x562288b14b60 2023-10-10T17:43:01.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9979ec9b80 10 client.0 _ll_register_callbacks cb 0x562288a25780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:01.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b20fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b20fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:36512/0 (socket says 172.21.15.78:36512) 2023-10-10T17:43:01.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 -- 172.21.15.78:0/470230450 learned_addr learned my addr 172.21.15.78:0/470230450 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f9966ffd700 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b1e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 -- 172.21.15.78:0/470230450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 msgr2=0x562288b1e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 0x562288b1e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 -- 172.21.15.78:0/470230450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 msgr2=0x562288b1e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b1e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 -- 172.21.15.78:0/470230450 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9958010c50 con 0x562288b14b60 2023-10-10T17:43:01.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.183+0000 7f99677fe700 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b20fd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9958010a40 tx=0x7f9958011dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29b74092d5044fef server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9946ffd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:43:01.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9946ffd700 1 -- 172.21.15.78:0/470230450 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995802c070 con 0x562288b14b60 2023-10-10T17:43:01.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9946ffd700 1 -- 172.21.15.78:0/470230450 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f9958017070 con 0x562288b14b60 2023-10-10T17:43:01.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562288b07b20 con 0x562288b14b60 2023-10-10T17:43:01.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9946ffd700 1 -- 172.21.15.78:0/470230450 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f995801e630 con 0x562288b14b60 2023-10-10T17:43:01.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9979ec9b80 10 client.25345 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:01.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562288a2c190 con 0x562288b14b60 2023-10-10T17:43:01.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9946ffd700 1 -- 172.21.15.78:0/470230450 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562288b07b20 con 0x562288b14b60 2023-10-10T17:43:01.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.184+0000 7f9946ffd700 1 -- 172.21.15.78:0/470230450 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f995801a3a0 con 0x562288b14b60 2023-10-10T17:43:01.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9979ec9b80 10 client.25345 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:01.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9979ec9b80 10 client.25345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:01.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9979ec9b80 20 client.25345 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:01.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b36ff0 0x562288b39400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9979ec9b80 4 client.25345 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:01.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562288a280a0 con 0x562288b36ff0 2023-10-10T17:43:01.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9966ffd700 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b36ff0 0x562288b39400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9946ffd700 10 client.25345 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.185+0000 7f9966ffd700 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b36ff0 0x562288b39400 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=391196a4ff2cb888 server_cookie=33cde648565c62e5 in_seq=0 out_seq=0 2023-10-10T17:43:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9946ffd700 1 -- 172.21.15.78:0/470230450 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562288a280a0 con 0x562288b36ff0 2023-10-10T17:43:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9946ffd700 10 client.25345 handle_command_reply: tid=0 2023-10-10T17:43:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 client.25345 shutdown 2023-10-10T17:43:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 msgr2=0x562288b20fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b20fd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9958010a40 tx=0x7f9958011dc0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 shutdown_connections 2023-10-10T17:43:01.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b20fd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b36ff0 0x562288b39400 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 0x562288b1e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/470230450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b1e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 >> 172.21.15.78:0/470230450 conn(0x562288b0a360 msgr2=0x562288b0ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9946ffd700 0 client.25345 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 shutdown_connections 2023-10-10T17:43:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.186+0000 7f9979ec9b80 1 -- 172.21.15.78:0/470230450 wait complete. 2023-10-10T17:43:01.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.187+0000 7f9979ec9b80 20 client.25345 trim_cache size 0 max 16384 2023-10-10T17:43:01.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 Processor -- start 2023-10-10T17:43:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 -- start start 2023-10-10T17:43:01.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b18a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b171a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b176e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288b09f10 con 0x562288b14b60 2023-10-10T17:43:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288b0a170 con 0x562288b15470 2023-10-10T17:43:01.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.215+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288b10950 con 0x562288b198b0 2023-10-10T17:43:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b18a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b18a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44990/0 (socket says 172.21.15.78:44990) 2023-10-10T17:43:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 -- 172.21.15.78:0/883366321 learned_addr learned my addr 172.21.15.78:0/883366321 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f9966ffd700 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b171a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f9967fff700 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b176e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 -- 172.21.15.78:0/883366321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 msgr2=0x562288b176e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b176e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 -- 172.21.15.78:0/883366321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 msgr2=0x562288b171a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b171a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 -- 172.21.15.78:0/883366321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562288a2bee0 con 0x562288b14b60 2023-10-10T17:43:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f9967fff700 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f99677fe700 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b18a10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f995801ed30 tx=0x7f9958011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bf603dd992e766 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f994e7fc700 1 -- 172.21.15.78:0/883366321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995802c070 con 0x562288b14b60 2023-10-10T17:43:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f994e7fc700 1 -- 172.21.15.78:0/883366321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f995801a9d0 con 0x562288b14b60 2023-10-10T17:43:01.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.216+0000 7f994e7fc700 1 -- 172.21.15.78:0/883366321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958031470 con 0x562288b14b60 2023-10-10T17:43:01.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.217+0000 7f9979ec9b80 1 -- 172.21.15.78:0/883366321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 msgr2=0x562288b18a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.217+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b18a10 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f995801ed30 tx=0x7f9958011b30 comp rx=0 tx=0).stop 2023-10-10T17:43:01.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.217+0000 7f9979ec9b80 1 -- 172.21.15.78:0/883366321 shutdown_connections 2023-10-10T17:43:01.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.217+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b198b0 0x562288b176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.217+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b14b60 0x562288b18a10 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.217+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/883366321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b15470 0x562288b171a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.218+0000 7f9979ec9b80 1 -- 172.21.15.78:0/883366321 >> 172.21.15.78:0/883366321 conn(0x562288b0a360 msgr2=0x562288b39000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.218+0000 7f9979ec9b80 1 -- 172.21.15.78:0/883366321 shutdown_connections 2023-10-10T17:43:01.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.218+0000 7f9979ec9b80 1 -- 172.21.15.78:0/883366321 wait complete. 2023-10-10T17:43:01.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.218+0000 7f9979ec9b80 1 -- start start 2023-10-10T17:43:01.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.218+0000 7f9979ec9b80 1 Processor -- start 2023-10-10T17:43:01.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b85e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b86390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9979ec9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 0x562288b868d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288b25f60 con 0x562288b15470 2023-10-10T17:43:01.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288b23990 con 0x562288b198b0 2023-10-10T17:43:01.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9979ec9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562288b23b50 con 0x562288b14b60 2023-10-10T17:43:01.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9979ec9b80 10 client.0 _ll_register_callbacks cb 0x7f99600cf360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:01.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b86390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b86390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:44998/0 (socket says 172.21.15.78:44998) 2023-10-10T17:43:01.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 -- 172.21.15.78:0/132745018 learned_addr learned my addr 172.21.15.78:0/132745018 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:01.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f99677fe700 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b85e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 -- 172.21.15.78:0/132745018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 msgr2=0x562288b85e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b85e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 -- 172.21.15.78:0/132745018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 msgr2=0x562288b868d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:01.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 0x562288b868d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 -- 172.21.15.78:0/132745018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562288a2c190 con 0x562288b15470 2023-10-10T17:43:01.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.219+0000 7f9966ffd700 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b86390 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x562288b1f800 tx=0x7f995c01d670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d49925195244306f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:01.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9946ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:01.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9946ffd700 1 -- 172.21.15.78:0/132745018 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995c016070 con 0x562288b15470 2023-10-10T17:43:01.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9946ffd700 1 -- 172.21.15.78:0/132745018 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f995c01dbd0 con 0x562288b15470 2023-10-10T17:43:01.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9958015070 con 0x562288b15470 2023-10-10T17:43:01.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9946ffd700 1 -- 172.21.15.78:0/132745018 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f995c01dd90 con 0x562288b15470 2023-10-10T17:43:01.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 10 client.15810 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:01.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56228887cc60 con 0x562288b15470 2023-10-10T17:43:01.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9946ffd700 1 -- 172.21.15.78:0/132745018 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9958015070 con 0x562288b15470 2023-10-10T17:43:01.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9946ffd700 1 -- 172.21.15.78:0/132745018 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f995c010790 con 0x562288b15470 2023-10-10T17:43:01.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 10 client.15810 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:01.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 10 client.15810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:01.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 20 client.15810 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:01.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b308a0 0x562288b32cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:01.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 4 client.15810 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:43:01.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.220+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562288a34660 con 0x562288b308a0 2023-10-10T17:43:01.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.221+0000 7f99677fe700 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b308a0 0x562288b32cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:01.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.221+0000 7f9946ffd700 10 client.15810 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.221+0000 7f99677fe700 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b308a0 0x562288b32cb0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f117fd6e77a296e1 server_cookie=15f0d95816d73398 in_seq=0 out_seq=0 2023-10-10T17:43:01.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.221+0000 7f9946ffd700 1 -- 172.21.15.78:0/132745018 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x562288a34660 con 0x562288b308a0 2023-10-10T17:43:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.221+0000 7f9946ffd700 10 client.15810 handle_command_reply: tid=0 2023-10-10T17:43:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 client.15810 shutdown 2023-10-10T17:43:01.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 msgr2=0x562288b86390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b86390 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x562288b1f800 tx=0x7f995c01d670 comp rx=0 tx=0).stop 2023-10-10T17:43:01.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 shutdown_connections 2023-10-10T17:43:01.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562288b14b60 0x562288b85e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562288b308a0 0x562288b32cb0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562288b198b0 0x562288b868d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9946ffd700 0 client.15810 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:01.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 --2- 172.21.15.78:0/132745018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562288b15470 0x562288b86390 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 >> 172.21.15.78:0/132745018 conn(0x562288b0a360 msgr2=0x562288b18a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 shutdown_connections 2023-10-10T17:43:01.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 1 -- 172.21.15.78:0/132745018 wait complete. 2023-10-10T17:43:01.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.222+0000 7f9979ec9b80 20 client.15810 trim_cache size 0 max 16384 2023-10-10T17:43:01.246 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:43:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:43:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:43:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: "4674d947-848d-4afb-9f1f-59ffba13c498": { 2023-10-10T17:43:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:43:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "4674d947-848d-4afb-9f1f-59ffba13c498", 2023-10-10T17:43:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:43:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:43:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:43:01.249 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:43:01.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99500a1350 msgr2=0x7f99500a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99500a1350 0x7f99500a3800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9958005950 tx=0x7f995800b410 comp rx=0 tx=0).stop 2023-10-10T17:43:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 msgr2=0x7f9968198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f9968198c10 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f9960006070 tx=0x7f9960004d70 comp rx=0 tx=0).stop 2023-10-10T17:43:01.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 shutdown_connections 2023-10-10T17:43:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9968101390 0x7f9968194270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99500a1350 0x7f99500a3800 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f996810a8b0 0x7f9968198c10 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 --2- 172.21.15.78:0/369543882 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9968101ca0 0x7f99681947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:01.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 >> 172.21.15.78:0/369543882 conn(0x7f99680770b0 msgr2=0x7f99680fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 shutdown_connections 2023-10-10T17:43:01.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:01.223+0000 7f996dd3d700 1 -- 172.21.15.78:0/369543882 wait complete. 2023-10-10T17:43:01.254 INFO:tasks.cephfs.filesystem:scrub status for tag:4674d947-848d-4afb-9f1f-59ffba13c498 - {'path': '/', 'tag': '4674d947-848d-4afb-9f1f-59ffba13c498', 'options': 'recursive,force'} 2023-10-10T17:43:02.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:02 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:43:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:02 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:02 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:43:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:02 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:02.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:02 smithi190 ceph-mon[101220]: pgmap v16: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:43:02.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:02 smithi190 ceph-mon[101220]: mgrmap e21: x(active, since 26s), standbys: y, z 2023-10-10T17:43:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:02 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:43:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:02 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:02.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:02 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:43:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:02 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:02 smithi157 ceph-mon[100896]: pgmap v16: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:43:02.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:02 smithi157 ceph-mon[100896]: mgrmap e21: x(active, since 26s), standbys: y, z 2023-10-10T17:43:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:02 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:43:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:02 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:02 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:43:03.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:02 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:03.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:02 smithi078 ceph-mon[99494]: pgmap v16: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 682 B/s wr, 6 op/s 2023-10-10T17:43:03.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:02 smithi078 ceph-mon[99494]: mgrmap e21: x(active, since 26s), standbys: y, z 2023-10-10T17:43:04.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.007+0000 7f597db1d700 1 -- 172.21.15.78:0/1704639139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978100ee0 msgr2=0x7f5978109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.007+0000 7f597db1d700 1 --2- 172.21.15.78:0/1704639139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978100ee0 0x7f5978109640 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f596c009990 tx=0x7f596c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.007+0000 7f597db1d700 1 -- 172.21.15.78:0/1704639139 shutdown_connections 2023-10-10T17:43:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.007+0000 7f597db1d700 1 --2- 172.21.15.78:0/1704639139 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59781005d0 0x7f59781009a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.007+0000 7f597db1d700 1 --2- 172.21.15.78:0/1704639139 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5978109b80 0x7f597810bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.007+0000 7f597db1d700 1 --2- 172.21.15.78:0/1704639139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978100ee0 0x7f5978109640 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.008+0000 7f597db1d700 1 -- 172.21.15.78:0/1704639139 >> 172.21.15.78:0/1704639139 conn(0x7f5978074a50 msgr2=0x7f5978074e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.008+0000 7f597db1d700 1 -- 172.21.15.78:0/1704639139 shutdown_connections 2023-10-10T17:43:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.008+0000 7f597db1d700 1 -- 172.21.15.78:0/1704639139 wait complete. 2023-10-10T17:43:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 Processor -- start 2023-10-10T17:43:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 -- start start 2023-10-10T17:43:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59781005d0 0x7f59781963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5978100ee0 0x7f5978196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978109b80 0x7f597819ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597819b3b0 con 0x7f5978109b80 2023-10-10T17:43:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597819b520 con 0x7f5978100ee0 2023-10-10T17:43:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.009+0000 7f597db1d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597819b6e0 con 0x7f59781005d0 2023-10-10T17:43:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978109b80 0x7f597819ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978109b80 0x7f597819ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45014/0 (socket says 172.21.15.78:45014) 2023-10-10T17:43:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 -- 172.21.15.78:0/2167345988 learned_addr learned my addr 172.21.15.78:0/2167345988 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5976ffd700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5978100ee0 0x7f5978196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f59777fe700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59781005d0 0x7f59781963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 -- 172.21.15.78:0/2167345988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59781005d0 msgr2=0x7f59781963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59781005d0 0x7f59781963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 -- 172.21.15.78:0/2167345988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5978100ee0 msgr2=0x7f5978196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5978100ee0 0x7f5978196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 -- 172.21.15.78:0/2167345988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c0095f0 con 0x7f5978109b80 2023-10-10T17:43:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5977fff700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978109b80 0x7f597819ad70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f59700094f0 tx=0x7f597000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.010+0000 7f5974ff9700 1 -- 172.21.15.78:0/2167345988 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970004db0 con 0x7f5978109b80 2023-10-10T17:43:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.011+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5978110380 con 0x7f5978109b80 2023-10-10T17:43:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.011+0000 7f5974ff9700 1 -- 172.21.15.78:0/2167345988 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f597000f040 con 0x7f5978109b80 2023-10-10T17:43:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.011+0000 7f5974ff9700 1 -- 172.21.15.78:0/2167345988 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970005980 con 0x7f5978109b80 2023-10-10T17:43:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.011+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59781101f0 con 0x7f5978109b80 2023-10-10T17:43:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.014+0000 7f5974ff9700 1 -- 172.21.15.78:0/2167345988 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f5970005020 con 0x7f5978109b80 2023-10-10T17:43:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.014+0000 7f5974ff9700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f59600a13e0 0x7f59600a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.015+0000 7f5974ff9700 1 -- 172.21.15.78:0/2167345988 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f59700ccbb0 con 0x7f5978109b80 2023-10-10T17:43:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.015+0000 7f5989ca9b80 1 Processor -- start 2023-10-10T17:43:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.015+0000 7f59777fe700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f59600a13e0 0x7f59600a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5989ca9b80 1 -- start start 2023-10-10T17:43:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 0x562fbadb21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 0x562fbadac560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadacb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbacb2000 con 0x562fbadb2b90 2023-10-10T17:43:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbad04840 con 0x562fbac086b0 2023-10-10T17:43:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbacfdcd0 con 0x562fbadb2720 2023-10-10T17:43:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadacb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadacb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45022/0 (socket says 172.21.15.78:45022) 2023-10-10T17:43:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5977fff700 1 -- 172.21.15.78:0/2734915097 learned_addr learned my addr 172.21.15.78:0/2734915097 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f5976ffd700 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 0x562fbadac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.016+0000 7f59777fe700 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 0x562fbadb21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f59777fe700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f59600a13e0 0x7f59600a3890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5968009c70 tx=0x7f5968011720 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f5977fff700 1 -- 172.21.15.78:0/2734915097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 msgr2=0x562fbadac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f5977fff700 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 0x562fbadac560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f5977fff700 1 -- 172.21.15.78:0/2734915097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 msgr2=0x562fbadb21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f5977fff700 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 0x562fbadb21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f5977fff700 1 -- 172.21.15.78:0/2734915097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fbab7f5d0 con 0x562fbadb2b90 2023-10-10T17:43:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f5977fff700 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadacb00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5970099440 tx=0x7f5970099510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f904e531805840 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f595dffb700 1 -- 172.21.15.78:0/2734915097 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59700990a0 con 0x562fbadb2b90 2023-10-10T17:43:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f595dffb700 1 -- 172.21.15.78:0/2734915097 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5970099740 con 0x562fbadb2b90 2023-10-10T17:43:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.017+0000 7f595dffb700 1 -- 172.21.15.78:0/2734915097 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970098e90 con 0x562fbadb2b90 2023-10-10T17:43:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 -- 172.21.15.78:0/2734915097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 msgr2=0x562fbadacb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadacb00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f5970099440 tx=0x7f5970099510 comp rx=0 tx=0).stop 2023-10-10T17:43:04.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 -- 172.21.15.78:0/2734915097 shutdown_connections 2023-10-10T17:43:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadacb00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 0x562fbadb21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/2734915097 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 0x562fbadac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 -- 172.21.15.78:0/2734915097 >> 172.21.15.78:0/2734915097 conn(0x562fbada1f50 msgr2=0x562fbada7270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 -- 172.21.15.78:0/2734915097 shutdown_connections 2023-10-10T17:43:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.018+0000 7f5989ca9b80 1 -- 172.21.15.78:0/2734915097 wait complete. 2023-10-10T17:43:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 -- start start 2023-10-10T17:43:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 Processor -- start 2023-10-10T17:43:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 0x562fbadb8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 0x562fbadb5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadb6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbac9f880 con 0x562fbadb2b90 2023-10-10T17:43:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbac9f520 con 0x562fbac086b0 2023-10-10T17:43:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbab818c0 con 0x562fbadb2720 2023-10-10T17:43:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5989ca9b80 10 client.0 _ll_register_callbacks cb 0x562fbacbd720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadb6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.019+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadb6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45026/0 (socket says 172.21.15.78:45026) 2023-10-10T17:43:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5977fff700 1 -- 172.21.15.78:0/1995851188 learned_addr learned my addr 172.21.15.78:0/1995851188 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5977fff700 1 -- 172.21.15.78:0/1995851188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 msgr2=0x562fbadb5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5977fff700 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 0x562fbadb5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5977fff700 1 -- 172.21.15.78:0/1995851188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 msgr2=0x562fbadb8b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T17:43:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5977fff700 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 0x562fbadb8b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5977fff700 1 -- 172.21.15.78:0/1995851188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59781101f0 con 0x562fbadb2b90 2023-10-10T17:43:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5977fff700 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadb6460 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5970009710 tx=0x7f5970098f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5d039ff1be10865 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5956ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5956ffd700 1 -- 172.21.15.78:0/1995851188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59700a1b90 con 0x562fbadb2b90 2023-10-10T17:43:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5956ffd700 1 -- 172.21.15.78:0/1995851188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f597009d770 con 0x562fbadb2b90 2023-10-10T17:43:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562fbad9f710 con 0x562fbadb2b90 2023-10-10T17:43:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5956ffd700 1 -- 172.21.15.78:0/1995851188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5970097ae0 con 0x562fbadb2b90 2023-10-10T17:43:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5989ca9b80 10 client.15822 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.020+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562fbacc4130 con 0x562fbadb2b90 2023-10-10T17:43:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5956ffd700 1 -- 172.21.15.78:0/1995851188 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562fbad9f710 con 0x562fbadb2b90 2023-10-10T17:43:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5956ffd700 1 -- 172.21.15.78:0/1995851188 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f597009aba0 con 0x562fbadb2b90 2023-10-10T17:43:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5989ca9b80 10 client.15822 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5989ca9b80 10 client.15822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5989ca9b80 20 client.15822 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbadce4e0 0x562fbadd08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5989ca9b80 4 client.15822 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562fbacc0040 con 0x562fbadce4e0 2023-10-10T17:43:04.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.021+0000 7f59777fe700 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbadce4e0 0x562fbadd08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.022+0000 7f5956ffd700 10 client.15822 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.022+0000 7f59777fe700 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbadce4e0 0x562fbadd08f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=956868601d99b08c server_cookie=b74d9b8a33099ec0 in_seq=0 out_seq=0 2023-10-10T17:43:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.022+0000 7f5956ffd700 1 -- 172.21.15.78:0/1995851188 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562fbacc0040 con 0x562fbadce4e0 2023-10-10T17:43:04.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.022+0000 7f5956ffd700 10 client.15822 handle_command_reply: tid=0 2023-10-10T17:43:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.022+0000 7f5989ca9b80 1 client.15822 shutdown 2023-10-10T17:43:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.022+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 msgr2=0x562fbadb6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadb6460 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5970009710 tx=0x7f5970098f90 comp rx=0 tx=0).stop 2023-10-10T17:43:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 shutdown_connections 2023-10-10T17:43:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbadce4e0 0x562fbadd08f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbadb2b90 0x562fbadb6460 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbac086b0 0x562fbadb8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1995851188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbadb2720 0x562fbadb5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 >> 172.21.15.78:0/1995851188 conn(0x562fbada1f50 msgr2=0x562fbada67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5956ffd700 0 client.15822 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 shutdown_connections 2023-10-10T17:43:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.023+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1995851188 wait complete. 2023-10-10T17:43:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.024+0000 7f5989ca9b80 20 client.15822 trim_cache size 0 max 16384 2023-10-10T17:43:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.051+0000 7f5989ca9b80 1 Processor -- start 2023-10-10T17:43:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.051+0000 7f5989ca9b80 1 -- start start 2023-10-10T17:43:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.051+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbadb5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.051+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbadbb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbada8c50 0x562fbada6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbada4000 con 0x562fbada8c50 2023-10-10T17:43:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbadb1df0 con 0x562fbadb2720 2023-10-10T17:43:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbadb09f0 con 0x562fbac086b0 2023-10-10T17:43:04.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbada8c50 0x562fbada6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbada8c50 0x562fbada6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45034/0 (socket says 172.21.15.78:45034) 2023-10-10T17:43:04.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 -- 172.21.15.78:0/3067568520 learned_addr learned my addr 172.21.15.78:0/3067568520 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:04.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f59777fe700 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbadb5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5976ffd700 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbadbb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 -- 172.21.15.78:0/3067568520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 msgr2=0x562fbadb5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbadb5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 -- 172.21.15.78:0/3067568520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 msgr2=0x562fbadbb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbadbb660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 -- 172.21.15.78:0/3067568520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fbadd5520 con 0x562fbada8c50 2023-10-10T17:43:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f59777fe700 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbadb5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5976ffd700 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbadbb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.052+0000 7f5977fff700 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbada8c50 0x562fbada6f60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f59700ca430 tx=0x7f5970098e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bfe5a95a0087b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.053+0000 7f595e7fc700 1 -- 172.21.15.78:0/3067568520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5970097ae0 con 0x562fbada8c50 2023-10-10T17:43:04.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.053+0000 7f595e7fc700 1 -- 172.21.15.78:0/3067568520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5970063020 con 0x562fbada8c50 2023-10-10T17:43:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.053+0000 7f595e7fc700 1 -- 172.21.15.78:0/3067568520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59700843e0 con 0x562fbada8c50 2023-10-10T17:43:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.053+0000 7f5989ca9b80 1 -- 172.21.15.78:0/3067568520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbada8c50 msgr2=0x562fbada6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.053+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbada8c50 0x562fbada6f60 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f59700ca430 tx=0x7f5970098e60 comp rx=0 tx=0).stop 2023-10-10T17:43:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 -- 172.21.15.78:0/3067568520 shutdown_connections 2023-10-10T17:43:04.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbadb5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbada8c50 0x562fbada6f60 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/3067568520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbadbb660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 -- 172.21.15.78:0/3067568520 >> 172.21.15.78:0/3067568520 conn(0x562fbada1f50 msgr2=0x562fbadb55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 -- 172.21.15.78:0/3067568520 shutdown_connections 2023-10-10T17:43:04.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 -- 172.21.15.78:0/3067568520 wait complete. 2023-10-10T17:43:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 -- start start 2023-10-10T17:43:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.054+0000 7f5989ca9b80 1 Processor -- start 2023-10-10T17:43:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbae18530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbae18a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5989ca9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbae213b0 0x562fbae23790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbae19210 con 0x562fbae213b0 2023-10-10T17:43:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbae23d00 con 0x562fbadb2720 2023-10-10T17:43:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5989ca9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fbae23ec0 con 0x562fbac086b0 2023-10-10T17:43:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5989ca9b80 10 client.0 _ll_register_callbacks cb 0x7f5968009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:04.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbae18530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5977fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbae213b0 0x562fbae23790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbae18530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:36568/0 (socket says 172.21.15.78:36568) 2023-10-10T17:43:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 -- 172.21.15.78:0/1809635859 learned_addr learned my addr 172.21.15.78:0/1809635859 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 -- 172.21.15.78:0/1809635859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 msgr2=0x562fbae18a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:04.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbae18a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 -- 172.21.15.78:0/1809635859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbae213b0 msgr2=0x562fbae23790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbae213b0 0x562fbae23790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f59777fe700 1 -- 172.21.15.78:0/1809635859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59780ff680 con 0x562fbac086b0 2023-10-10T17:43:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.055+0000 7f5977fff700 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbae213b0 0x562fbae23790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:04.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f59777fe700 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbae18530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5968011b20 tx=0x7f596801c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67bcaf5c0a48773f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f5956ffd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:43:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f5956ffd700 1 -- 172.21.15.78:0/1809635859 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968025040 con 0x562fbac086b0 2023-10-10T17:43:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f5956ffd700 1 -- 172.21.15.78:0/1809635859 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f59680253a0 con 0x562fbac086b0 2023-10-10T17:43:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5970099740 con 0x562fbac086b0 2023-10-10T17:43:04.066 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f5956ffd700 1 -- 172.21.15.78:0/1809635859 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5968025560 con 0x562fbac086b0 2023-10-10T17:43:04.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f5989ca9b80 10 client.25363 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:04.067 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.056+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562fbacc4130 con 0x562fbac086b0 2023-10-10T17:43:04.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5956ffd700 1 -- 172.21.15.78:0/1809635859 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5970099740 con 0x562fbac086b0 2023-10-10T17:43:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5956ffd700 1 -- 172.21.15.78:0/1809635859 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5968025ce0 con 0x562fbac086b0 2023-10-10T17:43:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5989ca9b80 10 client.25363 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5989ca9b80 10 client.25363 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5989ca9b80 20 client.25363 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:04.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbae25680 0x562fbae27a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5989ca9b80 4 client.25363 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:43:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562fbaccc600 con 0x562fbae25680 2023-10-10T17:43:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.057+0000 7f5976ffd700 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbae25680 0x562fbae27a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.058+0000 7f5956ffd700 10 client.25363 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:04.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.058+0000 7f5976ffd700 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbae25680 0x562fbae27a90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f5f7b9b6ddaaf48 server_cookie=f2ab56ea5304ea57 in_seq=0 out_seq=0 2023-10-10T17:43:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.064+0000 7f5956ffd700 1 -- 172.21.15.78:0/1809635859 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562fbaccc600 con 0x562fbae25680 2023-10-10T17:43:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.064+0000 7f5956ffd700 10 client.25363 handle_command_reply: tid=0 2023-10-10T17:43:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.064+0000 7f5989ca9b80 1 client.25363 shutdown 2023-10-10T17:43:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.064+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 msgr2=0x562fbae18530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.064+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbae18530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5968011b20 tx=0x7f596801c000 comp rx=0 tx=0).stop 2023-10-10T17:43:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 shutdown_connections 2023-10-10T17:43:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fbac086b0 0x562fbae18530 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fbae25680 0x562fbae27a90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fbae213b0 0x562fbae23790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 --2- 172.21.15.78:0/1809635859 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fbadb2720 0x562fbae18a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5956ffd700 0 client.25363 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 >> 172.21.15.78:0/1809635859 conn(0x562fbada1f50 msgr2=0x562fbadb0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 shutdown_connections 2023-10-10T17:43:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 1 -- 172.21.15.78:0/1809635859 wait complete. 2023-10-10T17:43:04.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f5989ca9b80 20 client.25363 trim_cache size 0 max 16384 2023-10-10T17:43:04.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:43:04.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:43:04.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:43:04.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:43:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f59600a13e0 msgr2=0x7f59600a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f597db1d700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f59600a13e0 0x7f59600a3890 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5968009c70 tx=0x7f5968011720 comp rx=0 tx=0).stop 2023-10-10T17:43:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978109b80 msgr2=0x7f597819ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f597db1d700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978109b80 0x7f597819ad70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f59700094f0 tx=0x7f597000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 shutdown_connections 2023-10-10T17:43:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f597db1d700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f59781005d0 0x7f59781963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.065+0000 7f597db1d700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f59600a13e0 0x7f59600a3890 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.066+0000 7f597db1d700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5978109b80 0x7f597819ad70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.066+0000 7f597db1d700 1 --2- 172.21.15.78:0/2167345988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5978100ee0 0x7f5978196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.066+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 >> 172.21.15.78:0/2167345988 conn(0x7f5978074a50 msgr2=0x7f59781015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.066+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 shutdown_connections 2023-10-10T17:43:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:04.066+0000 7f597db1d700 1 -- 172.21.15.78:0/2167345988 wait complete. 2023-10-10T17:43:04.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:04 smithi157 ceph-mon[100896]: pgmap v17: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:05.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:04 smithi078 ceph-mon[99494]: pgmap v17: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:04 smithi190 ceph-mon[101220]: pgmap v17: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:06.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:06 smithi157 ceph-mon[100896]: pgmap v18: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-10-10T17:43:06.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:06 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:43:07.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:06 smithi078 ceph-mon[99494]: pgmap v18: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-10-10T17:43:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:06 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:43:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:06 smithi190 ceph-mon[101220]: pgmap v18: 129 pgs: 129 active+clean; 1.2 MiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2023-10-10T17:43:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:06 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:43:08.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:08 smithi157 ceph-mon[100896]: pgmap v19: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:09.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:08 smithi078 ceph-mon[99494]: pgmap v19: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:08 smithi190 ceph-mon[101220]: pgmap v19: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:09.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:09 smithi157 ceph-mon[100896]: pgmap v20: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-10-10T17:43:10.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:09 smithi078 ceph-mon[99494]: pgmap v20: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-10-10T17:43:10.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:09 smithi190 ceph-mon[101220]: pgmap v20: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-10-10T17:43:12.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:12 smithi078 ceph-mon[99494]: pgmap v21: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-10-10T17:43:12.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:12 smithi190 ceph-mon[101220]: pgmap v21: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-10-10T17:43:12.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:12 smithi157 ceph-mon[100896]: pgmap v21: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 23 KiB/s wr, 18 op/s 2023-10-10T17:43:14.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:14 smithi078 ceph-mon[99494]: pgmap v22: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:14.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:14 smithi190 ceph-mon[101220]: pgmap v22: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:14.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:14 smithi157 ceph-mon[100896]: pgmap v22: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 21 op/s 2023-10-10T17:43:16.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:16 smithi078 ceph-mon[99494]: pgmap v23: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-10-10T17:43:16.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:16 smithi190 ceph-mon[101220]: pgmap v23: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-10-10T17:43:16.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:16 smithi157 ceph-mon[100896]: pgmap v23: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-10-10T17:43:17.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 -- 172.21.15.78:0/1760097949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38073740 msgr2=0x7f8c3810c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/1760097949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38073740 0x7f8c3810c630 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c009990 tx=0x7f8c2c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 -- 172.21.15.78:0/1760097949 shutdown_connections 2023-10-10T17:43:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/1760097949 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c38072e30 0x7f8c38073200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.181 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/1760097949 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c3810cb70 0x7f8c3810ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/1760097949 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38073740 0x7f8c3810c630 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 -- 172.21.15.78:0/1760097949 >> 172.21.15.78:0/1760097949 conn(0x7f8c380781e0 msgr2=0x7f8c380785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:17.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 -- 172.21.15.78:0/1760097949 shutdown_connections 2023-10-10T17:43:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.179+0000 7f8c3e16a700 1 -- 172.21.15.78:0/1760097949 wait complete. 2023-10-10T17:43:17.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.180+0000 7f8c3e16a700 1 Processor -- start 2023-10-10T17:43:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.180+0000 7f8c3e16a700 1 -- start start 2023-10-10T17:43:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38072e30 0x7f8c381985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c38073740 0x7f8c38198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c3810cb70 0x7f8c3819cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3819d590 con 0x7f8c38072e30 2023-10-10T17:43:17.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3819d700 con 0x7f8c3810cb70 2023-10-10T17:43:17.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c3819d8c0 con 0x7f8c38073740 2023-10-10T17:43:17.196 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38072e30 0x7f8c381985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38072e30 0x7f8c381985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45052/0 (socket says 172.21.15.78:45052) 2023-10-10T17:43:17.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 -- 172.21.15.78:0/3301334384 learned_addr learned my addr 172.21.15.78:0/3301334384 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:17.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c38073740 0x7f8c38198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 -- 172.21.15.78:0/3301334384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c38073740 msgr2=0x7f8c38198af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.197 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c38073740 0x7f8c38198af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c37fff700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c3810cb70 0x7f8c3819cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 -- 172.21.15.78:0/3301334384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c3810cb70 msgr2=0x7f8c3819cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c3810cb70 0x7f8c3819cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 -- 172.21.15.78:0/3301334384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2c0095f0 con 0x7f8c38072e30 2023-10-10T17:43:17.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c377fe700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38072e30 0x7f8c381985b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800d950 tx=0x7f8c2800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c34ff9700 1 -- 172.21.15.78:0/3301334384 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c28009940 con 0x7f8c38072e30 2023-10-10T17:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c34ff9700 1 -- 172.21.15.78:0/3301334384 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c28010460 con 0x7f8c38072e30 2023-10-10T17:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c381124a0 con 0x7f8c38072e30 2023-10-10T17:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c34ff9700 1 -- 172.21.15.78:0/3301334384 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2800f620 con 0x7f8c38072e30 2023-10-10T17:43:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.181+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c38112310 con 0x7f8c38072e30 2023-10-10T17:43:17.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.183+0000 7f8c34ff9700 1 -- 172.21.15.78:0/3301334384 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f8c280108d0 con 0x7f8c38072e30 2023-10-10T17:43:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.184+0000 7f8c34ff9700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8c200a13e0 0x7f8c200a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.184+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8c200a13e0 0x7f8c200a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.184+0000 7f8c34ff9700 1 -- 172.21.15.78:0/3301334384 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8c280cd140 con 0x7f8c38072e30 2023-10-10T17:43:17.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8c200a13e0 0x7f8c200a3890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8c38199bc0 tx=0x7f8c2c005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:17.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 Processor -- start 2023-10-10T17:43:17.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 -- start start 2023-10-10T17:43:17.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3720 0x55d4f70ed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3b90 0x55d4f70edb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f6ff3000 con 0x55d4f70f3b90 2023-10-10T17:43:17.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f7045840 con 0x55d4f6f496b0 2023-10-10T17:43:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.185+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f703ecd0 con 0x55d4f70f3720 2023-10-10T17:43:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3b90 0x55d4f70edb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3b90 0x55d4f70edb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45064/0 (socket says 172.21.15.78:45064) 2023-10-10T17:43:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 -- 172.21.15.78:0/2864625333 learned_addr learned my addr 172.21.15.78:0/2864625333 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:17.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c377fe700 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3720 0x55d4f70ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 -- 172.21.15.78:0/2864625333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3720 msgr2=0x55d4f70ed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3720 0x55d4f70ed560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 -- 172.21.15.78:0/2864625333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 msgr2=0x55d4f70f31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 -- 172.21.15.78:0/2864625333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4f6ec05d0 con 0x55d4f70f3b90 2023-10-10T17:43:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c37fff700 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3b90 0x55d4f70edb00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009cd0 tx=0x7f8c30007500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d196d673c55ac102 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c1dffb700 1 -- 172.21.15.78:0/2864625333 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30014400 con 0x55d4f70f3b90 2023-10-10T17:43:17.206 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c1dffb700 1 -- 172.21.15.78:0/2864625333 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c30014a20 con 0x55d4f70f3b90 2023-10-10T17:43:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.186+0000 7f8c1dffb700 1 -- 172.21.15.78:0/2864625333 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c300139e0 con 0x55d4f70f3b90 2023-10-10T17:43:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/2864625333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3b90 msgr2=0x55d4f70edb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3b90 0x55d4f70edb00 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f8c30009cd0 tx=0x7f8c30007500 comp rx=0 tx=0).stop 2023-10-10T17:43:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/2864625333 shutdown_connections 2023-10-10T17:43:17.207 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3b90 0x55d4f70edb00 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.208 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/2864625333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3720 0x55d4f70ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/2864625333 >> 172.21.15.78:0/2864625333 conn(0x55d4f70e2f50 msgr2=0x55d4f70e8270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:17.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/2864625333 shutdown_connections 2023-10-10T17:43:17.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/2864625333 wait complete. 2023-10-10T17:43:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.187+0000 7f8c4a2f6b80 1 -- start start 2023-10-10T17:43:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 1 Processor -- start 2023-10-10T17:43:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f6f496b0 0x55d4f70f9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f70f3720 0x55d4f70f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70f7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f6fe0880 con 0x55d4f6f496b0 2023-10-10T17:43:17.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f6fe0520 con 0x55d4f70f3720 2023-10-10T17:43:17.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f6ec28c0 con 0x55d4f70f3b90 2023-10-10T17:43:17.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c4a2f6b80 10 client.0 _ll_register_callbacks cb 0x55d4f6ffe720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:17.211 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f6f496b0 0x55d4f70f9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f6f496b0 0x55d4f70f9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45066/0 (socket says 172.21.15.78:45066) 2023-10-10T17:43:17.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 -- 172.21.15.78:0/806250224 learned_addr learned my addr 172.21.15.78:0/806250224 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:17.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f70f3720 0x55d4f70f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c37fff700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.212 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 -- 172.21.15.78:0/806250224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 msgr2=0x55d4f70f7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70f7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 -- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f70f3720 msgr2=0x55d4f70f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f70f3720 0x55d4f70f6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 -- 172.21.15.78:0/806250224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c30009900 con 0x55d4f6f496b0 2023-10-10T17:43:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c37fff700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c377fe700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f6f496b0 0x55d4f70f9b30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8c28096740 tx=0x7f8c2809ddf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da24a008cc7e4dda server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:17.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.188+0000 7f8c16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:17.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c16ffd700 1 -- 172.21.15.78:0/806250224 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2809c070 con 0x55d4f6f496b0 2023-10-10T17:43:17.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c16ffd700 1 -- 172.21.15.78:0/806250224 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8c280991c0 con 0x55d4f6f496b0 2023-10-10T17:43:17.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4f70e0710 con 0x55d4f6f496b0 2023-10-10T17:43:17.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c16ffd700 1 -- 172.21.15.78:0/806250224 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c28099380 con 0x55d4f6f496b0 2023-10-10T17:43:17.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 10 client.15840 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d4f7005130 con 0x55d4f6f496b0 2023-10-10T17:43:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c16ffd700 1 -- 172.21.15.78:0/806250224 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d4f70e0710 con 0x55d4f6f496b0 2023-10-10T17:43:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c16ffd700 1 -- 172.21.15.78:0/806250224 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8c280ca700 con 0x55d4f6f496b0 2023-10-10T17:43:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 10 client.15840 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:17.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 10 client.15840 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 20 client.15840 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f7106fe0 0x55d4f71093c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 4 client.15840 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:17.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4f7001040 con 0x55d4f7106fe0 2023-10-10T17:43:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.189+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f7106fe0 0x55d4f71093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.190+0000 7f8c16ffd700 10 client.15840 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.190+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f7106fe0 0x55d4f71093c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b29a8352b206aab server_cookie=6e92ab4a040e1b0e in_seq=0 out_seq=0 2023-10-10T17:43:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.190+0000 7f8c16ffd700 1 -- 172.21.15.78:0/806250224 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d4f7001040 con 0x55d4f7106fe0 2023-10-10T17:43:17.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.190+0000 7f8c16ffd700 10 client.15840 handle_command_reply: tid=0 2023-10-10T17:43:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.190+0000 7f8c4a2f6b80 1 client.15840 shutdown 2023-10-10T17:43:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f6f496b0 msgr2=0x55d4f70f9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f6f496b0 0x55d4f70f9b30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8c28096740 tx=0x7f8c2809ddf0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 shutdown_connections 2023-10-10T17:43:17.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f7106fe0 0x55d4f71093c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f6f496b0 0x55d4f70f9b30 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c16ffd700 0 client.15840 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/806250224 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f70f3720 0x55d4f70f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 >> 172.21.15.78:0/806250224 conn(0x55d4f70e2f50 msgr2=0x55d4f70e77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 shutdown_connections 2023-10-10T17:43:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/806250224 wait complete. 2023-10-10T17:43:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.191+0000 7f8c4a2f6b80 20 client.15840 trim_cache size 0 max 16384 2023-10-10T17:43:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 Processor -- start 2023-10-10T17:43:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 -- start start 2023-10-10T17:43:17.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70ed560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f70f7a80 con 0x55d4f70f3720 2023-10-10T17:43:17.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f70e93e0 con 0x55d4f6f496b0 2023-10-10T17:43:17.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f70f19f0 con 0x55d4f70f3b90 2023-10-10T17:43:17.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.216+0000 7f8c36ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70e7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45074/0 (socket says 172.21.15.78:45074) 2023-10-10T17:43:17.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1640950381 learned_addr learned my addr 172.21.15.78:0/1640950381 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:17.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c377fe700 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c37fff700 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70ed560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1640950381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 msgr2=0x55d4f70ed560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70ed560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1640950381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 msgr2=0x55d4f70f6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1640950381 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4f70e46f0 con 0x55d4f70f3720 2023-10-10T17:43:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70e7f60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00c010 tx=0x7f8c2c020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4918b31ac505dcf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c1e7fc700 1 -- 172.21.15.78:0/1640950381 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c00b7d0 con 0x55d4f70f3720 2023-10-10T17:43:17.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c1e7fc700 1 -- 172.21.15.78:0/1640950381 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c2c01e040 con 0x55d4f70f3720 2023-10-10T17:43:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.217+0000 7f8c1e7fc700 1 -- 172.21.15.78:0/1640950381 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c005380 con 0x55d4f70f3720 2023-10-10T17:43:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1640950381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 msgr2=0x55d4f70e7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70e7f60 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c00c010 tx=0x7f8c2c020000 comp rx=0 tx=0).stop 2023-10-10T17:43:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1640950381 shutdown_connections 2023-10-10T17:43:17.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70ed560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70f6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1640950381 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70e7f60 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1640950381 >> 172.21.15.78:0/1640950381 conn(0x55d4f70e2f50 msgr2=0x55d4f70f65e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1640950381 shutdown_connections 2023-10-10T17:43:17.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1640950381 wait complete. 2023-10-10T17:43:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 -- start start 2023-10-10T17:43:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.218+0000 7f8c4a2f6b80 1 Processor -- start 2023-10-10T17:43:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70fa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70faba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70fb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f7159e80 con 0x55d4f70f3720 2023-10-10T17:43:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f7159ff0 con 0x55d4f6f496b0 2023-10-10T17:43:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4f715a1b0 con 0x55d4f70f3b90 2023-10-10T17:43:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 10 client.0 _ll_register_callbacks cb 0x7f8c28099cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:17.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70faba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70faba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45082/0 (socket says 172.21.15.78:45082) 2023-10-10T17:43:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1258344267 learned_addr learned my addr 172.21.15.78:0/1258344267 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c377fe700 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70fa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c37fff700 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70fb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1258344267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 msgr2=0x55d4f70fb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70fb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 msgr2=0x55d4f70fa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70fa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 -- 172.21.15.78:0/1258344267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c3810fb50 con 0x55d4f70f3720 2023-10-10T17:43:17.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c36ffd700 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70faba0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c003730 tx=0x7f8c2c00b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1888434ad2b208 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c16ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c16ffd700 1 -- 172.21.15.78:0/1258344267 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c2c032c60 con 0x55d4f70f3720 2023-10-10T17:43:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c16ffd700 1 -- 172.21.15.78:0/1258344267 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8c2c032dc0 con 0x55d4f70f3720 2023-10-10T17:43:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.219+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8c30014a20 con 0x55d4f70f3720 2023-10-10T17:43:17.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c16ffd700 1 -- 172.21.15.78:0/1258344267 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c2c005070 con 0x55d4f70f3720 2023-10-10T17:43:17.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 10 client.15846 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:17.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d4f6e6cd80 con 0x55d4f70f3720 2023-10-10T17:43:17.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c16ffd700 1 -- 172.21.15.78:0/1258344267 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8c30014a20 con 0x55d4f70f3720 2023-10-10T17:43:17.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c16ffd700 1 -- 172.21.15.78:0/1258344267 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8c2c047070 con 0x55d4f70f3720 2023-10-10T17:43:17.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 10 client.15846 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 10 client.15846 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 20 client.15846 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f715b3c0 0x55d4f715d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 4 client.15846 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:43:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d4f700d600 con 0x55d4f715b3c0 2023-10-10T17:43:17.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.220+0000 7f8c377fe700 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f715b3c0 0x55d4f715d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.221+0000 7f8c16ffd700 10 client.15846 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:17.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.221+0000 7f8c377fe700 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f715b3c0 0x55d4f715d7d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1500a885686d5e49 server_cookie=bca6c54e5f2f02e4 in_seq=0 out_seq=0 2023-10-10T17:43:17.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c16ffd700 1 -- 172.21.15.78:0/1258344267 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d4f700d600 con 0x55d4f715b3c0 2023-10-10T17:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c16ffd700 10 client.15846 handle_command_reply: tid=0 2023-10-10T17:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 client.15846 shutdown 2023-10-10T17:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 msgr2=0x55d4f70faba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70faba0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c003730 tx=0x7f8c2c00b7f0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 shutdown_connections 2023-10-10T17:43:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d4f70f3b90 0x55d4f70fb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d4f715b3c0 0x55d4f715d7d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d4f6f496b0 0x55d4f70fa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c16ffd700 0 client.15846 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:17.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 --2- 172.21.15.78:0/1258344267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d4f70f3720 0x55d4f70faba0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.227+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 >> 172.21.15.78:0/1258344267 conn(0x55d4f70e2f50 msgr2=0x55d4f70f2690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 shutdown_connections 2023-10-10T17:43:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c4a2f6b80 1 -- 172.21.15.78:0/1258344267 wait complete. 2023-10-10T17:43:17.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c4a2f6b80 20 client.15846 trim_cache size 0 max 16384 2023-10-10T17:43:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:43:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:43:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:43:17.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:43:17.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8c200a13e0 msgr2=0x7f8c200a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8c200a13e0 0x7f8c200a3890 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8c38199bc0 tx=0x7f8c2c005fd0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38072e30 msgr2=0x7f8c381985b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38072e30 0x7f8c381985b0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800d950 tx=0x7f8c2800dc60 comp rx=0 tx=0).stop 2023-10-10T17:43:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 shutdown_connections 2023-10-10T17:43:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8c200a13e0 0x7f8c200a3890 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8c3810cb70 0x7f8c3819cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8c38072e30 0x7f8c381985b0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 --2- 172.21.15.78:0/3301334384 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8c38073740 0x7f8c38198af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 >> 172.21.15.78:0/3301334384 conn(0x7f8c380781e0 msgr2=0x7f8c381045d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 shutdown_connections 2023-10-10T17:43:17.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:17.228+0000 7f8c3e16a700 1 -- 172.21.15.78:0/3301334384 wait complete. 2023-10-10T17:43:17.512 INFO:tasks.workunit.client.0.smithi078.stderr:Note: switching to '48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf'. 2023-10-10T17:43:17.512 INFO:tasks.workunit.client.0.smithi078.stderr: 2023-10-10T17:43:17.512 INFO:tasks.workunit.client.0.smithi078.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-10T17:43:17.512 INFO:tasks.workunit.client.0.smithi078.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-10T17:43:17.513 INFO:tasks.workunit.client.0.smithi078.stderr:state without impacting any branches by switching back to a branch. 2023-10-10T17:43:17.513 INFO:tasks.workunit.client.0.smithi078.stderr: 2023-10-10T17:43:17.513 INFO:tasks.workunit.client.0.smithi078.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-10T17:43:17.513 INFO:tasks.workunit.client.0.smithi078.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-10T17:43:17.513 INFO:tasks.workunit.client.0.smithi078.stderr: 2023-10-10T17:43:17.514 INFO:tasks.workunit.client.0.smithi078.stderr: git switch -c 2023-10-10T17:43:17.514 INFO:tasks.workunit.client.0.smithi078.stderr: 2023-10-10T17:43:17.514 INFO:tasks.workunit.client.0.smithi078.stderr:Or undo this operation with: 2023-10-10T17:43:17.514 INFO:tasks.workunit.client.0.smithi078.stderr: 2023-10-10T17:43:17.515 INFO:tasks.workunit.client.0.smithi078.stderr: git switch - 2023-10-10T17:43:17.515 INFO:tasks.workunit.client.0.smithi078.stderr: 2023-10-10T17:43:17.515 INFO:tasks.workunit.client.0.smithi078.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-10T17:43:17.515 INFO:tasks.workunit.client.0.smithi078.stderr: 2023-10-10T17:43:17.515 INFO:tasks.workunit.client.0.smithi078.stderr:HEAD is now at 48b68c107f2 doc/cephfs: Update docs to match remove functionality and respective vxattrs 2023-10-10T17:43:17.517 DEBUG:teuthology.orchestra.run.smithi078:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-10T17:43:17.549 INFO:tasks.workunit.client.0.smithi078.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-10T17:43:17.550 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-10T17:43:17.551 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-10T17:43:18.076 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-10T17:43:18.128 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-10T17:43:18.165 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-10T17:43:18.166 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-10T17:43:18.166 INFO:tasks.workunit.client.0.smithi078.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-10T17:43:18.202 INFO:tasks.workunit.client.0.smithi078.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-10T17:43:18.204 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:43:18.205 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-10T17:43:18.262 INFO:tasks.workunit:Running workunits matching fs/misc on client.0... 2023-10-10T17:43:18.265 INFO:tasks.workunit:Running workunit fs/misc/acl.sh... 2023-10-10T17:43:18.266 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/acl.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/acl.sh 2023-10-10T17:43:18.325 INFO:tasks.workunit.client.0.smithi078.stderr:+ set -e 2023-10-10T17:43:18.326 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir -p testdir 2023-10-10T17:43:18.328 INFO:tasks.workunit.client.0.smithi078.stderr:+ cd testdir 2023-10-10T17:43:18.328 INFO:tasks.workunit.client.0.smithi078.stderr:+ set +e 2023-10-10T17:43:18.328 INFO:tasks.workunit.client.0.smithi078.stderr:+ setfacl -d -m u:nobody:rw . 2023-10-10T17:43:18.337 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 0 '!=' 0 2023-10-10T17:43:18.337 INFO:tasks.workunit.client.0.smithi078.stderr:+ set -e 2023-10-10T17:43:18.337 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=0 2023-10-10T17:43:18.338 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 0 -lt 100 ']' 2023-10-10T17:43:18.338 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 0 + 1 2023-10-10T17:43:18.338 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=1 2023-10-10T17:43:18.339 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:18.341 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:18.341 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:18.345 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:18.345 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:18.345 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:18.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:18 smithi078 ceph-mon[99494]: pgmap v24: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2023-10-10T17:43:18.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:18 smithi190 ceph-mon[101220]: pgmap v24: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2023-10-10T17:43:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:18 smithi157 ceph-mon[100896]: pgmap v24: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2023-10-10T17:43:19.066 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:19.066 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:19.253 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:19.253 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:19.270 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:19.270 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:19.270 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 1 -lt 100 ']' 2023-10-10T17:43:19.270 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 1 + 1 2023-10-10T17:43:19.306 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=2 2023-10-10T17:43:19.306 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:19.325 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:19.326 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:19.327 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:19.327 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:19.327 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:19.774 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:19.775 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:19.894 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:19.895 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:19.903 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:19.903 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:19.904 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -lt 100 ']' 2023-10-10T17:43:19.904 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 2 + 1 2023-10-10T17:43:19.934 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=3 2023-10-10T17:43:19.934 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:19.960 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:19.961 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:19.961 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:19.961 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:19.962 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:20.274 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:20.274 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:20.461 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:20.461 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:20.489 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:20.490 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:20.490 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 3 -lt 100 ']' 2023-10-10T17:43:20.490 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 3 + 1 2023-10-10T17:43:20.517 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=4 2023-10-10T17:43:20.517 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:20.543 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:20.544 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:20.544 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:20.545 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:20.545 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:20.859 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:20 smithi078 ceph-mon[99494]: pgmap v25: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2023-10-10T17:43:20.899 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:20.899 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:20.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:20 smithi190 ceph-mon[101220]: pgmap v25: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2023-10-10T17:43:20.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:20 smithi157 ceph-mon[100896]: pgmap v25: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2023-10-10T17:43:21.128 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:21.128 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:21.136 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:21.136 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:21.137 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 4 -lt 100 ']' 2023-10-10T17:43:21.137 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 4 + 1 2023-10-10T17:43:21.166 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=5 2023-10-10T17:43:21.167 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:21.217 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:21.217 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:21.218 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:21.218 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:21.219 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:21.582 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:21.583 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:21.802 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:21.803 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:21.810 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:21.810 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:21.810 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 5 -lt 100 ']' 2023-10-10T17:43:21.811 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 5 + 1 2023-10-10T17:43:21.833 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=6 2023-10-10T17:43:21.833 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:21.859 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:21.859 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:21.860 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:21.860 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:21.860 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:22.282 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:22.283 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:22.435 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:22.436 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:22.444 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:22.444 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:22.444 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 6 -lt 100 ']' 2023-10-10T17:43:22.445 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 6 + 1 2023-10-10T17:43:22.466 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=7 2023-10-10T17:43:22.467 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:22.497 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:22.497 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:22.498 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:22.498 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:22.498 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:22.874 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:22.874 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:22.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:22 smithi078 ceph-mon[99494]: pgmap v26: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2023-10-10T17:43:22.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:22 smithi190 ceph-mon[101220]: pgmap v26: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2023-10-10T17:43:22.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:22 smithi157 ceph-mon[100896]: pgmap v26: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 5 op/s 2023-10-10T17:43:23.010 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:23.011 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:23.018 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:23.018 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:23.019 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 7 -lt 100 ']' 2023-10-10T17:43:23.019 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 7 + 1 2023-10-10T17:43:23.041 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=8 2023-10-10T17:43:23.041 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:23.067 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:23.067 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:23.068 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:23.068 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:23.069 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:23.382 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:23.382 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:23.519 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:23.519 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:23.527 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:23.527 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:23.528 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 8 -lt 100 ']' 2023-10-10T17:43:23.528 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 8 + 1 2023-10-10T17:43:23.550 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=9 2023-10-10T17:43:23.550 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:23.575 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:23.575 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:23.576 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:23.576 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:23.576 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:23.974 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:23.974 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:24.110 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:24.111 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:24.118 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:24.119 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:24.119 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 9 -lt 100 ']' 2023-10-10T17:43:24.119 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 9 + 1 2023-10-10T17:43:24.141 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=10 2023-10-10T17:43:24.142 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:24.167 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:24.168 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:24.169 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:24.169 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:24.170 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:24.582 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:24.582 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:24.744 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:24.744 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:24.752 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:24.752 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:24.752 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 10 -lt 100 ']' 2023-10-10T17:43:24.753 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 10 + 1 2023-10-10T17:43:24.775 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=11 2023-10-10T17:43:24.775 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:24.804 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:24.805 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:24.806 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:24.806 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:24.806 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:24.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:24 smithi078 ceph-mon[99494]: pgmap v27: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2023-10-10T17:43:24.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:24 smithi190 ceph-mon[101220]: pgmap v27: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2023-10-10T17:43:24.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:24 smithi157 ceph-mon[100896]: pgmap v27: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.4 KiB/s wr, 9 op/s 2023-10-10T17:43:25.182 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:25.182 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:25.318 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:25.319 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:25.335 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:25.335 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:25.335 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 11 -lt 100 ']' 2023-10-10T17:43:25.336 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 11 + 1 2023-10-10T17:43:25.371 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=12 2023-10-10T17:43:25.372 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:25.398 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:25.398 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:25.399 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:25.399 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:25.399 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:25.858 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:25.858 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:26.002 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:26.002 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:26.018 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:26.018 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:26.019 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 12 -lt 100 ']' 2023-10-10T17:43:26.019 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 12 + 1 2023-10-10T17:43:26.041 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=13 2023-10-10T17:43:26.041 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:26.068 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:26.068 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:26.069 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:26.069 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:26.069 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:26.457 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:26.457 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:26.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:26 smithi078 ceph-mon[99494]: pgmap v28: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2023-10-10T17:43:26.635 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:26.635 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:26.643 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:26.643 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:26.644 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 13 -lt 100 ']' 2023-10-10T17:43:26.644 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 13 + 1 2023-10-10T17:43:26.666 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=14 2023-10-10T17:43:26.666 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:26.692 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:26.692 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:26.693 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:26.693 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:26.693 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:26.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:26 smithi190 ceph-mon[101220]: pgmap v28: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2023-10-10T17:43:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:26 smithi157 ceph-mon[100896]: pgmap v28: 129 pgs: 129 active+clean; 1.2 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2023-10-10T17:43:27.223 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:27.224 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:27.377 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:27.377 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:27.385 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:27.385 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:27.386 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 14 -lt 100 ']' 2023-10-10T17:43:27.386 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 14 + 1 2023-10-10T17:43:27.407 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=15 2023-10-10T17:43:27.407 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:27.458 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:27.458 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:27.459 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:27.459 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:27.459 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:27.881 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:27.882 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:28.018 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:28.019 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:28.035 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:28.035 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:28.035 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 15 -lt 100 ']' 2023-10-10T17:43:28.035 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 15 + 1 2023-10-10T17:43:28.079 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=16 2023-10-10T17:43:28.079 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:28.134 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:28.134 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:28.135 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:28.135 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:28.135 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:28.523 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:28.523 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:28.676 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:28.677 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:28.693 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:28.693 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:28.694 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 16 -lt 100 ']' 2023-10-10T17:43:28.694 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 16 + 1 2023-10-10T17:43:28.716 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=17 2023-10-10T17:43:28.716 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:28.744 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:28.745 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:28.745 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:28.745 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:28.745 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:28.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:28 smithi078 ceph-mon[99494]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 10 KiB/s wr, 9 op/s 2023-10-10T17:43:28.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:28 smithi190 ceph-mon[101220]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 10 KiB/s wr, 9 op/s 2023-10-10T17:43:28.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:28 smithi157 ceph-mon[100896]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 10 KiB/s wr, 9 op/s 2023-10-10T17:43:29.114 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:29.115 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:29.234 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:29.235 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:29.243 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:29.243 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:29.243 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 17 -lt 100 ']' 2023-10-10T17:43:29.244 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 17 + 1 2023-10-10T17:43:29.265 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=18 2023-10-10T17:43:29.266 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:29.297 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:29.297 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:29.298 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:29.298 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:29.298 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:29.665 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:29.665 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:29.810 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:29.810 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:29.818 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:29.819 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:29.819 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 18 -lt 100 ']' 2023-10-10T17:43:29.819 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 18 + 1 2023-10-10T17:43:29.849 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=19 2023-10-10T17:43:29.849 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:29.883 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:29.883 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:29.884 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:29.884 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:29.884 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:30.348 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:30.348 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:30.468 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:30.468 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:30.476 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:30.476 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:30.476 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 19 -lt 100 ']' 2023-10-10T17:43:30.476 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 19 + 1 2023-10-10T17:43:30.512 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=20 2023-10-10T17:43:30.512 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:30.541 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:30.541 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:30.542 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:30.542 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:30.542 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:30.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:30 smithi078 ceph-mon[99494]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-10-10T17:43:30.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:30 smithi190 ceph-mon[101220]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-10-10T17:43:30.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:30 smithi157 ceph-mon[100896]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-10-10T17:43:31.156 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:31.156 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:31.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:43:31.378 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:31.379 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:31.384 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:31.385 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:31.385 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 20 -lt 100 ']' 2023-10-10T17:43:31.385 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 20 + 1 2023-10-10T17:43:31.462 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=21 2023-10-10T17:43:31.462 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:31.470 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:31.471 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:31.471 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:31.471 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:31.471 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:32.381 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:32.382 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:32.512 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:32.512 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:32.535 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:32.535 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:32.535 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 21 -lt 100 ']' 2023-10-10T17:43:32.536 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 21 + 1 2023-10-10T17:43:32.595 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=22 2023-10-10T17:43:32.596 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:32.691 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:32.691 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:32.692 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:32.692 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:32.692 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:32.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:32 smithi078 ceph-mon[99494]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-10-10T17:43:32.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:32 smithi190 ceph-mon[101220]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-10-10T17:43:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 -- 172.21.15.78:0/2442991747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b600ffdd0 msgr2=0x7f9b60103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:32.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 --2- 172.21.15.78:0/2442991747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b600ffdd0 0x7f9b60103d90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9b480099c0 tx=0x7f9b48009cd0 comp rx=0 tx=0).stop 2023-10-10T17:43:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 -- 172.21.15.78:0/2442991747 shutdown_connections 2023-10-10T17:43:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 --2- 172.21.15.78:0/2442991747 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b60104470 0x7f9b6010c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 --2- 172.21.15.78:0/2442991747 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b600ff430 0x7f9b600ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 --2- 172.21.15.78:0/2442991747 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b600ffdd0 0x7f9b60103d90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:32.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 -- 172.21.15.78:0/2442991747 >> 172.21.15.78:0/2442991747 conn(0x7f9b60076370 msgr2=0x7f9b60076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 -- 172.21.15.78:0/2442991747 shutdown_connections 2023-10-10T17:43:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.967+0000 7f9b653e2700 1 -- 172.21.15.78:0/2442991747 wait complete. 2023-10-10T17:43:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 Processor -- start 2023-10-10T17:43:32.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 -- start start 2023-10-10T17:43:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b600ff430 0x7f9b601971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b600ffdd0 0x7f9b601976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:32.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b60104470 0x7f9b601a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b60197ff0 con 0x7f9b60104470 2023-10-10T17:43:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b601a2760 con 0x7f9b600ff430 2023-10-10T17:43:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b653e2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b601a28d0 con 0x7f9b600ffdd0 2023-10-10T17:43:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b60104470 0x7f9b601a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b60104470 0x7f9b601a2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45100/0 (socket says 172.21.15.78:45100) 2023-10-10T17:43:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1734696804 learned_addr learned my addr 172.21.15.78:0/1734696804 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.968+0000 7f9b5effd700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b600ff430 0x7f9b601971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b567fc700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b600ffdd0 0x7f9b601976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1734696804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b600ffdd0 msgr2=0x7f9b601976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b600ffdd0 0x7f9b601976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:32.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1734696804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b600ff430 msgr2=0x7f9b601971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b600ff430 0x7f9b601971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1734696804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b480095f0 con 0x7f9b60104470 2023-10-10T17:43:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b567fc700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b600ffdd0 0x7f9b601976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5effd700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b600ff430 0x7f9b601971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:32.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b60104470 0x7f9b601a2220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9b58007ee0 tx=0x7f9b5800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5cff9700 1 -- 172.21.15.78:0/1734696804 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5800ddd0 con 0x7f9b60104470 2023-10-10T17:43:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5cff9700 1 -- 172.21.15.78:0/1734696804 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b5800f040 con 0x7f9b60104470 2023-10-10T17:43:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6010f1f0 con 0x7f9b60104470 2023-10-10T17:43:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b5cff9700 1 -- 172.21.15.78:0/1734696804 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b58015400 con 0x7f9b60104470 2023-10-10T17:43:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.969+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6010f060 con 0x7f9b60104470 2023-10-10T17:43:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.971+0000 7f9b5cff9700 1 -- 172.21.15.78:0/1734696804 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f9b58004020 con 0x7f9b60104470 2023-10-10T17:43:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.972+0000 7f9b5cff9700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b4c0a13e0 0x7f9b4c0a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.972+0000 7f9b5cff9700 1 -- 172.21.15.78:0/1734696804 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f9b580ccc10 con 0x7f9b60104470 2023-10-10T17:43:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.972+0000 7f9b5effd700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b4c0a13e0 0x7f9b4c0a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:32.972+0000 7f9b5effd700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b4c0a13e0 0x7f9b4c0a3890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b50005950 tx=0x7f9b500058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:32.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:32 smithi157 ceph-mon[100896]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-10-10T17:43:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.053+0000 7f9b7156eb80 1 Processor -- start 2023-10-10T17:43:33.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b7156eb80 1 -- start start 2023-10-10T17:43:33.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54ea9a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.057 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea9ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54e9a0000 con 0x55f54eaa0b90 2023-10-10T17:43:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54e9f2840 con 0x55f54e8f66b0 2023-10-10T17:43:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54e9ebcd0 con 0x55f54eaa0720 2023-10-10T17:43:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea9ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea9ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45108/0 (socket says 172.21.15.78:45108) 2023-10-10T17:43:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/782129008 learned_addr learned my addr 172.21.15.78:0/782129008 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5effd700 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b567fc700 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54ea9a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/782129008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 msgr2=0x55f54ea9a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54ea9a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/782129008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 msgr2=0x55f54eaa01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.054+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.055+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/782129008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f54e86d5d0 con 0x55f54eaa0b90 2023-10-10T17:43:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.055+0000 7f9b567fc700 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54ea9a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:33.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.055+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea9ab00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9b580ca670 tx=0x7f9b58095c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad33148f344bdd2f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.055+0000 7f9b557fa700 1 -- 172.21.15.78:0/782129008 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b580997a0 con 0x55f54eaa0b90 2023-10-10T17:43:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.055+0000 7f9b557fa700 1 -- 172.21.15.78:0/782129008 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b5809d7d0 con 0x55f54eaa0b90 2023-10-10T17:43:33.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.055+0000 7f9b557fa700 1 -- 172.21.15.78:0/782129008 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b58098c90 con 0x55f54eaa0b90 2023-10-10T17:43:33.068 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 -- 172.21.15.78:0/782129008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 msgr2=0x55f54ea9ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea9ab00 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9b580ca670 tx=0x7f9b58095c00 comp rx=0 tx=0).stop 2023-10-10T17:43:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 -- 172.21.15.78:0/782129008 shutdown_connections 2023-10-10T17:43:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea9ab00 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.069 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/782129008 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54ea9a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 -- 172.21.15.78:0/782129008 >> 172.21.15.78:0/782129008 conn(0x55f54ea8ff50 msgr2=0x55f54ea95270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 -- 172.21.15.78:0/782129008 shutdown_connections 2023-10-10T17:43:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 -- 172.21.15.78:0/782129008 wait complete. 2023-10-10T17:43:33.070 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 -- start start 2023-10-10T17:43:33.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 Processor -- start 2023-10-10T17:43:33.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54e8f66b0 0x55f54eaa6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0720 0x55f54eaa3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.056+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54eaa0b90 0x55f54eaa43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54e98d880 con 0x55f54eaa0720 2023-10-10T17:43:33.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54e98d520 con 0x55f54eaa0b90 2023-10-10T17:43:33.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54e86f8c0 con 0x55f54e8f66b0 2023-10-10T17:43:33.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b7156eb80 10 client.0 _ll_register_callbacks cb 0x55f54e9ab720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:33.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0720 0x55f54eaa3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0720 0x55f54eaa3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45110/0 (socket says 172.21.15.78:45110) 2023-10-10T17:43:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 -- 172.21.15.78:0/2496900151 learned_addr learned my addr 172.21.15.78:0/2496900151 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54eaa0b90 0x55f54eaa43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b5effd700 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54e8f66b0 0x55f54eaa6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 -- 172.21.15.78:0/2496900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54e8f66b0 msgr2=0x55f54eaa6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54e8f66b0 0x55f54eaa6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 -- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54eaa0b90 msgr2=0x55f54eaa43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54eaa0b90 0x55f54eaa43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 -- 172.21.15.78:0/2496900151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6010f060 con 0x55f54eaa0720 2023-10-10T17:43:33.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b567fc700 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0720 0x55f54eaa3d00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800b580 tx=0x7f9b480057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d947222a3515786 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b467fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b467fc700 1 -- 172.21.15.78:0/2496900151 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4800bba0 con 0x55f54eaa0720 2023-10-10T17:43:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b467fc700 1 -- 172.21.15.78:0/2496900151 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f9b4800bd00 con 0x55f54eaa0720 2023-10-10T17:43:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f54ea8d710 con 0x55f54eaa0720 2023-10-10T17:43:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b467fc700 1 -- 172.21.15.78:0/2496900151 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b4800bee0 con 0x55f54eaa0720 2023-10-10T17:43:33.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.057+0000 7f9b7156eb80 10 client.15858 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f54e9b2130 con 0x55f54eaa0720 2023-10-10T17:43:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b467fc700 1 -- 172.21.15.78:0/2496900151 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f54ea8d710 con 0x55f54eaa0720 2023-10-10T17:43:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b467fc700 1 -- 172.21.15.78:0/2496900151 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9b480046c0 con 0x55f54eaa0720 2023-10-10T17:43:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b7156eb80 10 client.15858 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:33.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b7156eb80 10 client.15858 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b7156eb80 20 client.15858 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eab3e70 0x55f54eab6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b7156eb80 4 client.15858 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f54e9ae040 con 0x55f54eab3e70 2023-10-10T17:43:33.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.058+0000 7f9b5effd700 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eab3e70 0x55f54eab6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b467fc700 10 client.15858 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b5effd700 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eab3e70 0x55f54eab6280 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12d8109c02f4f613 server_cookie=1e8c239a8abd4bb in_seq=0 out_seq=0 2023-10-10T17:43:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b467fc700 1 -- 172.21.15.78:0/2496900151 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f54e9ae040 con 0x55f54eab3e70 2023-10-10T17:43:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b467fc700 10 client.15858 handle_command_reply: tid=0 2023-10-10T17:43:33.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b7156eb80 1 client.15858 shutdown 2023-10-10T17:43:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0720 msgr2=0x55f54eaa3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0720 0x55f54eaa3d00 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800b580 tx=0x7f9b480057d0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 shutdown_connections 2023-10-10T17:43:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54e8f66b0 0x55f54eaa6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.059+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eab3e70 0x55f54eab6280 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.060+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54eaa0b90 0x55f54eaa43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.060+0000 7f9b467fc700 0 client.15858 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.060+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/2496900151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0720 0x55f54eaa3d00 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.060+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 >> 172.21.15.78:0/2496900151 conn(0x55f54ea8ff50 msgr2=0x55f54ea947e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:33.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.060+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 shutdown_connections 2023-10-10T17:43:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.060+0000 7f9b7156eb80 1 -- 172.21.15.78:0/2496900151 wait complete. 2023-10-10T17:43:33.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.060+0000 7f9b7156eb80 20 client.15858 trim_cache size 0 max 16384 2023-10-10T17:43:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.074+0000 7f9b7156eb80 1 Processor -- start 2023-10-10T17:43:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.074+0000 7f9b7156eb80 1 -- start start 2023-10-10T17:43:33.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54ea9a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea94f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54ea963e0 con 0x55f54eaa0b90 2023-10-10T17:43:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54ea96c50 con 0x55f54e8f66b0 2023-10-10T17:43:33.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54eaa01e0 con 0x55f54eaa0720 2023-10-10T17:43:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea94f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea94f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45122/0 (socket says 172.21.15.78:45122) 2023-10-10T17:43:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1757427935 learned_addr learned my addr 172.21.15.78:0/1757427935 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:33.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b567fc700 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1757427935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 msgr2=0x55f54eaa3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5effd700 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54ea9a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1757427935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 msgr2=0x55f54ea9a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54ea9a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/1757427935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f54e9b1e80 con 0x55f54eaa0b90 2023-10-10T17:43:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b567fc700 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea94f60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9b580996e0 tx=0x7f9b58084040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9455c0344226d060 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.075+0000 7f9b55ffb700 1 -- 172.21.15.78:0/1757427935 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b580997a0 con 0x55f54eaa0b90 2023-10-10T17:43:33.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b55ffb700 1 -- 172.21.15.78:0/1757427935 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b580a1bf0 con 0x55f54eaa0b90 2023-10-10T17:43:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b55ffb700 1 -- 172.21.15.78:0/1757427935 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b58099c60 con 0x55f54eaa0b90 2023-10-10T17:43:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 -- 172.21.15.78:0/1757427935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 msgr2=0x55f54ea94f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea94f60 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f9b580996e0 tx=0x7f9b58084040 comp rx=0 tx=0).stop 2023-10-10T17:43:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 -- 172.21.15.78:0/1757427935 shutdown_connections 2023-10-10T17:43:33.091 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54ea94f60 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54ea9a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/1757427935 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 -- 172.21.15.78:0/1757427935 >> 172.21.15.78:0/1757427935 conn(0x55f54ea8ff50 msgr2=0x55f54eaa35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 -- 172.21.15.78:0/1757427935 shutdown_connections 2023-10-10T17:43:33.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.076+0000 7f9b7156eb80 1 -- 172.21.15.78:0/1757427935 wait complete. 2023-10-10T17:43:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 -- start start 2023-10-10T17:43:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 Processor -- start 2023-10-10T17:43:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54eb06d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54eaa8390 con 0x55f54eaa0b90 2023-10-10T17:43:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54eb07270 con 0x55f54e8f66b0 2023-10-10T17:43:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54eb073b0 con 0x55f54eaa0720 2023-10-10T17:43:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b7156eb80 10 client.0 _ll_register_callbacks cb 0x7f9b48009810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:33.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54eb06d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b5f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54eb06d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45128/0 (socket says 172.21.15.78:45128) 2023-10-10T17:43:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/3755021983 learned_addr learned my addr 172.21.15.78:0/3755021983 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b5effd700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.077+0000 7f9b567fc700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/3755021983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 msgr2=0x55f54eaa7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 msgr2=0x55f54eaa7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b5f7fe700 1 -- 172.21.15.78:0/3755021983 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f54e86d5d0 con 0x55f54eaa0b90 2023-10-10T17:43:33.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b5effd700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b5f7fe700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54eb06d30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9b58004460 tx=0x7f9b580b9280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa788f55f8352ce9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b467fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b467fc700 1 -- 172.21.15.78:0/3755021983 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b580997a0 con 0x55f54eaa0b90 2023-10-10T17:43:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b467fc700 1 -- 172.21.15.78:0/3755021983 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f9b58099c10 con 0x55f54eaa0b90 2023-10-10T17:43:33.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b5809d7d0 con 0x55f54eaa0b90 2023-10-10T17:43:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b467fc700 1 -- 172.21.15.78:0/3755021983 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b58004bd0 con 0x55f54eaa0b90 2023-10-10T17:43:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b7156eb80 10 client.15864 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55f54e802c00 con 0x55f54eaa0b90 2023-10-10T17:43:33.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b467fc700 1 -- 172.21.15.78:0/3755021983 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b5809d7d0 con 0x55f54eaa0b90 2023-10-10T17:43:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b467fc700 1 -- 172.21.15.78:0/3755021983 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9b580ca340 con 0x55f54eaa0b90 2023-10-10T17:43:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b7156eb80 10 client.15864 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b7156eb80 10 client.15864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b7156eb80 20 client.15864 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:33.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.078+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eb08580 0x55f54eb0a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.079+0000 7f9b7156eb80 4 client.15864 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:43:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.079+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f54e9ba600 con 0x55f54eb08580 2023-10-10T17:43:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.079+0000 7f9b5effd700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eb08580 0x55f54eb0a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.079+0000 7f9b467fc700 10 client.15864 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:33.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.079+0000 7f9b5effd700 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eb08580 0x55f54eb0a960 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cbd853909f675ac server_cookie=19b2ea2e7b8dd8f4 in_seq=0 out_seq=0 2023-10-10T17:43:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b467fc700 1 -- 172.21.15.78:0/3755021983 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f54e9ba600 con 0x55f54eb08580 2023-10-10T17:43:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b467fc700 10 client.15864 handle_command_reply: tid=0 2023-10-10T17:43:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 client.15864 shutdown 2023-10-10T17:43:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 msgr2=0x55f54eb06d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:33.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54eb06d30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9b58004460 tx=0x7f9b580b9280 comp rx=0 tx=0).stop 2023-10-10T17:43:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 shutdown_connections 2023-10-10T17:43:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f54eb08580 0x55f54eb0a960 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f54eaa0b90 0x55f54eb06d30 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f54e8f66b0 0x55f54eaa7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b467fc700 0 client.15864 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 --2- 172.21.15.78:0/3755021983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f54eaa0720 0x55f54eaa7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 >> 172.21.15.78:0/3755021983 conn(0x55f54ea8ff50 msgr2=0x55f54ea9ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 shutdown_connections 2023-10-10T17:43:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.097+0000 7f9b7156eb80 1 -- 172.21.15.78:0/3755021983 wait complete. 2023-10-10T17:43:33.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b7156eb80 20 client.15864 trim_cache size 0 max 16384 2023-10-10T17:43:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:43:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:43:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:43:33.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:43:33.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b4c0a13e0 msgr2=0x7f9b4c0a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b4c0a13e0 0x7f9b4c0a3890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b50005950 tx=0x7f9b500058e0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b60104470 msgr2=0x7f9b601a2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b60104470 0x7f9b601a2220 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f9b58007ee0 tx=0x7f9b5800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 shutdown_connections 2023-10-10T17:43:33.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b4c0a13e0 0x7f9b4c0a3890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b60104470 0x7f9b601a2220 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b600ff430 0x7f9b601971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 --2- 172.21.15.78:0/1734696804 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b600ffdd0 0x7f9b601976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 >> 172.21.15.78:0/1734696804 conn(0x7f9b60076370 msgr2=0x7f9b600fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:33.108 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 shutdown_connections 2023-10-10T17:43:33.109 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:33.098+0000 7f9b653e2700 1 -- 172.21.15.78:0/1734696804 wait complete. 2023-10-10T17:43:33.564 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:33.564 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:33.726 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:33.726 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:33.743 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:33.743 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:33.743 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 22 -lt 100 ']' 2023-10-10T17:43:33.744 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 22 + 1 2023-10-10T17:43:33.837 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=23 2023-10-10T17:43:33.837 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:33.879 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:33.879 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:33.880 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:33.880 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:33.880 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:34.456 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:34.456 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:34.709 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:34.709 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:34.727 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:34.728 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:34.728 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 23 -lt 100 ']' 2023-10-10T17:43:34.728 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 23 + 1 2023-10-10T17:43:34.806 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=24 2023-10-10T17:43:34.807 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:34.884 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:34.884 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:34.885 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:34.885 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:34.885 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:34.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 -- 172.21.15.78:0/2556919241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0ff300 msgr2=0x7f4d1c10b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:34.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/2556919241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0ff300 0x7f4d1c10b790 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f4d1400b0a0 tx=0x7f4d1400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 -- 172.21.15.78:0/2556919241 shutdown_connections 2023-10-10T17:43:34.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/2556919241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0ff300 0x7f4d1c10b790 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/2556919241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4d1c104960 0x7f4d1c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/2556919241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d1c0fe950 0x7f4d1c0fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 -- 172.21.15.78:0/2556919241 >> 172.21.15.78:0/2556919241 conn(0x7f4d1c075920 msgr2=0x7f4d1c075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:34.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 -- 172.21.15.78:0/2556919241 shutdown_connections 2023-10-10T17:43:34.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.901+0000 7f4d20b0b700 1 -- 172.21.15.78:0/2556919241 wait complete. 2023-10-10T17:43:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.902+0000 7f4d20b0b700 1 Processor -- start 2023-10-10T17:43:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.902+0000 7f4d20b0b700 1 -- start start 2023-10-10T17:43:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d20b0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0fe950 0x7f4d1c100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d20b0b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4d1c0ff300 0x7f4d1c100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d20b0b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d1c104960 0x7f4d1c1a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d20b0b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c1012d0 con 0x7f4d1c0fe950 2023-10-10T17:43:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d20b0b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c1a4b20 con 0x7f4d1c0ff300 2023-10-10T17:43:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d20b0b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c1a4c60 con 0x7f4d1c104960 2023-10-10T17:43:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0fe950 0x7f4d1c100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0fe950 0x7f4d1c100560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45138/0 (socket says 172.21.15.78:45138) 2023-10-10T17:43:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1263590079 learned_addr learned my addr 172.21.15.78:0/1263590079 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d1c104960 0x7f4d1c1a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4d1c0ff300 0x7f4d1c100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1263590079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d1c104960 msgr2=0x7f4d1c1a45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:34.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d1c104960 0x7f4d1c1a45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1263590079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4d1c0ff300 msgr2=0x7f4d1c100aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4d1c0ff300 0x7f4d1c100aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1263590079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d14009d00 con 0x7f4d1c0fe950 2023-10-10T17:43:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0fe950 0x7f4d1c100560 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00e970 tx=0x7f4d0c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:34.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.903+0000 7f4d0b7fe700 1 -- 172.21.15.78:0/1263590079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c009790 con 0x7f4d1c0fe950 2023-10-10T17:43:34.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.904+0000 7f4d0b7fe700 1 -- 172.21.15.78:0/1263590079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d0c00c7e0 con 0x7f4d1c0fe950 2023-10-10T17:43:34.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.904+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c10ea40 con 0x7f4d1c0fe950 2023-10-10T17:43:34.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.904+0000 7f4d0b7fe700 1 -- 172.21.15.78:0/1263590079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c003c70 con 0x7f4d1c0fe950 2023-10-10T17:43:34.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.904+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c10e8b0 con 0x7f4d1c0fe950 2023-10-10T17:43:34.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.906+0000 7f4d0b7fe700 1 -- 172.21.15.78:0/1263590079 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f4d0c00cc50 con 0x7f4d1c0fe950 2023-10-10T17:43:34.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.906+0000 7f4d0b7fe700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4d040a13e0 0x7f4d040a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.907+0000 7f4d0b7fe700 1 -- 172.21.15.78:0/1263590079 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4d0c014070 con 0x7f4d1c0fe950 2023-10-10T17:43:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.907+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4d040a13e0 0x7f4d040a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.907+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4d040a13e0 0x7f4d040a3890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c101d50 tx=0x7f4d1000a380 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:34.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:34 smithi190 ceph-mon[101220]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-10-10T17:43:34.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:34 smithi157 ceph-mon[100896]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-10-10T17:43:34.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 Processor -- start 2023-10-10T17:43:34.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 -- start start 2023-10-10T17:43:34.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e3f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e3fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9d40060 con 0x55d9e9e3ab60 2023-10-10T17:43:34.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9d928a0 con 0x55d9e9e3b470 2023-10-10T17:43:34.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.977+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9d8bd30 con 0x55d9e9e3f8b0 2023-10-10T17:43:34.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45146/0 (socket says 172.21.15.78:45146) 2023-10-10T17:43:34.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1750584393 learned_addr learned my addr 172.21.15.78:0/1750584393 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:34.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e3f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e3fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1750584393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3f8b0 msgr2=0x55d9e9e3fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:34.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e3fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1750584393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 msgr2=0x55d9e9e3f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:34.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e3f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 -- 172.21.15.78:0/1750584393 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9e9c0d630 con 0x55d9e9e3ab60 2023-10-10T17:43:34.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e3fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:34.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3af30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c096510 tx=0x7f4d0c099c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8b047b397aec84 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:34.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d08ff9700 1 -- 172.21.15.78:0/1750584393 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c09e040 con 0x55d9e9e3ab60 2023-10-10T17:43:34.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d08ff9700 1 -- 172.21.15.78:0/1750584393 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d0c0a2460 con 0x55d9e9e3ab60 2023-10-10T17:43:34.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.978+0000 7f4d08ff9700 1 -- 172.21.15.78:0/1750584393 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c099500 con 0x55d9e9e3ab60 2023-10-10T17:43:34.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/1750584393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 msgr2=0x55d9e9e3af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:34.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3af30 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c096510 tx=0x7f4d0c099c80 comp rx=0 tx=0).stop 2023-10-10T17:43:34.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/1750584393 shutdown_connections 2023-10-10T17:43:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e3fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3af30 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/1750584393 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e3f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/1750584393 >> 172.21.15.78:0/1750584393 conn(0x55d9e9e30360 msgr2=0x55d9e9e35b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/1750584393 shutdown_connections 2023-10-10T17:43:34.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.979+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/1750584393 wait complete. 2023-10-10T17:43:34.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 -- start start 2023-10-10T17:43:34.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 Processor -- start 2023-10-10T17:43:34.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e470b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e44330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e44a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:34.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9d2d8e0 con 0x55d9e9e3ab60 2023-10-10T17:43:34.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9d2d580 con 0x55d9e9e3f8b0 2023-10-10T17:43:34.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9c0f920 con 0x55d9e9e3b470 2023-10-10T17:43:34.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d2cc97b80 10 client.0 _ll_register_callbacks cb 0x55d9e9d4b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:34.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e470b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e470b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45152/0 (socket says 172.21.15.78:45152) 2023-10-10T17:43:34.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d1a59c700 1 -- 172.21.15.78:0/76333818 learned_addr learned my addr 172.21.15.78:0/76333818 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:34.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.980+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e44a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e44330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:34.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d1a59c700 1 -- 172.21.15.78:0/76333818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 msgr2=0x55d9e9e44330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:34.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e44330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d1a59c700 1 -- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 msgr2=0x55d9e9e44a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:34.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e44a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:34.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d1a59c700 1 -- 172.21.15.78:0/76333818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c10e8b0 con 0x55d9e9e3ab60 2023-10-10T17:43:34.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e470b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00efd0 tx=0x7f4d0c0cd370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9addd6c7490a9691 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:34.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d01ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:34.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d01ffb700 1 -- 172.21.15.78:0/76333818 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c09a270 con 0x55d9e9e3ab60 2023-10-10T17:43:34.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d01ffb700 1 -- 172.21.15.78:0/76333818 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4d0c09a3d0 con 0x55d9e9e3ab60 2023-10-10T17:43:34.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d9e9e2db20 con 0x55d9e9e3ab60 2023-10-10T17:43:34.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d01ffb700 1 -- 172.21.15.78:0/76333818 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d0c0a1fd0 con 0x55d9e9e3ab60 2023-10-10T17:43:34.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d2cc97b80 10 client.15876 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d9e9d52190 con 0x55d9e9e3ab60 2023-10-10T17:43:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.981+0000 7f4d01ffb700 1 -- 172.21.15.78:0/76333818 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d9e9e2db20 con 0x55d9e9e3ab60 2023-10-10T17:43:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d01ffb700 1 -- 172.21.15.78:0/76333818 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4d0c0caa40 con 0x55d9e9e3ab60 2023-10-10T17:43:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d2cc97b80 10 client.15876 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d2cc97b80 10 client.15876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d2cc97b80 20 client.15876 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9e54460 0x55d9e9e56840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d2cc97b80 4 client.15876 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d9e9d4e0a0 con 0x55d9e9e54460 2023-10-10T17:43:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9e54460 0x55d9e9e56840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9e54460 0x55d9e9e56840 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69b0aafcf49298ca server_cookie=d90425f25aa17eb1 in_seq=0 out_seq=0 2023-10-10T17:43:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.982+0000 7f4d01ffb700 10 client.15876 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d01ffb700 1 -- 172.21.15.78:0/76333818 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d9e9d4e0a0 con 0x55d9e9e54460 2023-10-10T17:43:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d01ffb700 10 client.15876 handle_command_reply: tid=0 2023-10-10T17:43:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 client.15876 shutdown 2023-10-10T17:43:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 msgr2=0x55d9e9e470b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e470b0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00efd0 tx=0x7f4d0c0cd370 comp rx=0 tx=0).stop 2023-10-10T17:43:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 shutdown_connections 2023-10-10T17:43:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9e54460 0x55d9e9e56840 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e44a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d01ffb700 0 client.15876 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e470b0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/76333818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e44330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 >> 172.21.15.78:0/76333818 conn(0x55d9e9e30360 msgr2=0x55d9e9e34de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 shutdown_connections 2023-10-10T17:43:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.983+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/76333818 wait complete. 2023-10-10T17:43:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:34.984+0000 7f4d2cc97b80 20 client.15876 trim_cache size 0 max 16384 2023-10-10T17:43:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.001+0000 7f4d2cc97b80 1 Processor -- start 2023-10-10T17:43:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.001+0000 7f4d2cc97b80 1 -- start start 2023-10-10T17:43:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e46be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e43bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9e2ff50 con 0x55d9e9e3ab60 2023-10-10T17:43:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9d4e0d0 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9e2d660 con 0x55d9e9e3b470 2023-10-10T17:43:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45160/0 (socket says 172.21.15.78:45160) 2023-10-10T17:43:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 -- 172.21.15.78:0/3506961879 learned_addr learned my addr 172.21.15.78:0/3506961879 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e46be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e43bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 -- 172.21.15.78:0/3506961879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 msgr2=0x55d9e9e46be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e46be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 -- 172.21.15.78:0/3506961879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 msgr2=0x55d9e9e43bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e43bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 -- 172.21.15.78:0/3506961879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d9e9e31b00 con 0x55d9e9e3ab60 2023-10-10T17:43:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e46be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.002+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3e1a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c099fa0 tx=0x7f4d0c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e37eac5395eefa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d097fa700 1 -- 172.21.15.78:0/3506961879 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c095ff0 con 0x55d9e9e3ab60 2023-10-10T17:43:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d097fa700 1 -- 172.21.15.78:0/3506961879 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d0c09e040 con 0x55d9e9e3ab60 2023-10-10T17:43:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d097fa700 1 -- 172.21.15.78:0/3506961879 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d0c09a270 con 0x55d9e9e3ab60 2023-10-10T17:43:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/3506961879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 msgr2=0x55d9e9e3e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3e1a0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c099fa0 tx=0x7f4d0c003680 comp rx=0 tx=0).stop 2023-10-10T17:43:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/3506961879 shutdown_connections 2023-10-10T17:43:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e43bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e3e1a0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/3506961879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3b470 0x55d9e9e46be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/3506961879 >> 172.21.15.78:0/3506961879 conn(0x55d9e9e30360 msgr2=0x55d9e9e3fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/3506961879 shutdown_connections 2023-10-10T17:43:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.003+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/3506961879 wait complete. 2023-10-10T17:43:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 -- start start 2023-10-10T17:43:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 Processor -- start 2023-10-10T17:43:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e47980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e47ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e48400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9ea72e0 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9ea7420 con 0x55d9e9e3b470 2023-10-10T17:43:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d9e9ea75b0 con 0x55d9e9e3ab60 2023-10-10T17:43:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d2cc97b80 10 client.0 _ll_register_callbacks cb 0x7f4d0c099b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d1ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e48400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d1ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e48400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45170/0 (socket says 172.21.15.78:45170) 2023-10-10T17:43:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d1ad9d700 1 -- 172.21.15.78:0/2664336080 learned_addr learned my addr 172.21.15.78:0/2664336080 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.004+0000 7f4d19d9b700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e47ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e47980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1ad9d700 1 -- 172.21.15.78:0/2664336080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3ab60 msgr2=0x55d9e9e47980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e47980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1ad9d700 1 -- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 msgr2=0x55d9e9e47ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e47ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1ad9d700 1 -- 172.21.15.78:0/2664336080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d9e9d52190 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e47980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d1ad9d700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e48400 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4d14015000 tx=0x7f4d14004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88ae8678d17829d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d01ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d01ffb700 1 -- 172.21.15.78:0/2664336080 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1400e040 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d01ffb700 1 -- 172.21.15.78:0/2664336080 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4d140076e0 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4d0c0a2460 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d01ffb700 1 -- 172.21.15.78:0/2664336080 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d14004e60 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d2cc97b80 10 client.15882 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d9e9bb9de0 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.005+0000 7f4d01ffb700 1 -- 172.21.15.78:0/2664336080 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d0c0a2460 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d01ffb700 1 -- 172.21.15.78:0/2664336080 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4d1400b0d0 con 0x55d9e9e3f8b0 2023-10-10T17:43:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d2cc97b80 10 client.15882 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d2cc97b80 10 client.15882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d2cc97b80 20 client.15882 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9ea8c60 0x55d9e9eab070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d2cc97b80 4 client.15882 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:43:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d9e9d5a660 con 0x55d9e9ea8c60 2023-10-10T17:43:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9ea8c60 0x55d9e9eab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d01ffb700 10 client.15882 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d1a59c700 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9ea8c60 0x55d9e9eab070 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7716537ed92fe21c server_cookie=55ed7e3f5e425b98 in_seq=0 out_seq=0 2023-10-10T17:43:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d01ffb700 1 -- 172.21.15.78:0/2664336080 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d9e9d5a660 con 0x55d9e9ea8c60 2023-10-10T17:43:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d01ffb700 10 client.15882 handle_command_reply: tid=0 2023-10-10T17:43:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.006+0000 7f4d2cc97b80 1 client.15882 shutdown 2023-10-10T17:43:35.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3f8b0 msgr2=0x55d9e9e48400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:35.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e48400 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4d14015000 tx=0x7f4d14004c40 comp rx=0 tx=0).stop 2023-10-10T17:43:35.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 shutdown_connections 2023-10-10T17:43:35.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d9e9e3ab60 0x55d9e9e47980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d9e9ea8c60 0x55d9e9eab070 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d9e9e3f8b0 0x55d9e9e48400 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 --2- 172.21.15.78:0/2664336080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d9e9e3b470 0x55d9e9e47ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d01ffb700 0 client.15882 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:35.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 >> 172.21.15.78:0/2664336080 conn(0x55d9e9e30360 msgr2=0x55d9e9e43bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:35.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 shutdown_connections 2023-10-10T17:43:35.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 1 -- 172.21.15.78:0/2664336080 wait complete. 2023-10-10T17:43:35.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d2cc97b80 20 client.15882 trim_cache size 0 max 16384 2023-10-10T17:43:35.029 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:43:35.029 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:43:35.030 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:43:35.030 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:43:35.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4d040a13e0 msgr2=0x7f4d040a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:35.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4d040a13e0 0x7f4d040a3890 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c101d50 tx=0x7f4d1000a380 comp rx=0 tx=0).stop 2023-10-10T17:43:35.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0fe950 msgr2=0x7f4d1c100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:35.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0fe950 0x7f4d1c100560 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f4d0c00e970 tx=0x7f4d0c00ec80 comp rx=0 tx=0).stop 2023-10-10T17:43:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 shutdown_connections 2023-10-10T17:43:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4d1c104960 0x7f4d1c1a45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4d040a13e0 0x7f4d040a3890 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d1c0fe950 0x7f4d1c100560 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 --2- 172.21.15.78:0/1263590079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4d1c0ff300 0x7f4d1c100aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.007+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 >> 172.21.15.78:0/1263590079 conn(0x7f4d1c075920 msgr2=0x7f4d1c067580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.008+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 shutdown_connections 2023-10-10T17:43:35.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:35.008+0000 7f4d20b0b700 1 -- 172.21.15.78:0/1263590079 wait complete. 2023-10-10T17:43:35.053 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:34 smithi078 ceph-mon[99494]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-10-10T17:43:35.054 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:43:35.055 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:43:35.414 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:35.414 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:35.457 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:35.457 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:35.460 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:35.461 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:35.461 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 24 -lt 100 ']' 2023-10-10T17:43:35.461 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 24 + 1 2023-10-10T17:43:35.520 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=25 2023-10-10T17:43:35.520 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:35.548 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:35.549 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:35.549 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:35.549 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:35.550 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:35.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:35 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2023-10-10T17:43:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:35 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2023-10-10T17:43:35.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:35 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-10-10T17:43:35.950 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:35 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2023-10-10T17:43:35.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:35 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2023-10-10T17:43:35.951 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:35 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-10-10T17:43:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:35 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]: dispatch 2023-10-10T17:43:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:35 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]: dispatch 2023-10-10T17:43:35.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:35 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2023-10-10T17:43:36.139 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:36.139 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:36.350 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:36.351 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:36.368 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:36.368 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:36.368 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 25 -lt 100 ']' 2023-10-10T17:43:36.369 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 25 + 1 2023-10-10T17:43:36.403 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=26 2023-10-10T17:43:36.403 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:36.434 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:36.434 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:36.435 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:36.435 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:36.435 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:36.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:36 smithi190 ceph-mon[101220]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-10-10T17:43:36.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2023-10-10T17:43:36.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2023-10-10T17:43:36.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:36 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-10-10T17:43:36.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:36 smithi190 ceph-mon[101220]: osdmap e93: 12 total, 12 up, 12 in 2023-10-10T17:43:36.968 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:36 smithi078 ceph-mon[99494]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-10-10T17:43:36.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2023-10-10T17:43:36.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2023-10-10T17:43:36.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:36 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-10-10T17:43:36.969 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:36 smithi078 ceph-mon[99494]: osdmap e93: 12 total, 12 up, 12 in 2023-10-10T17:43:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:36 smithi157 ceph-mon[100896]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-10-10T17:43:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [2, 4, 8, 11]}]': finished 2023-10-10T17:43:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 1]}]': finished 2023-10-10T17:43:36.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:36 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2023-10-10T17:43:36.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:36 smithi157 ceph-mon[100896]: osdmap e93: 12 total, 12 up, 12 in 2023-10-10T17:43:37.205 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:37.206 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:37.417 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:37.417 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:37.425 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:37.426 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:37.426 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 26 -lt 100 ']' 2023-10-10T17:43:37.426 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 26 + 1 2023-10-10T17:43:37.486 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=27 2023-10-10T17:43:37.486 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:37.523 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:37.524 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:37.524 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:37.525 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:37.525 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 -- 172.21.15.78:0/896285261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 msgr2=0x7fc8401066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 --2- 172.21.15.78:0/896285261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401066a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc83800b0a0 tx=0x7fc83800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 -- 172.21.15.78:0/896285261 shutdown_connections 2023-10-10T17:43:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 --2- 172.21.15.78:0/896285261 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc8401013e0 0x7fc84010d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 --2- 172.21.15.78:0/896285261 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401066a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 --2- 172.21.15.78:0/896285261 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc840100a50 0x7fc840100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 -- 172.21.15.78:0/896285261 >> 172.21.15.78:0/896285261 conn(0x7fc8400774a0 msgr2=0x7fc8400778a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 -- 172.21.15.78:0/896285261 shutdown_connections 2023-10-10T17:43:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.775+0000 7fc845763700 1 -- 172.21.15.78:0/896285261 wait complete. 2023-10-10T17:43:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 Processor -- start 2023-10-10T17:43:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 -- start start 2023-10-10T17:43:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc840100a50 0x7fc840072df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8401013e0 0x7fc840073330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc840073b60 con 0x7fc8401062d0 2023-10-10T17:43:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8401a4c50 con 0x7fc840100a50 2023-10-10T17:43:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.776+0000 7fc845763700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8401a4d90 con 0x7fc8401013e0 2023-10-10T17:43:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401a4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45184/0 (socket says 172.21.15.78:45184) 2023-10-10T17:43:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 -- 172.21.15.78:0/3518733227 learned_addr learned my addr 172.21.15.78:0/3518733227 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83e7fc700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8401013e0 0x7fc840073330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83effd700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc840100a50 0x7fc840072df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 -- 172.21.15.78:0/3518733227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8401013e0 msgr2=0x7fc840073330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8401013e0 0x7fc840073330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 -- 172.21.15.78:0/3518733227 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc840100a50 msgr2=0x7fc840072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc840100a50 0x7fc840072df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:37.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 -- 172.21.15.78:0/3518733227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc838009d00 con 0x7fc8401062d0 2023-10-10T17:43:37.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc83f7fe700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401a4710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc83000b550 tx=0x7fc83000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:37.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc837fff700 1 -- 172.21.15.78:0/3518733227 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc830011830 con 0x7fc8401062d0 2023-10-10T17:43:37.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc837fff700 1 -- 172.21.15.78:0/3518733227 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc830011e50 con 0x7fc8401062d0 2023-10-10T17:43:37.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc840110340 con 0x7fc8401062d0 2023-10-10T17:43:37.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc837fff700 1 -- 172.21.15.78:0/3518733227 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc83000f2e0 con 0x7fc8401062d0 2023-10-10T17:43:37.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.777+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8401101b0 con 0x7fc8401062d0 2023-10-10T17:43:37.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.778+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8401104d0 con 0x7fc8401062d0 2023-10-10T17:43:37.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.779+0000 7fc837fff700 1 -- 172.21.15.78:0/3518733227 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7fc8300105a0 con 0x7fc8401062d0 2023-10-10T17:43:37.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.780+0000 7fc837fff700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc82c0a13e0 0x7fc82c0a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:37.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.780+0000 7fc83effd700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc82c0a13e0 0x7fc82c0a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:37.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.780+0000 7fc837fff700 1 -- 172.21.15.78:0/3518733227 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fc8300cd160 con 0x7fc8401062d0 2023-10-10T17:43:37.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.780+0000 7fc83effd700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc82c0a13e0 0x7fc82c0a3890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc838006130 tx=0x7fc8380060a0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:37.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:37.784+0000 7fc837fff700 1 -- 172.21.15.78:0/3518733227 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc830096100 con 0x7fc8401062d0 2023-10-10T17:43:37.908 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:37 smithi078 ceph-mon[99494]: osdmap e94: 12 total, 12 up, 12 in 2023-10-10T17:43:37.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:37 smithi190 ceph-mon[101220]: osdmap e94: 12 total, 12 up, 12 in 2023-10-10T17:43:37.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:37 smithi157 ceph-mon[100896]: osdmap e94: 12 total, 12 up, 12 in 2023-10-10T17:43:38.013 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:38.014 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:38.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.029+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc840074320 con 0x7fc8401062d0 2023-10-10T17:43:38.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.030+0000 7fc837fff700 1 -- 172.21.15.78:0/3518733227 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7fc830015090 con 0x7fc8401062d0 2023-10-10T17:43:38.031 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:43:38.031 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:43:38.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc82c0a13e0 msgr2=0x7fc82c0a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:38.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc82c0a13e0 0x7fc82c0a3890 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc838006130 tx=0x7fc8380060a0 comp rx=0 tx=0).stop 2023-10-10T17:43:38.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 msgr2=0x7fc8401a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:38.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401a4710 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fc83000b550 tx=0x7fc83000b860 comp rx=0 tx=0).stop 2023-10-10T17:43:38.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 shutdown_connections 2023-10-10T17:43:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc82c0a13e0 0x7fc82c0a3890 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc8401062d0 0x7fc8401a4710 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc840100a50 0x7fc840072df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 --2- 172.21.15.78:0/3518733227 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc8401013e0 0x7fc840073330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:38.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 >> 172.21.15.78:0/3518733227 conn(0x7fc8400774a0 msgr2=0x7fc8400ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 shutdown_connections 2023-10-10T17:43:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:38.032+0000 7fc845763700 1 -- 172.21.15.78:0/3518733227 wait complete. 2023-10-10T17:43:38.036 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:43:38.129 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:43:38.227 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:38.228 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:38.242 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:38.242 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:38.243 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 27 -lt 100 ']' 2023-10-10T17:43:38.243 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 27 + 1 2023-10-10T17:43:38.269 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=28 2023-10-10T17:43:38.270 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:38.286 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:38.286 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:38.287 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:38.287 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:38.287 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:38.897 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:38.897 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:38 smithi190 ceph-mon[101220]: pgmap v36: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 20 KiB/s wr, 14 op/s 2023-10-10T17:43:38.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:38 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3518733227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:43:38.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:38 smithi157 ceph-mon[100896]: pgmap v36: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 20 KiB/s wr, 14 op/s 2023-10-10T17:43:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:38 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3518733227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:43:39.005 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:39.006 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:39.068 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:39.068 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:39.068 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 28 -lt 100 ']' 2023-10-10T17:43:39.069 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 28 + 1 2023-10-10T17:43:39.121 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=29 2023-10-10T17:43:39.121 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:39.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:38 smithi078 ceph-mon[99494]: pgmap v36: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 20 KiB/s wr, 14 op/s 2023-10-10T17:43:39.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:38 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3518733227' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:43:39.159 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:39.159 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:39.160 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:39.160 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:39.161 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:39.671 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:39.672 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:39.800 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:39.800 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:39.817 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:39.817 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:39.817 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 29 -lt 100 ']' 2023-10-10T17:43:39.818 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 29 + 1 2023-10-10T17:43:39.861 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=30 2023-10-10T17:43:39.861 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:39.921 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:39.921 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:39.922 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:39.922 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:39.923 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:40.638 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:40.638 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:40.783 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:40.784 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:40.811 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:40.811 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:40.811 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 30 -lt 100 ']' 2023-10-10T17:43:40.812 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 30 + 1 2023-10-10T17:43:40.852 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=31 2023-10-10T17:43:40.853 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:40.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:40 smithi078 ceph-mon[99494]: pgmap v37: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 20 KiB/s wr, 14 op/s 2023-10-10T17:43:40.892 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:40.893 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:40.893 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:40.894 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:40.894 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:40.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:40 smithi190 ceph-mon[101220]: pgmap v37: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 20 KiB/s wr, 14 op/s 2023-10-10T17:43:40.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:40 smithi157 ceph-mon[100896]: pgmap v37: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 20 KiB/s wr, 14 op/s 2023-10-10T17:43:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/3374135708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08100530 msgr2=0x7f9a0810c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/3374135708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08100530 0x7f9a0810c040 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b0a0 tx=0x7f9a0000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/3374135708 shutdown_connections 2023-10-10T17:43:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/3374135708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08100530 0x7f9a0810c040 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/3374135708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a08105210 0x7f9a081055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/3374135708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a080ffba0 0x7f9a080ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/3374135708 >> 172.21.15.78:0/3374135708 conn(0x7f9a080748f0 msgr2=0x7f9a08074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:41.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/3374135708 shutdown_connections 2023-10-10T17:43:41.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.214+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/3374135708 wait complete. 2023-10-10T17:43:41.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 Processor -- start 2023-10-10T17:43:41.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 -- start start 2023-10-10T17:43:41.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a080ffba0 0x7f9a08197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a08100530 0x7f9a081978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08105210 0x7f9a081a23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a081981e0 con 0x7f9a08105210 2023-10-10T17:43:41.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a081a2920 con 0x7f9a080ffba0 2023-10-10T17:43:41.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0fa8c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a081a2ae0 con 0x7f9a08100530 2023-10-10T17:43:41.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08105210 0x7f9a081a23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08105210 0x7f9a081a23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45198/0 (socket says 172.21.15.78:45198) 2023-10-10T17:43:41.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 -- 172.21.15.78:0/2709900829 learned_addr learned my addr 172.21.15.78:0/2709900829 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:41.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0d828700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a080ffba0 0x7f9a08197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0d027700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a08100530 0x7f9a081978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 -- 172.21.15.78:0/2709900829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a08100530 msgr2=0x7f9a081978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a08100530 0x7f9a081978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 -- 172.21.15.78:0/2709900829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a080ffba0 msgr2=0x7f9a08197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a080ffba0 0x7f9a08197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.215+0000 7f9a0e029700 1 -- 172.21.15.78:0/2709900829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a00009d00 con 0x7f9a08105210 2023-10-10T17:43:41.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.216+0000 7f9a0e029700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08105210 0x7f9a081a23e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b830 tx=0x7f9a000079b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:41.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.216+0000 7f99feffd700 1 -- 172.21.15.78:0/2709900829 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000e040 con 0x7f9a08105210 2023-10-10T17:43:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.216+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0810f210 con 0x7f9a08105210 2023-10-10T17:43:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.216+0000 7f99feffd700 1 -- 172.21.15.78:0/2709900829 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a00004ce0 con 0x7f9a08105210 2023-10-10T17:43:41.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.216+0000 7f99feffd700 1 -- 172.21.15.78:0/2709900829 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0001d480 con 0x7f9a08105210 2023-10-10T17:43:41.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.216+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0810f080 con 0x7f9a08105210 2023-10-10T17:43:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.218+0000 7f99feffd700 1 -- 172.21.15.78:0/2709900829 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f9a00003a00 con 0x7f9a08105210 2023-10-10T17:43:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.218+0000 7f99feffd700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99f40a13e0 0x7f99f40a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.218+0000 7f9a0d828700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99f40a13e0 0x7f99f40a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.218+0000 7f99feffd700 1 -- 172.21.15.78:0/2709900829 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f9a000ce9c0 con 0x7f9a08105210 2023-10-10T17:43:41.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.219+0000 7f9a0d828700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99f40a13e0 0x7f99f40a3890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9a04007b60 tx=0x7f9a040073d0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 Processor -- start 2023-10-10T17:43:41.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 -- start start 2023-10-10T17:43:41.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd63f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd64470 0x55905cd68370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd68d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cc69060 con 0x55905cd688b0 2023-10-10T17:43:41.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905ccbb8a0 con 0x55905cd64470 2023-10-10T17:43:41.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.317+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905ccb4d30 con 0x55905cd63b60 2023-10-10T17:43:41.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd68d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd68d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45206/0 (socket says 172.21.15.78:45206) 2023-10-10T17:43:41.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 -- 172.21.15.78:0/1465999269 learned_addr learned my addr 172.21.15.78:0/1465999269 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:41.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0d828700 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd63f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0d027700 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd64470 0x55905cd68370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 -- 172.21.15.78:0/1465999269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 msgr2=0x55905cd63f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd63f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 -- 172.21.15.78:0/1465999269 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd64470 msgr2=0x55905cd68370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd64470 0x55905cd68370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 -- 172.21.15.78:0/1465999269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55905cb36630 con 0x55905cd688b0 2023-10-10T17:43:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0d828700 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd63f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:41.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f9a0e029700 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd68d00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9a000062a0 tx=0x7f9a000043f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aabdd2fbfdd8ad2c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f99f3fff700 1 -- 172.21.15.78:0/1465999269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0009af00 con 0x55905cd688b0 2023-10-10T17:43:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f99f3fff700 1 -- 172.21.15.78:0/1465999269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a00004e50 con 0x55905cd688b0 2023-10-10T17:43:41.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.318+0000 7f99f3fff700 1 -- 172.21.15.78:0/1465999269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a000860b0 con 0x55905cd688b0 2023-10-10T17:43:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/1465999269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 msgr2=0x55905cd68d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd68d00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f9a000062a0 tx=0x7f9a000043f0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/1465999269 shutdown_connections 2023-10-10T17:43:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd63f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd68d00 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/1465999269 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd64470 0x55905cd68370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/1465999269 >> 172.21.15.78:0/1465999269 conn(0x55905cd59360 msgr2=0x55905cd5eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/1465999269 shutdown_connections 2023-10-10T17:43:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.319+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/1465999269 wait complete. 2023-10-10T17:43:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 -- start start 2023-10-10T17:43:41.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 Processor -- start 2023-10-10T17:43:41.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cd70060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cd6d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd6d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cc568e0 con 0x55905cd688b0 2023-10-10T17:43:41.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cc56580 con 0x55905cd63b60 2023-10-10T17:43:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cb38920 con 0x55905cd64470 2023-10-10T17:43:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a1bc18b80 10 client.0 _ll_register_callbacks cb 0x55905cc74780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd6d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd6d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45212/0 (socket says 172.21.15.78:45212) 2023-10-10T17:43:41.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a0e029700 1 -- 172.21.15.78:0/407979565 learned_addr learned my addr 172.21.15.78:0/407979565 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:41.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a0d828700 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cd70060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.320+0000 7f9a0d027700 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cd6d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a0e029700 1 -- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 msgr2=0x55905cd6d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a0e029700 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cd6d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a0e029700 1 -- 172.21.15.78:0/407979565 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 msgr2=0x55905cd70060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a0e029700 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cd70060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a0e029700 1 -- 172.21.15.78:0/407979565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0810f080 con 0x55905cd688b0 2023-10-10T17:43:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a0e029700 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd6d980 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9a00097fc0 tx=0x7f9a00004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ab68b3ee20a98d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:41.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f99f17fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:41.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f99f17fa700 1 -- 172.21.15.78:0/407979565 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0009af00 con 0x55905cd688b0 2023-10-10T17:43:41.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f99f17fa700 1 -- 172.21.15.78:0/407979565 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f9a0009b5a0 con 0x55905cd688b0 2023-10-10T17:43:41.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55905cd56b20 con 0x55905cd688b0 2023-10-10T17:43:41.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f99f17fa700 1 -- 172.21.15.78:0/407979565 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a000cbba0 con 0x55905cd688b0 2023-10-10T17:43:41.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 10 client.15900 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:41.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55905cc7b190 con 0x55905cd688b0 2023-10-10T17:43:41.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f99f17fa700 1 -- 172.21.15.78:0/407979565 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a000c9070 con 0x55905cd688b0 2023-10-10T17:43:41.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f99f17fa700 1 -- 172.21.15.78:0/407979565 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9a000b1070 con 0x55905cd688b0 2023-10-10T17:43:41.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 10 client.15900 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:41.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:43:41.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 20 client.15900 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:41.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cd7d480 0x55905cd7f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 4 client.15900 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.321+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55905cc770a0 con 0x55905cd7d480 2023-10-10T17:43:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.322+0000 7f9a0d828700 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cd7d480 0x55905cd7f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.322+0000 7f99f17fa700 10 client.15900 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:43:41.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.322+0000 7f9a0d828700 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cd7d480 0x55905cd7f890 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3ea543262d22e030 server_cookie=545ab5dc4d49c45d in_seq=0 out_seq=0 2023-10-10T17:43:41.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.322+0000 7f99f17fa700 1 -- 172.21.15.78:0/407979565 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55905cc770a0 con 0x55905cd7d480 2023-10-10T17:43:41.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f99f17fa700 10 client.15900 handle_command_reply: tid=0 2023-10-10T17:43:41.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 client.15900 shutdown 2023-10-10T17:43:41.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 msgr2=0x55905cd6d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd6d980 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9a00097fc0 tx=0x7f9a00004360 comp rx=0 tx=0).stop 2023-10-10T17:43:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 shutdown_connections 2023-10-10T17:43:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cd6d980 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cd70060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cd7d480 0x55905cd7f890 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/407979565 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cd6d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f99f17fa700 0 client.15900 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:43:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 >> 172.21.15.78:0/407979565 conn(0x55905cd59360 msgr2=0x55905cd69d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 shutdown_connections 2023-10-10T17:43:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/407979565 wait complete. 2023-10-10T17:43:41.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.323+0000 7f9a1bc18b80 20 client.15900 trim_cache size 0 max 16384 2023-10-10T17:43:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 Processor -- start 2023-10-10T17:43:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 -- start start 2023-10-10T17:43:41.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd5ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd64470 0x55905cd79a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd688b0 0x55905cd6fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cc770d0 con 0x55905cd64470 2023-10-10T17:43:41.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cd5f8e0 con 0x55905cd688b0 2023-10-10T17:43:41.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cd6e8c0 con 0x55905cd63b60 2023-10-10T17:43:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd64470 0x55905cd79a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd64470 0x55905cd79a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45218/0 (socket says 172.21.15.78:45218) 2023-10-10T17:43:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 -- 172.21.15.78:0/273122498 learned_addr learned my addr 172.21.15.78:0/273122498 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d828700 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd5ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0e029700 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd688b0 0x55905cd6fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 -- 172.21.15.78:0/273122498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 msgr2=0x55905cd5ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd5ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 -- 172.21.15.78:0/273122498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd688b0 msgr2=0x55905cd6fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd688b0 0x55905cd6fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.336+0000 7f9a0d027700 1 -- 172.21.15.78:0/273122498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55905cc7aee0 con 0x55905cd64470 2023-10-10T17:43:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.337+0000 7f9a0d027700 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd64470 0x55905cd79a60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f99f80045d0 tx=0x7f99f800bba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f5aec55b75fbfb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.337+0000 7f99fcdac700 1 -- 172.21.15.78:0/273122498 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8016070 con 0x55905cd64470 2023-10-10T17:43:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.337+0000 7f99fcdac700 1 -- 172.21.15.78:0/273122498 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f99f8017040 con 0x55905cd64470 2023-10-10T17:43:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.337+0000 7f99fcdac700 1 -- 172.21.15.78:0/273122498 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8011d10 con 0x55905cd64470 2023-10-10T17:43:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/273122498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd64470 msgr2=0x55905cd79a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd64470 0x55905cd79a60 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f99f80045d0 tx=0x7f99f800bba0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/273122498 shutdown_connections 2023-10-10T17:43:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd63b60 0x55905cd5ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd688b0 0x55905cd6fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/273122498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd64470 0x55905cd79a60 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/273122498 >> 172.21.15.78:0/273122498 conn(0x55905cd59360 msgr2=0x55905cd6acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:41.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/273122498 shutdown_connections 2023-10-10T17:43:41.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/273122498 wait complete. 2023-10-10T17:43:41.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- start start 2023-10-10T17:43:41.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 Processor -- start 2023-10-10T17:43:41.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cdc5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cdc59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cdd0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cdc60d0 con 0x55905cd688b0 2023-10-10T17:43:41.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cdd0850 con 0x55905cd63b60 2023-10-10T17:43:41.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55905cdd09c0 con 0x55905cd64470 2023-10-10T17:43:41.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.338+0000 7f9a1bc18b80 10 client.0 _ll_register_callbacks cb 0x7f99f800b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:41.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cdd0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cdd0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45226/0 (socket says 172.21.15.78:45226) 2023-10-10T17:43:41.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 -- 172.21.15.78:0/2742622519 learned_addr learned my addr 172.21.15.78:0/2742622519 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:41.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0d828700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cdc5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0d027700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cdc59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 -- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 msgr2=0x55905cdc59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cdc59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 -- 172.21.15.78:0/2742622519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 msgr2=0x55905cdc5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:41.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cdc5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 -- 172.21.15.78:0/2742622519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f80042b0 con 0x55905cd688b0 2023-10-10T17:43:41.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0d027700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cdc59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:41.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a0e029700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cdd0310 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9a0009be50 tx=0x7f9a00009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9027c0b2060d59 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:41.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f99f17fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:41.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f99f17fa700 1 -- 172.21.15.78:0/2742622519 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0009af00 con 0x55905cd688b0 2023-10-10T17:43:41.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f99f17fa700 1 -- 172.21.15.78:0/2742622519 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f9a00024360 con 0x55905cd688b0 2023-10-10T17:43:41.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9a00004e50 con 0x55905cd688b0 2023-10-10T17:43:41.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f99f17fa700 1 -- 172.21.15.78:0/2742622519 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9a000978e0 con 0x55905cd688b0 2023-10-10T17:43:41.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a1bc18b80 10 client.15906 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:41.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55905cacbc60 con 0x55905cd688b0 2023-10-10T17:43:41.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f99f17fa700 1 -- 172.21.15.78:0/2742622519 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a00004e50 con 0x55905cd688b0 2023-10-10T17:43:41.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.339+0000 7f99f17fa700 1 -- 172.21.15.78:0/2742622519 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f9a00019070 con 0x55905cd688b0 2023-10-10T17:43:41.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.340+0000 7f9a1bc18b80 10 client.15906 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:41.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.340+0000 7f9a1bc18b80 10 client.15906 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:43:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.340+0000 7f9a1bc18b80 20 client.15906 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.340+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cdd1c10 0x55905cdd3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.340+0000 7f9a1bc18b80 4 client.15906 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:43:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.340+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55905cc83660 con 0x55905cdd1c10 2023-10-10T17:43:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.340+0000 7f9a0d828700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cdd1c10 0x55905cdd3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a0d828700 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cdd1c10 0x55905cdd3ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3eeaa63dc2e68bc2 server_cookie=e4e03b93e751700a in_seq=0 out_seq=0 2023-10-10T17:43:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f99f17fa700 10 client.15906 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:43:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f99f17fa700 1 -- 172.21.15.78:0/2742622519 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55905cc83660 con 0x55905cdd1c10 2023-10-10T17:43:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f99f17fa700 10 client.15906 handle_command_reply: tid=0 2023-10-10T17:43:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 client.15906 shutdown 2023-10-10T17:43:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 msgr2=0x55905cdd0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cdd0310 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9a0009be50 tx=0x7f9a00009c90 comp rx=0 tx=0).stop 2023-10-10T17:43:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 shutdown_connections 2023-10-10T17:43:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55905cd688b0 0x55905cdd0310 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55905cd63b60 0x55905cdc5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55905cdd1c10 0x55905cdd3ff0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 --2- 172.21.15.78:0/2742622519 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55905cd64470 0x55905cdc59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f99f17fa700 0 client.15906 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:43:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 >> 172.21.15.78:0/2742622519 conn(0x55905cd59360 msgr2=0x55905cd668f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 shutdown_connections 2023-10-10T17:43:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 1 -- 172.21.15.78:0/2742622519 wait complete. 2023-10-10T17:43:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.341+0000 7f9a1bc18b80 20 client.15906 trim_cache size 0 max 16384 2023-10-10T17:43:41.368 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:43:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99f40a13e0 msgr2=0x7f99f40a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99f40a13e0 0x7f99f40a3890 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9a04007b60 tx=0x7f9a040073d0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08105210 msgr2=0x7f9a081a23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08105210 0x7f9a081a23e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f9a0000b830 tx=0x7f9a000079b0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 shutdown_connections 2023-10-10T17:43:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f99f40a13e0 0x7f99f40a3890 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9a08105210 0x7f9a081a23e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9a080ffba0 0x7f9a08197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 --2- 172.21.15.78:0/2709900829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9a08100530 0x7f9a081978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 >> 172.21.15.78:0/2709900829 conn(0x7f9a080748f0 msgr2=0x7f9a080fcf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 shutdown_connections 2023-10-10T17:43:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:41.342+0000 7f9a0fa8c700 1 -- 172.21.15.78:0/2709900829 wait complete. 2023-10-10T17:43:41.396 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:43:41.721 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:41.722 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:41.843 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:41.844 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:41.855 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:41.856 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:41.856 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 31 -lt 100 ']' 2023-10-10T17:43:41.856 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 31 + 1 2023-10-10T17:43:41.889 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=32 2023-10-10T17:43:41.890 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:41.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:41 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:43:41.977 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:41.977 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:41.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:41 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:43:41.978 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:41.979 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:41.979 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:42.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:41 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:43:42.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:42 smithi190 ceph-mon[101220]: pgmap v38: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 KiB/s wr, 9 op/s 2023-10-10T17:43:42.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:42 smithi078 ceph-mon[99494]: pgmap v38: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 KiB/s wr, 9 op/s 2023-10-10T17:43:42.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:42 smithi157 ceph-mon[100896]: pgmap v38: 129 pgs: 3 peering, 126 active+clean; 1.5 MiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 10 KiB/s wr, 9 op/s 2023-10-10T17:43:43.446 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:43.447 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:43.641 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:43.641 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:43.651 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:43.652 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:43.652 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 32 -lt 100 ']' 2023-10-10T17:43:43.652 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 32 + 1 2023-10-10T17:43:43.735 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=33 2023-10-10T17:43:43.736 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:43.790 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:43.791 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:43.792 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:43.792 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:43.792 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:44.671 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:44.671 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:44.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:44 smithi078 ceph-mon[99494]: pgmap v39: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 15 op/s; 239 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:44.949 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:44.950 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:44 smithi157 ceph-mon[100896]: pgmap v39: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 15 op/s; 239 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:45.000 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:45.000 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:45.000 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 33 -lt 100 ']' 2023-10-10T17:43:45.000 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 33 + 1 2023-10-10T17:43:45.102 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=34 2023-10-10T17:43:45.102 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:45.156 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:45.156 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:45.157 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:45.157 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:45.157 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:44 smithi190 ceph-mon[101220]: pgmap v39: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 15 op/s; 239 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:45.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 -- 172.21.15.78:0/3380906539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc069700 msgr2=0x7f5ccc10d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 --2- 172.21.15.78:0/3380906539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc069700 0x7f5ccc10d250 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400b0a0 tx=0x7f5cc400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 -- 172.21.15.78:0/3380906539 shutdown_connections 2023-10-10T17:43:45.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 --2- 172.21.15.78:0/3380906539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ccc106420 0x7f5ccc1067f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 --2- 172.21.15.78:0/3380906539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc069700 0x7f5ccc10d250 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 --2- 172.21.15.78:0/3380906539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ccc068d70 0x7f5ccc0691c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 -- 172.21.15.78:0/3380906539 >> 172.21.15.78:0/3380906539 conn(0x7f5ccc075170 msgr2=0x7f5ccc075570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:45.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 -- 172.21.15.78:0/3380906539 shutdown_connections 2023-10-10T17:43:45.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 -- 172.21.15.78:0/3380906539 wait complete. 2023-10-10T17:43:45.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 Processor -- start 2023-10-10T17:43:45.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.582+0000 7f5cd3756700 1 -- start start 2023-10-10T17:43:45.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd3756700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ccc068d70 0x7f5ccc1965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd3756700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ccc069700 0x7f5ccc196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd3756700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc106420 0x7f5ccc19af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd3756700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc19b580 con 0x7f5ccc106420 2023-10-10T17:43:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd3756700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc19b6f0 con 0x7f5ccc069700 2023-10-10T17:43:45.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd3756700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc19b8b0 con 0x7f5ccc068d70 2023-10-10T17:43:45.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc106420 0x7f5ccc19af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc106420 0x7f5ccc19af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45240/0 (socket says 172.21.15.78:45240) 2023-10-10T17:43:45.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/812414200 learned_addr learned my addr 172.21.15.78:0/812414200 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:45.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ccc068d70 0x7f5ccc1965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ccc069700 0x7f5ccc196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/812414200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ccc068d70 msgr2=0x7f5ccc1965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ccc068d70 0x7f5ccc1965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/812414200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ccc069700 msgr2=0x7f5ccc196ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ccc069700 0x7f5ccc196ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/812414200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cc4009d00 con 0x7f5ccc106420 2023-10-10T17:43:45.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ccc069700 0x7f5ccc196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:45.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc106420 0x7f5ccc19af40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400b780 tx=0x7f5cc40078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:45.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.583+0000 7f5cc27fc700 1 -- 172.21.15.78:0/812414200 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc400e040 con 0x7f5ccc106420 2023-10-10T17:43:45.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.584+0000 7f5cc27fc700 1 -- 172.21.15.78:0/812414200 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cc401d470 con 0x7f5ccc106420 2023-10-10T17:43:45.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.584+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc110420 con 0x7f5ccc106420 2023-10-10T17:43:45.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.584+0000 7f5cc27fc700 1 -- 172.21.15.78:0/812414200 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40052a0 con 0x7f5ccc106420 2023-10-10T17:43:45.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.584+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc110290 con 0x7f5ccc106420 2023-10-10T17:43:45.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.586+0000 7f5cc27fc700 1 -- 172.21.15.78:0/812414200 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f5cc401d5e0 con 0x7f5ccc106420 2023-10-10T17:43:45.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.586+0000 7f5cc27fc700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5cb80a1460 0x7f5cb80a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.587+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5cb80a1460 0x7f5cb80a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.587+0000 7f5cc27fc700 1 -- 172.21.15.78:0/812414200 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5cc40ce610 con 0x7f5ccc106420 2023-10-10T17:43:45.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.587+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5cb80a1460 0x7f5cb80a3910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009be0 tx=0x7f5cbc009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.676+0000 7f5cdf8e2b80 1 Processor -- start 2023-10-10T17:43:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.676+0000 7f5cdf8e2b80 1 -- start start 2023-10-10T17:43:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.676+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380396ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.676+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803973370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.676+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x563803973d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.676+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563803874060 con 0x5638039738b0 2023-10-10T17:43:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638038c68a0 con 0x56380396f470 2023-10-10T17:43:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638038bfd30 con 0x56380396eb60 2023-10-10T17:43:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x563803973d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x563803973d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45248/0 (socket says 172.21.15.78:45248) 2023-10-10T17:43:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4209219947 learned_addr learned my addr 172.21.15.78:0/4209219947 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380396ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803973370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4209219947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 msgr2=0x56380396ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380396ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4209219947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 msgr2=0x563803973370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803973370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4209219947 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563803741630 con 0x5638039738b0 2023-10-10T17:43:45.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803973370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:45.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x563803973d00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40062a0 tx=0x7f5cc400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d200ab870c13b03 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:45.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cb77fe700 1 -- 172.21.15.78:0/4209219947 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc409f1e0 con 0x5638039738b0 2023-10-10T17:43:45.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cb77fe700 1 -- 172.21.15.78:0/4209219947 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cc40cd070 con 0x5638039738b0 2023-10-10T17:43:45.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.677+0000 7f5cb77fe700 1 -- 172.21.15.78:0/4209219947 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc409b610 con 0x5638039738b0 2023-10-10T17:43:45.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4209219947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 msgr2=0x563803973d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x563803973d00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40062a0 tx=0x7f5cc400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4209219947 shutdown_connections 2023-10-10T17:43:45.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380396ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x563803973d00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4209219947 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803973370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4209219947 >> 172.21.15.78:0/4209219947 conn(0x563803964360 msgr2=0x563803969b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:45.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4209219947 shutdown_connections 2023-10-10T17:43:45.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4209219947 wait complete. 2023-10-10T17:43:45.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.678+0000 7f5cdf8e2b80 1 -- start start 2023-10-10T17:43:45.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 1 Processor -- start 2023-10-10T17:43:45.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x5638039782b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x5638039789f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638038618e0 con 0x5638039738b0 2023-10-10T17:43:45.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563803861580 con 0x56380396f470 2023-10-10T17:43:45.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563803743920 con 0x56380396eb60 2023-10-10T17:43:45.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cdf8e2b80 10 client.0 _ll_register_callbacks cb 0x56380387f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:45.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x5638039789f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x5638039789f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45254/0 (socket says 172.21.15.78:45254) 2023-10-10T17:43:45.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4146870691 learned_addr learned my addr 172.21.15.78:0/4146870691 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:45.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x5638039782b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 msgr2=0x56380397b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4146870691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 msgr2=0x5638039782b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.679+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x5638039782b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/4146870691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc110290 con 0x5638039738b0 2023-10-10T17:43:45.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x5638039782b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:45.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x5638039789f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009cd0 tx=0x7f5cc40a1110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df69d412bc207c26 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:45.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cb4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:45.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4146870691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc409f1e0 con 0x5638039738b0 2023-10-10T17:43:45.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4146870691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5cc409a6a0 con 0x5638039738b0 2023-10-10T17:43:45.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563803961b20 con 0x5638039738b0 2023-10-10T17:43:45.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4146870691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cc40b9540 con 0x5638039738b0 2023-10-10T17:43:45.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 10 client.15918 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:45.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x563803886190 con 0x5638039738b0 2023-10-10T17:43:45.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4146870691 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cc40c9080 con 0x5638039738b0 2023-10-10T17:43:45.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4146870691 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5cc40a30e0 con 0x5638039738b0 2023-10-10T17:43:45.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 10 client.15918 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:45.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 10 client.15918 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:43:45.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 20 client.15918 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x563803988250 0x56380398a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 4 client.15918 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.680+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5638038820a0 con 0x563803988250 2023-10-10T17:43:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.681+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x563803988250 0x56380398a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.681+0000 7f5cb4ff9700 10 client.15918 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:43:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.681+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x563803988250 0x56380398a630 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4f46821bff2fb2b server_cookie=156d5e5a28e5aba5 in_seq=0 out_seq=0 2023-10-10T17:43:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4146870691 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5638038820a0 con 0x563803988250 2023-10-10T17:43:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cb4ff9700 10 client.15918 handle_command_reply: tid=0 2023-10-10T17:43:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 client.15918 shutdown 2023-10-10T17:43:45.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 msgr2=0x5638039789f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x5638039789f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f5cc4009cd0 tx=0x7f5cc40a1110 comp rx=0 tx=0).stop 2023-10-10T17:43:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 shutdown_connections 2023-10-10T17:43:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x563803988250 0x56380398a630 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cb4ff9700 0 client.15918 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:43:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x5638039789f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4146870691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x5638039782b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 >> 172.21.15.78:0/4146870691 conn(0x563803964360 msgr2=0x563803974d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 shutdown_connections 2023-10-10T17:43:45.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4146870691 wait complete. 2023-10-10T17:43:45.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.682+0000 7f5cdf8e2b80 20 client.15918 trim_cache size 0 max 16384 2023-10-10T17:43:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 Processor -- start 2023-10-10T17:43:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 -- start start 2023-10-10T17:43:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803984b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x56380397abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380398ac90 con 0x5638039738b0 2023-10-10T17:43:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563803972400 con 0x56380396f470 2023-10-10T17:43:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.695+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563803972180 con 0x56380396eb60 2023-10-10T17:43:45.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x56380397abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x56380397abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45260/0 (socket says 172.21.15.78:45260) 2023-10-10T17:43:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/2711276569 learned_addr learned my addr 172.21.15.78:0/2711276569 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/2711276569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 msgr2=0x56380397f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/2711276569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 msgr2=0x563803984b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803984b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 -- 172.21.15.78:0/2711276569 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563803965b00 con 0x5638039738b0 2023-10-10T17:43:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:45.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x56380397abd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40062a0 tx=0x7f5cc400b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b257d57bffa4edf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cb7fff700 1 -- 172.21.15.78:0/2711276569 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc409b680 con 0x5638039738b0 2023-10-10T17:43:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cb7fff700 1 -- 172.21.15.78:0/2711276569 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563803961b20 con 0x5638039738b0 2023-10-10T17:43:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.696+0000 7f5cb7fff700 1 -- 172.21.15.78:0/2711276569 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc40cbba0 con 0x5638039738b0 2023-10-10T17:43:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/2711276569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 msgr2=0x56380397abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x56380397abd0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40062a0 tx=0x7f5cc400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/2711276569 shutdown_connections 2023-10-10T17:43:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396eb60 0x56380397f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5638039738b0 0x56380397abd0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/2711276569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56380396f470 0x563803984b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/2711276569 >> 172.21.15.78:0/2711276569 conn(0x563803964360 msgr2=0x563803975c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:45.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/2711276569 shutdown_connections 2023-10-10T17:43:45.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/2711276569 wait complete. 2023-10-10T17:43:45.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 -- start start 2023-10-10T17:43:45.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.697+0000 7f5cdf8e2b80 1 Processor -- start 2023-10-10T17:43:45.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56380396eb60 0x56380397fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396f470 0x56380397d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5638039738b0 0x56380397dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56380397e680 con 0x56380396eb60 2023-10-10T17:43:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638039df4b0 con 0x5638039738b0 2023-10-10T17:43:45.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5638039df5f0 con 0x56380396f470 2023-10-10T17:43:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 10 client.0 _ll_register_callbacks cb 0x56380387f780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56380396eb60 0x56380397fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56380396eb60 0x56380397fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45264/0 (socket says 172.21.15.78:45264) 2023-10-10T17:43:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 -- 172.21.15.78:0/4081539059 learned_addr learned my addr 172.21.15.78:0/4081539059 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:45.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396f470 0x56380397d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5638039738b0 0x56380397dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 -- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396f470 msgr2=0x56380397d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396f470 0x56380397d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 -- 172.21.15.78:0/4081539059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5638039738b0 msgr2=0x56380397dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:45.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5638039738b0 0x56380397dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 -- 172.21.15.78:0/4081539059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563803886190 con 0x56380396eb60 2023-10-10T17:43:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd1cf3700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5638039738b0 0x56380397dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cd14f2700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56380396eb60 0x56380397fa80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x56380398af00 tx=0x7f5cbc0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ae4e0ae0529982 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cb4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4081539059 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc024040 con 0x56380396eb60 2023-10-10T17:43:45.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4081539059 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5cbc028540 con 0x56380396eb60 2023-10-10T17:43:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.698+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5cc40cd070 con 0x56380396eb60 2023-10-10T17:43:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4081539059 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cbc028780 con 0x56380396eb60 2023-10-10T17:43:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 10 client.15924 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5638036edde0 con 0x56380396eb60 2023-10-10T17:43:45.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4081539059 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5cc40cd070 con 0x56380396eb60 2023-10-10T17:43:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4081539059 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5cbc01f430 con 0x56380396eb60 2023-10-10T17:43:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 10 client.15924 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 10 client.15924 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:43:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 20 client.15924 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:45.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5638039e0ca0 0x5638039e30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 4 client.15924 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:43:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56380388e660 con 0x5638039e0ca0 2023-10-10T17:43:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.699+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5638039e0ca0 0x5638039e30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cb4ff9700 10 client.15924 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:43:45.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cd0cf1700 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5638039e0ca0 0x5638039e30b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9502d3d6853a63e5 server_cookie=b4b98825ae4f938e in_seq=0 out_seq=0 2023-10-10T17:43:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cb4ff9700 1 -- 172.21.15.78:0/4081539059 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56380388e660 con 0x5638039e0ca0 2023-10-10T17:43:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cb4ff9700 10 client.15924 handle_command_reply: tid=0 2023-10-10T17:43:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 client.15924 shutdown 2023-10-10T17:43:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56380396eb60 msgr2=0x56380397fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56380396eb60 0x56380397fa80 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x56380398af00 tx=0x7f5cbc0047c0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 shutdown_connections 2023-10-10T17:43:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5638039738b0 0x56380397dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56380396eb60 0x56380397fa80 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5638039e0ca0 0x5638039e30b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 --2- 172.21.15.78:0/4081539059 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56380396f470 0x56380397d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cb4ff9700 0 client.15924 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:43:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 >> 172.21.15.78:0/4081539059 conn(0x563803964360 msgr2=0x563803984b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 shutdown_connections 2023-10-10T17:43:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 1 -- 172.21.15.78:0/4081539059 wait complete. 2023-10-10T17:43:45.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.700+0000 7f5cdf8e2b80 20 client.15924 trim_cache size 0 max 16384 2023-10-10T17:43:45.727 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:43:45.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5cb80a1460 msgr2=0x7f5cb80a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5cb80a1460 0x7f5cb80a3910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc009be0 tx=0x7f5cbc009260 comp rx=0 tx=0).stop 2023-10-10T17:43:45.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc106420 msgr2=0x7f5ccc19af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:45.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc106420 0x7f5ccc19af40 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400b780 tx=0x7f5cc40078e0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 shutdown_connections 2023-10-10T17:43:45.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5ccc068d70 0x7f5ccc1965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5cb80a1460 0x7f5cb80a3910 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ccc106420 0x7f5ccc19af40 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 --2- 172.21.15.78:0/812414200 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5ccc069700 0x7f5ccc196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:45.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 >> 172.21.15.78:0/812414200 conn(0x7f5ccc075170 msgr2=0x7f5ccc0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:45.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 shutdown_connections 2023-10-10T17:43:45.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:45.701+0000 7f5cd3756700 1 -- 172.21.15.78:0/812414200 wait complete. 2023-10-10T17:43:45.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:43:46.212 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:46.213 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:46.526 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:46.527 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:46.651 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:46.651 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:46.651 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 34 -lt 100 ']' 2023-10-10T17:43:46.651 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 34 + 1 2023-10-10T17:43:46.652 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=35 2023-10-10T17:43:46.652 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:46.706 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:46.707 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:46.707 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:46.708 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:46.708 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:46.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:46 smithi157 ceph-mon[100896]: pgmap v40: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 12 op/s; 199 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:47.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:46 smithi078 ceph-mon[99494]: pgmap v40: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 12 op/s; 199 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:46 smithi190 ceph-mon[101220]: pgmap v40: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 12 op/s; 199 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:47.526 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:47.688 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:47.901 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:47.902 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:47.958 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:47.958 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:47.958 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 35 -lt 100 ']' 2023-10-10T17:43:47.959 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 35 + 1 2023-10-10T17:43:48.085 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=36 2023-10-10T17:43:48.085 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:48.164 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:48.164 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:48.165 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:48.165 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:48.165 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:48 smithi157 ceph-mon[100896]: pgmap v41: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s; 180 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:49.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:48 smithi078 ceph-mon[99494]: pgmap v41: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s; 180 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:48 smithi190 ceph-mon[101220]: pgmap v41: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s; 180 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:49.621 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:49.621 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:49.893 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:49.893 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:49.907 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:49.907 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:49.908 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 36 -lt 100 ']' 2023-10-10T17:43:49.908 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 36 + 1 2023-10-10T17:43:49.938 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=37 2023-10-10T17:43:49.938 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:49.950 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:49.951 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:49.951 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:49.952 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:49.952 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:49.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:49 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:43:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:49 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:43:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:49 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:43:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:49 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:43:49.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:49 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:43:49.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:49 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:43:49.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:49 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:43:50.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:49 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:43:50.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:49 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:43:50.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:49 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:43:50.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:49 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:43:50.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:49 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:43:50.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:49 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:43:50.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:49 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:43:50.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:49 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:43:50.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:49 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:43:50.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:49 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:43:50.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:49 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:43:50.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:49 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:43:50.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:49 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:43:50.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:49 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:43:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:50 smithi157 ceph-mon[100896]: pgmap v42: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 6 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:51.015 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:50 smithi078 ceph-mon[99494]: pgmap v42: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 6 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:50 smithi190 ceph-mon[101220]: pgmap v42: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 6 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:51.854 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:51.854 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:52.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:52.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:52.265 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:52.265 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:52.266 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 37 -lt 100 ']' 2023-10-10T17:43:52.266 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 37 + 1 2023-10-10T17:43:52.392 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=38 2023-10-10T17:43:52.393 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:52.457 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:52.457 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:52.458 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:52.458 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:52.458 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1904526579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 msgr2=0x7f7cf40fed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.634 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1904526579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf40fed30 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009990 tx=0x7f7ce8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 -- 172.21.15.78:0/3095174331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834101660 msgr2=0x7f8834104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 --2- 172.21.15.78:0/3095174331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834101660 0x7f8834104ef0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8820009920 tx=0x7f8820009c30 comp rx=0 tx=0).stop 2023-10-10T17:43:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1904526579 shutdown_connections 2023-10-10T17:43:52.635 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1904526579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cf40ff270 0x7f7cf410b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1904526579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cf41048d0 0x7f7cf4104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1904526579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf40fed30 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.636 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1904526579 >> 172.21.15.78:0/1904526579 conn(0x7f7cf4075810 msgr2=0x7f7cf4075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 -- 172.21.15.78:0/3095174331 shutdown_connections 2023-10-10T17:43:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 --2- 172.21.15.78:0/3095174331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8834105620 0x7f883410d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 --2- 172.21.15.78:0/3095174331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834100cc0 0x7f8834101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 --2- 172.21.15.78:0/3095174331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834101660 0x7f8834104ef0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 -- 172.21.15.78:0/3095174331 >> 172.21.15.78:0/3095174331 conn(0x7f88340775c0 msgr2=0x7f88340779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1904526579 shutdown_connections 2023-10-10T17:43:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1904526579 wait complete. 2023-10-10T17:43:52.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 -- 172.21.15.78:0/3095174331 shutdown_connections 2023-10-10T17:43:52.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.633+0000 7f883aff8700 1 -- 172.21.15.78:0/3095174331 wait complete. 2023-10-10T17:43:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f7cf9aeb700 1 Processor -- start 2023-10-10T17:43:52.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 Processor -- start 2023-10-10T17:43:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 -- start start 2023-10-10T17:43:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8834100cc0 0x7f88341965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834101660 0x7f8834196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f7cf9aeb700 1 -- start start 2023-10-10T17:43:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834105620 0x7f883419af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883419b5b0 con 0x7f8834101660 2023-10-10T17:43:52.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883419b720 con 0x7f8834100cc0 2023-10-10T17:43:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.634+0000 7f883aff8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f883419b8e0 con 0x7f8834105620 2023-10-10T17:43:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834101660 0x7f8834196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834101660 0x7f8834196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45286/0 (socket says 172.21.15.78:45286) 2023-10-10T17:43:52.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 -- 172.21.15.78:0/2772340376 learned_addr learned my addr 172.21.15.78:0/2772340376 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf9aeb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf4196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf9aeb700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cf40ff270 0x7f7cf41968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.647 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf9aeb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cf41048d0 0x7f7cf419ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf9aeb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf419b370 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf9aeb700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf419b4e0 con 0x7f7cf40ff270 2023-10-10T17:43:52.648 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf9aeb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cf419b6a0 con 0x7f7cf41048d0 2023-10-10T17:43:52.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8838d94700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8834100cc0 0x7f88341965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf4196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf4196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45290/0 (socket says 172.21.15.78:45290) 2023-10-10T17:43:52.649 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 -- 172.21.15.78:0/1762369977 learned_addr learned my addr 172.21.15.78:0/1762369977 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8839595700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834105620 0x7f883419af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.653 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cf40ff270 0x7f7cf41968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 -- 172.21.15.78:0/2772340376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834105620 msgr2=0x7f883419af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834105620 0x7f883419af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 -- 172.21.15.78:0/2772340376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8834100cc0 msgr2=0x7f88341965d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8834100cc0 0x7f88341965d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 -- 172.21.15.78:0/2772340376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c009c40 con 0x7f8834101660 2023-10-10T17:43:52.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf3fff700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cf41048d0 0x7f7cf419ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8838d94700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8834100cc0 0x7f88341965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 -- 172.21.15.78:0/1762369977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cf41048d0 msgr2=0x7f7cf419ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cf41048d0 0x7f7cf419ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 -- 172.21.15.78:0/1762369977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cf40ff270 msgr2=0x7f7cf41968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cf40ff270 0x7f7cf41968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf37fe700 1 -- 172.21.15.78:0/1762369977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce80095f0 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f8833fff700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834101660 0x7f8834196b10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8820009230 tx=0x7f882000bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.635+0000 7f7cf3fff700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cf41048d0 0x7f7cf419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:52.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf4196390 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400e970 tx=0x7f7ce400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f8831ffb700 1 -- 172.21.15.78:0/2772340376 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882001c070 con 0x7f8834101660 2023-10-10T17:43:52.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f8831ffb700 1 -- 172.21.15.78:0/2772340376 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f882000bbd0 con 0x7f8834101660 2023-10-10T17:43:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8820009520 con 0x7f8834101660 2023-10-10T17:43:52.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f8831ffb700 1 -- 172.21.15.78:0/2772340376 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820004930 con 0x7f8834101660 2023-10-10T17:43:52.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/1762369977 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4009790 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/1762369977 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce400eec0 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf410e9b0 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/1762369977 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4003cf0 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf410e820 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.636+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88341103a0 con 0x7f8834101660 2023-10-10T17:43:52.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.638+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/1762369977 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f7ce400ccd0 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.639+0000 7f8831ffb700 1 -- 172.21.15.78:0/2772340376 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f8820020070 con 0x7f8834101660 2023-10-10T17:43:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.639+0000 7f7cf0ff9700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7cdc0a13e0 0x7f7cdc0a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.639+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7cdc0a13e0 0x7f7cdc0a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.665 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.639+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/1762369977 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f7ce4014070 con 0x7f7cf40fe8c0 2023-10-10T17:43:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.640+0000 7f8831ffb700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88240a1460 0x7f88240a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.640+0000 7f8838d94700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88240a1460 0x7f88240a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.640+0000 7f8831ffb700 1 -- 172.21.15.78:0/2772340376 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f88200ce020 con 0x7f8834101660 2023-10-10T17:43:52.666 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.640+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7cdc0a13e0 0x7f7cdc0a3890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cf41979a0 tx=0x7f7ce8005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.640+0000 7f8838d94700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88240a1460 0x7f88240a3910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8828005fd0 tx=0x7f8828009310 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 Processor -- start 2023-10-10T17:43:52.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 Processor -- start 2023-10-10T17:43:52.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 -- start start 2023-10-10T17:43:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 -- start start 2023-10-10T17:43:52.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48d2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b8b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8b720 0x5636a1b85560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48d7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b85b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1a8b000 con 0x5636a19e16b0 2023-10-10T17:43:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1add840 con 0x5636a1b8b720 2023-10-10T17:43:52.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1ad6cd0 con 0x5636a1b8bb90 2023-10-10T17:43:52.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd47d8060 con 0x55bdd48d2b60 2023-10-10T17:43:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd482a8a0 con 0x55bdd48d78b0 2023-10-10T17:43:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8847184b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd4823d30 con 0x55bdd48d3470 2023-10-10T17:43:52.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b8b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8838d94700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48d2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8838d94700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48d2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45300/0 (socket says 172.21.15.78:45300) 2023-10-10T17:43:52.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f8838d94700 1 -- 172.21.15.78:0/1927590769 learned_addr learned my addr 172.21.15.78:0/1927590769 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b8b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45302/0 (socket says 172.21.15.78:45302) 2023-10-10T17:43:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.717+0000 7f7cf37fe700 1 -- 172.21.15.78:0/2134435697 learned_addr learned my addr 172.21.15.78:0/2134435697 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8b720 0x5636a1b85560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.726 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf3fff700 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b85b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8839595700 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8833fff700 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48d7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8838d94700 1 -- 172.21.15.78:0/1927590769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 msgr2=0x55bdd48d7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8838d94700 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48d7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8838d94700 1 -- 172.21.15.78:0/1927590769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 msgr2=0x55bdd48d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8838d94700 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48d7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8838d94700 1 -- 172.21.15.78:0/1927590769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdd46a5630 con 0x55bdd48d2b60 2023-10-10T17:43:52.728 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2134435697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 msgr2=0x5636a1b85b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b85b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2134435697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 msgr2=0x5636a1b8b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b8b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2134435697 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636a19585d0 con 0x5636a1b8b720 2023-10-10T17:43:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8839595700 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f8838d94700 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48d2f30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8828011f80 tx=0x7f8828014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4622043d2e01bf2b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.718+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8b720 0x5636a1b85560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800b2e0 tx=0x7f7ce8020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73c057a262733b73 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.722+0000 7f7cd9ffb700 1 -- 172.21.15.78:0/2134435697 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce802b070 con 0x5636a1b8b720 2023-10-10T17:43:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.722+0000 7f7cd9ffb700 1 -- 172.21.15.78:0/2134435697 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce801e040 con 0x5636a1b8b720 2023-10-10T17:43:52.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.722+0000 7f7cd9ffb700 1 -- 172.21.15.78:0/2134435697 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce800f6c0 con 0x5636a1b8b720 2023-10-10T17:43:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.722+0000 7f881effd700 1 -- 172.21.15.78:0/1927590769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8828009ea0 con 0x55bdd48d2b60 2023-10-10T17:43:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.722+0000 7f881effd700 1 -- 172.21.15.78:0/1927590769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8828016070 con 0x55bdd48d2b60 2023-10-10T17:43:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.722+0000 7f881effd700 1 -- 172.21.15.78:0/1927590769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882801d5f0 con 0x55bdd48d2b60 2023-10-10T17:43:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2134435697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8b720 msgr2=0x5636a1b85560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8b720 0x5636a1b85560 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ce800b2e0 tx=0x7f7ce8020000 comp rx=0 tx=0).stop 2023-10-10T17:43:52.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2134435697 shutdown_connections 2023-10-10T17:43:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b85b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b8b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2134435697 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8b720 0x5636a1b85560 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2134435697 >> 172.21.15.78:0/2134435697 conn(0x5636a1b7af50 msgr2=0x5636a1b80270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2134435697 shutdown_connections 2023-10-10T17:43:52.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 -- 172.21.15.78:0/1927590769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 msgr2=0x55bdd48d2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48d2f30 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f8828011f80 tx=0x7f8828014000 comp rx=0 tx=0).stop 2023-10-10T17:43:52.736 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2134435697 wait complete. 2023-10-10T17:43:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 -- 172.21.15.78:0/1927590769 shutdown_connections 2023-10-10T17:43:52.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48d2f30 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 --2- 172.21.15.78:0/1927590769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48d7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 -- 172.21.15.78:0/1927590769 >> 172.21.15.78:0/1927590769 conn(0x55bdd48c8360 msgr2=0x55bdd48cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 -- 172.21.15.78:0/1927590769 shutdown_connections 2023-10-10T17:43:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.723+0000 7f8847184b80 1 -- 172.21.15.78:0/1927590769 wait complete. 2023-10-10T17:43:52.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 -- start start 2023-10-10T17:43:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f8847184b80 1 -- start start 2023-10-10T17:43:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f8847184b80 1 Processor -- start 2023-10-10T17:43:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 Processor -- start 2023-10-10T17:43:52.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b94aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b94fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1a78880 con 0x5636a19e16b0 2023-10-10T17:43:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1a78520 con 0x5636a1b8bb90 2023-10-10T17:43:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a195a8c0 con 0x5636a1b8b720 2023-10-10T17:43:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7d05c77b80 10 client.0 _ll_register_callbacks cb 0x5636a1a96720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:52.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b94aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b94aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45314/0 (socket says 172.21.15.78:45314) 2023-10-10T17:43:52.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f7cf37fe700 1 -- 172.21.15.78:0/2153138805 learned_addr learned my addr 172.21.15.78:0/2153138805 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.724+0000 7f8847184b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48defe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8847184b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48dc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8847184b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48dc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8847184b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd47c58e0 con 0x55bdd48d2b60 2023-10-10T17:43:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8847184b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd47c5580 con 0x55bdd48d78b0 2023-10-10T17:43:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8847184b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd46a7920 con 0x55bdd48d3470 2023-10-10T17:43:52.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf3fff700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b94fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8847184b80 10 client.0 _ll_register_callbacks cb 0x55bdd47e3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:52.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48defe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48defe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45320/0 (socket says 172.21.15.78:45320) 2023-10-10T17:43:52.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 -- 172.21.15.78:0/1677570450 learned_addr learned my addr 172.21.15.78:0/1677570450 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8839595700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48dc870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf37fe700 1 -- 172.21.15.78:0/2153138805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 msgr2=0x5636a1b94fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b94fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf37fe700 1 -- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8bb90 msgr2=0x5636a1b8ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8833fff700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48dc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf37fe700 1 -- 172.21.15.78:0/2153138805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf410c000 con 0x5636a19e16b0 2023-10-10T17:43:52.751 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b94fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:52.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 -- 172.21.15.78:0/1677570450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 msgr2=0x55bdd48dc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48dc260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 -- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 msgr2=0x55bdd48dc870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48dc870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 -- 172.21.15.78:0/1677570450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8828011c60 con 0x55bdd48d2b60 2023-10-10T17:43:52.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8839595700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48dc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b94aa0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400c5b0 tx=0x7f7ce4016420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da249f67cc2bbc50 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.754 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f7cd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:52.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8838d94700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48defe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f882801f680 tx=0x7f882801fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f062998ff13af7a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.725+0000 7f8813fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2153138805 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4099710 con 0x5636a19e16b0 2023-10-10T17:43:52.755 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2153138805 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f7ce409e050 con 0x5636a19e16b0 2023-10-10T17:43:52.756 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636a1b78710 con 0x5636a19e16b0 2023-10-10T17:43:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8813fff700 1 -- 172.21.15.78:0/1677570450 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882802d070 con 0x55bdd48d2b60 2023-10-10T17:43:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8813fff700 1 -- 172.21.15.78:0/1677570450 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f8828018070 con 0x55bdd48d2b60 2023-10-10T17:43:52.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bdd48c5b20 con 0x55bdd48d2b60 2023-10-10T17:43:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2153138805 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ce40cd2c0 con 0x5636a19e16b0 2023-10-10T17:43:52.757 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 10 client.15945 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:52.758 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5636a1a9d130 con 0x5636a19e16b0 2023-10-10T17:43:52.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8813fff700 1 -- 172.21.15.78:0/1677570450 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f882801fea0 con 0x55bdd48d2b60 2023-10-10T17:43:52.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8847184b80 10 client.15948 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:52.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bdd47ea190 con 0x55bdd48d2b60 2023-10-10T17:43:52.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2153138805 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce405fe90 con 0x5636a19e16b0 2023-10-10T17:43:52.759 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2153138805 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ce40add20 con 0x5636a19e16b0 2023-10-10T17:43:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 10 client.15945 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 10 client.15945 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:52.760 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 20 client.15945 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:52.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8813fff700 1 -- 172.21.15.78:0/1677570450 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bdd48c5b20 con 0x55bdd48d2b60 2023-10-10T17:43:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1b9ef00 0x5636a1ba12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.761 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 4 client.15945 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:52.762 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636a1a99040 con 0x5636a1b9ef00 2023-10-10T17:43:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8813fff700 1 -- 172.21.15.78:0/1677570450 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f88280049f0 con 0x55bdd48d2b60 2023-10-10T17:43:52.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8847184b80 10 client.15948 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:52.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.726+0000 7f8847184b80 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:52.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f8847184b80 20 client.15948 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:52.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f8847184b80 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd48ec740 0x55bdd48eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f8847184b80 4 client.15948 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdd47e60a0 con 0x55bdd48ec740 2023-10-10T17:43:52.764 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1b9ef00 0x5636a1ba12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f8833fff700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd48ec740 0x55bdd48eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f7cd2ffd700 10 client.15945 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.765 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1b9ef00 0x5636a1ba12e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f75681ddc379f404 server_cookie=72d502c431fed6b0 in_seq=0 out_seq=0 2023-10-10T17:43:52.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f8813fff700 10 client.15948 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.727+0000 7f8833fff700 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd48ec740 0x55bdd48eeb50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=783d81b1bb275709 server_cookie=4275fa19151660ff in_seq=0 out_seq=0 2023-10-10T17:43:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2153138805 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5636a1a99040 con 0x5636a1b9ef00 2023-10-10T17:43:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7cd2ffd700 10 client.15945 handle_command_reply: tid=0 2023-10-10T17:43:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 client.15945 shutdown 2023-10-10T17:43:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 msgr2=0x5636a1b94aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.767 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b94aa0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400c5b0 tx=0x7f7ce4016420 comp rx=0 tx=0).stop 2023-10-10T17:43:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8813fff700 1 -- 172.21.15.78:0/1677570450 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bdd47e60a0 con 0x55bdd48ec740 2023-10-10T17:43:52.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8813fff700 10 client.15948 handle_command_reply: tid=0 2023-10-10T17:43:52.768 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 shutdown_connections 2023-10-10T17:43:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1b9ef00 0x5636a1ba12e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a19e16b0 0x5636a1b94aa0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7cd2ffd700 0 client.15945 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2153138805 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b94fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 >> 172.21.15.78:0/2153138805 conn(0x5636a1b7af50 msgr2=0x5636a1b7f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 client.15948 shutdown 2023-10-10T17:43:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 shutdown_connections 2023-10-10T17:43:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2153138805 wait complete. 2023-10-10T17:43:52.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 msgr2=0x55bdd48defe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48defe0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f882801f680 tx=0x7f882801fb40 comp rx=0 tx=0).stop 2023-10-10T17:43:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 shutdown_connections 2023-10-10T17:43:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd48ec740 0x55bdd48eeb50 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48dc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d2b60 0x55bdd48defe0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8813fff700 0 client.15948 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 --2- 172.21.15.78:0/1677570450 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d3470 0x55bdd48dc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 >> 172.21.15.78:0/1677570450 conn(0x55bdd48c8360 msgr2=0x55bdd48d8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.774 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.728+0000 7f7d05c77b80 20 client.15945 trim_cache size 0 max 16384 2023-10-10T17:43:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.729+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 shutdown_connections 2023-10-10T17:43:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.729+0000 7f8847184b80 1 -- 172.21.15.78:0/1677570450 wait complete. 2023-10-10T17:43:52.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.729+0000 7f8847184b80 20 client.15948 trim_cache size 0 max 16384 2023-10-10T17:43:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 Processor -- start 2023-10-10T17:43:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 -- start start 2023-10-10T17:43:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e39b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d3470 0x55bdd48e8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48deb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd48ce8b0 con 0x55bdd48d3470 2023-10-10T17:43:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd48dd840 con 0x55bdd48d78b0 2023-10-10T17:43:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8847184b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd48c7ee0 con 0x55bdd48d2b60 2023-10-10T17:43:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8833fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d3470 0x55bdd48e8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8833fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d3470 0x55bdd48e8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45332/0 (socket says 172.21.15.78:45332) 2023-10-10T17:43:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.742+0000 7f8833fff700 1 -- 172.21.15.78:0/3910704426 learned_addr learned my addr 172.21.15.78:0/3910704426 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8838d94700 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8839595700 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48deb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8833fff700 1 -- 172.21.15.78:0/3910704426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 msgr2=0x55bdd48e39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8833fff700 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8833fff700 1 -- 172.21.15.78:0/3910704426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 msgr2=0x55bdd48deb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8833fff700 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48deb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8833fff700 1 -- 172.21.15.78:0/3910704426 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdd47e9ee0 con 0x55bdd48d3470 2023-10-10T17:43:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8838d94700 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:43:52.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f8833fff700 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d3470 0x55bdd48e8b80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f88200098f0 tx=0x7f8820000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330feec8e9cc98c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f881f7fe700 1 -- 172.21.15.78:0/3910704426 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8820098fa0 con 0x55bdd48d3470 2023-10-10T17:43:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f881f7fe700 1 -- 172.21.15.78:0/3910704426 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f882009ee90 con 0x55bdd48d3470 2023-10-10T17:43:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.743+0000 7f881f7fe700 1 -- 172.21.15.78:0/3910704426 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882009a0f0 con 0x55bdd48d3470 2023-10-10T17:43:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 -- 172.21.15.78:0/3910704426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d3470 msgr2=0x55bdd48e8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d3470 0x55bdd48e8b80 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f88200098f0 tx=0x7f8820000bc0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 -- 172.21.15.78:0/3910704426 shutdown_connections 2023-10-10T17:43:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d78b0 0x55bdd48deb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 --2- 172.21.15.78:0/3910704426 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d3470 0x55bdd48e8b80 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 -- 172.21.15.78:0/3910704426 >> 172.21.15.78:0/3910704426 conn(0x55bdd48c8360 msgr2=0x55bdd48d9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 -- 172.21.15.78:0/3910704426 shutdown_connections 2023-10-10T17:43:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 -- 172.21.15.78:0/3910704426 wait complete. 2023-10-10T17:43:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 -- start start 2023-10-10T17:43:52.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.744+0000 7f8847184b80 1 Processor -- start 2023-10-10T17:43:52.783 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 Processor -- start 2023-10-10T17:43:52.784 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 -- start start 2023-10-10T17:43:52.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8847184b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.785 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b7ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8847184b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d3470 0x55bdd48e3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8847184b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d78b0 0x55bdd4937d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8847184b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd4938340 con 0x55bdd48d78b0 2023-10-10T17:43:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8847184b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd49384b0 con 0x55bdd48d3470 2023-10-10T17:43:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8847184b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdd4938670 con 0x55bdd48d2b60 2023-10-10T17:43:52.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8847184b80 10 client.0 _ll_register_callbacks cb 0x7f882c009f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b8e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.787 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1b7abe0 con 0x5636a1b8bb90 2023-10-10T17:43:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1b78280 con 0x5636a19e16b0 2023-10-10T17:43:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1b899d0 con 0x5636a1b8b720 2023-10-10T17:43:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b7ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.788 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b7ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47060/0 (socket says 172.21.15.78:47060) 2023-10-10T17:43:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7cf2ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b8e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.789 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7cf37fe700 1 -- 172.21.15.78:0/2514693219 learned_addr learned my addr 172.21.15.78:0/2514693219 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8833fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d3470 0x55bdd48e3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8833fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d3470 0x55bdd48e3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47062/0 (socket says 172.21.15.78:47062) 2023-10-10T17:43:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8833fff700 1 -- 172.21.15.78:0/3329580012 learned_addr learned my addr 172.21.15.78:0/3329580012 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8838d94700 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.790 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f7cf3fff700 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.745+0000 7f8839595700 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d78b0 0x55bdd4937d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2514693219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 msgr2=0x5636a1b7ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.791 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b7ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2514693219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8bb90 msgr2=0x5636a1b8ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8833fff700 1 -- 172.21.15.78:0/3329580012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 msgr2=0x55bdd48e32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8833fff700 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8833fff700 1 -- 172.21.15.78:0/3329580012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d78b0 msgr2=0x55bdd4937d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8833fff700 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d78b0 0x55bdd4937d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8833fff700 1 -- 172.21.15.78:0/3329580012 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88341103a0 con 0x55bdd48d3470 2023-10-10T17:43:52.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2514693219 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636a1b7c6f0 con 0x5636a1b8b720 2023-10-10T17:43:52.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b8e590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8000c00 tx=0x7f7ce8023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77d0128f1ba64d7a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8833fff700 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d3470 0x55bdd48e3830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f88200a2a90 tx=0x7f8820009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31b4ca10e1a9d233 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8813fff700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:43:52.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8813fff700 1 -- 172.21.15.78:0/3329580012 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882009a2f0 con 0x55bdd48d3470 2023-10-10T17:43:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cda7fc700 1 -- 172.21.15.78:0/2514693219 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce802b040 con 0x5636a1b8b720 2023-10-10T17:43:52.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cda7fc700 1 -- 172.21.15.78:0/2514693219 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce8038e30 con 0x5636a1b8b720 2023-10-10T17:43:52.796 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f7cda7fc700 1 -- 172.21.15.78:0/2514693219 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8004a40 con 0x5636a1b8b720 2023-10-10T17:43:52.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8828016070 con 0x55bdd48d3470 2023-10-10T17:43:52.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8813fff700 1 -- 172.21.15.78:0/3329580012 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f882009ae60 con 0x55bdd48d3470 2023-10-10T17:43:52.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8813fff700 1 -- 172.21.15.78:0/3329580012 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88200053c0 con 0x55bdd48d3470 2023-10-10T17:43:52.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8847184b80 10 client.25412 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:52.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.746+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55bdd463ac60 con 0x55bdd48d3470 2023-10-10T17:43:52.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8813fff700 1 -- 172.21.15.78:0/3329580012 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8828016070 con 0x55bdd48d3470 2023-10-10T17:43:52.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8813fff700 1 -- 172.21.15.78:0/3329580012 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f88200cbc20 con 0x55bdd48d3470 2023-10-10T17:43:52.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8847184b80 10 client.25412 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:52.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8847184b80 10 client.25412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8847184b80 20 client.25412 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8847184b80 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd4940ca0 0x55bdd49430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8847184b80 4 client.25412 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:43:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bdd47f2660 con 0x55bdd4940ca0 2023-10-10T17:43:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2514693219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 msgr2=0x5636a1b8e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b8e590 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8000c00 tx=0x7f7ce8023000 comp rx=0 tx=0).stop 2023-10-10T17:43:52.800 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2514693219 shutdown_connections 2023-10-10T17:43:52.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8bb90 0x5636a1b8ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b7ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2514693219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8b720 0x5636a1b8e590 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.801 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2514693219 >> 172.21.15.78:0/2514693219 conn(0x5636a1b7af50 msgr2=0x5636a1b88e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f8838d94700 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd4940ca0 0x55bdd49430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.802 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2514693219 shutdown_connections 2023-10-10T17:43:52.802 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2514693219 wait complete. 2023-10-10T17:43:52.803 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.747+0000 7f7d05c77b80 1 -- start start 2023-10-10T17:43:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8813fff700 10 client.25412 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8838d94700 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd4940ca0 0x55bdd49430b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b33bc918e732b144 server_cookie=b716c1ed91d36025 in_seq=0 out_seq=0 2023-10-10T17:43:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8813fff700 1 -- 172.21.15.78:0/3329580012 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bdd47f2660 con 0x55bdd4940ca0 2023-10-10T17:43:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8813fff700 10 client.25412 handle_command_reply: tid=0 2023-10-10T17:43:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 client.25412 shutdown 2023-10-10T17:43:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d3470 msgr2=0x55bdd48e3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d3470 0x55bdd48e3830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f88200a2a90 tx=0x7f8820009c30 comp rx=0 tx=0).stop 2023-10-10T17:43:52.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 1 Processor -- start 2023-10-10T17:43:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 shutdown_connections 2023-10-10T17:43:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bdd48d2b60 0x55bdd48e32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bdd4940ca0 0x55bdd49430b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bdd48d78b0 0x55bdd4937d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8813fff700 0 client.25412 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 --2- 172.21.15.78:0/3329580012 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bdd48d3470 0x55bdd48e3830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 >> 172.21.15.78:0/3329580012 conn(0x55bdd48c8360 msgr2=0x55bdd48d9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 shutdown_connections 2023-10-10T17:43:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 1 -- 172.21.15.78:0/3329580012 wait complete. 2023-10-10T17:43:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f8847184b80 20 client.25412 trim_cache size 0 max 16384 2023-10-10T17:43:52.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b92610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.808 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:43:52.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8b720 0x5636a1b92b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b93090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1b95310 con 0x5636a1b8b720 2023-10-10T17:43:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1bfa5a0 con 0x5636a19e16b0 2023-10-10T17:43:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636a1bfa6e0 con 0x5636a1b8bb90 2023-10-10T17:43:52.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7d05c77b80 10 client.0 _ll_register_callbacks cb 0x5636a1a96720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.748+0000 7f7cf2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8b720 0x5636a1b92b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b92610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b92610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47072/0 (socket says 172.21.15.78:47072) 2023-10-10T17:43:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf37fe700 1 -- 172.21.15.78:0/2975737479 learned_addr learned my addr 172.21.15.78:0/2975737479 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:52.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf3fff700 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b93090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2975737479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 msgr2=0x5636a1b93090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b93090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 msgr2=0x5636a1b92610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b92610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2975737479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636a19585d0 con 0x5636a1b8b720 2023-10-10T17:43:52.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88240a1460 msgr2=0x7f88240a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88240a1460 0x7f88240a3910 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8828005fd0 tx=0x7f8828009310 comp rx=0 tx=0).stop 2023-10-10T17:43:52.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834101660 msgr2=0x7f8834196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834101660 0x7f8834196b10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8820009230 tx=0x7f882000bf60 comp rx=0 tx=0).stop 2023-10-10T17:43:52.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b92610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:52.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 shutdown_connections 2023-10-10T17:43:52.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8834105620 0x7f883419af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88240a1460 0x7f88240a3910 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8834100cc0 0x7f88341965d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 --2- 172.21.15.78:0/2772340376 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8834101660 0x7f8834196b10 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8b720 0x5636a1b92b50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x5636a1b8e4d0 tx=0x7f7ce800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e035744a2a04bdb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:52.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 >> 172.21.15.78:0/2772340376 conn(0x7f88340775c0 msgr2=0x7f8834100280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:52.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 shutdown_connections 2023-10-10T17:43:52.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f883aff8700 1 -- 172.21.15.78:0/2772340376 wait complete. 2023-10-10T17:43:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2975737479 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce802b040 con 0x5636a1b8b720 2023-10-10T17:43:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2975737479 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f7ce8038b70 con 0x5636a1b8b720 2023-10-10T17:43:52.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.749+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7ce801e040 con 0x5636a1b8b720 2023-10-10T17:43:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2975737479 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ce8004500 con 0x5636a1b8b720 2023-10-10T17:43:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 10 client.15960 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5636a1904d80 con 0x5636a1b8b720 2023-10-10T17:43:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2975737479 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ce801e040 con 0x5636a1b8b720 2023-10-10T17:43:52.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2975737479 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7ce80428e0 con 0x5636a1b8b720 2023-10-10T17:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 10 client.15960 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 20 client.15960 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1bfb830 0x5636a1bfdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:52.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 4 client.15960 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:43:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5636a1aa5600 con 0x5636a1bfb830 2023-10-10T17:43:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.750+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1bfb830 0x5636a1bfdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.751+0000 7f7cd2ffd700 10 client.15960 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.751+0000 7f7cf37fe700 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1bfb830 0x5636a1bfdc10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1780c9b6b2734da9 server_cookie=7a92f2ee20cc0a6b in_seq=0 out_seq=0 2023-10-10T17:43:52.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.764+0000 7f7cd2ffd700 1 -- 172.21.15.78:0/2975737479 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5636a1aa5600 con 0x5636a1bfb830 2023-10-10T17:43:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.764+0000 7f7cd2ffd700 10 client.15960 handle_command_reply: tid=0 2023-10-10T17:43:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.764+0000 7f7d05c77b80 1 client.15960 shutdown 2023-10-10T17:43:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8b720 msgr2=0x5636a1b92b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8b720 0x5636a1b92b50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x5636a1b8e4d0 tx=0x7f7ce800be00 comp rx=0 tx=0).stop 2023-10-10T17:43:52.823 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 shutdown_connections 2023-10-10T17:43:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5636a1b8bb90 0x5636a1b93090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5636a1bfb830 0x5636a1bfdc10 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5636a19e16b0 0x5636a1b92610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 --2- 172.21.15.78:0/2975737479 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5636a1b8b720 0x5636a1b92b50 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cd2ffd700 0 client.15960 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:52.824 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 >> 172.21.15.78:0/2975737479 conn(0x5636a1b7af50 msgr2=0x5636a1b89db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 shutdown_connections 2023-10-10T17:43:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 1 -- 172.21.15.78:0/2975737479 wait complete. 2023-10-10T17:43:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7d05c77b80 20 client.15960 trim_cache size 0 max 16384 2023-10-10T17:43:52.825 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:43:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:43:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:43:52.826 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:43:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7cdc0a13e0 msgr2=0x7f7cdc0a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7cdc0a13e0 0x7f7cdc0a3890 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7cf41979a0 tx=0x7f7ce8005fd0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 msgr2=0x7f7cf4196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf4196390 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400e970 tx=0x7f7ce400ec80 comp rx=0 tx=0).stop 2023-10-10T17:43:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 shutdown_connections 2023-10-10T17:43:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7cf41048d0 0x7f7cf419ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f7cdc0a13e0 0x7f7cdc0a3890 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cf40fe8c0 0x7f7cf4196390 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 --2- 172.21.15.78:0/1762369977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7cf40ff270 0x7f7cf41968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:52.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.765+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 >> 172.21.15.78:0/1762369977 conn(0x7f7cf4075810 msgr2=0x7f7cf40fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:52.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.766+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 shutdown_connections 2023-10-10T17:43:52.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:43:52.766+0000 7f7cf9aeb700 1 -- 172.21.15.78:0/1762369977 wait complete. 2023-10-10T17:43:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:52 smithi157 ceph-mon[100896]: pgmap v43: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 6 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:52.998 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:52 smithi078 ceph-mon[99494]: pgmap v43: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 6 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:53.070 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:53.070 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:52 smithi190 ceph-mon[101220]: pgmap v43: 129 pgs: 129 active+clean; 1.5 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.0 KiB/s wr, 6 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:53.248 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:53.248 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:53.273 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:53.274 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:53.274 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 38 -lt 100 ']' 2023-10-10T17:43:53.275 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 38 + 1 2023-10-10T17:43:53.284 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=39 2023-10-10T17:43:53.285 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:53.338 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:53.339 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:53.340 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:53.340 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:53.340 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:53.703 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:53.704 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:53.831 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:43:53.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:43:53.856 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:53.857 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:53.865 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:53.866 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:53.866 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 39 -lt 100 ']' 2023-10-10T17:43:53.866 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 39 + 1 2023-10-10T17:43:53.892 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=40 2023-10-10T17:43:53.893 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:53.908 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:53.909 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:53.910 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:53.910 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:53.910 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:54.319 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:54.320 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:54.408 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:54.408 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:54.423 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:54.423 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:54.423 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 40 -lt 100 ']' 2023-10-10T17:43:54.424 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 40 + 1 2023-10-10T17:43:54.509 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=41 2023-10-10T17:43:54.509 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:54.537 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:54.537 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:54.538 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:54.539 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:54.539 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:54.976 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:54 smithi078 ceph-mon[99494]: pgmap v44: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:54.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:54 smithi157 ceph-mon[100896]: pgmap v44: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:54 smithi190 ceph-mon[101220]: pgmap v44: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 10 op/s; 159 B/s, 0 keys/s, 1 objects/s recovering 2023-10-10T17:43:55.178 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:55.178 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:55.306 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:55.307 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:55.340 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:55.340 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:55.341 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 41 -lt 100 ']' 2023-10-10T17:43:55.341 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 41 + 1 2023-10-10T17:43:55.376 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=42 2023-10-10T17:43:55.377 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:55.463 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:55.464 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:55.465 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:55.465 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:55.465 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:56.361 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:56.361 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:56.548 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:56.548 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:56.556 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:56.556 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:56.556 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 42 -lt 100 ']' 2023-10-10T17:43:56.557 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 42 + 1 2023-10-10T17:43:56.584 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=43 2023-10-10T17:43:56.584 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:56.658 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:56.659 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:56.659 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:56.659 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:56.660 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:56.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:56 smithi157 ceph-mon[100896]: pgmap v45: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 KiB/s wr, 6 op/s 2023-10-10T17:43:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.043+0000 7f38f53f0700 1 -- 172.21.15.78:0/2803775092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 msgr2=0x7f38f00fedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.043+0000 7f38f53f0700 1 --2- 172.21.15.78:0/2803775092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f00fedd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009990 tx=0x7f38d8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 -- 172.21.15.78:0/2803775092 shutdown_connections 2023-10-10T17:43:57.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 --2- 172.21.15.78:0/2803775092 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f0104970 0x7f38f0104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 --2- 172.21.15.78:0/2803775092 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38f00ff310 0x7f38f010b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 --2- 172.21.15.78:0/2803775092 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f00fedd0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 -- 172.21.15.78:0/2803775092 >> 172.21.15.78:0/2803775092 conn(0x7f38f0075890 msgr2=0x7f38f0075c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:57.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 -- 172.21.15.78:0/2803775092 shutdown_connections 2023-10-10T17:43:57.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 -- 172.21.15.78:0/2803775092 wait complete. 2023-10-10T17:43:57.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 Processor -- start 2023-10-10T17:43:57.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.044+0000 7f38f53f0700 1 -- start start 2023-10-10T17:43:57.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38f53f0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f01952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38f53f0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38f00ff310 0x7f38f0195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38f53f0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f0104970 0x7f38f0199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38f53f0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f019a290 con 0x7f38f00fe960 2023-10-10T17:43:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38f53f0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f019a400 con 0x7f38f00ff310 2023-10-10T17:43:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38f53f0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f019a5c0 con 0x7f38f0104970 2023-10-10T17:43:57.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f01952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f01952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45362/0 (socket says 172.21.15.78:45362) 2023-10-10T17:43:57.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 -- 172.21.15.78:0/471237360 learned_addr learned my addr 172.21.15.78:0/471237360 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:57.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f0104970 0x7f38f0199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38e7fff700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38f00ff310 0x7f38f0195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 -- 172.21.15.78:0/471237360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f0104970 msgr2=0x7f38f0199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f0104970 0x7f38f0199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 -- 172.21.15.78:0/471237360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38f00ff310 msgr2=0x7f38f0195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38f00ff310 0x7f38f0195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38eeffd700 1 -- 172.21.15.78:0/471237360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d80095f0 con 0x7f38f00fe960 2023-10-10T17:43:57.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.045+0000 7f38e7fff700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38f00ff310 0x7f38f0195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.046+0000 7f38eeffd700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f01952c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38e000d980 tx=0x7f38e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.046+0000 7f38ecff9700 1 -- 172.21.15.78:0/471237360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e000b950 con 0x7f38f00fe960 2023-10-10T17:43:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.046+0000 7f38ecff9700 1 -- 172.21.15.78:0/471237360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38e0007620 con 0x7f38f00fe960 2023-10-10T17:43:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.046+0000 7f38ecff9700 1 -- 172.21.15.78:0/471237360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e000f3c0 con 0x7f38f00fe960 2023-10-10T17:43:57.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.046+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f010e9f0 con 0x7f38f00fe960 2023-10-10T17:43:57.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.046+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f010e860 con 0x7f38f00fe960 2023-10-10T17:43:57.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.048+0000 7f38ecff9700 1 -- 172.21.15.78:0/471237360 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f38e0010640 con 0x7f38f00fe960 2023-10-10T17:43:57.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.049+0000 7f38ecff9700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f38d00a1460 0x7f38d00a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.049+0000 7f38e7fff700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f38d00a1460 0x7f38d00a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.049+0000 7f38ecff9700 1 -- 172.21.15.78:0/471237360 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f38e00cd0b0 con 0x7f38f00fe960 2023-10-10T17:43:57.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.050+0000 7f38e7fff700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f38d00a1460 0x7f38d00a3910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38d800b2e0 tx=0x7f38d8005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:57.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:56 smithi078 ceph-mon[99494]: pgmap v45: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 KiB/s wr, 6 op/s 2023-10-10T17:43:57.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 Processor -- start 2023-10-10T17:43:57.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 -- start start 2023-10-10T17:43:57.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af899fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af89e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89e930 0x5643af89ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af79f0e0 con 0x5643af899be0 2023-10-10T17:43:57.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af7f1920 con 0x5643af89a4f0 2023-10-10T17:43:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f390157cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af7eadb0 con 0x5643af89e930 2023-10-10T17:43:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.142+0000 7f38eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af899fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af899fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45370/0 (socket says 172.21.15.78:45370) 2023-10-10T17:43:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 -- 172.21.15.78:0/1718509562 learned_addr learned my addr 172.21.15.78:0/1718509562 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38e7fff700 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af89e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89e930 0x5643af89ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 -- 172.21.15.78:0/1718509562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89e930 msgr2=0x5643af89ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89e930 0x5643af89ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 -- 172.21.15.78:0/1718509562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 msgr2=0x5643af89e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af89e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 -- 172.21.15.78:0/1718509562 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643af66c6b0 con 0x5643af899be0 2023-10-10T17:43:57.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38e7fff700 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af89e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:57.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af899fb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f38e0096670 tx=0x7f38e00104e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30bb34efe26aace server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:57.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38e57fa700 1 -- 172.21.15.78:0/1718509562 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e00a2050 con 0x5643af899be0 2023-10-10T17:43:57.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38e57fa700 1 -- 172.21.15.78:0/1718509562 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38e00cce30 con 0x5643af899be0 2023-10-10T17:43:57.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.143+0000 7f38e57fa700 1 -- 172.21.15.78:0/1718509562 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e00a1bc0 con 0x5643af899be0 2023-10-10T17:43:57.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 -- 172.21.15.78:0/1718509562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 msgr2=0x5643af899fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af899fb0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f38e0096670 tx=0x7f38e00104e0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 -- 172.21.15.78:0/1718509562 shutdown_connections 2023-10-10T17:43:57.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89e930 0x5643af89ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af899fb0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 --2- 172.21.15.78:0/1718509562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af89e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 -- 172.21.15.78:0/1718509562 >> 172.21.15.78:0/1718509562 conn(0x5643af88f3e0 msgr2=0x5643af894bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 -- 172.21.15.78:0/1718509562 shutdown_connections 2023-10-10T17:43:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.144+0000 7f390157cb80 1 -- 172.21.15.78:0/1718509562 wait complete. 2023-10-10T17:43:57.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.145+0000 7f390157cb80 1 -- start start 2023-10-10T17:43:57.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.145+0000 7f390157cb80 1 Processor -- start 2023-10-10T17:43:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.145+0000 7f390157cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8a6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f390157cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af8a3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f390157cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89e930 0x5643af8a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f390157cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af78c960 con 0x5643af89e930 2023-10-10T17:43:57.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f390157cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af78c600 con 0x5643af89a4f0 2023-10-10T17:43:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f390157cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af66e9a0 con 0x5643af899be0 2023-10-10T17:43:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f390157cb80 10 client.0 _ll_register_callbacks cb 0x5643af7aa800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89e930 0x5643af8a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89e930 0x5643af8a3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45380/0 (socket says 172.21.15.78:45380) 2023-10-10T17:43:57.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 -- 172.21.15.78:0/1100003026 learned_addr learned my addr 172.21.15.78:0/1100003026 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38eeffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38e7fff700 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af8a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 -- 172.21.15.78:0/1100003026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 msgr2=0x5643af8a6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8a6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 -- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 msgr2=0x5643af8a3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af8a3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 -- 172.21.15.78:0/1100003026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f010e860 con 0x5643af89e930 2023-10-10T17:43:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8a6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:43:57.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89e930 0x5643af8a3ac0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009790 tx=0x7f38e8007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=898c1b42f4758300 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:57.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.146+0000 7f38de7fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:57.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f38de7fc700 1 -- 172.21.15.78:0/1100003026 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e800e040 con 0x5643af89e930 2023-10-10T17:43:57.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f38de7fc700 1 -- 172.21.15.78:0/1100003026 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f38e80138c0 con 0x5643af89e930 2023-10-10T17:43:57.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643af88cba0 con 0x5643af89e930 2023-10-10T17:43:57.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f38de7fc700 1 -- 172.21.15.78:0/1100003026 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38e8013a80 con 0x5643af89e930 2023-10-10T17:43:57.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 10 client.15972 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:57.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5643af7b1210 con 0x5643af89e930 2023-10-10T17:43:57.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f38de7fc700 1 -- 172.21.15.78:0/1100003026 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5643af88cba0 con 0x5643af89e930 2023-10-10T17:43:57.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f38de7fc700 1 -- 172.21.15.78:0/1100003026 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f38e8016c90 con 0x5643af89e930 2023-10-10T17:43:57.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 10 client.15972 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:57.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:57.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 20 client.15972 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:57.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af8b3900 0x5643af8b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 4 client.15972 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:43:57.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.147+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5643af7ad120 con 0x5643af8b3900 2023-10-10T17:43:57.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.148+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af8b3900 0x5643af8b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.148+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af8b3900 0x5643af8b5d10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa077e1b7bb5420c server_cookie=88aef416897c69c0 in_seq=0 out_seq=0 2023-10-10T17:43:57.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.148+0000 7f38de7fc700 10 client.15972 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:57.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f38de7fc700 1 -- 172.21.15.78:0/1100003026 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5643af7ad120 con 0x5643af8b3900 2023-10-10T17:43:57.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f38de7fc700 10 client.15972 handle_command_reply: tid=0 2023-10-10T17:43:57.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 client.15972 shutdown 2023-10-10T17:43:57.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89e930 msgr2=0x5643af8a3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89e930 0x5643af8a3ac0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f38e8009790 tx=0x7f38e8007c30 comp rx=0 tx=0).stop 2023-10-10T17:43:57.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 shutdown_connections 2023-10-10T17:43:57.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8a6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af8b3900 0x5643af8b5d10 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89e930 0x5643af8a3ac0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f38de7fc700 0 client.15972 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:57.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 --2- 172.21.15.78:0/1100003026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89a4f0 0x5643af8a3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 >> 172.21.15.78:0/1100003026 conn(0x5643af88f3e0 msgr2=0x5643af893e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:57.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 shutdown_connections 2023-10-10T17:43:57.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.149+0000 7f390157cb80 1 -- 172.21.15.78:0/1100003026 wait complete. 2023-10-10T17:43:57.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.150+0000 7f390157cb80 20 client.15972 trim_cache size 0 max 16384 2023-10-10T17:43:57.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:56 smithi190 ceph-mon[101220]: pgmap v45: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.0 KiB/s wr, 6 op/s 2023-10-10T17:43:57.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 Processor -- start 2023-10-10T17:43:57.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 -- start start 2023-10-10T17:43:57.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8990f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89a4f0 0x5643af8a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af89c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af88ef60 con 0x5643af89a4f0 2023-10-10T17:43:57.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af895790 con 0x5643af89e930 2023-10-10T17:43:57.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f390157cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af8959d0 con 0x5643af899be0 2023-10-10T17:43:57.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f38e7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89a4f0 0x5643af8a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f38e7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89a4f0 0x5643af8a5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45386/0 (socket says 172.21.15.78:45386) 2023-10-10T17:43:57.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.169+0000 7f38e7fff700 1 -- 172.21.15.78:0/2786933420 learned_addr learned my addr 172.21.15.78:0/2786933420 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:57.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38eeffd700 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af89c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e7fff700 1 -- 172.21.15.78:0/2786933420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 msgr2=0x5643af8990f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e7fff700 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8990f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e7fff700 1 -- 172.21.15.78:0/2786933420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 msgr2=0x5643af89c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e7fff700 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af89c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e7fff700 1 -- 172.21.15.78:0/2786933420 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643af7b0f60 con 0x5643af89a4f0 2023-10-10T17:43:57.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af89c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:43:57.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e7fff700 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89a4f0 0x5643af8a5c30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009990 tx=0x7f38d801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d119fa8dbcbc7e06 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:57.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e5ffb700 1 -- 172.21.15.78:0/2786933420 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d8004500 con 0x5643af89a4f0 2023-10-10T17:43:57.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e5ffb700 1 -- 172.21.15.78:0/2786933420 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38d802c040 con 0x5643af89a4f0 2023-10-10T17:43:57.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.170+0000 7f38e5ffb700 1 -- 172.21.15.78:0/2786933420 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d800f860 con 0x5643af89a4f0 2023-10-10T17:43:57.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 -- 172.21.15.78:0/2786933420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89a4f0 msgr2=0x5643af8a5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89a4f0 0x5643af8a5c30 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f38d8009990 tx=0x7f38d801f000 comp rx=0 tx=0).stop 2023-10-10T17:43:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 -- 172.21.15.78:0/2786933420 shutdown_connections 2023-10-10T17:43:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af899be0 0x5643af8990f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af89c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 --2- 172.21.15.78:0/2786933420 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af89a4f0 0x5643af8a5c30 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 -- 172.21.15.78:0/2786933420 >> 172.21.15.78:0/2786933420 conn(0x5643af88f3e0 msgr2=0x5643af8989e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 -- 172.21.15.78:0/2786933420 shutdown_connections 2023-10-10T17:43:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 -- 172.21.15.78:0/2786933420 wait complete. 2023-10-10T17:43:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.171+0000 7f390157cb80 1 -- start start 2023-10-10T17:43:57.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 1 Processor -- start 2023-10-10T17:43:57.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af8a2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89a4f0 0x5643af8a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af8a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af907800 con 0x5643af899be0 2023-10-10T17:43:57.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af907970 con 0x5643af89e930 2023-10-10T17:43:57.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643af907b30 con 0x5643af89a4f0 2023-10-10T17:43:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f390157cb80 10 client.0 _ll_register_callbacks cb 0x7f38e8007590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:43:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f38eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af8a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f38eeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af8a2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45390/0 (socket says 172.21.15.78:45390) 2023-10-10T17:43:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f38eeffd700 1 -- 172.21.15.78:0/1000659962 learned_addr learned my addr 172.21.15.78:0/1000659962 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:43:57.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f38e7fff700 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89a4f0 0x5643af8a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.172+0000 7f38ef7fe700 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af8a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38eeffd700 1 -- 172.21.15.78:0/1000659962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89a4f0 msgr2=0x5643af8a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89a4f0 0x5643af8a2540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38eeffd700 1 -- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 msgr2=0x5643af8a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:43:57.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af8a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38eeffd700 1 -- 172.21.15.78:0/1000659962 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f010c0a0 con 0x5643af899be0 2023-10-10T17:43:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38eeffd700 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af8a2000 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f38e000b540 tx=0x7f38e0010460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60845c7aa1971628 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:43:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38de7fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:43:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38de7fc700 1 -- 172.21.15.78:0/1000659962 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e00a2050 con 0x5643af899be0 2023-10-10T17:43:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38de7fc700 1 -- 172.21.15.78:0/1000659962 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f38e00b7040 con 0x5643af899be0 2023-10-10T17:43:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38e00cce30 con 0x5643af899be0 2023-10-10T17:43:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38de7fc700 1 -- 172.21.15.78:0/1000659962 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38e0099e60 con 0x5643af899be0 2023-10-10T17:43:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f390157cb80 10 client.15978 fetch_fsmap learned FSMap version 24 2023-10-10T17:43:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5643af601ce0 con 0x5643af899be0 2023-10-10T17:43:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.173+0000 7f38de7fc700 1 -- 172.21.15.78:0/1000659962 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f38e0083030 con 0x5643af899be0 2023-10-10T17:43:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f38de7fc700 1 -- 172.21.15.78:0/1000659962 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f38e00a1bc0 con 0x5643af899be0 2023-10-10T17:43:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f390157cb80 10 client.15978 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:43:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f390157cb80 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:43:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f390157cb80 20 client.15978 populate_metadata read hostname 'smithi078' 2023-10-10T17:43:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f390157cb80 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af908e10 0x5643af90b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:43:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f390157cb80 4 client.15978 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:43:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5643af7b96e0 con 0x5643af908e10 2023-10-10T17:43:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f38e7fff700 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af908e10 0x5643af90b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:43:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f38e7fff700 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af908e10 0x5643af90b1f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b58c17b72aacff6 server_cookie=4e2bbbcf79c0cf43 in_seq=0 out_seq=0 2023-10-10T17:43:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.174+0000 7f38de7fc700 10 client.15978 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f38de7fc700 1 -- 172.21.15.78:0/1000659962 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5643af7b96e0 con 0x5643af908e10 2023-10-10T17:43:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f38de7fc700 10 client.15978 handle_command_reply: tid=0 2023-10-10T17:43:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 client.15978 shutdown 2023-10-10T17:43:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 msgr2=0x5643af8a2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af8a2000 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f38e000b540 tx=0x7f38e0010460 comp rx=0 tx=0).stop 2023-10-10T17:43:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 shutdown_connections 2023-10-10T17:43:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5643af908e10 0x5643af90b1f0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5643af89e930 0x5643af8a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5643af899be0 0x5643af8a2000 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f38de7fc700 0 client.15978 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:43:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 --2- 172.21.15.78:0/1000659962 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5643af89a4f0 0x5643af8a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 >> 172.21.15.78:0/1000659962 conn(0x5643af88f3e0 msgr2=0x5643af899270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 shutdown_connections 2023-10-10T17:43:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 1 -- 172.21.15.78:0/1000659962 wait complete. 2023-10-10T17:43:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.176+0000 7f390157cb80 20 client.15978 trim_cache size 0 max 16384 2023-10-10T17:43:57.193 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:43:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:43:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "7a838c33-ca0d-4be7-b908-f785a2885899", 2023-10-10T17:43:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:43:57.194 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:43:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f38d00a1460 msgr2=0x7f38d00a3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f38d00a1460 0x7f38d00a3910 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f38d800b2e0 tx=0x7f38d8005fd0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 msgr2=0x7f38f01952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:43:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f01952c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f38e000d980 tx=0x7f38e000dc90 comp rx=0 tx=0).stop 2023-10-10T17:43:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 shutdown_connections 2023-10-10T17:43:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f38f0104970 0x7f38f0199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f38d00a1460 0x7f38d00a3910 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f38f00fe960 0x7f38f01952c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 --2- 172.21.15.78:0/471237360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f38f00ff310 0x7f38f0195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:43:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 >> 172.21.15.78:0/471237360 conn(0x7f38f0075890 msgr2=0x7f38f00fced0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:43:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 shutdown_connections 2023-10-10T17:43:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:43:57.177+0000 7f38f53f0700 1 -- 172.21.15.78:0/471237360 wait complete. 2023-10-10T17:43:57.286 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:57.286 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:57.288 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:43:57.500 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:57.500 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:57.506 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:57.506 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:57.506 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 43 -lt 100 ']' 2023-10-10T17:43:57.506 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 43 + 1 2023-10-10T17:43:57.533 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=44 2023-10-10T17:43:57.534 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:57.550 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:57.550 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:57.551 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:57.551 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:57.551 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:58.102 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:58.103 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:58.216 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:58.217 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:58.231 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:58.232 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:58.232 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 44 -lt 100 ']' 2023-10-10T17:43:58.232 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 44 + 1 2023-10-10T17:43:58.270 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=45 2023-10-10T17:43:58.270 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:58.330 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:58.330 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:58.331 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:58.331 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:58.331 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:58.835 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:58.836 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:58.989 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:43:58.989 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:43:58.997 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:58.998 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:43:58.998 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 45 -lt 100 ']' 2023-10-10T17:43:58.998 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 45 + 1 2023-10-10T17:43:59.033 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=46 2023-10-10T17:43:59.034 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:43:59.076 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:59.076 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:43:59.077 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:43:59.077 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:43:59.077 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:43:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:58 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:43:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:58 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:58 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:43:59.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:58 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:59.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:43:58 smithi078 ceph-mon[99494]: pgmap v46: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 14 op/s 2023-10-10T17:43:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:58 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:43:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:58 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:58 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:43:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:58 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:43:58 smithi190 ceph-mon[101220]: pgmap v46: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 14 op/s 2023-10-10T17:43:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:58 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:43:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:58 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:58 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:43:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:58 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:43:59.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:43:58 smithi157 ceph-mon[100896]: pgmap v46: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 13 KiB/s wr, 14 op/s 2023-10-10T17:43:59.885 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:43:59.886 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:00.164 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:00.164 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:00.173 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:00.173 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:00.173 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 46 -lt 100 ']' 2023-10-10T17:44:00.174 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 46 + 1 2023-10-10T17:44:00.233 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=47 2023-10-10T17:44:00.233 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:00.275 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:00.276 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:00.277 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:00.277 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:00.277 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:00.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 -- 172.21.15.78:0/3554264662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6728101370 msgr2=0x7f672810d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 --2- 172.21.15.78:0/3554264662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6728101370 0x7f672810d1d0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f672000b0a0 tx=0x7f672000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 -- 172.21.15.78:0/3554264662 shutdown_connections 2023-10-10T17:44:00.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 --2- 172.21.15.78:0/3554264662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6728106410 0x7f67281067e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 --2- 172.21.15.78:0/3554264662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6728101370 0x7f672810d1d0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 --2- 172.21.15.78:0/3554264662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f67281009e0 0x7f6728100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 -- 172.21.15.78:0/3554264662 >> 172.21.15.78:0/3554264662 conn(0x7f67280775c0 msgr2=0x7f67280779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 -- 172.21.15.78:0/3554264662 shutdown_connections 2023-10-10T17:44:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.582+0000 7f672eeef700 1 -- 172.21.15.78:0/3554264662 wait complete. 2023-10-10T17:44:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 Processor -- start 2023-10-10T17:44:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 -- start start 2023-10-10T17:44:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67281009e0 0x7f67281965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6728101370 0x7f6728196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6728106410 0x7f672819af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672819b590 con 0x7f67281009e0 2023-10-10T17:44:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672819b700 con 0x7f6728106410 2023-10-10T17:44:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672eeef700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672819b8c0 con 0x7f6728101370 2023-10-10T17:44:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67281009e0 0x7f67281965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67281009e0 0x7f67281965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45404/0 (socket says 172.21.15.78:45404) 2023-10-10T17:44:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 -- 172.21.15.78:0/1314755085 learned_addr learned my addr 172.21.15.78:0/1314755085 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672d48c700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6728106410 0x7f672819af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f6727fff700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6728101370 0x7f6728196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 -- 172.21.15.78:0/1314755085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6728101370 msgr2=0x7f6728196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6728101370 0x7f6728196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 -- 172.21.15.78:0/1314755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6728106410 msgr2=0x7f672819af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6728106410 0x7f672819af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672cc8b700 1 -- 172.21.15.78:0/1314755085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6720009d00 con 0x7f67281009e0 2023-10-10T17:44:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f6727fff700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6728101370 0x7f6728196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.583+0000 7f672d48c700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6728106410 0x7f672819af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.584+0000 7f672cc8b700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67281009e0 0x7f67281965b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f671800c9b0 tx=0x7f671800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.584+0000 7f6725ffb700 1 -- 172.21.15.78:0/1314755085 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718007b00 con 0x7f67281009e0 2023-10-10T17:44:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.584+0000 7f6725ffb700 1 -- 172.21.15.78:0/1314755085 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6718007c60 con 0x7f67281009e0 2023-10-10T17:44:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.584+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6728110450 con 0x7f67281009e0 2023-10-10T17:44:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.584+0000 7f6725ffb700 1 -- 172.21.15.78:0/1314755085 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6718018660 con 0x7f67281009e0 2023-10-10T17:44:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.584+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67281102c0 con 0x7f67281009e0 2023-10-10T17:44:00.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.586+0000 7f6725ffb700 1 -- 172.21.15.78:0/1314755085 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 21) v1 ==== 135948+0+0 (secure 0 0 0) 0x7f6718018880 con 0x7f67281009e0 2023-10-10T17:44:00.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.586+0000 7f6725ffb700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f67140a13e0 0x7f67140a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.587+0000 7f6727fff700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f67140a13e0 0x7f67140a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.587+0000 7f6725ffb700 1 -- 172.21.15.78:0/1314755085 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f67180cd3b0 con 0x7f67281009e0 2023-10-10T17:44:00.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.587+0000 7f6727fff700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f67140a13e0 0x7f67140a3890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6710005fd0 tx=0x7f6710005dc0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:00.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 Processor -- start 2023-10-10T17:44:00.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 -- start start 2023-10-10T17:44:00.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065f8b60 0x5589065f8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f9470 0x5589065fd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x5589065fdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589064fe060 con 0x5589065fd8b0 2023-10-10T17:44:00.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589065508a0 con 0x5589065f9470 2023-10-10T17:44:00.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f673b07bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558906549d30 con 0x5589065f8b60 2023-10-10T17:44:00.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f672d48c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x5589065fdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f672d48c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x5589065fdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45416/0 (socket says 172.21.15.78:45416) 2023-10-10T17:44:00.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.657+0000 7f672d48c700 1 -- 172.21.15.78:0/1453318839 learned_addr learned my addr 172.21.15.78:0/1453318839 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:00.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672cc8b700 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065f8b60 0x5589065f8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f6727fff700 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f9470 0x5589065fd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672d48c700 1 -- 172.21.15.78:0/1453318839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065f8b60 msgr2=0x5589065f8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672d48c700 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065f8b60 0x5589065f8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672d48c700 1 -- 172.21.15.78:0/1453318839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f9470 msgr2=0x5589065fd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672d48c700 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f9470 0x5589065fd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672d48c700 1 -- 172.21.15.78:0/1453318839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5589063cb630 con 0x5589065fd8b0 2023-10-10T17:44:00.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672cc8b700 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065f8b60 0x5589065f8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:00.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f672d48c700 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x5589065fdd00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f672000b830 tx=0x7f6720009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d39c7c723f04cce server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:00.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f671effd700 1 -- 172.21.15.78:0/1453318839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672000e050 con 0x5589065fd8b0 2023-10-10T17:44:00.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f671effd700 1 -- 172.21.15.78:0/1453318839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67200079a0 con 0x5589065fd8b0 2023-10-10T17:44:00.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.658+0000 7f671effd700 1 -- 172.21.15.78:0/1453318839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67200056d0 con 0x5589065fd8b0 2023-10-10T17:44:00.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 -- 172.21.15.78:0/1453318839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 msgr2=0x5589065fdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x5589065fdd00 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f672000b830 tx=0x7f6720009290 comp rx=0 tx=0).stop 2023-10-10T17:44:00.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 -- 172.21.15.78:0/1453318839 shutdown_connections 2023-10-10T17:44:00.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065f8b60 0x5589065f8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x5589065fdd00 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 --2- 172.21.15.78:0/1453318839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f9470 0x5589065fd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 -- 172.21.15.78:0/1453318839 >> 172.21.15.78:0/1453318839 conn(0x5589065ee360 msgr2=0x5589065f3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:00.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 -- 172.21.15.78:0/1453318839 shutdown_connections 2023-10-10T17:44:00.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 -- 172.21.15.78:0/1453318839 wait complete. 2023-10-10T17:44:00.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 -- start start 2023-10-10T17:44:00.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.659+0000 7f673b07bb80 1 Processor -- start 2023-10-10T17:44:00.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906604fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065f9470 0x558906602250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906602880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f673b07bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589064eb8e0 con 0x5589065f9470 2023-10-10T17:44:00.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f673b07bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589064eb580 con 0x5589065f8b60 2023-10-10T17:44:00.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f673b07bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589063cd920 con 0x5589065fd8b0 2023-10-10T17:44:00.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f673b07bb80 10 client.0 _ll_register_callbacks cb 0x558906509780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:00.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065f9470 0x558906602250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065f9470 0x558906602250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45420/0 (socket says 172.21.15.78:45420) 2023-10-10T17:44:00.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 -- 172.21.15.78:0/4010547401 learned_addr learned my addr 172.21.15.78:0/4010547401 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:00.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f672cc8b700 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906604fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f672d48c700 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906602880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 -- 172.21.15.78:0/4010547401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 msgr2=0x558906602880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906602880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 -- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 msgr2=0x558906604fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906604fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 -- 172.21.15.78:0/4010547401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672810dac0 con 0x5589065f9470 2023-10-10T17:44:00.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6727fff700 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065f9470 0x558906602250 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f671000f6d0 tx=0x7f6710004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3340676ac96a46 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:00.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.660+0000 7f6707fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:00.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f6707fff700 1 -- 172.21.15.78:0/4010547401 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671001d630 con 0x5589065f9470 2023-10-10T17:44:00.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f6707fff700 1 -- 172.21.15.78:0/4010547401 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f671001d790 con 0x5589065f9470 2023-10-10T17:44:00.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5589065ebb20 con 0x5589065f9470 2023-10-10T17:44:00.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f6707fff700 1 -- 172.21.15.78:0/4010547401 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f671001d950 con 0x5589065f9470 2023-10-10T17:44:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 10 client.15990 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x558906510190 con 0x5589065f9470 2023-10-10T17:44:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f6707fff700 1 -- 172.21.15.78:0/4010547401 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5589065ebb20 con 0x5589065f9470 2023-10-10T17:44:00.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f6707fff700 1 -- 172.21.15.78:0/4010547401 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6710023020 con 0x5589065f9470 2023-10-10T17:44:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 10 client.15990 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 10 client.15990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 20 client.15990 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906612800 0x558906614c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 4 client.15990 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:00.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.661+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55890650c0a0 con 0x558906612800 2023-10-10T17:44:00.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.662+0000 7f672cc8b700 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906612800 0x558906614c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.662+0000 7f6707fff700 10 client.15990 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:00.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.662+0000 7f672cc8b700 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906612800 0x558906614c10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=698e3d10cc5eccc9 server_cookie=b3166f833f8f4b67 in_seq=0 out_seq=0 2023-10-10T17:44:00.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f6707fff700 1 -- 172.21.15.78:0/4010547401 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55890650c0a0 con 0x558906612800 2023-10-10T17:44:00.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f6707fff700 10 client.15990 handle_command_reply: tid=0 2023-10-10T17:44:00.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 client.15990 shutdown 2023-10-10T17:44:00.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065f9470 msgr2=0x558906602250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065f9470 0x558906602250 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f671000f6d0 tx=0x7f6710004750 comp rx=0 tx=0).stop 2023-10-10T17:44:00.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 shutdown_connections 2023-10-10T17:44:00.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906602880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906612800 0x558906614c10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906604fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 --2- 172.21.15.78:0/4010547401 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065f9470 0x558906602250 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f6707fff700 0 client.15990 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 >> 172.21.15.78:0/4010547401 conn(0x5589065ee360 msgr2=0x5589065f2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 shutdown_connections 2023-10-10T17:44:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 1 -- 172.21.15.78:0/4010547401 wait complete. 2023-10-10T17:44:00.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.663+0000 7f673b07bb80 20 client.15990 trim_cache size 0 max 16384 2023-10-10T17:44:00.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 Processor -- start 2023-10-10T17:44:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 -- start start 2023-10-10T17:44:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x558906604b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589066050b0 0x5589065f3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589065edee0 con 0x5589065fd8b0 2023-10-10T17:44:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589065f51c0 con 0x5589065f8b60 2023-10-10T17:44:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f673b07bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558906600800 con 0x5589066050b0 2023-10-10T17:44:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f6727fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x558906604b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f6727fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x558906604b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45428/0 (socket says 172.21.15.78:45428) 2023-10-10T17:44:00.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f6727fff700 1 -- 172.21.15.78:0/3864181177 learned_addr learned my addr 172.21.15.78:0/3864181177 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:00.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.681+0000 7f672cc8b700 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f6727fff700 1 -- 172.21.15.78:0/3864181177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589066050b0 msgr2=0x5589065f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f672d48c700 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589066050b0 0x5589065f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f6727fff700 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589066050b0 0x5589065f3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f6727fff700 1 -- 172.21.15.78:0/3864181177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 msgr2=0x558906609970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f6727fff700 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f6727fff700 1 -- 172.21.15.78:0/3864181177 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55890650fee0 con 0x5589065fd8b0 2023-10-10T17:44:00.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f6727fff700 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x558906604b70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6710025ba0 tx=0x7f6710005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae5ec6d6e72bd65 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:00.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f671f7fe700 1 -- 172.21.15.78:0/3864181177 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6710025600 con 0x5589065fd8b0 2023-10-10T17:44:00.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f671f7fe700 1 -- 172.21.15.78:0/3864181177 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f671000fb40 con 0x5589065fd8b0 2023-10-10T17:44:00.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.682+0000 7f671f7fe700 1 -- 172.21.15.78:0/3864181177 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f671001d430 con 0x5589065fd8b0 2023-10-10T17:44:00.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 -- 172.21.15.78:0/3864181177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 msgr2=0x558906604b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x558906604b70 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f6710025ba0 tx=0x7f6710005bb0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 -- 172.21.15.78:0/3864181177 shutdown_connections 2023-10-10T17:44:00.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589066050b0 0x5589065f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 --2- 172.21.15.78:0/3864181177 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589065fd8b0 0x558906604b70 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 -- 172.21.15.78:0/3864181177 >> 172.21.15.78:0/3864181177 conn(0x5589065ee360 msgr2=0x5589065fb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:00.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 -- 172.21.15.78:0/3864181177 shutdown_connections 2023-10-10T17:44:00.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 -- 172.21.15.78:0/3864181177 wait complete. 2023-10-10T17:44:00.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 -- start start 2023-10-10T17:44:00.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.683+0000 7f673b07bb80 1 Processor -- start 2023-10-10T17:44:00.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906605bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589066050b0 0x5589066060f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558906606670 con 0x5589066050b0 2023-10-10T17:44:00.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589066067e0 con 0x5589065f8b60 2023-10-10T17:44:00.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589066069a0 con 0x5589065fd8b0 2023-10-10T17:44:00.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 10 client.0 _ll_register_callbacks cb 0x7f671000dc20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:00.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589066050b0 0x5589066060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589066050b0 0x5589066060f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45436/0 (socket says 172.21.15.78:45436) 2023-10-10T17:44:00.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 -- 172.21.15.78:0/857545085 learned_addr learned my addr 172.21.15.78:0/857545085 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:00.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672cc8b700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f6727fff700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906605bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 -- 172.21.15.78:0/857545085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 msgr2=0x558906605bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906605bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 -- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 msgr2=0x558906609800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:00.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 -- 172.21.15.78:0/857545085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558906510190 con 0x5589066050b0 2023-10-10T17:44:00.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672cc8b700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:00.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f672d48c700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589066050b0 0x5589066060f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f672000b830 tx=0x7f6720004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19e92985cda5eef7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:00.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f6707fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:00.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f6707fff700 1 -- 172.21.15.78:0/857545085 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6720005450 con 0x5589066050b0 2023-10-10T17:44:00.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f6707fff700 1 -- 172.21.15.78:0/857545085 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f67200090d0 con 0x5589066050b0 2023-10-10T17:44:00.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.684+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67200079a0 con 0x5589066050b0 2023-10-10T17:44:00.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f6707fff700 1 -- 172.21.15.78:0/857545085 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f672000e020 con 0x5589066050b0 2023-10-10T17:44:00.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 10 client.15996 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:00.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x558906360c60 con 0x5589066050b0 2023-10-10T17:44:00.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f6707fff700 1 -- 172.21.15.78:0/857545085 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67200079a0 con 0x5589066050b0 2023-10-10T17:44:00.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f6707fff700 1 -- 172.21.15.78:0/857545085 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f672000b0d0 con 0x5589066050b0 2023-10-10T17:44:00.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 10 client.15996 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:00.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:00.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 20 client.15996 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:00.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906667820 0x558906669c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:00.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 4 client.15996 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:44:00.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558906518660 con 0x558906667820 2023-10-10T17:44:00.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.685+0000 7f672cc8b700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906667820 0x558906669c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:00.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f6707fff700 10 client.15996 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:00.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f672cc8b700 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906667820 0x558906669c00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86edb595a4a453a9 server_cookie=717779e2efea2140 in_seq=0 out_seq=0 2023-10-10T17:44:00.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f6707fff700 1 -- 172.21.15.78:0/857545085 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558906518660 con 0x558906667820 2023-10-10T17:44:00.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f6707fff700 10 client.15996 handle_command_reply: tid=0 2023-10-10T17:44:00.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 client.15996 shutdown 2023-10-10T17:44:00.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589066050b0 msgr2=0x5589066060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589066050b0 0x5589066060f0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f672000b830 tx=0x7f6720004d10 comp rx=0 tx=0).stop 2023-10-10T17:44:00.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 shutdown_connections 2023-10-10T17:44:00.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558906667820 0x558906669c00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5589066050b0 0x5589066060f0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5589065f8b60 0x558906609800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 --2- 172.21.15.78:0/857545085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5589065fd8b0 0x558906605bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f6707fff700 0 client.15996 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:00.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 >> 172.21.15.78:0/857545085 conn(0x5589065ee360 msgr2=0x5589065f7960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:00.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 shutdown_connections 2023-10-10T17:44:00.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 1 -- 172.21.15.78:0/857545085 wait complete. 2023-10-10T17:44:00.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.686+0000 7f673b07bb80 20 client.15996 trim_cache size 0 max 16384 2023-10-10T17:44:00.709 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:44:00.709 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:44:00.710 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:44:00.710 INFO:teuthology.orchestra.run.smithi078.stdout: "7a838c33-ca0d-4be7-b908-f785a2885899": { 2023-10-10T17:44:00.710 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:44:00.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "7a838c33-ca0d-4be7-b908-f785a2885899", 2023-10-10T17:44:00.710 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:44:00.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:44:00.711 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:44:00.711 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:44:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f67140a13e0 msgr2=0x7f67140a3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f67140a13e0 0x7f67140a3890 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6710005fd0 tx=0x7f6710005dc0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67281009e0 msgr2=0x7f67281965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:00.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67281009e0 0x7f67281965b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f671800c9b0 tx=0x7f671800cd70 comp rx=0 tx=0).stop 2023-10-10T17:44:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 shutdown_connections 2023-10-10T17:44:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f67140a13e0 0x7f67140a3890 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6728106410 0x7f672819af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f67281009e0 0x7f67281965b0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 --2- 172.21.15.78:0/1314755085 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6728101370 0x7f6728196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 >> 172.21.15.78:0/1314755085 conn(0x7f67280775c0 msgr2=0x7f67280ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 shutdown_connections 2023-10-10T17:44:00.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:00.687+0000 7f672eeef700 1 -- 172.21.15.78:0/1314755085 wait complete. 2023-10-10T17:44:00.717 INFO:tasks.cephfs.filesystem:scrub status for tag:7a838c33-ca0d-4be7-b908-f785a2885899 - {'path': '/', 'tag': '7a838c33-ca0d-4be7-b908-f785a2885899', 'options': 'recursive,force'} 2023-10-10T17:44:00.927 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:00.927 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:01.114 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:01.114 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:01.122 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:01.122 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:01.122 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 47 -lt 100 ']' 2023-10-10T17:44:01.122 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 47 + 1 2023-10-10T17:44:01.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:00 smithi078 ceph-mon[99494]: pgmap v47: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2023-10-10T17:44:01.144 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=48 2023-10-10T17:44:01.145 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:00 smithi190 ceph-mon[101220]: pgmap v47: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2023-10-10T17:44:01.186 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:01.186 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:01.187 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:01.187 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:01.187 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:00 smithi157 ceph-mon[100896]: pgmap v47: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2023-10-10T17:44:01.519 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:01.519 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:01.680 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:01.681 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:01.697 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:01.697 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:01.698 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 48 -lt 100 ']' 2023-10-10T17:44:01.698 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 48 + 1 2023-10-10T17:44:01.720 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=49 2023-10-10T17:44:01.720 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:01.744 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:01.745 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:01.746 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:01.746 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:01.747 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:02.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:01 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:44:02.085 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:02.085 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:02.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:01 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:44:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:01 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:44:02.264 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:02.264 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:02.281 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:02.281 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:02.281 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 49 -lt 100 ']' 2023-10-10T17:44:02.281 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 49 + 1 2023-10-10T17:44:02.311 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=50 2023-10-10T17:44:02.311 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:02.337 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:02.337 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:02.338 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:02.338 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:02.338 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:02.793 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:02.794 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:03.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:02 smithi078 ceph-mon[99494]: pgmap v48: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2023-10-10T17:44:03.138 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:03.139 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:03.155 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:03.156 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:03.156 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 50 -lt 100 ']' 2023-10-10T17:44:03.156 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 50 + 1 2023-10-10T17:44:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:02 smithi190 ceph-mon[101220]: pgmap v48: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2023-10-10T17:44:03.179 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=51 2023-10-10T17:44:03.179 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:03.209 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:03.209 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:03.210 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:03.210 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:03.210 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:03.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:02 smithi157 ceph-mon[100896]: pgmap v48: 129 pgs: 129 active+clean; 1.6 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 KiB/s wr, 11 op/s 2023-10-10T17:44:03.610 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:03.611 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:03.797 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:03.797 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:03.814 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:03.814 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:03.814 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 51 -lt 100 ']' 2023-10-10T17:44:03.814 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 51 + 1 2023-10-10T17:44:03.844 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=52 2023-10-10T17:44:03.844 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:03.875 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:03.876 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:03.877 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:03.877 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:03.877 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:04.218 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:04.218 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:04.330 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:04.331 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:04.339 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:04.339 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:04.339 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 52 -lt 100 ']' 2023-10-10T17:44:04.340 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 52 + 1 2023-10-10T17:44:04.361 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=53 2023-10-10T17:44:04.361 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:04.387 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:04.387 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:04.388 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:04.388 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:04.388 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:04.760 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:04.760 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:04.897 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:04.897 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:04.905 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:04.905 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:04.906 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 53 -lt 100 ']' 2023-10-10T17:44:04.906 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 53 + 1 2023-10-10T17:44:04.936 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=54 2023-10-10T17:44:04.937 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:04.962 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:04.962 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:04.963 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:04.963 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:04.964 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:05.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:04 smithi078 ceph-mon[99494]: pgmap v49: 129 pgs: 129 active+clean; 1.7 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 38 KiB/s wr, 27 op/s 2023-10-10T17:44:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:04 smithi190 ceph-mon[101220]: pgmap v49: 129 pgs: 129 active+clean; 1.7 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 38 KiB/s wr, 27 op/s 2023-10-10T17:44:05.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:04 smithi157 ceph-mon[100896]: pgmap v49: 129 pgs: 129 active+clean; 1.7 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 38 KiB/s wr, 27 op/s 2023-10-10T17:44:05.301 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:05.302 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:05.438 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:05.438 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:05.446 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:05.447 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:05.447 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 54 -lt 100 ']' 2023-10-10T17:44:05.447 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 54 + 1 2023-10-10T17:44:05.477 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=55 2023-10-10T17:44:05.477 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:05.508 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:05.509 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:05.510 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:05.510 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:05.510 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:05.884 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:05.885 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:06.038 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:06.039 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:06.046 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:06.047 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:06.047 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 55 -lt 100 ']' 2023-10-10T17:44:06.047 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 55 + 1 2023-10-10T17:44:06.078 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=56 2023-10-10T17:44:06.078 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:06.119 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:06.119 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:06.120 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:06.120 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:06.120 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:06.560 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:06.561 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:06.705 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:06.705 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:06.714 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:06.714 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:06.714 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 56 -lt 100 ']' 2023-10-10T17:44:06.715 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 56 + 1 2023-10-10T17:44:06.758 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=57 2023-10-10T17:44:06.758 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:06.789 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:06.790 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:06.791 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:06.791 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:06.791 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:07.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:06 smithi078 ceph-mon[99494]: pgmap v50: 129 pgs: 129 active+clean; 1.7 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 34 KiB/s wr, 24 op/s 2023-10-10T17:44:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:06 smithi190 ceph-mon[101220]: pgmap v50: 129 pgs: 129 active+clean; 1.7 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 34 KiB/s wr, 24 op/s 2023-10-10T17:44:07.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:06 smithi157 ceph-mon[100896]: pgmap v50: 129 pgs: 129 active+clean; 1.7 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 34 KiB/s wr, 24 op/s 2023-10-10T17:44:07.509 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:07.510 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:07.796 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:07.797 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:07.813 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:07.814 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:07.814 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 57 -lt 100 ']' 2023-10-10T17:44:07.814 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 57 + 1 2023-10-10T17:44:07.882 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=58 2023-10-10T17:44:07.882 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:07.967 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:07.967 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:07.969 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:07.969 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:07.969 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:08.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:07 smithi078 ceph-mon[99494]: mgrmap e22: x(active, since 91s), standbys: y, z 2023-10-10T17:44:08.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:07 smithi190 ceph-mon[101220]: mgrmap e22: x(active, since 91s), standbys: y, z 2023-10-10T17:44:08.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:07 smithi157 ceph-mon[100896]: mgrmap e22: x(active, since 91s), standbys: y, z 2023-10-10T17:44:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 -- 172.21.15.78:0/1453370269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 msgr2=0x7f614c1016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 --2- 172.21.15.78:0/1453370269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c1016b0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f614400b0a0 tx=0x7f614400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 -- 172.21.15.78:0/1453370269 shutdown_connections 2023-10-10T17:44:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 --2- 172.21.15.78:0/1453370269 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c10a800 0x7f614c10cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 --2- 172.21.15.78:0/1453370269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c1016b0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 --2- 172.21.15.78:0/1453370269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f614c101bf0 0x7f614c10a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 -- 172.21.15.78:0/1453370269 >> 172.21.15.78:0/1453370269 conn(0x7f614c0763a0 msgr2=0x7f614c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:08.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.302+0000 7f6153042700 1 -- 172.21.15.78:0/1453370269 shutdown_connections 2023-10-10T17:44:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 -- 172.21.15.78:0/1453370269 wait complete. 2023-10-10T17:44:08.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 Processor -- start 2023-10-10T17:44:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 -- start start 2023-10-10T17:44:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c0ff110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f614c101bf0 0x7f614c0ff650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c10a800 0x7f614c1a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c0ffe80 con 0x7f614c1012e0 2023-10-10T17:44:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c1a4be0 con 0x7f614c10a800 2023-10-10T17:44:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6153042700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c1a4d20 con 0x7f614c101bf0 2023-10-10T17:44:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.303+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c0ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c0ff110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45446/0 (socket says 172.21.15.78:45446) 2023-10-10T17:44:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 -- 172.21.15.78:0/516215919 learned_addr learned my addr 172.21.15.78:0/516215919 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f614bfff700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f614c101bf0 0x7f614c0ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f61515df700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c10a800 0x7f614c1a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 -- 172.21.15.78:0/516215919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f614c101bf0 msgr2=0x7f614c0ff650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f614c101bf0 0x7f614c0ff650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 -- 172.21.15.78:0/516215919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c10a800 msgr2=0x7f614c1a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c10a800 0x7f614c1a46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 -- 172.21.15.78:0/516215919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6144009d00 con 0x7f614c1012e0 2023-10-10T17:44:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6150dde700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c0ff110 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6144006070 tx=0x7f61440079b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6149ffb700 1 -- 172.21.15.78:0/516215919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614400e040 con 0x7f614c1012e0 2023-10-10T17:44:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c110270 con 0x7f614c1012e0 2023-10-10T17:44:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6149ffb700 1 -- 172.21.15.78:0/516215919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6144004ce0 con 0x7f614c1012e0 2023-10-10T17:44:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6149ffb700 1 -- 172.21.15.78:0/516215919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614401d480 con 0x7f614c1012e0 2023-10-10T17:44:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.304+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c1100e0 con 0x7f614c1012e0 2023-10-10T17:44:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.307+0000 7f6149ffb700 1 -- 172.21.15.78:0/516215919 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6144003a00 con 0x7f614c1012e0 2023-10-10T17:44:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.307+0000 7f6149ffb700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f61340a12c0 0x7f61340a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.307+0000 7f6149ffb700 1 -- 172.21.15.78:0/516215919 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f61440ce520 con 0x7f614c1012e0 2023-10-10T17:44:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.307+0000 7f614bfff700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f61340a12c0 0x7f61340a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.308+0000 7f614bfff700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f61340a12c0 0x7f61340a3770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f614c100900 tx=0x7f614000b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:08.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.381+0000 7f615f1ceb80 1 Processor -- start 2023-10-10T17:44:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.381+0000 7f615f1ceb80 1 -- start start 2023-10-10T17:44:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.381+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad411e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.381+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41720 0x55b69ad3b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.381+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41b90 0x55b69ad3bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ac41000 con 0x55b69ab976b0 2023-10-10T17:44:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ac93840 con 0x55b69ad41b90 2023-10-10T17:44:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ac8ccd0 con 0x55b69ad41720 2023-10-10T17:44:08.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad411e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad411e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45454/0 (socket says 172.21.15.78:45454) 2023-10-10T17:44:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 -- 172.21.15.78:0/2963145818 learned_addr learned my addr 172.21.15.78:0/2963145818 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f614bfff700 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41720 0x55b69ad3b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f61515df700 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41b90 0x55b69ad3bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 -- 172.21.15.78:0/2963145818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41720 msgr2=0x55b69ad3b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41720 0x55b69ad3b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 -- 172.21.15.78:0/2963145818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41b90 msgr2=0x55b69ad3bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41b90 0x55b69ad3bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 -- 172.21.15.78:0/2963145818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b69ab0e5d0 con 0x55b69ab976b0 2023-10-10T17:44:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f61515df700 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41b90 0x55b69ad3bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6150dde700 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad411e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6144097dc0 tx=0x7f6144004020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d015dfaf57ccdd2a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6132ffd700 1 -- 172.21.15.78:0/2963145818 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440ce070 con 0x55b69ab976b0 2023-10-10T17:44:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6132ffd700 1 -- 172.21.15.78:0/2963145818 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6144004e50 con 0x55b69ab976b0 2023-10-10T17:44:08.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.382+0000 7f6132ffd700 1 -- 172.21.15.78:0/2963145818 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440cbbc0 con 0x55b69ab976b0 2023-10-10T17:44:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 -- 172.21.15.78:0/2963145818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 msgr2=0x55b69ad411e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad411e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6144097dc0 tx=0x7f6144004020 comp rx=0 tx=0).stop 2023-10-10T17:44:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 -- 172.21.15.78:0/2963145818 shutdown_connections 2023-10-10T17:44:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41b90 0x55b69ad3bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad411e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/2963145818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41720 0x55b69ad3b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 -- 172.21.15.78:0/2963145818 >> 172.21.15.78:0/2963145818 conn(0x55b69ad30f50 msgr2=0x55b69ad36270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 -- 172.21.15.78:0/2963145818 shutdown_connections 2023-10-10T17:44:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.383+0000 7f615f1ceb80 1 -- 172.21.15.78:0/2963145818 wait complete. 2023-10-10T17:44:08.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 -- start start 2023-10-10T17:44:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 Processor -- start 2023-10-10T17:44:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad4a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad4ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad44c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ac2e880 con 0x55b69ab976b0 2023-10-10T17:44:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ac2e520 con 0x55b69ad41720 2023-10-10T17:44:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ab108c0 con 0x55b69ad41b90 2023-10-10T17:44:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f615f1ceb80 10 client.0 _ll_register_callbacks cb 0x55b69ac4c720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad4a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad4a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45460/0 (socket says 172.21.15.78:45460) 2023-10-10T17:44:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.384+0000 7f6150dde700 1 -- 172.21.15.78:0/20496974 learned_addr learned my addr 172.21.15.78:0/20496974 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f614bfff700 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad4ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f61515df700 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad44c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f6150dde700 1 -- 172.21.15.78:0/20496974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 msgr2=0x55b69ad44c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f6150dde700 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad44c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f6150dde700 1 -- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 msgr2=0x55b69ad4ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f6150dde700 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad4ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.403 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f6150dde700 1 -- 172.21.15.78:0/20496974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c1100e0 con 0x55b69ab976b0 2023-10-10T17:44:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f6150dde700 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad4a920 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f614409b050 tx=0x7f6144007550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474ad74367cb13bd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f612bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f612bfff700 1 -- 172.21.15.78:0/20496974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440ce040 con 0x55b69ab976b0 2023-10-10T17:44:08.404 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f612bfff700 1 -- 172.21.15.78:0/20496974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f614409b350 con 0x55b69ab976b0 2023-10-10T17:44:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b69ad2e710 con 0x55b69ab976b0 2023-10-10T17:44:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f612bfff700 1 -- 172.21.15.78:0/20496974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61440a3710 con 0x55b69ab976b0 2023-10-10T17:44:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f615f1ceb80 10 client.16008 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:08.405 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b69ac53130 con 0x55b69ab976b0 2023-10-10T17:44:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.385+0000 7f612bfff700 1 -- 172.21.15.78:0/20496974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b69ad2e710 con 0x55b69ab976b0 2023-10-10T17:44:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f612bfff700 1 -- 172.21.15.78:0/20496974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f61440ce1a0 con 0x55b69ab976b0 2023-10-10T17:44:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f615f1ceb80 10 client.16008 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:08.406 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f615f1ceb80 10 client.16008 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f615f1ceb80 20 client.16008 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ad54940 0x55b69ad56d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f615f1ceb80 4 client.16008 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b69ac4f040 con 0x55b69ad54940 2023-10-10T17:44:08.407 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f614bfff700 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ad54940 0x55b69ad56d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f612bfff700 10 client.16008 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.386+0000 7f614bfff700 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ad54940 0x55b69ad56d20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52c43532d3f1d53d server_cookie=fcf772f7bf5ff94e in_seq=0 out_seq=0 2023-10-10T17:44:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f612bfff700 1 -- 172.21.15.78:0/20496974 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b69ac4f040 con 0x55b69ad54940 2023-10-10T17:44:08.408 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f612bfff700 10 client.16008 handle_command_reply: tid=0 2023-10-10T17:44:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 client.16008 shutdown 2023-10-10T17:44:08.409 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 msgr2=0x55b69ad4a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad4a920 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f614409b050 tx=0x7f6144007550 comp rx=0 tx=0).stop 2023-10-10T17:44:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 shutdown_connections 2023-10-10T17:44:08.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad44c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ad54940 0x55b69ad56d20 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad4a920 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/20496974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad4ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f612bfff700 0 client.16008 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:08.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 >> 172.21.15.78:0/20496974 conn(0x55b69ad30f50 msgr2=0x55b69ad357e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 shutdown_connections 2023-10-10T17:44:08.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.387+0000 7f615f1ceb80 1 -- 172.21.15.78:0/20496974 wait complete. 2023-10-10T17:44:08.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.388+0000 7f615f1ceb80 20 client.16008 trim_cache size 0 max 16384 2023-10-10T17:44:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 Processor -- start 2023-10-10T17:44:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 -- start start 2023-10-10T17:44:08.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad35f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad444c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad44a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ad33020 con 0x55b69ab976b0 2023-10-10T17:44:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ad38080 con 0x55b69ad41720 2023-10-10T17:44:08.423 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ad36fe0 con 0x55b69ad41b90 2023-10-10T17:44:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad35f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad35f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45468/0 (socket says 172.21.15.78:45468) 2023-10-10T17:44:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.402+0000 7f6150dde700 1 -- 172.21.15.78:0/3957480195 learned_addr learned my addr 172.21.15.78:0/3957480195 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f6150dde700 1 -- 172.21.15.78:0/3957480195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 msgr2=0x55b69ad44a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f6150dde700 1 --2- 172.21.15.78:0/3957480195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad44a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f6150dde700 1 -- 172.21.15.78:0/3957480195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 msgr2=0x55b69ad444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f6150dde700 1 --2- 172.21.15.78:0/3957480195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f6150dde700 1 -- 172.21.15.78:0/3957480195 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b69ac52e80 con 0x55b69ab976b0 2023-10-10T17:44:08.425 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f6150dde700 1 --2- 172.21.15.78:0/3957480195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad35f60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6144097850 tx=0x7f6144004480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23068dbefa6e5d70 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f61337fe700 1 -- 172.21.15.78:0/3957480195 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614409b7a0 con 0x55b69ab976b0 2023-10-10T17:44:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f61337fe700 1 -- 172.21.15.78:0/3957480195 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f614409ae90 con 0x55b69ab976b0 2023-10-10T17:44:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.403+0000 7f61337fe700 1 -- 172.21.15.78:0/3957480195 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440a37a0 con 0x55b69ab976b0 2023-10-10T17:44:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3957480195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 msgr2=0x55b69ad35f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.426 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3957480195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad35f60 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6144097850 tx=0x7f6144004480 comp rx=0 tx=0).stop 2023-10-10T17:44:08.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3957480195 shutdown_connections 2023-10-10T17:44:08.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3957480195 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad44a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3957480195 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad35f60 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3957480195 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.427 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3957480195 >> 172.21.15.78:0/3957480195 conn(0x55b69ad30f50 msgr2=0x55b69ad454f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:08.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3957480195 shutdown_connections 2023-10-10T17:44:08.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3957480195 wait complete. 2023-10-10T17:44:08.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 -- start start 2023-10-10T17:44:08.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.404+0000 7f615f1ceb80 1 Processor -- start 2023-10-10T17:44:08.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad484d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.428 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad48a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad48f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ada77a0 con 0x55b69ab976b0 2023-10-10T17:44:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ada78e0 con 0x55b69ad41720 2023-10-10T17:44:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b69ada7aa0 con 0x55b69ad41b90 2023-10-10T17:44:08.429 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 10 client.0 _ll_register_callbacks cb 0x7f6144097780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad484d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45474/0 (socket says 172.21.15.78:45474) 2023-10-10T17:44:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 -- 172.21.15.78:0/3819504424 learned_addr learned my addr 172.21.15.78:0/3819504424 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:08.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f614bfff700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad48a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f61515df700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad48f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 -- 172.21.15.78:0/3819504424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 msgr2=0x55b69ad48f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad48f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 -- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 msgr2=0x55b69ad48a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:08.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad48a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 -- 172.21.15.78:0/3819504424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b69ac53130 con 0x55b69ab976b0 2023-10-10T17:44:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f61515df700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad48f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f6150dde700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad484d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6144097dc0 tx=0x7f614409acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ec9c1b0c4745de server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f612bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:08.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f612bfff700 1 -- 172.21.15.78:0/3819504424 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61440ce030 con 0x55b69ab976b0 2023-10-10T17:44:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f612bfff700 1 -- 172.21.15.78:0/3819504424 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f614409b350 con 0x55b69ab976b0 2023-10-10T17:44:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6144004e50 con 0x55b69ab976b0 2023-10-10T17:44:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f612bfff700 1 -- 172.21.15.78:0/3819504424 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61440a3710 con 0x55b69ab976b0 2023-10-10T17:44:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 10 client.16014 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.405+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b69aaa3c00 con 0x55b69ab976b0 2023-10-10T17:44:08.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f612bfff700 1 -- 172.21.15.78:0/3819504424 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6144004e50 con 0x55b69ab976b0 2023-10-10T17:44:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f612bfff700 1 -- 172.21.15.78:0/3819504424 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f61440ce190 con 0x55b69ab976b0 2023-10-10T17:44:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f615f1ceb80 10 client.16014 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f615f1ceb80 10 client.16014 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f615f1ceb80 20 client.16014 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:08.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ada8d00 0x55b69adab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f615f1ceb80 4 client.16014 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:44:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b69ac5b600 con 0x55b69ada8d00 2023-10-10T17:44:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f614bfff700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ada8d00 0x55b69adab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f612bfff700 10 client.16014 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:08.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.406+0000 7f614bfff700 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ada8d00 0x55b69adab0e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78e949efd446f3a5 server_cookie=715de07d4f04d94f in_seq=0 out_seq=0 2023-10-10T17:44:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f612bfff700 1 -- 172.21.15.78:0/3819504424 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b69ac5b600 con 0x55b69ada8d00 2023-10-10T17:44:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f612bfff700 10 client.16014 handle_command_reply: tid=0 2023-10-10T17:44:08.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 client.16014 shutdown 2023-10-10T17:44:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 msgr2=0x55b69ad484d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad484d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6144097dc0 tx=0x7f614409acb0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 shutdown_connections 2023-10-10T17:44:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b69ad41b90 0x55b69ad48f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b69ada8d00 0x55b69adab0e0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b69ab976b0 0x55b69ad484d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f612bfff700 0 client.16014 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 --2- 172.21.15.78:0/3819504424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b69ad41720 0x55b69ad48a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.425+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 >> 172.21.15.78:0/3819504424 conn(0x55b69ad30f50 msgr2=0x55b69ad44490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:08.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 shutdown_connections 2023-10-10T17:44:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f615f1ceb80 1 -- 172.21.15.78:0/3819504424 wait complete. 2023-10-10T17:44:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f615f1ceb80 20 client.16014 trim_cache size 0 max 16384 2023-10-10T17:44:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:44:08.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:44:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:44:08.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:44:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f61340a12c0 msgr2=0x7f61340a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f61340a12c0 0x7f61340a3770 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f614c100900 tx=0x7f614000b410 comp rx=0 tx=0).stop 2023-10-10T17:44:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 msgr2=0x7f614c0ff110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:08.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c0ff110 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f6144006070 tx=0x7f61440079b0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 shutdown_connections 2023-10-10T17:44:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f61340a12c0 0x7f61340a3770 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f614c10a800 0x7f614c1a46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f614c1012e0 0x7f614c0ff110 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 --2- 172.21.15.78:0/516215919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f614c101bf0 0x7f614c0ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:08.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 >> 172.21.15.78:0/516215919 conn(0x7f614c0763a0 msgr2=0x7f614c10b070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:08.444 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.426+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 shutdown_connections 2023-10-10T17:44:08.444 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:08.427+0000 7f6153042700 1 -- 172.21.15.78:0/516215919 wait complete. 2023-10-10T17:44:08.801 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:08.801 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:08.988 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:08.988 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:09.013 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.013 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.013 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 58 -lt 100 ']' 2023-10-10T17:44:09.014 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 58 + 1 2023-10-10T17:44:09.043 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=59 2023-10-10T17:44:09.044 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:09.074 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:09.075 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:09.075 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:09.076 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:09.076 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:09.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:08 smithi078 ceph-mon[99494]: pgmap v51: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 42 KiB/s wr, 28 op/s 2023-10-10T17:44:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:08 smithi190 ceph-mon[101220]: pgmap v51: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 42 KiB/s wr, 28 op/s 2023-10-10T17:44:09.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:08 smithi157 ceph-mon[100896]: pgmap v51: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 42 KiB/s wr, 28 op/s 2023-10-10T17:44:09.353 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:09.353 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:09.410 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:09.411 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:09.414 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.414 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.414 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 59 -lt 100 ']' 2023-10-10T17:44:09.414 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 59 + 1 2023-10-10T17:44:09.423 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=60 2023-10-10T17:44:09.424 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:09.431 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:09.431 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:09.432 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:09.432 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:09.432 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:09.568 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:09.568 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:09.607 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:09.608 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:09.611 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.611 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.611 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 60 -lt 100 ']' 2023-10-10T17:44:09.612 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 60 + 1 2023-10-10T17:44:09.623 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=61 2023-10-10T17:44:09.624 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:09.634 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:09.634 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:09.635 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:09.635 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:09.635 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:09.861 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:09.861 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:09.956 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:09.956 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:09.963 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.963 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:09.963 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 61 -lt 100 ']' 2023-10-10T17:44:09.963 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 61 + 1 2023-10-10T17:44:09.993 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=62 2023-10-10T17:44:09.994 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:09.999 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:10.000 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:10.000 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:10.001 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:10.001 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:10.401 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:10.401 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:10.512 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:10.513 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:10.523 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:10.523 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:10.523 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 62 -lt 100 ']' 2023-10-10T17:44:10.524 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 62 + 1 2023-10-10T17:44:10.552 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=63 2023-10-10T17:44:10.552 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:10.583 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:10.583 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:10.584 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:10.584 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:10.584 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:10.926 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:10.926 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:11.087 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:11.088 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:11.095 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:11.095 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:11.096 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 63 -lt 100 ']' 2023-10-10T17:44:11.096 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 63 + 1 2023-10-10T17:44:11.118 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=64 2023-10-10T17:44:11.118 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:11.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:10 smithi078 ceph-mon[99494]: pgmap v52: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 20 op/s 2023-10-10T17:44:11.149 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:11.150 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:11.150 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:11.151 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:11.151 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:11.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:10 smithi190 ceph-mon[101220]: pgmap v52: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 20 op/s 2023-10-10T17:44:11.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:10 smithi157 ceph-mon[100896]: pgmap v52: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 20 op/s 2023-10-10T17:44:11.517 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:11.518 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:11.637 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:11.638 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:11.646 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:11.647 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:11.647 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 64 -lt 100 ']' 2023-10-10T17:44:11.647 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 64 + 1 2023-10-10T17:44:11.676 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=65 2023-10-10T17:44:11.677 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:11.703 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:11.703 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:11.704 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:11.704 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:11.704 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:12.026 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:12.027 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:12.154 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:12.154 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:12.162 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:12.162 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:12.163 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 65 -lt 100 ']' 2023-10-10T17:44:12.163 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 65 + 1 2023-10-10T17:44:12.193 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=66 2023-10-10T17:44:12.194 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:12.219 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:12.219 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:12.220 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:12.220 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:12.221 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:12.617 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:12.617 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:12.737 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:12.738 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:12.746 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:12.746 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:12.747 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 66 -lt 100 ']' 2023-10-10T17:44:12.747 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 66 + 1 2023-10-10T17:44:12.776 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=67 2023-10-10T17:44:12.777 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:12.803 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:12.803 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:12.804 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:12.804 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:12.804 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:12.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:12 smithi078 ceph-mon[99494]: pgmap v53: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 20 op/s 2023-10-10T17:44:13.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:12 smithi190 ceph-mon[101220]: pgmap v53: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 20 op/s 2023-10-10T17:44:13.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:12 smithi157 ceph-mon[100896]: pgmap v53: 129 pgs: 129 active+clean; 1.8 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 36 KiB/s wr, 20 op/s 2023-10-10T17:44:13.342 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:13.343 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:13.472 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:13.474 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:13.478 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:13.479 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:13.479 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 67 -lt 100 ']' 2023-10-10T17:44:13.479 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 67 + 1 2023-10-10T17:44:13.501 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=68 2023-10-10T17:44:13.501 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:13.535 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:13.536 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:13.536 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:13.537 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:13.537 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:13.917 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:13.917 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:14.037 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:14.037 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:14.046 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:14.046 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:14.046 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 68 -lt 100 ']' 2023-10-10T17:44:14.046 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 68 + 1 2023-10-10T17:44:14.076 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=69 2023-10-10T17:44:14.076 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:14.107 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:14.108 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:14.108 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:14.109 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:14.109 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:14.492 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:14.492 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:14.620 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:14.621 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:14.629 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:14.629 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:14.630 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 69 -lt 100 ']' 2023-10-10T17:44:14.630 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 69 + 1 2023-10-10T17:44:14.652 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=70 2023-10-10T17:44:14.652 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:14.682 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:14.683 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:14.683 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:14.684 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:14.684 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:15.036 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:14 smithi078 ceph-mon[99494]: pgmap v54: 129 pgs: 129 active+clean; 1.9 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 47 KiB/s wr, 30 op/s 2023-10-10T17:44:15.075 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:15.075 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:14 smithi190 ceph-mon[101220]: pgmap v54: 129 pgs: 129 active+clean; 1.9 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 47 KiB/s wr, 30 op/s 2023-10-10T17:44:15.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:14 smithi157 ceph-mon[100896]: pgmap v54: 129 pgs: 129 active+clean; 1.9 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 47 KiB/s wr, 30 op/s 2023-10-10T17:44:15.237 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:15.237 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:15.245 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:15.246 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:15.246 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 70 -lt 100 ']' 2023-10-10T17:44:15.246 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 70 + 1 2023-10-10T17:44:15.268 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=71 2023-10-10T17:44:15.268 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:15.294 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:15.294 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:15.295 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:15.295 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:15.295 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:15.608 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:15.609 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:15.737 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:15.737 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:15.745 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:15.745 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:15.745 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 71 -lt 100 ']' 2023-10-10T17:44:15.746 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 71 + 1 2023-10-10T17:44:15.768 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=72 2023-10-10T17:44:15.768 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:15.794 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:15.794 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:15.795 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:15.795 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:15.795 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:16.175 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:16.175 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:16.312 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:16.312 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:16.329 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:16.329 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:16.329 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 72 -lt 100 ']' 2023-10-10T17:44:16.329 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 72 + 1 2023-10-10T17:44:16.351 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=73 2023-10-10T17:44:16.351 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:16.377 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:16.377 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:16.378 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:16.378 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:16.379 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:16.783 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:16.784 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:16.903 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:16.904 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:16.911 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:16.911 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:16.912 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 73 -lt 100 ']' 2023-10-10T17:44:16.912 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 73 + 1 2023-10-10T17:44:16.934 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=74 2023-10-10T17:44:16.934 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:16.965 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:16.965 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:16.966 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:16.966 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:16.966 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:17.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:16 smithi078 ceph-mon[99494]: pgmap v55: 129 pgs: 129 active+clean; 1.9 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:17.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:16 smithi190 ceph-mon[101220]: pgmap v55: 129 pgs: 129 active+clean; 1.9 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:17.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:16 smithi157 ceph-mon[100896]: pgmap v55: 129 pgs: 129 active+clean; 1.9 MiB data, 84 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:17.317 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:17.317 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:17.478 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:17.479 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:17.495 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:17.496 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:17.496 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 74 -lt 100 ']' 2023-10-10T17:44:17.496 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 74 + 1 2023-10-10T17:44:17.517 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=75 2023-10-10T17:44:17.518 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:17.549 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:17.549 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:17.550 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:17.550 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:17.550 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:17.900 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:17.900 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:18.037 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:18.037 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:18.045 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:18.119 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:18.120 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 75 -lt 100 ']' 2023-10-10T17:44:18.120 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 75 + 1 2023-10-10T17:44:18.121 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=76 2023-10-10T17:44:18.121 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:18.127 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:18.127 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:18.128 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:18.128 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:18.128 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:19.108 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:19.108 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:19.261 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:19.262 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:19.278 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:19.279 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:19.279 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 76 -lt 100 ']' 2023-10-10T17:44:19.279 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 76 + 1 2023-10-10T17:44:19.301 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=77 2023-10-10T17:44:19.301 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:19.326 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:19.327 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:19.327 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:19.328 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:19.328 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:19.733 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:19.734 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:19.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:19 smithi078 ceph-mon[99494]: pgmap v56: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 29 KiB/s wr, 17 op/s 2023-10-10T17:44:19.911 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:19.912 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:19.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:19 smithi190 ceph-mon[101220]: pgmap v56: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 29 KiB/s wr, 17 op/s 2023-10-10T17:44:19.919 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:19.920 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:19.920 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 77 -lt 100 ']' 2023-10-10T17:44:19.920 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 77 + 1 2023-10-10T17:44:19.950 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=78 2023-10-10T17:44:19.951 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:19.976 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:19.976 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:19.977 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:19.977 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:19.978 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:19.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:19 smithi157 ceph-mon[100896]: pgmap v56: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 29 KiB/s wr, 17 op/s 2023-10-10T17:44:20.349 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:20.350 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:20.519 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:20.520 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:20.528 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:20.529 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:20.529 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 78 -lt 100 ']' 2023-10-10T17:44:20.529 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 78 + 1 2023-10-10T17:44:20.559 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=79 2023-10-10T17:44:20.560 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:20.585 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:20.585 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:20.586 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:20.586 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:20.586 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:20.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:20 smithi078 ceph-mon[99494]: pgmap v57: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:20.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:20 smithi190 ceph-mon[101220]: pgmap v57: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:20.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:20 smithi157 ceph-mon[100896]: pgmap v57: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:21.016 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:21.017 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:21.161 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:21.162 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:21.169 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:21.170 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:21.170 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 79 -lt 100 ']' 2023-10-10T17:44:21.170 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 79 + 1 2023-10-10T17:44:21.201 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=80 2023-10-10T17:44:21.201 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:21.227 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:21.227 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:21.228 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:21.229 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:21.229 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:21.608 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:21.609 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:21.844 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:21.845 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:21.871 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:21.871 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:21.871 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 80 -lt 100 ']' 2023-10-10T17:44:21.872 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 80 + 1 2023-10-10T17:44:21.917 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=81 2023-10-10T17:44:21.917 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:21.959 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:21.960 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:21.960 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:21.961 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:21.961 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:22.510 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:22.510 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:22.753 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:22.753 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:22.769 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:22.769 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:22.769 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 81 -lt 100 ']' 2023-10-10T17:44:22.769 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 81 + 1 2023-10-10T17:44:22.813 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=82 2023-10-10T17:44:22.814 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:22.860 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:22.860 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:22.861 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:22.861 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:22.862 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:22.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:22 smithi078 ceph-mon[99494]: pgmap v58: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:22.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:22 smithi190 ceph-mon[101220]: pgmap v58: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:22.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:22 smithi157 ceph-mon[100896]: pgmap v58: 129 pgs: 129 active+clean; 2.0 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 20 KiB/s wr, 13 op/s 2023-10-10T17:44:23.525 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:23.525 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:23.719 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:23.720 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:23.869 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:23.870 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:23.870 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 82 -lt 100 ']' 2023-10-10T17:44:23.870 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 82 + 1 2023-10-10T17:44:23.967 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=83 2023-10-10T17:44:23.967 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 -- 172.21.15.78:0/4083935654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4100ec0 msgr2=0x7ff2d4109620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 --2- 172.21.15.78:0/4083935654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4100ec0 0x7ff2d4109620 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc009990 tx=0x7ff2bc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 -- 172.21.15.78:0/4083935654 shutdown_connections 2023-10-10T17:44:24.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 --2- 172.21.15.78:0/4083935654 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2d41005b0 0x7ff2d4100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 --2- 172.21.15.78:0/4083935654 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d4109b60 0x7ff2d410bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 --2- 172.21.15.78:0/4083935654 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4100ec0 0x7ff2d4109620 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 -- 172.21.15.78:0/4083935654 >> 172.21.15.78:0/4083935654 conn(0x7ff2d4074a50 msgr2=0x7ff2d4074e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:24.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 -- 172.21.15.78:0/4083935654 shutdown_connections 2023-10-10T17:44:24.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.011+0000 7ff2d978f700 1 -- 172.21.15.78:0/4083935654 wait complete. 2023-10-10T17:44:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 Processor -- start 2023-10-10T17:44:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 -- start start 2023-10-10T17:44:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d41005b0 0x7ff2d41963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2d4100ec0 0x7ff2d4196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.017 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4109b60 0x7ff2d419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d419b3a0 con 0x7ff2d4109b60 2023-10-10T17:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d419b510 con 0x7ff2d41005b0 2023-10-10T17:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d978f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d419b6d0 con 0x7ff2d4100ec0 2023-10-10T17:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4109b60 0x7ff2d419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.018 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4109b60 0x7ff2d419ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45492/0 (socket says 172.21.15.78:45492) 2023-10-10T17:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 -- 172.21.15.78:0/623201833 learned_addr learned my addr 172.21.15.78:0/623201833 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2cbfff700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2d4100ec0 0x7ff2d4196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d41005b0 0x7ff2d41963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 -- 172.21.15.78:0/623201833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2d4100ec0 msgr2=0x7ff2d4196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2d4100ec0 0x7ff2d4196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 -- 172.21.15.78:0/623201833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d41005b0 msgr2=0x7ff2d41963c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d41005b0 0x7ff2d41963c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d37fe700 1 -- 172.21.15.78:0/623201833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2bc0095f0 con 0x7ff2d4109b60 2023-10-10T17:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.012+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d41005b0 0x7ff2d41963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:24.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.013+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4109b60 0x7ff2d419ad60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0094f0 tx=0x7ff2cc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.013+0000 7ff2d0ff9700 1 -- 172.21.15.78:0/623201833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc004db0 con 0x7ff2d4109b60 2023-10-10T17:44:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.013+0000 7ff2d0ff9700 1 -- 172.21.15.78:0/623201833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2cc00f040 con 0x7ff2d4109b60 2023-10-10T17:44:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.013+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2d410f1d0 con 0x7ff2d4109b60 2023-10-10T17:44:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.013+0000 7ff2d0ff9700 1 -- 172.21.15.78:0/623201833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc0059e0 con 0x7ff2d4109b60 2023-10-10T17:44:24.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.013+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2d410f040 con 0x7ff2d4109b60 2023-10-10T17:44:24.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.015+0000 7ff2d0ff9700 1 -- 172.21.15.78:0/623201833 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff2cc00db60 con 0x7ff2d4109b60 2023-10-10T17:44:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.016+0000 7ff2d0ff9700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff2c00a12c0 0x7ff2c00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.016+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff2c00a12c0 0x7ff2c00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.016+0000 7ff2d0ff9700 1 -- 172.21.15.78:0/623201833 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7ff2cc0cca70 con 0x7ff2d4109b60 2023-10-10T17:44:24.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.016+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff2c00a12c0 0x7ff2c00a3770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4005fd0 tx=0x7ff2c4005ee0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:24.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.209+0000 7ff2e591bb80 1 Processor -- start 2023-10-10T17:44:24.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.209+0000 7ff2e591bb80 1 -- start start 2023-10-10T17:44:24.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc738b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7385560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 0x555dc7385b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc728b000 con 0x555dc71e16b0 2023-10-10T17:44:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc72dd840 con 0x555dc738bb90 2023-10-10T17:44:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc72d6cd0 con 0x555dc738b720 2023-10-10T17:44:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc738b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc738b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45496/0 (socket says 172.21.15.78:45496) 2023-10-10T17:44:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/3521664531 learned_addr learned my addr 172.21.15.78:0/3521664531 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2cbfff700 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7385560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/3521664531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 msgr2=0x555dc7385560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 0x555dc7385b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7385560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/3521664531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 msgr2=0x555dc7385b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 0x555dc7385b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/3521664531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555dc71585d0 con 0x555dc71e16b0 2023-10-10T17:44:24.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc738b1e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400edd0 tx=0x7ff2c4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b298dc4258e8771c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2c97fa700 1 -- 172.21.15.78:0/3521664531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4022070 con 0x555dc71e16b0 2023-10-10T17:44:24.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2c97fa700 1 -- 172.21.15.78:0/3521664531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2c400ee00 con 0x555dc71e16b0 2023-10-10T17:44:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.210+0000 7ff2c97fa700 1 -- 172.21.15.78:0/3521664531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4005480 con 0x555dc71e16b0 2023-10-10T17:44:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 -- 172.21.15.78:0/3521664531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 msgr2=0x555dc738b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc738b1e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400edd0 tx=0x7ff2c4009f90 comp rx=0 tx=0).stop 2023-10-10T17:44:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 -- 172.21.15.78:0/3521664531 shutdown_connections 2023-10-10T17:44:24.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 0x555dc7385b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc738b1e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/3521664531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7385560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 -- 172.21.15.78:0/3521664531 >> 172.21.15.78:0/3521664531 conn(0x555dc737af50 msgr2=0x555dc7380270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:24.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 -- 172.21.15.78:0/3521664531 shutdown_connections 2023-10-10T17:44:24.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.211+0000 7ff2e591bb80 1 -- 172.21.15.78:0/3521664531 wait complete. 2023-10-10T17:44:24.223 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 -- start start 2023-10-10T17:44:24.223 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 Processor -- start 2023-10-10T17:44:24.223 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc71e16b0 0x555dc7391b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.223 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc738ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.223 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc738f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc7278880 con 0x555dc738bb90 2023-10-10T17:44:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc7278520 con 0x555dc71e16b0 2023-10-10T17:44:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc715a8c0 con 0x555dc738b720 2023-10-10T17:44:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2e591bb80 10 client.0 _ll_register_callbacks cb 0x555dc7296720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:24.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc738f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc738f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45506/0 (socket says 172.21.15.78:45506) 2023-10-10T17:44:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 -- 172.21.15.78:0/638698232 learned_addr learned my addr 172.21.15.78:0/638698232 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2cbfff700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc738ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc71e16b0 0x555dc7391b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 -- 172.21.15.78:0/638698232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 msgr2=0x555dc738ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc738ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 -- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc71e16b0 msgr2=0x555dc7391b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc71e16b0 0x555dc7391b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d37fe700 1 -- 172.21.15.78:0/638698232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c400eab0 con 0x555dc738bb90 2023-10-10T17:44:24.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2cbfff700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc738ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.212+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc71e16b0 0x555dc7391b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc738f480 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc095cd0 tx=0x7ff2cc099440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926be7548a937da9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2ba7fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/638698232 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc0a1c20 con 0x555dc738bb90 2023-10-10T17:44:24.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/638698232 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7ff2cc099c40 con 0x555dc738bb90 2023-10-10T17:44:24.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555dc7378710 con 0x555dc738bb90 2023-10-10T17:44:24.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/638698232 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2cc098cc0 con 0x555dc738bb90 2023-10-10T17:44:24.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 10 client.16026 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:24.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555dc729d130 con 0x555dc738bb90 2023-10-10T17:44:24.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/638698232 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555dc7378710 con 0x555dc738bb90 2023-10-10T17:44:24.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/638698232 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff2cc0b5020 con 0x555dc738bb90 2023-10-10T17:44:24.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 10 client.16026 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:24.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 10 client.16026 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:24.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 20 client.16026 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:24.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc739ef90 0x555dc73a13a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 4 client.16026 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555dc7299040 con 0x555dc739ef90 2023-10-10T17:44:24.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.213+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc739ef90 0x555dc73a13a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.214+0000 7ff2ba7fc700 10 client.16026 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:24.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.214+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc739ef90 0x555dc73a13a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84ea82aaeca99a50 server_cookie=e20d68b4b4785baf in_seq=0 out_seq=0 2023-10-10T17:44:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/638698232 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555dc7299040 con 0x555dc739ef90 2023-10-10T17:44:24.232 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2ba7fc700 10 client.16026 handle_command_reply: tid=0 2023-10-10T17:44:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 client.16026 shutdown 2023-10-10T17:44:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 msgr2=0x555dc738f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc738f480 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc095cd0 tx=0x7ff2cc099440 comp rx=0 tx=0).stop 2023-10-10T17:44:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 shutdown_connections 2023-10-10T17:44:24.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc739ef90 0x555dc73a13a0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc738f480 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2ba7fc700 0 client.16026 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc71e16b0 0x555dc7391b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/638698232 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc738ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 >> 172.21.15.78:0/638698232 conn(0x555dc737af50 msgr2=0x555dc737f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 shutdown_connections 2023-10-10T17:44:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 1 -- 172.21.15.78:0/638698232 wait complete. 2023-10-10T17:44:24.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.215+0000 7ff2e591bb80 20 client.16026 trim_cache size 0 max 16384 2023-10-10T17:44:24.236 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:24.236 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:24.236 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:24.236 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:24.237 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.230+0000 7ff2e591bb80 1 Processor -- start 2023-10-10T17:44:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.230+0000 7ff2e591bb80 1 -- start start 2023-10-10T17:44:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc737ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7388e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 0x555dc73893a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc7386660 con 0x555dc71e16b0 2023-10-10T17:44:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc737cfd0 con 0x555dc738bb90 2023-10-10T17:44:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc738f7e0 con 0x555dc738b720 2023-10-10T17:44:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc737ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc737ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45510/0 (socket says 172.21.15.78:45510) 2023-10-10T17:44:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/2738051020 learned_addr learned my addr 172.21.15.78:0/2738051020 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2cbfff700 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7388e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/2738051020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 msgr2=0x555dc7388e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7388e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/2738051020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 msgr2=0x555dc73893a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 0x555dc73893a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 -- 172.21.15.78:0/2738051020 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555dc729ce80 con 0x555dc71e16b0 2023-10-10T17:44:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2cbfff700 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7388e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc737ff60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400e680 tx=0x7ff2c4011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61a5ee52231652d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:24.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.231+0000 7ff2c9ffb700 1 -- 172.21.15.78:0/2738051020 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c402d070 con 0x555dc71e16b0 2023-10-10T17:44:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2c9ffb700 1 -- 172.21.15.78:0/2738051020 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2c402ecb0 con 0x555dc71e16b0 2023-10-10T17:44:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2c9ffb700 1 -- 172.21.15.78:0/2738051020 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4022040 con 0x555dc71e16b0 2023-10-10T17:44:24.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 -- 172.21.15.78:0/2738051020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 msgr2=0x555dc737ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc737ff60 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ff2c400e680 tx=0x7ff2c4011d30 comp rx=0 tx=0).stop 2023-10-10T17:44:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 -- 172.21.15.78:0/2738051020 shutdown_connections 2023-10-10T17:44:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738bb90 0x555dc73893a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc71e16b0 0x555dc737ff60 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/2738051020 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc738b720 0x555dc7388e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 -- 172.21.15.78:0/2738051020 >> 172.21.15.78:0/2738051020 conn(0x555dc737af50 msgr2=0x555dc7381a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 -- 172.21.15.78:0/2738051020 shutdown_connections 2023-10-10T17:44:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.232+0000 7ff2e591bb80 1 -- 172.21.15.78:0/2738051020 wait complete. 2023-10-10T17:44:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 -- start start 2023-10-10T17:44:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 Processor -- start 2023-10-10T17:44:24.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc71e16b0 0x555dc7392850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738b720 0x555dc7392d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc73f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc73f2390 con 0x555dc738bb90 2023-10-10T17:44:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc73f24d0 con 0x555dc738b720 2023-10-10T17:44:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dc73f2610 con 0x555dc71e16b0 2023-10-10T17:44:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2e591bb80 10 client.0 _ll_register_callbacks cb 0x7ff2cc095ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc73f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc73f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45520/0 (socket says 172.21.15.78:45520) 2023-10-10T17:44:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 -- 172.21.15.78:0/1379390438 learned_addr learned my addr 172.21.15.78:0/1379390438 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc71e16b0 0x555dc7392850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 -- 172.21.15.78:0/1379390438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc71e16b0 msgr2=0x555dc7392850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc71e16b0 0x555dc7392850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 -- 172.21.15.78:0/1379390438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738b720 msgr2=0x555dc7392d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738b720 0x555dc7392d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 -- 172.21.15.78:0/1379390438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555dc71585d0 con 0x555dc738bb90 2023-10-10T17:44:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc71e16b0 0x555dc7392850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2d37fe700 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc73f1e50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc082fd0 tx=0x7ff2cc0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36d1ea7343e5ec0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.233+0000 7ff2ba7fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/1379390438 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc095690 con 0x555dc738bb90 2023-10-10T17:44:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/1379390438 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7ff2cc0a1c20 con 0x555dc738bb90 2023-10-10T17:44:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff2c400ee00 con 0x555dc738bb90 2023-10-10T17:44:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/1379390438 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2cc0a1790 con 0x555dc738bb90 2023-10-10T17:44:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 10 client.16032 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:24.253 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x555dc70edc00 con 0x555dc738bb90 2023-10-10T17:44:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/1379390438 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff2cc063080 con 0x555dc738bb90 2023-10-10T17:44:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/1379390438 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ff2cc0b5070 con 0x555dc738bb90 2023-10-10T17:44:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 10 client.16032 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 10 client.16032 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:24.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 20 client.16032 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc73f3880 0x555dc73f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 4 client.16032 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:44:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555dc72a5600 con 0x555dc73f3880 2023-10-10T17:44:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.234+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc73f3880 0x555dc73f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:24.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.235+0000 7ff2ba7fc700 10 client.16032 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:24.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.235+0000 7ff2d2ffd700 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc73f3880 0x555dc73f5c60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6471645d0f5ae85 server_cookie=a77f2c9e7e9aba51 in_seq=0 out_seq=0 2023-10-10T17:44:24.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2ba7fc700 1 -- 172.21.15.78:0/1379390438 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555dc72a5600 con 0x555dc73f3880 2023-10-10T17:44:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2ba7fc700 10 client.16032 handle_command_reply: tid=0 2023-10-10T17:44:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 client.16032 shutdown 2023-10-10T17:44:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 msgr2=0x555dc73f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc73f1e50 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc082fd0 tx=0x7ff2cc0093f0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 shutdown_connections 2023-10-10T17:44:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555dc71e16b0 0x555dc7392850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555dc73f3880 0x555dc73f5c60 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555dc738bb90 0x555dc73f1e50 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 --2- 172.21.15.78:0/1379390438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555dc738b720 0x555dc7392d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2ba7fc700 0 client.16032 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.255+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 >> 172.21.15.78:0/1379390438 conn(0x555dc737af50 msgr2=0x555dc7388e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 shutdown_connections 2023-10-10T17:44:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2e591bb80 1 -- 172.21.15.78:0/1379390438 wait complete. 2023-10-10T17:44:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2e591bb80 20 client.16032 trim_cache size 0 max 16384 2023-10-10T17:44:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:44:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:44:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:44:24.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:44:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff2c00a12c0 msgr2=0x7ff2c00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff2c00a12c0 0x7ff2c00a3770 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4005fd0 tx=0x7ff2c4005ee0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4109b60 msgr2=0x7ff2d419ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:24.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4109b60 0x7ff2d419ad60 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0094f0 tx=0x7ff2cc007f90 comp rx=0 tx=0).stop 2023-10-10T17:44:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 shutdown_connections 2023-10-10T17:44:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff2c00a12c0 0x7ff2c00a3770 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff2d4109b60 0x7ff2d419ad60 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff2d41005b0 0x7ff2d41963c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 --2- 172.21.15.78:0/623201833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff2d4100ec0 0x7ff2d4196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 >> 172.21.15.78:0/623201833 conn(0x7ff2d4074a50 msgr2=0x7ff2d40fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 shutdown_connections 2023-10-10T17:44:24.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:24.256+0000 7ff2d978f700 1 -- 172.21.15.78:0/623201833 wait complete. 2023-10-10T17:44:24.607 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:24.608 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:24.786 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:24.786 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:24.802 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:24.802 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:24.802 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 83 -lt 100 ']' 2023-10-10T17:44:24.802 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 83 + 1 2023-10-10T17:44:24.833 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=84 2023-10-10T17:44:24.833 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:24.864 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:24.864 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:24.865 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:24.865 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:24.865 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:24.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:24 smithi078 ceph-mon[99494]: pgmap v59: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 16 op/s 2023-10-10T17:44:24.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:24 smithi190 ceph-mon[101220]: pgmap v59: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 16 op/s 2023-10-10T17:44:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:24 smithi157 ceph-mon[100896]: pgmap v59: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 27 KiB/s wr, 16 op/s 2023-10-10T17:44:25.240 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:25.241 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:25.371 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:25.372 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:25.375 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:25.375 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:25.376 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 84 -lt 100 ']' 2023-10-10T17:44:25.376 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 84 + 1 2023-10-10T17:44:25.388 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=85 2023-10-10T17:44:25.388 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:25.398 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:25.399 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:25.399 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:25.400 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:25.400 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:25.732 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:25.732 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:25.861 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:25.861 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:25.877 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:25.878 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:25.878 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 85 -lt 100 ']' 2023-10-10T17:44:25.879 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 85 + 1 2023-10-10T17:44:25.908 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=86 2023-10-10T17:44:25.909 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:25.934 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:25.934 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:25.935 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:25.935 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:25.935 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:26.274 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:26.274 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:26.435 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:26.436 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:26.452 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:26.452 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:26.453 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 86 -lt 100 ']' 2023-10-10T17:44:26.453 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 86 + 1 2023-10-10T17:44:26.483 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=87 2023-10-10T17:44:26.483 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:26.509 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:26.509 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:26.510 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:26.510 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:26.510 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:26.840 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:26.841 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:26.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:26 smithi190 ceph-mon[101220]: pgmap v60: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-10-10T17:44:26.960 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:26.961 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:26.968 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:26.968 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:26.969 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 87 -lt 100 ']' 2023-10-10T17:44:26.969 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 87 + 1 2023-10-10T17:44:26.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:26 smithi157 ceph-mon[100896]: pgmap v60: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-10-10T17:44:27.000 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=88 2023-10-10T17:44:27.000 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:27.031 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:27.031 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:27.032 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:27.032 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:27.032 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:27.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:26 smithi078 ceph-mon[99494]: pgmap v60: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-10-10T17:44:27.432 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:27.432 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:27.560 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:27.561 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:27.577 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:27.578 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:27.578 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 88 -lt 100 ']' 2023-10-10T17:44:27.578 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 88 + 1 2023-10-10T17:44:27.608 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=89 2023-10-10T17:44:27.608 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:27.633 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:27.633 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:27.633 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:27.634 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:27.634 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:27.974 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:27.974 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:28.127 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:28.127 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:28.152 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:28.152 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:28.153 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 89 -lt 100 ']' 2023-10-10T17:44:28.153 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 89 + 1 2023-10-10T17:44:28.174 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=90 2023-10-10T17:44:28.175 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:28.206 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:28.206 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:28.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:28.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:28.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:28.565 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:28.565 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:28.719 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:28.719 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:28.735 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:28.736 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:28.736 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 90 -lt 100 ']' 2023-10-10T17:44:28.736 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 90 + 1 2023-10-10T17:44:28.766 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=91 2023-10-10T17:44:28.766 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:28.800 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:28.800 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:28.801 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:28.801 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:28.801 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:28.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:28 smithi078 ceph-mon[99494]: pgmap v61: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 14 op/s 2023-10-10T17:44:28.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:28 smithi190 ceph-mon[101220]: pgmap v61: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 14 op/s 2023-10-10T17:44:28.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:28 smithi157 ceph-mon[100896]: pgmap v61: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 14 op/s 2023-10-10T17:44:29.265 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:29.265 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:29.411 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:29.411 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:29.427 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:29.427 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:29.427 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 91 -lt 100 ']' 2023-10-10T17:44:29.428 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 91 + 1 2023-10-10T17:44:29.457 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=92 2023-10-10T17:44:29.458 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:29.482 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:29.483 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:29.484 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:29.484 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:29.484 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:30.165 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:30.165 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:30.293 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:30.293 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:30.310 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:30.311 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:30.311 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 92 -lt 100 ']' 2023-10-10T17:44:30.311 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 92 + 1 2023-10-10T17:44:30.341 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=93 2023-10-10T17:44:30.341 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:30.372 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:30.372 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:30.373 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:30.373 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:30.374 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:30.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:44:30.765 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:30.765 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:30.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:30 smithi190 ceph-mon[101220]: pgmap v62: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 11 op/s 2023-10-10T17:44:30.920 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:30.920 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:30.934 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:30.935 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:30.935 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 93 -lt 100 ']' 2023-10-10T17:44:30.935 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 93 + 1 2023-10-10T17:44:30.962 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=94 2023-10-10T17:44:30.963 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:30.970 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:30.971 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:30.971 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:30.971 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:30.972 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:30.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:30 smithi157 ceph-mon[100896]: pgmap v62: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 11 op/s 2023-10-10T17:44:31.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:30 smithi078 ceph-mon[99494]: pgmap v62: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 11 op/s 2023-10-10T17:44:31.373 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:31.373 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:31.487 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:31.487 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:31.511 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:31.511 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:31.511 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 94 -lt 100 ']' 2023-10-10T17:44:31.512 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 94 + 1 2023-10-10T17:44:31.537 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=95 2023-10-10T17:44:31.538 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:31.575 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:31.576 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:31.576 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:31.577 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:31.577 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:32.164 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:32.165 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:32.310 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:32.310 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:32.322 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:32.322 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:32.323 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 95 -lt 100 ']' 2023-10-10T17:44:32.323 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 95 + 1 2023-10-10T17:44:32.354 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=96 2023-10-10T17:44:32.354 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:32.418 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:32.418 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:32.419 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:32.419 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:32.419 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:32.897 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:32 smithi078 ceph-mon[99494]: pgmap v63: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 11 op/s 2023-10-10T17:44:32.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:32 smithi190 ceph-mon[101220]: pgmap v63: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 11 op/s 2023-10-10T17:44:32.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:32 smithi157 ceph-mon[100896]: pgmap v63: 129 pgs: 129 active+clean; 2.1 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 11 op/s 2023-10-10T17:44:33.073 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:33.073 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:33.284 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:33.285 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:33.326 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:33.327 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:33.327 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 96 -lt 100 ']' 2023-10-10T17:44:33.327 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 96 + 1 2023-10-10T17:44:33.357 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=97 2023-10-10T17:44:33.357 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:33.399 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:33.400 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:33.401 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:33.401 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:33.401 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:33.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.631+0000 7fae963d5700 1 -- 172.21.15.78:0/2714795055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90100340 msgr2=0x7fae9010c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.631+0000 7fae963d5700 1 --2- 172.21.15.78:0/2714795055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90100340 0x7fae9010c010 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fae8800b0d0 tx=0x7fae8800b3e0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.631+0000 7fae963d5700 1 -- 172.21.15.78:0/2714795055 shutdown_connections 2023-10-10T17:44:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.631+0000 7fae963d5700 1 --2- 172.21.15.78:0/2714795055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae90104180 0x7fae90104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.631+0000 7fae963d5700 1 --2- 172.21.15.78:0/2714795055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae90100340 0x7fae9010c010 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.631+0000 7fae963d5700 1 --2- 172.21.15.78:0/2714795055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae900ff990 0x7fae900ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.631+0000 7fae963d5700 1 -- 172.21.15.78:0/2714795055 >> 172.21.15.78:0/2714795055 conn(0x7fae90075920 msgr2=0x7fae90075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:33.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.632+0000 7fae963d5700 1 -- 172.21.15.78:0/2714795055 shutdown_connections 2023-10-10T17:44:33.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.632+0000 7fae963d5700 1 -- 172.21.15.78:0/2714795055 wait complete. 2023-10-10T17:44:33.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.632+0000 7fae963d5700 1 Processor -- start 2023-10-10T17:44:33.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.632+0000 7fae963d5700 1 -- start start 2023-10-10T17:44:33.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae963d5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae900ff990 0x7fae901972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae963d5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae90100340 0x7fae90197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae963d5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae90104180 0x7fae901a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae963d5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae90198170 con 0x7fae900ff990 2023-10-10T17:44:33.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae963d5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae901a28b0 con 0x7fae90100340 2023-10-10T17:44:33.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae963d5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae901a2a70 con 0x7fae90104180 2023-10-10T17:44:33.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae900ff990 0x7fae901972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae900ff990 0x7fae901972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45530/0 (socket says 172.21.15.78:45530) 2023-10-10T17:44:33.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 -- 172.21.15.78:0/4235369058 learned_addr learned my addr 172.21.15.78:0/4235369058 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:33.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae94972700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae90104180 0x7fae901a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae90100340 0x7fae90197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 -- 172.21.15.78:0/4235369058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae90104180 msgr2=0x7fae901a2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae90104180 0x7fae901a2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 -- 172.21.15.78:0/4235369058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae90100340 msgr2=0x7fae90197830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae90100340 0x7fae90197830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 -- 172.21.15.78:0/4235369058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88009d00 con 0x7fae900ff990 2023-10-10T17:44:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8ffff700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae900ff990 0x7fae901972f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fae8000d950 tx=0x7fae8000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8d7fa700 1 -- 172.21.15.78:0/4235369058 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80009b40 con 0x7fae900ff990 2023-10-10T17:44:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8d7fa700 1 -- 172.21.15.78:0/4235369058 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae80009ca0 con 0x7fae900ff990 2023-10-10T17:44:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae8d7fa700 1 -- 172.21.15.78:0/4235369058 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8000f620 con 0x7fae900ff990 2023-10-10T17:44:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.633+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9010f260 con 0x7fae900ff990 2023-10-10T17:44:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.634+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9010f0d0 con 0x7fae900ff990 2023-10-10T17:44:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.636+0000 7fae8d7fa700 1 -- 172.21.15.78:0/4235369058 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fae80010460 con 0x7fae900ff990 2023-10-10T17:44:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.636+0000 7fae8d7fa700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fae780a1340 0x7fae780a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.636+0000 7fae8d7fa700 1 -- 172.21.15.78:0/4235369058 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fae800cd2e0 con 0x7fae900ff990 2023-10-10T17:44:33.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.636+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fae780a1340 0x7fae780a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.637+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fae780a1340 0x7fae780a37f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae84005ea0 tx=0x7fae84005e30 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:33.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 Processor -- start 2023-10-10T17:44:33.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 -- start start 2023-10-10T17:44:33.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc3b60 0x562c7acc3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc4470 0x562c7acc8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acc8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7abc9060 con 0x562c7acc4470 2023-10-10T17:44:33.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7ac1b8a0 con 0x562c7acc3b60 2023-10-10T17:44:33.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.710+0000 7faea2561b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7ac14d30 con 0x562c7acc88b0 2023-10-10T17:44:33.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc4470 0x562c7acc8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc4470 0x562c7acc8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45540/0 (socket says 172.21.15.78:45540) 2023-10-10T17:44:33.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 -- 172.21.15.78:0/2562858166 learned_addr learned my addr 172.21.15.78:0/2562858166 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:33.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8ffff700 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc3b60 0x562c7acc3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae94972700 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acc8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 -- 172.21.15.78:0/2562858166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 msgr2=0x562c7acc8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acc8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 -- 172.21.15.78:0/2562858166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc3b60 msgr2=0x562c7acc3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc3b60 0x562c7acc3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 -- 172.21.15.78:0/2562858166 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c7aa96630 con 0x562c7acc4470 2023-10-10T17:44:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc4470 0x562c7acc8370 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fae8400ec80 tx=0x7fae84011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9520bdcadc98e73 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae767fc700 1 -- 172.21.15.78:0/2562858166 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae84010d90 con 0x562c7acc4470 2023-10-10T17:44:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae767fc700 1 -- 172.21.15.78:0/2562858166 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae8401d910 con 0x562c7acc4470 2023-10-10T17:44:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.711+0000 7fae767fc700 1 -- 172.21.15.78:0/2562858166 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8400f690 con 0x562c7acc4470 2023-10-10T17:44:33.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 -- 172.21.15.78:0/2562858166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc4470 msgr2=0x562c7acc8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc4470 0x562c7acc8370 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fae8400ec80 tx=0x7fae84011000 comp rx=0 tx=0).stop 2023-10-10T17:44:33.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 -- 172.21.15.78:0/2562858166 shutdown_connections 2023-10-10T17:44:33.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acc8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc3b60 0x562c7acc3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 --2- 172.21.15.78:0/2562858166 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc4470 0x562c7acc8370 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 -- 172.21.15.78:0/2562858166 >> 172.21.15.78:0/2562858166 conn(0x562c7acb9360 msgr2=0x562c7acbeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:33.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 -- 172.21.15.78:0/2562858166 shutdown_connections 2023-10-10T17:44:33.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.712+0000 7faea2561b80 1 -- 172.21.15.78:0/2562858166 wait complete. 2023-10-10T17:44:33.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 -- start start 2023-10-10T17:44:33.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 Processor -- start 2023-10-10T17:44:33.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7accffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc4470 0x562c7accd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc88b0 0x562c7accd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7abb68e0 con 0x562c7acc3b60 2023-10-10T17:44:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7abb6580 con 0x562c7acc88b0 2023-10-10T17:44:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7aa98920 con 0x562c7acc4470 2023-10-10T17:44:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7faea2561b80 10 client.0 _ll_register_callbacks cb 0x562c7abd4780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7fae8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7accffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7fae8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7accffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45544/0 (socket says 172.21.15.78:45544) 2023-10-10T17:44:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7fae8ffff700 1 -- 172.21.15.78:0/2134139157 learned_addr learned my addr 172.21.15.78:0/2134139157 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.713+0000 7fae94972700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc88b0 0x562c7accd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc4470 0x562c7accd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8ffff700 1 -- 172.21.15.78:0/2134139157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc4470 msgr2=0x562c7accd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8ffff700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc4470 0x562c7accd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8ffff700 1 -- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc88b0 msgr2=0x562c7accd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8ffff700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc88b0 0x562c7accd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8ffff700 1 -- 172.21.15.78:0/2134139157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8400e960 con 0x562c7acc3b60 2023-10-10T17:44:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc4470 0x562c7accd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae8ffff700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7accffc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fae80096af0 tx=0x7fae8009a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402500a47a0763cc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae6f7fe700 1 -- 172.21.15.78:0/2134139157 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80010910 con 0x562c7acc3b60 2023-10-10T17:44:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae6f7fe700 1 -- 172.21.15.78:0/2134139157 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fae800a24d0 con 0x562c7acc3b60 2023-10-10T17:44:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562c7acb6b20 con 0x562c7acc3b60 2023-10-10T17:44:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae6f7fe700 1 -- 172.21.15.78:0/2134139157 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae8009a2e0 con 0x562c7acc3b60 2023-10-10T17:44:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7faea2561b80 10 client.16044 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562c7abdb190 con 0x562c7acc3b60 2023-10-10T17:44:33.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae6f7fe700 1 -- 172.21.15.78:0/2134139157 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562c7acb6b20 con 0x562c7acc3b60 2023-10-10T17:44:33.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.714+0000 7fae6f7fe700 1 -- 172.21.15.78:0/2134139157 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fae800a2040 con 0x562c7acc3b60 2023-10-10T17:44:33.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7faea2561b80 10 client.16044 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:33.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7faea2561b80 10 client.16044 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:33.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7faea2561b80 20 client.16044 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:33.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7faea2561b80 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7acdd740 0x562c7acdfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7faea2561b80 4 client.16044 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:33.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c7abd70a0 con 0x562c7acdd740 2023-10-10T17:44:33.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7acdd740 0x562c7acdfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7fae6f7fe700 10 client.16044 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:33.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.715+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7acdd740 0x562c7acdfb50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=688a731a90ed88a3 server_cookie=3833f39ad4af860 in_seq=0 out_seq=0 2023-10-10T17:44:33.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7fae6f7fe700 1 -- 172.21.15.78:0/2134139157 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562c7abd70a0 con 0x562c7acdd740 2023-10-10T17:44:33.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7fae6f7fe700 10 client.16044 handle_command_reply: tid=0 2023-10-10T17:44:33.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 client.16044 shutdown 2023-10-10T17:44:33.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 msgr2=0x562c7accffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7accffc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fae80096af0 tx=0x7fae8009a100 comp rx=0 tx=0).stop 2023-10-10T17:44:33.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 shutdown_connections 2023-10-10T17:44:33.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7acdd740 0x562c7acdfb50 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc88b0 0x562c7accd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7accffc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7fae6f7fe700 0 client.16044 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:33.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 --2- 172.21.15.78:0/2134139157 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc4470 0x562c7accd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 >> 172.21.15.78:0/2134139157 conn(0x562c7acb9360 msgr2=0x562c7acbdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:33.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 shutdown_connections 2023-10-10T17:44:33.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 1 -- 172.21.15.78:0/2134139157 wait complete. 2023-10-10T17:44:33.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.716+0000 7faea2561b80 20 client.16044 trim_cache size 0 max 16384 2023-10-10T17:44:33.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.733+0000 7faea2561b80 1 Processor -- start 2023-10-10T17:44:33.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.733+0000 7faea2561b80 1 -- start start 2023-10-10T17:44:33.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7faea2561b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc3b60 0x562c7acd49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7faea2561b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acc3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7faea2561b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc88b0 0x562c7accfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7faea2561b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7acb8f30 con 0x562c7acc88b0 2023-10-10T17:44:33.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7faea2561b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7acce850 con 0x562c7acc4470 2023-10-10T17:44:33.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7faea2561b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7acda230 con 0x562c7acc3b60 2023-10-10T17:44:33.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc88b0 0x562c7accfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc88b0 0x562c7accfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45556/0 (socket says 172.21.15.78:45556) 2023-10-10T17:44:33.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 -- 172.21.15.78:0/426608422 learned_addr learned my addr 172.21.15.78:0/426608422 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:33.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acc3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae8ffff700 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc3b60 0x562c7acd49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 -- 172.21.15.78:0/426608422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc3b60 msgr2=0x562c7acd49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc3b60 0x562c7acd49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 -- 172.21.15.78:0/426608422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 msgr2=0x562c7acc3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acc3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 -- 172.21.15.78:0/426608422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c7acc8f50 con 0x562c7acc88b0 2023-10-10T17:44:33.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae8ffff700 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc3b60 0x562c7acd49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:33.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae94972700 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc88b0 0x562c7accfb60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fae880093b0 tx=0x7fae880093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54793cd3752d2ce2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:33.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae76ffd700 1 -- 172.21.15.78:0/426608422 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8800e040 con 0x562c7acc88b0 2023-10-10T17:44:33.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae76ffd700 1 -- 172.21.15.78:0/426608422 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae880079a0 con 0x562c7acc88b0 2023-10-10T17:44:33.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.734+0000 7fae76ffd700 1 -- 172.21.15.78:0/426608422 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88026ba0 con 0x562c7acc88b0 2023-10-10T17:44:33.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 -- 172.21.15.78:0/426608422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc88b0 msgr2=0x562c7accfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc88b0 0x562c7accfb60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fae880093b0 tx=0x7fae880093e0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 -- 172.21.15.78:0/426608422 shutdown_connections 2023-10-10T17:44:33.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc3b60 0x562c7acd49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc88b0 0x562c7accfb60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 --2- 172.21.15.78:0/426608422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acc3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 -- 172.21.15.78:0/426608422 >> 172.21.15.78:0/426608422 conn(0x562c7acb9360 msgr2=0x562c7acc2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:33.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 -- 172.21.15.78:0/426608422 shutdown_connections 2023-10-10T17:44:33.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 -- 172.21.15.78:0/426608422 wait complete. 2023-10-10T17:44:33.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.735+0000 7faea2561b80 1 -- start start 2023-10-10T17:44:33.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 1 Processor -- start 2023-10-10T17:44:33.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7acd4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acd4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acd0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7acd1230 con 0x562c7acc3b60 2023-10-10T17:44:33.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7acd13a0 con 0x562c7acc4470 2023-10-10T17:44:33.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c7acd1560 con 0x562c7acc88b0 2023-10-10T17:44:33.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7faea2561b80 10 client.0 _ll_register_callbacks cb 0x7fae80099bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:33.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7acd4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7acd4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45558/0 (socket says 172.21.15.78:45558) 2023-10-10T17:44:33.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 -- 172.21.15.78:0/3122862818 learned_addr learned my addr 172.21.15.78:0/3122862818 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:33.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acd4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae94972700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acd0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 -- 172.21.15.78:0/3122862818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 msgr2=0x562c7acd0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acd0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 -- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 msgr2=0x562c7acd4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:33.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acd4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 -- 172.21.15.78:0/3122862818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9010c8a0 con 0x562c7acc3b60 2023-10-10T17:44:33.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acd4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:33.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae8ffff700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7acd4620 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fae800b6040 tx=0x7fae80005910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c61e556ccc4644 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:33.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.736+0000 7fae6f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:33.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7fae6f7fe700 1 -- 172.21.15.78:0/3122862818 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80014070 con 0x562c7acc3b60 2023-10-10T17:44:33.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7fae6f7fe700 1 -- 172.21.15.78:0/3122862818 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fae800a24d0 con 0x562c7acc3b60 2023-10-10T17:44:33.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae8401d910 con 0x562c7acc3b60 2023-10-10T17:44:33.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7fae6f7fe700 1 -- 172.21.15.78:0/3122862818 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae800a2250 con 0x562c7acc3b60 2023-10-10T17:44:33.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 10 client.16050 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:33.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562c7aa42de0 con 0x562c7acc3b60 2023-10-10T17:44:33.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7fae6f7fe700 1 -- 172.21.15.78:0/3122862818 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae8401d910 con 0x562c7acc3b60 2023-10-10T17:44:33.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7fae6f7fe700 1 -- 172.21.15.78:0/3122862818 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fae800ca310 con 0x562c7acc3b60 2023-10-10T17:44:33.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 10 client.16050 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 20 client.16050 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7ad324f0 0x562c7ad34900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:33.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 4 client.16050 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:44:33.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562c7abe3660 con 0x562c7ad324f0 2023-10-10T17:44:33.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.737+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7ad324f0 0x562c7ad34900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:33.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7fae6f7fe700 10 client.16050 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:33.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7fae8f7fe700 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7ad324f0 0x562c7ad34900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b6be90225700d31 server_cookie=1dd909f730d6e850 in_seq=0 out_seq=0 2023-10-10T17:44:33.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7fae6f7fe700 1 -- 172.21.15.78:0/3122862818 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x562c7abe3660 con 0x562c7ad324f0 2023-10-10T17:44:33.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7fae6f7fe700 10 client.16050 handle_command_reply: tid=0 2023-10-10T17:44:33.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 client.16050 shutdown 2023-10-10T17:44:33.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 msgr2=0x562c7acd4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7acd4620 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fae800b6040 tx=0x7fae80005910 comp rx=0 tx=0).stop 2023-10-10T17:44:33.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 shutdown_connections 2023-10-10T17:44:33.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562c7acc88b0 0x562c7acd0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562c7ad324f0 0x562c7ad34900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562c7acc3b60 0x562c7acd4620 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7fae6f7fe700 0 client.16050 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:33.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 --2- 172.21.15.78:0/3122862818 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562c7acc4470 0x562c7acd4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 >> 172.21.15.78:0/3122862818 conn(0x562c7acb9360 msgr2=0x562c7acc31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:33.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 shutdown_connections 2023-10-10T17:44:33.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 1 -- 172.21.15.78:0/3122862818 wait complete. 2023-10-10T17:44:33.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.738+0000 7faea2561b80 20 client.16050 trim_cache size 0 max 16384 2023-10-10T17:44:33.759 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:44:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:44:33.759 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:44:33.759 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:44:33.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fae780a1340 msgr2=0x7fae780a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fae780a1340 0x7fae780a37f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae84005ea0 tx=0x7fae84005e30 comp rx=0 tx=0).stop 2023-10-10T17:44:33.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae900ff990 msgr2=0x7fae901972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:33.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae900ff990 0x7fae901972f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fae8000d950 tx=0x7fae8000dd10 comp rx=0 tx=0).stop 2023-10-10T17:44:33.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 shutdown_connections 2023-10-10T17:44:33.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fae90104180 0x7fae901a2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fae780a1340 0x7fae780a37f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fae900ff990 0x7fae901972f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 --2- 172.21.15.78:0/4235369058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fae90100340 0x7fae90197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:33.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 >> 172.21.15.78:0/4235369058 conn(0x7fae90075920 msgr2=0x7fae900fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:33.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 shutdown_connections 2023-10-10T17:44:33.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:33.739+0000 7fae963d5700 1 -- 172.21.15.78:0/4235369058 wait complete. 2023-10-10T17:44:33.805 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:44:33.805 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:44:34.115 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:34.115 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:34.245 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:34.245 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:34.278 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:34.278 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:34.278 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 97 -lt 100 ']' 2023-10-10T17:44:34.279 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 97 + 1 2023-10-10T17:44:34.304 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=98 2023-10-10T17:44:34.304 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:34.311 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:34.313 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:34.314 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:34.314 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:34.314 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:34.698 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:34.698 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:34.803 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:34.803 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:34.818 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:34.818 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:34.819 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 98 -lt 100 ']' 2023-10-10T17:44:34.819 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 98 + 1 2023-10-10T17:44:34.870 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=99 2023-10-10T17:44:34.870 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:34.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:34 smithi078 ceph-mon[99494]: pgmap v64: 129 pgs: 129 active+clean; 2.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 15 op/s 2023-10-10T17:44:34.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:34 smithi190 ceph-mon[101220]: pgmap v64: 129 pgs: 129 active+clean; 2.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 15 op/s 2023-10-10T17:44:34.940 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:34.941 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:34.941 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:34.942 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:34.942 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:34.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:34 smithi157 ceph-mon[100896]: pgmap v64: 129 pgs: 129 active+clean; 2.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 15 op/s 2023-10-10T17:44:35.756 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:35.756 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:35.917 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:35.918 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:35.935 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:35.935 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:35.936 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 99 -lt 100 ']' 2023-10-10T17:44:35.936 INFO:tasks.workunit.client.0.smithi078.stderr:++ expr 99 + 1 2023-10-10T17:44:35.995 INFO:tasks.workunit.client.0.smithi078.stderr:+ c=100 2023-10-10T17:44:35.996 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:36.082 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:36.083 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:36.084 INFO:tasks.workunit.client.0.smithi078.stderr:+ c1=2 2023-10-10T17:44:36.084 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo 3 2023-10-10T17:44:36.084 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo tee /proc/sys/vm/drop_caches 2023-10-10T17:44:36.764 INFO:tasks.workunit.client.0.smithi078.stderr:++ getfacl d1 2023-10-10T17:44:36.764 INFO:tasks.workunit.client.0.smithi078.stderr:++ grep -c nobody:rw 2023-10-10T17:44:36.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:36 smithi078 ceph-mon[99494]: pgmap v65: 129 pgs: 129 active+clean; 2.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 12 op/s 2023-10-10T17:44:36.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:36 smithi190 ceph-mon[101220]: pgmap v65: 129 pgs: 129 active+clean; 2.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 12 op/s 2023-10-10T17:44:36.967 INFO:tasks.workunit.client.0.smithi078.stderr:+ c2=2 2023-10-10T17:44:36.968 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:36.977 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:36.977 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 2 -ne 2 ']' 2023-10-10T17:44:36.977 INFO:tasks.workunit.client.0.smithi078.stderr:+ '[' 100 -lt 100 ']' 2023-10-10T17:44:36.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:36 smithi157 ceph-mon[100896]: pgmap v65: 129 pgs: 129 active+clean; 2.2 MiB data, 85 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 18 KiB/s wr, 12 op/s 2023-10-10T17:44:36.989 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir d1 2023-10-10T17:44:37.043 INFO:tasks.workunit.client.0.smithi078.stderr:+ setfattr -n system.posix_acl_access -v 0x02000000 d1 2023-10-10T17:44:37.057 INFO:tasks.workunit.client.0.smithi078.stderr:+ setfattr -n system.posix_acl_default -v 0x02000000 . 2023-10-10T17:44:37.061 INFO:tasks.workunit.client.0.smithi078.stderr:+ expect_failure getfattr -n system.posix_acl_access d1 2023-10-10T17:44:37.061 INFO:tasks.workunit.client.0.smithi078.stderr:+ getfattr -n system.posix_acl_access d1 2023-10-10T17:44:37.084 INFO:tasks.workunit.client.0.smithi078.stderr:d1: system.posix_acl_access: No such attribute 2023-10-10T17:44:37.084 INFO:tasks.workunit.client.0.smithi078.stderr:+ return 0 2023-10-10T17:44:37.084 INFO:tasks.workunit.client.0.smithi078.stderr:+ expect_failure getfattr -n system.posix_acl_default . 2023-10-10T17:44:37.084 INFO:tasks.workunit.client.0.smithi078.stderr:+ getfattr -n system.posix_acl_default . 2023-10-10T17:44:37.085 INFO:tasks.workunit.client.0.smithi078.stderr:.: system.posix_acl_default: No such attribute 2023-10-10T17:44:37.085 INFO:tasks.workunit.client.0.smithi078.stderr:+ return 0 2023-10-10T17:44:37.085 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir d1 2023-10-10T17:44:37.086 INFO:tasks.workunit.client.0.smithi078.stderr:+ cd .. 2023-10-10T17:44:37.087 INFO:tasks.workunit.client.0.smithi078.stderr:+ rmdir testdir 2023-10-10T17:44:37.096 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo OK 2023-10-10T17:44:37.097 INFO:tasks.workunit.client.0.smithi078.stdout:OK 2023-10-10T17:44:37.097 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T17:44:37.097 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T17:44:37.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 -- 172.21.15.78:0/99863219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 msgr2=0x7f93201008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 --2- 172.21.15.78:0/99863219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201008e0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f93100099c0 tx=0x7f9310009cd0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 -- 172.21.15.78:0/99863219 shutdown_connections 2023-10-10T17:44:37.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 --2- 172.21.15.78:0/99863219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9320105550 0x7f932010d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 --2- 172.21.15.78:0/99863219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201008e0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 --2- 172.21.15.78:0/99863219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320100eb0 0x7f9320104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 -- 172.21.15.78:0/99863219 >> 172.21.15.78:0/99863219 conn(0x7f9320077050 msgr2=0x7f9320077450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 -- 172.21.15.78:0/99863219 shutdown_connections 2023-10-10T17:44:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.409+0000 7f93280b7700 1 -- 172.21.15.78:0/99863219 wait complete. 2023-10-10T17:44:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 Processor -- start 2023-10-10T17:44:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 -- start start 2023-10-10T17:44:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201963c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320100eb0 0x7f9320196900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9320105550 0x7f932019ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932019b3a0 con 0x7f9320100510 2023-10-10T17:44:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932019b510 con 0x7f9320105550 2023-10-10T17:44:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f93280b7700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932019b6d0 con 0x7f9320100eb0 2023-10-10T17:44:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201963c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201963c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45570/0 (socket says 172.21.15.78:45570) 2023-10-10T17:44:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 -- 172.21.15.78:0/4176982345 learned_addr learned my addr 172.21.15.78:0/4176982345 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325652700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320100eb0 0x7f9320196900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9326654700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9320105550 0x7f932019ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 -- 172.21.15.78:0/4176982345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320100eb0 msgr2=0x7f9320196900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320100eb0 0x7f9320196900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 -- 172.21.15.78:0/4176982345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9320105550 msgr2=0x7f932019ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9320105550 0x7f932019ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 -- 172.21.15.78:0/4176982345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93100095f0 con 0x7f9320100510 2023-10-10T17:44:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9326654700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9320105550 0x7f932019ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.410+0000 7f9325e53700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201963c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9310004730 tx=0x7f9310004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.411+0000 7f9316ffd700 1 -- 172.21.15.78:0/4176982345 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931001d070 con 0x7f9320100510 2023-10-10T17:44:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.411+0000 7f9316ffd700 1 -- 172.21.15.78:0/4176982345 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9310004c90 con 0x7f9320100510 2023-10-10T17:44:37.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.411+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9320110360 con 0x7f9320100510 2023-10-10T17:44:37.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.411+0000 7f9316ffd700 1 -- 172.21.15.78:0/4176982345 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9310005720 con 0x7f9320100510 2023-10-10T17:44:37.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.411+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93201101d0 con 0x7f9320100510 2023-10-10T17:44:37.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.412+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93201104f0 con 0x7f9320100510 2023-10-10T17:44:37.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.413+0000 7f9316ffd700 1 -- 172.21.15.78:0/4176982345 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f9310005000 con 0x7f9320100510 2023-10-10T17:44:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.414+0000 7f9316ffd700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f930c0a12c0 0x7f930c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.414+0000 7f9325652700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f930c0a12c0 0x7f930c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.414+0000 7f9316ffd700 1 -- 172.21.15.78:0/4176982345 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f93100cdeb0 con 0x7f9320100510 2023-10-10T17:44:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.414+0000 7f9325652700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f930c0a12c0 0x7f930c0a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f93201979d0 tx=0x7f931c00b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:37.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.417+0000 7f9316ffd700 1 -- 172.21.15.78:0/4176982345 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9310097020 con 0x7f9320100510 2023-10-10T17:44:37.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.656+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9320067a60 con 0x7f9320100510 2023-10-10T17:44:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.657+0000 7f9316ffd700 1 -- 172.21.15.78:0/4176982345 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10807 (secure 0 0 0) 0x7f931009a640 con 0x7f9320100510 2023-10-10T17:44:37.659 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:44:37.659 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:41:01.590420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:44:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f930c0a12c0 msgr2=0x7f930c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f930c0a12c0 0x7f930c0a3770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f93201979d0 tx=0x7f931c00b410 comp rx=0 tx=0).stop 2023-10-10T17:44:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 msgr2=0x7f93201963c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201963c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f9310004730 tx=0x7f9310004760 comp rx=0 tx=0).stop 2023-10-10T17:44:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 shutdown_connections 2023-10-10T17:44:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f930c0a12c0 0x7f930c0a3770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9320105550 0x7f932019ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9320100510 0x7f93201963c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 --2- 172.21.15.78:0/4176982345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9320100eb0 0x7f9320196900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 >> 172.21.15.78:0/4176982345 conn(0x7f9320077050 msgr2=0x7f93200fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 shutdown_connections 2023-10-10T17:44:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.659+0000 7f93280b7700 1 -- 172.21.15.78:0/4176982345 wait complete. 2023-10-10T17:44:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-10-10T17:44:37.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:44:37.752 INFO:tasks.workunit:Running workunit fs/misc/chmod.sh... 2023-10-10T17:44:37.753 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/chmod.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh 2023-10-10T17:44:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 -- 172.21.15.78:0/4143541129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d8104560 msgr2=0x7fb1d810c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/4143541129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d8104560 0x7fb1d810c130 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb1d000b0d0 tx=0x7fb1d000b3e0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 -- 172.21.15.78:0/4143541129 shutdown_connections 2023-10-10T17:44:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/4143541129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d8104560 0x7fb1d810c130 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/4143541129 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1d80ff520 0x7fb1d80ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/4143541129 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1d80ffec0 0x7fb1d8103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 -- 172.21.15.78:0/4143541129 >> 172.21.15.78:0/4143541129 conn(0x7fb1d80760d0 msgr2=0x7fb1d80764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:37.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 -- 172.21.15.78:0/1697278228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc10a840 msgr2=0x7fd7dc10cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1697278228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc10a840 0x7fd7dc10cca0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd7d400b0a0 tx=0x7fd7d400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 -- 172.21.15.78:0/4143541129 shutdown_connections 2023-10-10T17:44:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fb1df2a1700 1 -- 172.21.15.78:0/4143541129 wait complete. 2023-10-10T17:44:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 -- 172.21.15.78:0/1697278228 shutdown_connections 2023-10-10T17:44:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1697278228 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc101500 0x7fd7dc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1697278228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc10a840 0x7fd7dc10cca0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1697278228 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc101e10 0x7fd7dc10a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 -- 172.21.15.78:0/1697278228 >> 172.21.15.78:0/1697278228 conn(0x7fd7dc075fd0 msgr2=0x7fd7dc0763d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.984+0000 7fd7e1934700 1 -- 172.21.15.78:0/1697278228 shutdown_connections 2023-10-10T17:44:37.990 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fd7e1934700 1 -- 172.21.15.78:0/1697278228 wait complete. 2023-10-10T17:44:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 Processor -- start 2023-10-10T17:44:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 -- start start 2023-10-10T17:44:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d80ff520 0x7fb1d8195170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1d80ffec0 0x7fb1d81956b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1d8104560 0x7fb1d8195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d81a49b0 con 0x7fb1d80ff520 2023-10-10T17:44:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d81a4b20 con 0x7fb1d80ffec0 2023-10-10T17:44:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1df2a1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d81a4ce0 con 0x7fb1d8104560 2023-10-10T17:44:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d80ff520 0x7fb1d8195170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dc83c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1d80ffec0 0x7fb1d81956b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d80ff520 0x7fb1d8195170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45590/0 (socket says 172.21.15.78:45590) 2023-10-10T17:44:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 -- 172.21.15.78:0/3831101062 learned_addr learned my addr 172.21.15.78:0/3831101062 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:37.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fd7e1934700 1 Processor -- start 2023-10-10T17:44:37.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1d8104560 0x7fb1d8195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:37.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 -- 172.21.15.78:0/3831101062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1d8104560 msgr2=0x7fb1d8195bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1d8104560 0x7fb1d8195bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 -- 172.21.15.78:0/3831101062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1d80ffec0 msgr2=0x7fb1d81956b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:37.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1d80ffec0 0x7fb1d81956b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:37.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fb1dd03d700 1 -- 172.21.15.78:0/3831101062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0009d00 con 0x7fb1d80ff520 2023-10-10T17:44:37.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.985+0000 7fd7e1934700 1 -- start start 2023-10-10T17:44:37.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d80ff520 0x7fb1d8195170 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400e9a0 tx=0x7fb1d400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7e1934700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101500 0x7fd7dc0ff150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7e1934700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc101e10 0x7fd7dc0ff690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7e1934700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc10a840 0x7fd7dc1a4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:37.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fb1ce7fc700 1 -- 172.21.15.78:0/3831101062 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4009790 con 0x7fb1d80ff520 2023-10-10T17:44:37.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d810f380 con 0x7fb1d80ff520 2023-10-10T17:44:37.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fb1ce7fc700 1 -- 172.21.15.78:0/3831101062 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1d4010430 con 0x7fb1d80ff520 2023-10-10T17:44:37.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7e1934700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc0ffec0 con 0x7fd7dc101500 2023-10-10T17:44:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7e1934700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc1a4c80 con 0x7fd7dc10a840 2023-10-10T17:44:37.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7e1934700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7dc1a4dc0 con 0x7fd7dc101e10 2023-10-10T17:44:37.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fb1ce7fc700 1 -- 172.21.15.78:0/3831101062 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d4018680 con 0x7fb1d80ff520 2023-10-10T17:44:38.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d810f1f0 con 0x7fb1d80ff520 2023-10-10T17:44:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101500 0x7fd7dc0ff150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101500 0x7fd7dc0ff150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45596/0 (socket says 172.21.15.78:45596) 2023-10-10T17:44:38.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 -- 172.21.15.78:0/1169365958 learned_addr learned my addr 172.21.15.78:0/1169365958 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7db7fe700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc10a840 0x7fd7dc1a4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc101e10 0x7fd7dc0ff690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 -- 172.21.15.78:0/1169365958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc101e10 msgr2=0x7fd7dc0ff690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc101e10 0x7fd7dc0ff690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 -- 172.21.15.78:0/1169365958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc10a840 msgr2=0x7fd7dc1a4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc10a840 0x7fd7dc1a4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 -- 172.21.15.78:0/1169365958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d4009d00 con 0x7fd7dc101500 2023-10-10T17:44:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.986+0000 7fd7daffd700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101500 0x7fd7dc0ff150 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00e9a0 tx=0x7fd7cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.987+0000 7fd7e0932700 1 -- 172.21.15.78:0/1169365958 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc009790 con 0x7fd7dc101500 2023-10-10T17:44:38.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.987+0000 7fd7e0932700 1 -- 172.21.15.78:0/1169365958 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7cc010430 con 0x7fd7dc101500 2023-10-10T17:44:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.987+0000 7fd7e0932700 1 -- 172.21.15.78:0/1169365958 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc018680 con 0x7fd7dc101500 2023-10-10T17:44:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.987+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7dc110320 con 0x7fd7dc101500 2023-10-10T17:44:38.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.987+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc110190 con 0x7fd7dc101500 2023-10-10T17:44:38.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.989+0000 7fb1ce7fc700 1 -- 172.21.15.78:0/3831101062 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fb1d40105a0 con 0x7fb1d80ff520 2023-10-10T17:44:38.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.989+0000 7fd7e0932700 1 -- 172.21.15.78:0/1169365958 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fd7cc0105a0 con 0x7fd7dc101500 2023-10-10T17:44:38.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.989+0000 7fb1ce7fc700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb1c40a12c0 0x7fb1c40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.989+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb1c40a12c0 0x7fb1c40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.989+0000 7fb1ce7fc700 1 -- 172.21.15.78:0/3831101062 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fb1d4014070 con 0x7fb1d80ff520 2023-10-10T17:44:38.006 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.990+0000 7fd7e0932700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd7c40a1340 0x7fd7c40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.990+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd7c40a1340 0x7fd7c40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.990+0000 7fd7e0932700 1 -- 172.21.15.78:0/1169365958 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd7cc014070 con 0x7fd7dc101500 2023-10-10T17:44:38.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:37.990+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb1c40a12c0 0x7fb1c40a3770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8069770 tx=0x7fb1c8009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:37.990+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd7c40a1340 0x7fd7c40a37f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc100940 tx=0x7fd7d0009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.076+0000 7fd7edac0b80 1 Processor -- start 2023-10-10T17:44:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 Processor -- start 2023-10-10T17:44:38.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7edac0b80 1 -- start start 2023-10-10T17:44:38.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 -- start start 2023-10-10T17:44:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263bdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594343fc6b0 0x5594345a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e6263c2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6262c3060 con 0x55e6263c28b0 2023-10-10T17:44:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6263158a0 con 0x55e6263bdb60 2023-10-10T17:44:38.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e62630ed30 con 0x55e6263be470 2023-10-10T17:44:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 0x5594345a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594344a6000 con 0x5594343fc6b0 2023-10-10T17:44:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594344f8840 con 0x5594345a6b90 2023-10-10T17:44:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594344f1cd0 con 0x5594345a6720 2023-10-10T17:44:38.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594343fc6b0 0x5594345a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594343fc6b0 0x5594345a61e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45612/0 (socket says 172.21.15.78:45612) 2023-10-10T17:44:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 -- 172.21.15.78:0/3323786851 learned_addr learned my addr 172.21.15.78:0/3323786851 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e6263c2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e6263c2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45610/0 (socket says 172.21.15.78:45610) 2023-10-10T17:44:38.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 -- 172.21.15.78:0/4213269444 learned_addr learned my addr 172.21.15.78:0/4213269444 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7db7fe700 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 0x5594345a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263bdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 -- 172.21.15.78:0/3323786851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 msgr2=0x5594345a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 -- 172.21.15.78:0/3323786851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 msgr2=0x5594345a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 0x5594345a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.092 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fd7daffd700 1 -- 172.21.15.78:0/3323786851 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594343735d0 con 0x5594343fc6b0 2023-10-10T17:44:38.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 -- 172.21.15.78:0/4213269444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 msgr2=0x55e6263c2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 -- 172.21.15.78:0/4213269444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 msgr2=0x55e6263bdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263bdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.077+0000 7fb1dd83e700 1 -- 172.21.15.78:0/4213269444 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e626190630 con 0x55e6263c28b0 2023-10-10T17:44:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:44:38.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263bdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fd7daffd700 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594343fc6b0 0x5594345a61e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0a1e60 tx=0x7fd7cc099b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb24b2dbc04c409 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e6263c2d00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009c90 tx=0x7fb1d0005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69cf08438715a97 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fd7c1ffb700 1 -- 172.21.15.78:0/3323786851 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc09df70 con 0x5594343fc6b0 2023-10-10T17:44:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fd7c1ffb700 1 -- 172.21.15.78:0/3323786851 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7cc0a2390 con 0x5594343fc6b0 2023-10-10T17:44:38.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fd7c1ffb700 1 -- 172.21.15.78:0/3323786851 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc099630 con 0x5594343fc6b0 2023-10-10T17:44:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fb1c37fe700 1 -- 172.21.15.78:0/4213269444 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000e040 con 0x55e6263c28b0 2023-10-10T17:44:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fb1c37fe700 1 -- 172.21.15.78:0/4213269444 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1d00074e0 con 0x55e6263c28b0 2023-10-10T17:44:38.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fb1c37fe700 1 -- 172.21.15.78:0/4213269444 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0003e40 con 0x55e6263c28b0 2023-10-10T17:44:38.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3323786851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594343fc6b0 msgr2=0x5594345a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.078+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594343fc6b0 0x5594345a61e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0a1e60 tx=0x7fd7cc099b70 comp rx=0 tx=0).stop 2023-10-10T17:44:38.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3323786851 shutdown_connections 2023-10-10T17:44:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 0x5594345a0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594343fc6b0 0x5594345a61e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3323786851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3323786851 >> 172.21.15.78:0/3323786851 conn(0x559434595f50 msgr2=0x55943459b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3323786851 shutdown_connections 2023-10-10T17:44:38.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3323786851 wait complete. 2023-10-10T17:44:38.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4213269444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 msgr2=0x55e6263c2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e6263c2d00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0009c90 tx=0x7fb1d0005a20 comp rx=0 tx=0).stop 2023-10-10T17:44:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4213269444 shutdown_connections 2023-10-10T17:44:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e6263c2d00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263bdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4213269444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4213269444 >> 172.21.15.78:0/4213269444 conn(0x55e6263b3360 msgr2=0x55e6263b8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4213269444 shutdown_connections 2023-10-10T17:44:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4213269444 wait complete. 2023-10-10T17:44:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 -- start start 2023-10-10T17:44:38.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fb1eb42db80 1 Processor -- start 2023-10-10T17:44:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 -- start start 2023-10-10T17:44:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.079+0000 7fd7edac0b80 1 Processor -- start 2023-10-10T17:44:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345acb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6b90 0x5594345aa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559434493880 con 0x5594345a6b90 2023-10-10T17:44:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559434493520 con 0x5594343fc6b0 2023-10-10T17:44:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594343758c0 con 0x5594345a6720 2023-10-10T17:44:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7edac0b80 10 client.0 _ll_register_callbacks cb 0x5594344b1720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7db7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6b90 0x5594345aa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7db7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6b90 0x5594345aa4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45624/0 (socket says 172.21.15.78:45624) 2023-10-10T17:44:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7db7fe700 1 -- 172.21.15.78:0/2556909849 learned_addr learned my addr 172.21.15.78:0/2556909849 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7daffd700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345acb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263bdb60 0x55e6263c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 0x55e6263c7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6262b08e0 con 0x55e6263bdb60 2023-10-10T17:44:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6262b0580 con 0x55e6263c28b0 2023-10-10T17:44:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e626192920 con 0x55e6263be470 2023-10-10T17:44:38.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1eb42db80 10 client.0 _ll_register_callbacks cb 0x55e6262ce780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dd03d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263bdb60 0x55e6263c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dd03d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263bdb60 0x55e6263c9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45626/0 (socket says 172.21.15.78:45626) 2023-10-10T17:44:38.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dc83c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dd03d700 1 -- 172.21.15.78:0/2565118915 learned_addr learned my addr 172.21.15.78:0/2565118915 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7da7fc700 1 -- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 msgr2=0x5594345acb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345acb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7da7fc700 1 -- 172.21.15.78:0/2556909849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6b90 msgr2=0x5594345aa4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.111 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6b90 0x5594345aa4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 0x55e6263c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.112 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7da7fc700 1 -- 172.21.15.78:0/2556909849 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7dc110190 con 0x5594345a6720 2023-10-10T17:44:38.113 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7daffd700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dc83c700 1 -- 172.21.15.78:0/2565118915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 msgr2=0x55e6263c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 0x55e6263c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dc83c700 1 -- 172.21.15.78:0/2565118915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263bdb60 msgr2=0x55e6263c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263bdb60 0x55e6263c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dc83c700 1 -- 172.21.15.78:0/2565118915 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d810c970 con 0x55e6263be470 2023-10-10T17:44:38.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263bdb60 0x55e6263c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:44:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.080+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a9da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0005950 tx=0x7fd7d00047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9843e59ec257336 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7baffd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:44:38.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c7260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8069800 tx=0x7fb1c80047c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d13109c710310f97 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:44:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7baffd700 1 -- 172.21.15.78:0/2556909849 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0004260 con 0x5594345a6720 2023-10-10T17:44:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559434593710 con 0x5594345a6720 2023-10-10T17:44:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7baffd700 1 -- 172.21.15.78:0/2556909849 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd7d00043c0 con 0x5594345a6720 2023-10-10T17:44:38.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/2565118915 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8004240 con 0x55e6263be470 2023-10-10T17:44:38.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e6263b0b20 con 0x55e6263be470 2023-10-10T17:44:38.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/2565118915 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fb1c80043a0 con 0x55e6263be470 2023-10-10T17:44:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7baffd700 1 -- 172.21.15.78:0/2556909849 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7d0004580 con 0x5594345a6720 2023-10-10T17:44:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7edac0b80 10 client.25552 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5594344b8130 con 0x5594345a6720 2023-10-10T17:44:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/2565118915 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1c8004560 con 0x55e6263be470 2023-10-10T17:44:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1eb42db80 10 client.25555 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e6262d5190 con 0x55e6263be470 2023-10-10T17:44:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7baffd700 1 -- 172.21.15.78:0/2556909849 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559434593710 con 0x5594345a6720 2023-10-10T17:44:38.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/2565118915 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e6263b0b20 con 0x55e6263be470 2023-10-10T17:44:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7baffd700 1 -- 172.21.15.78:0/2556909849 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd7d001d970 con 0x5594345a6720 2023-10-10T17:44:38.123 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7edac0b80 10 client.25552 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7edac0b80 10 client.25552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:38.124 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7edac0b80 20 client.25552 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.081+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5594345ba070 0x5594345bc480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fd7edac0b80 4 client.25552 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:38.125 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594344b4040 con 0x5594345ba070 2023-10-10T17:44:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/2565118915 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb1c8027840 con 0x55e6263be470 2023-10-10T17:44:38.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1eb42db80 10 client.25555 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1eb42db80 10 client.25555 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:44:38.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1eb42db80 20 client.25555 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e6263e0030 0x55e6263e2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1eb42db80 4 client.25555 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:38.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6262d10a0 con 0x55e6263e0030 2023-10-10T17:44:38.130 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fd7daffd700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5594345ba070 0x5594345bc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e6263e0030 0x55e6263e2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.131 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fd7baffd700 10 client.25552 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.132 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fd7daffd700 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5594345ba070 0x5594345bc480 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b409e6259ba47d34 server_cookie=9681af47de30e8bd in_seq=0 out_seq=0 2023-10-10T17:44:38.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1c0ff9700 10 client.25555 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:44:38.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.082+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e6263e0030 0x55e6263e2440 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d79c590ed4c7d8fd server_cookie=a7f9fab879919c53 in_seq=0 out_seq=0 2023-10-10T17:44:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7baffd700 1 -- 172.21.15.78:0/2556909849 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5594344b4040 con 0x5594345ba070 2023-10-10T17:44:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7baffd700 10 client.25552 handle_command_reply: tid=0 2023-10-10T17:44:38.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/2565118915 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e6262d10a0 con 0x55e6263e0030 2023-10-10T17:44:38.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1c0ff9700 10 client.25555 handle_command_reply: tid=0 2023-10-10T17:44:38.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 client.25555 shutdown 2023-10-10T17:44:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 client.25552 shutdown 2023-10-10T17:44:38.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 msgr2=0x55e6263c7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c7260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8069800 tx=0x7fb1c80047c0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 shutdown_connections 2023-10-10T17:44:38.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 0x55e6263c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263bdb60 0x55e6263c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e6263e0030 0x55e6263e2440 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/2565118915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263c7260 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 >> 172.21.15.78:0/2565118915 conn(0x55e6263b3360 msgr2=0x55e6263c3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1c0ff9700 0 client.25555 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:44:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 msgr2=0x5594345a9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.141 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a9da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7d0005950 tx=0x7fd7d00047c0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 shutdown_connections 2023-10-10T17:44:38.142 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 shutdown_connections 2023-10-10T17:44:38.143 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5594345ba070 0x5594345bc480 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.143 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6b90 0x5594345aa4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.143 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7baffd700 0 client.25552 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345acb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/2556909849 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6720 0x5594345a9da0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 >> 172.21.15.78:0/2556909849 conn(0x559434595f50 msgr2=0x55943459a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.144 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 shutdown_connections 2023-10-10T17:44:38.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 1 -- 172.21.15.78:0/2565118915 wait complete. 2023-10-10T17:44:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 1 -- 172.21.15.78:0/2556909849 wait complete. 2023-10-10T17:44:38.146 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fd7edac0b80 20 client.25552 trim_cache size 0 max 16384 2023-10-10T17:44:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.083+0000 7fb1eb42db80 20 client.25555 trim_cache size 0 max 16384 2023-10-10T17:44:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 Processor -- start 2023-10-10T17:44:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 -- start start 2023-10-10T17:44:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263bdb60 0x55e6263b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263be470 0x55e6263ce940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 0x55e6263c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6262d10a0 con 0x55e6263be470 2023-10-10T17:44:38.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6263b09c0 con 0x55e6263c28b0 2023-10-10T17:44:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6263c57c0 con 0x55e6263bdb60 2023-10-10T17:44:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263be470 0x55e6263ce940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263be470 0x55e6263ce940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45638/0 (socket says 172.21.15.78:45638) 2023-10-10T17:44:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 -- 172.21.15.78:0/4134959377 learned_addr learned my addr 172.21.15.78:0/4134959377 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263bdb60 0x55e6263b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 -- 172.21.15.78:0/4134959377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263bdb60 msgr2=0x55e6263b8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263bdb60 0x55e6263b8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 -- 172.21.15.78:0/4134959377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 msgr2=0x55e6263c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 0x55e6263c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 -- 172.21.15.78:0/4134959377 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6263b7820 con 0x55e6263be470 2023-10-10T17:44:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.108+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263be470 0x55e6263ce940 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8016040 tx=0x7fb1c8027d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=293d4d5eca50666a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1c3fff700 1 -- 172.21.15.78:0/4134959377 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c801fdc0 con 0x55e6263be470 2023-10-10T17:44:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1c3fff700 1 -- 172.21.15.78:0/4134959377 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1c8009040 con 0x55e6263be470 2023-10-10T17:44:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1c3fff700 1 -- 172.21.15.78:0/4134959377 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8004a90 con 0x55e6263be470 2023-10-10T17:44:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4134959377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263be470 msgr2=0x55e6263ce940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263be470 0x55e6263ce940 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8016040 tx=0x7fb1c8027d70 comp rx=0 tx=0).stop 2023-10-10T17:44:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4134959377 shutdown_connections 2023-10-10T17:44:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263bdb60 0x55e6263b8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263c28b0 0x55e6263c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/4134959377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263be470 0x55e6263ce940 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4134959377 >> 172.21.15.78:0/4134959377 conn(0x55e6263b3360 msgr2=0x55e6263c4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4134959377 shutdown_connections 2023-10-10T17:44:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.109+0000 7fb1eb42db80 1 -- 172.21.15.78:0/4134959377 wait complete. 2023-10-10T17:44:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 -- start start 2023-10-10T17:44:38.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 Processor -- start 2023-10-10T17:44:38.156 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 Processor -- start 2023-10-10T17:44:38.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263cd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e626432ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e626433430 con 0x55e6263c28b0 2023-10-10T17:44:38.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e626433570 con 0x55e6263bdb60 2023-10-10T17:44:38.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6264336b0 con 0x55e6263be470 2023-10-10T17:44:38.158 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 -- start start 2023-10-10T17:44:38.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1eb42db80 10 client.0 _ll_register_callbacks cb 0x7fb1c801f430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1dd83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e626432ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1dd83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e626432ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45646/0 (socket says 172.21.15.78:45646) 2023-10-10T17:44:38.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1dd83e700 1 -- 172.21.15.78:0/3215926821 learned_addr learned my addr 172.21.15.78:0/3215926821 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.160 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6b90 0x5594345a0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fb1dc83c700 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.161 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559434595b70 con 0x5594345a6720 2023-10-10T17:44:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594345a8360 con 0x5594343fc6b0 2023-10-10T17:44:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.110+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594345a4ab0 con 0x5594345a6b90 2023-10-10T17:44:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7da7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.162 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1dd83e700 1 -- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 msgr2=0x55e6263cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263cd060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1dd83e700 1 -- 172.21.15.78:0/3215926821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 msgr2=0x55e6263ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1dd83e700 1 -- 172.21.15.78:0/3215926821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1c802c040 con 0x55e6263c28b0 2023-10-10T17:44:38.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7db7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6b90 0x5594345a0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345a55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47370/0 (socket says 172.21.15.78:47370) 2023-10-10T17:44:38.164 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7da7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45650/0 (socket says 172.21.15.78:45650) 2023-10-10T17:44:38.165 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 -- 172.21.15.78:0/1645161696 learned_addr learned my addr 172.21.15.78:0/1645161696 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1dd83e700 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e626432ef0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0005a90 tx=0x7fb1d0006250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207ec827451274e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 -- 172.21.15.78:0/1645161696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6b90 msgr2=0x5594345a0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6b90 0x5594345a0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 -- 172.21.15.78:0/1645161696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 msgr2=0x5594345a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.166 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 -- 172.21.15.78:0/1645161696 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594344b7e80 con 0x5594343fc6b0 2023-10-10T17:44:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7db7fe700 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6b90 0x5594345a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:44:38.167 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:44:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/3215926821 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d000e040 con 0x55e6263c28b0 2023-10-10T17:44:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/3215926821 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fb1d0003c00 con 0x55e6263c28b0 2023-10-10T17:44:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb1d00074e0 con 0x55e6263c28b0 2023-10-10T17:44:38.169 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7daffd700 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345a55d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc099c40 tx=0x7fd7cc0cab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89422111040b9a40 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/3215926821 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1d0003dc0 con 0x55e6263c28b0 2023-10-10T17:44:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1eb42db80 10 client.16086 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55e62613cde0 con 0x55e6263c28b0 2023-10-10T17:44:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7c27fc700 1 -- 172.21.15.78:0/1645161696 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0b1070 con 0x5594343fc6b0 2023-10-10T17:44:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7c27fc700 1 -- 172.21.15.78:0/1645161696 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7cc010880 con 0x5594343fc6b0 2023-10-10T17:44:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fd7c27fc700 1 -- 172.21.15.78:0/1645161696 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc0a1f00 con 0x5594343fc6b0 2023-10-10T17:44:38.171 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 -- 172.21.15.78:0/1645161696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 msgr2=0x5594345a55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345a55d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc099c40 tx=0x7fd7cc0cab40 comp rx=0 tx=0).stop 2023-10-10T17:44:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 -- 172.21.15.78:0/1645161696 shutdown_connections 2023-10-10T17:44:38.172 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594345a6b90 0x5594345a0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594343fc6b0 0x5594345a55d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.173 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/1645161696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.111+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/3215926821 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb1d00074e0 con 0x55e6263c28b0 2023-10-10T17:44:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/3215926821 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb1d000b100 con 0x55e6263c28b0 2023-10-10T17:44:38.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1eb42db80 10 client.16086 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1eb42db80 10 client.16086 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:44:38.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1eb42db80 20 client.16086 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e626434890 0x55e626436c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1eb42db80 4 client.16086 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:44:38.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e6262dd660 con 0x55e626434890 2023-10-10T17:44:38.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e626434890 0x55e626436c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.175 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 -- 172.21.15.78:0/1645161696 >> 172.21.15.78:0/1645161696 conn(0x559434595f50 msgr2=0x5594345a4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 -- 172.21.15.78:0/1645161696 shutdown_connections 2023-10-10T17:44:38.176 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fd7edac0b80 1 -- 172.21.15.78:0/1645161696 wait complete. 2023-10-10T17:44:38.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1c0ff9700 10 client.16086 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:44:38.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.112+0000 7fb1dd03d700 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e626434890 0x55e626436c70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a584f039df5d9417 server_cookie=68c09eeed903b022 in_seq=0 out_seq=0 2023-10-10T17:44:38.177 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 -- start start 2023-10-10T17:44:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1c0ff9700 1 -- 172.21.15.78:0/3215926821 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e6262dd660 con 0x55e626434890 2023-10-10T17:44:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1c0ff9700 10 client.16086 handle_command_reply: tid=0 2023-10-10T17:44:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 client.16086 shutdown 2023-10-10T17:44:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 msgr2=0x55e626432ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e626432ef0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb1d0005a90 tx=0x7fb1d0006250 comp rx=0 tx=0).stop 2023-10-10T17:44:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 shutdown_connections 2023-10-10T17:44:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e6263c28b0 0x55e626432ef0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e6263bdb60 0x55e6263ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55e626434890 0x55e626436c70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 --2- 172.21.15.78:0/3215926821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e6263be470 0x55e6263cd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1c0ff9700 0 client.16086 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:44:38.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 >> 172.21.15.78:0/3215926821 conn(0x55e6263b3360 msgr2=0x55e6263c08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.180 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 Processor -- start 2023-10-10T17:44:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 shutdown_connections 2023-10-10T17:44:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 1 -- 172.21.15.78:0/3215926821 wait complete. 2023-10-10T17:44:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fb1eb42db80 20 client.16086 trim_cache size 0 max 16384 2023-10-10T17:44:38.182 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594343fc6b0 0x5594345ad060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 0x55943460cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594345add30 con 0x5594345a6720 2023-10-10T17:44:38.183 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594345adea0 con 0x5594345a6b90 2023-10-10T17:44:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943460d470 con 0x5594343fc6b0 2023-10-10T17:44:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7edac0b80 10 client.0 _ll_register_callbacks cb 0x7fd7d001f4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7da7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7da7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345ad5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45658/0 (socket says 172.21.15.78:45658) 2023-10-10T17:44:38.184 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7da7fc700 1 -- 172.21.15.78:0/3464754394 learned_addr learned my addr 172.21.15.78:0/3464754394 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.185 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:44:38.185 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.113+0000 7fd7daffd700 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594343fc6b0 0x5594345ad060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7da7fc700 1 -- 172.21.15.78:0/3464754394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594343fc6b0 msgr2=0x5594345ad060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594343fc6b0 0x5594345ad060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7da7fc700 1 -- 172.21.15.78:0/3464754394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 msgr2=0x55943460cf30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:44:38.186 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 0x55943460cf30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.187 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7da7fc700 1 -- 172.21.15.78:0/3464754394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594344b8130 con 0x5594345a6720 2023-10-10T17:44:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb1c40a12c0 msgr2=0x7fb1c40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb1c40a12c0 0x7fb1c40a3770 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb1d8069770 tx=0x7fb1c8009260 comp rx=0 tx=0).stop 2023-10-10T17:44:38.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d80ff520 msgr2=0x7fb1d8195170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d80ff520 0x7fb1d8195170 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb1d400e9a0 tx=0x7fb1d400ecb0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.188 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7da7fc700 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345ad5a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd7d001dd70 tx=0x7fd7d0005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ffa33f786ef49c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 shutdown_connections 2023-10-10T17:44:38.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb1d8104560 0x7fb1d8195bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb1c40a12c0 0x7fb1c40a3770 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb1d80ff520 0x7fb1d8195170 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 --2- 172.21.15.78:0/3831101062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb1d80ffec0 0x7fb1d81956b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 >> 172.21.15.78:0/3831101062 conn(0x7fb1d80760d0 msgr2=0x7fb1d80fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.190 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7baffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:38.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 shutdown_connections 2023-10-10T17:44:38.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fb1df2a1700 1 -- 172.21.15.78:0/3831101062 wait complete. 2023-10-10T17:44:38.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7baffd700 1 -- 172.21.15.78:0/3464754394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d001d5b0 con 0x5594345a6720 2023-10-10T17:44:38.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7baffd700 1 -- 172.21.15.78:0/3464754394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd7d001d710 con 0x5594345a6720 2023-10-10T17:44:38.192 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7cc0a2390 con 0x5594345a6720 2023-10-10T17:44:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7baffd700 1 -- 172.21.15.78:0/3464754394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7d001d8d0 con 0x5594345a6720 2023-10-10T17:44:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7edac0b80 10 client.16092 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x7fd7dc10d960 con 0x5594345a6720 2023-10-10T17:44:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.114+0000 7fd7baffd700 1 -- 172.21.15.78:0/3464754394 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7cc0a2390 con 0x5594345a6720 2023-10-10T17:44:38.193 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7baffd700 1 -- 172.21.15.78:0/3464754394 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd7d00286e0 con 0x5594345a6720 2023-10-10T17:44:38.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7edac0b80 10 client.16092 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7edac0b80 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:38.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7edac0b80 20 client.16092 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55943460ebf0 0x559434611000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.194 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7edac0b80 4 client.16092 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:44:38.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5594344c0600 con 0x55943460ebf0 2023-10-10T17:44:38.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7daffd700 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55943460ebf0 0x559434611000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7baffd700 10 client.16092 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.195 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.115+0000 7fd7daffd700 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55943460ebf0 0x559434611000 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a4b141304fbf04d server_cookie=c6660d189236ae94 in_seq=0 out_seq=0 2023-10-10T17:44:38.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:44:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.130+0000 7fd7baffd700 1 -- 172.21.15.78:0/3464754394 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5594344c0600 con 0x55943460ebf0 2023-10-10T17:44:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.130+0000 7fd7baffd700 10 client.16092 handle_command_reply: tid=0 2023-10-10T17:44:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.130+0000 7fd7edac0b80 1 client.16092 shutdown 2023-10-10T17:44:38.198 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 msgr2=0x5594345ad5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345ad5a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd7d001dd70 tx=0x7fd7d0005d00 comp rx=0 tx=0).stop 2023-10-10T17:44:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 shutdown_connections 2023-10-10T17:44:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5594343fc6b0 0x5594345ad060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55943460ebf0 0x559434611000 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5594345a6b90 0x55943460cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 --2- 172.21.15.78:0/3464754394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5594345a6720 0x5594345ad5a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7baffd700 0 client.16092 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 >> 172.21.15.78:0/3464754394 conn(0x559434595f50 msgr2=0x5594345a77f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 shutdown_connections 2023-10-10T17:44:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 1 -- 172.21.15.78:0/3464754394 wait complete. 2023-10-10T17:44:38.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7edac0b80 20 client.16092 trim_cache size 0 max 16384 2023-10-10T17:44:38.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:44:38.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:44:38.201 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:44:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:44:38.202 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd7c40a1340 msgr2=0x7fd7c40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd7c40a1340 0x7fd7c40a37f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc100940 tx=0x7fd7d0009260 comp rx=0 tx=0).stop 2023-10-10T17:44:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101500 msgr2=0x7fd7dc0ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101500 0x7fd7dc0ff150 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc00e9a0 tx=0x7fd7cc00ecb0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.131+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 shutdown_connections 2023-10-10T17:44:38.203 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.132+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd7c40a1340 0x7fd7c40a37f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.132+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd7dc10a840 0x7fd7dc1a4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.132+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd7dc101500 0x7fd7dc0ff150 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.132+0000 7fd7e1934700 1 --2- 172.21.15.78:0/1169365958 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd7dc101e10 0x7fd7dc0ff690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.132+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 >> 172.21.15.78:0/1169365958 conn(0x7fd7dc075fd0 msgr2=0x7fd7dc104370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.132+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 shutdown_connections 2023-10-10T17:44:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:38.132+0000 7fd7e1934700 1 -- 172.21.15.78:0/1169365958 wait complete. 2023-10-10T17:44:38.206 INFO:tasks.workunit.client.0.smithi078.stderr:+ set -e 2023-10-10T17:44:38.206 INFO:tasks.workunit.client.0.smithi078.stderr:+ file=test_chmod.155593 2023-10-10T17:44:38.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo foo 2023-10-10T17:44:38.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 0 '!=' 0 2023-10-10T17:44:38.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ chmod 400 test_chmod.155593 2023-10-10T17:44:38.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 0 '!=' 0 2023-10-10T17:44:38.207 INFO:tasks.workunit.client.0.smithi078.stderr:+ check_perms test_chmod.155593 -r-------- 2023-10-10T17:44:38.208 INFO:tasks.workunit.client.0.smithi078.stderr:+ file=test_chmod.155593 2023-10-10T17:44:38.208 INFO:tasks.workunit.client.0.smithi078.stderr:++ ls -la test_chmod.155593 2023-10-10T17:44:38.208 INFO:tasks.workunit.client.0.smithi078.stderr:+ r='-r--------. 1 ubuntu ubuntu 4 Oct 10 17:44 test_chmod.155593' 2023-10-10T17:44:38.208 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 0 '!=' 0 2023-10-10T17:44:38.208 INFO:tasks.workunit.client.0.smithi078.stderr:+ perms=-r-------- 2023-10-10T17:44:38.209 INFO:tasks.workunit.client.0.smithi078.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Oct 10 17:44 test_chmod.155593 2023-10-10T17:44:38.209 INFO:tasks.workunit.client.0.smithi078.stderr:++ awk '{print $1}' 2023-10-10T17:44:38.246 INFO:tasks.workunit.client.0.smithi078.stderr:+ test -r-------- '!=' -r--------. 2023-10-10T17:44:38.246 INFO:tasks.workunit.client.0.smithi078.stderr:++ echo -r--------. 1 ubuntu ubuntu 4 Oct 10 17:44 test_chmod.155593 2023-10-10T17:44:38.246 INFO:tasks.workunit.client.0.smithi078.stderr:++ awk '{print $1}' 2023-10-10T17:44:38.250 INFO:tasks.workunit.client.0.smithi078.stderr:+ test -r--------. '!=' -r--------. 2023-10-10T17:44:38.250 INFO:tasks.workunit.client.0.smithi078.stderr:+ set +e 2023-10-10T17:44:38.251 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo bar 2023-10-10T17:44:38.251 INFO:tasks.workunit.client.0.smithi078.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/chmod.sh: line 40: test_chmod.155593: Permission denied 2023-10-10T17:44:38.251 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 1 = 0 2023-10-10T17:44:38.251 INFO:tasks.workunit.client.0.smithi078.stderr:+ set -e 2023-10-10T17:44:38.251 INFO:tasks.workunit.client.0.smithi078.stderr:+ chmod 600 test_chmod.155593 2023-10-10T17:44:38.252 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo bar 2023-10-10T17:44:38.252 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 0 '!=' 0 2023-10-10T17:44:38.252 INFO:tasks.workunit.client.0.smithi078.stderr:+ check_perms test_chmod.155593 -rw------- 2023-10-10T17:44:38.252 INFO:tasks.workunit.client.0.smithi078.stderr:+ file=test_chmod.155593 2023-10-10T17:44:38.252 INFO:tasks.workunit.client.0.smithi078.stderr:++ ls -la test_chmod.155593 2023-10-10T17:44:38.253 INFO:tasks.workunit.client.0.smithi078.stderr:+ r='-rw-------. 1 ubuntu ubuntu 8 Oct 10 17:44 test_chmod.155593' 2023-10-10T17:44:38.253 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 0 '!=' 0 2023-10-10T17:44:38.253 INFO:tasks.workunit.client.0.smithi078.stderr:+ perms=-rw------- 2023-10-10T17:44:38.253 INFO:tasks.workunit.client.0.smithi078.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Oct 10 17:44 test_chmod.155593 2023-10-10T17:44:38.253 INFO:tasks.workunit.client.0.smithi078.stderr:++ awk '{print $1}' 2023-10-10T17:44:38.255 INFO:tasks.workunit.client.0.smithi078.stderr:+ test -rw------- '!=' -rw-------. 2023-10-10T17:44:38.255 INFO:tasks.workunit.client.0.smithi078.stderr:++ echo -rw-------. 1 ubuntu ubuntu 8 Oct 10 17:44 test_chmod.155593 2023-10-10T17:44:38.255 INFO:tasks.workunit.client.0.smithi078.stderr:++ awk '{print $1}' 2023-10-10T17:44:38.255 INFO:tasks.workunit.client.0.smithi078.stderr:+ test -rw-------. '!=' -rw-------. 2023-10-10T17:44:38.256 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo foo 2023-10-10T17:44:38.256 INFO:tasks.workunit.client.0.smithi078.stderr:+ test 0 '!=' 0 2023-10-10T17:44:38.256 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T17:44:38.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T17:44:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 -- 172.21.15.78:0/1975846682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c101400 msgr2=0x7f871c10d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1975846682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c101400 0x7f871c10d0a0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f871400b0a0 tx=0x7f871400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 -- 172.21.15.78:0/1975846682 shutdown_connections 2023-10-10T17:44:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1975846682 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c101400 0x7f871c10d0a0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1975846682 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f871c1063f0 0x7f871c1067c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1975846682 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f871c100a50 0x7f871c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 -- 172.21.15.78:0/1975846682 >> 172.21.15.78:0/1975846682 conn(0x7f871c077590 msgr2=0x7f871c077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 -- 172.21.15.78:0/1975846682 shutdown_connections 2023-10-10T17:44:38.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.330+0000 7f8720fbd700 1 -- 172.21.15.78:0/1975846682 wait complete. 2023-10-10T17:44:38.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 Processor -- start 2023-10-10T17:44:38.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 -- start start 2023-10-10T17:44:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f871c100a50 0x7f871c19e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f871c101400 0x7f871c19eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c1063f0 0x7f871c1985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c198c40 con 0x7f871c1063f0 2023-10-10T17:44:38.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c198db0 con 0x7f871c101400 2023-10-10T17:44:38.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8720fbd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f871c198f70 con 0x7f871c100a50 2023-10-10T17:44:38.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f871a59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f871c100a50 0x7f871c19e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f871a59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f871c100a50 0x7f871c19e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37202/0 (socket says 172.21.15.78:37202) 2023-10-10T17:44:38.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f871a59c700 1 -- 172.21.15.78:0/1125374846 learned_addr learned my addr 172.21.15.78:0/1125374846 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f871ad9d700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c1063f0 0x7f871c1985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.331+0000 7f8719d9b700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f871c101400 0x7f871c19eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f871ad9d700 1 -- 172.21.15.78:0/1125374846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f871c100a50 msgr2=0x7f871c19e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f871ad9d700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f871c100a50 0x7f871c19e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f871ad9d700 1 -- 172.21.15.78:0/1125374846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f871c101400 msgr2=0x7f871c19eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f871ad9d700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f871c101400 0x7f871c19eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f871ad9d700 1 -- 172.21.15.78:0/1125374846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8714009d00 con 0x7f871c1063f0 2023-10-10T17:44:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f871ad9d700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c1063f0 0x7f871c1985d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8714009fd0 tx=0x7f871400ce50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f870b7fe700 1 -- 172.21.15.78:0/1125374846 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871400e040 con 0x7f871c1063f0 2023-10-10T17:44:38.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f871c110410 con 0x7f871c1063f0 2023-10-10T17:44:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f870b7fe700 1 -- 172.21.15.78:0/1125374846 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f871401d940 con 0x7f871c1063f0 2023-10-10T17:44:38.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f870b7fe700 1 -- 172.21.15.78:0/1125374846 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87140055e0 con 0x7f871c1063f0 2023-10-10T17:44:38.341 INFO:tasks.workunit:Running workunit fs/misc/dac_override.sh... 2023-10-10T17:44:38.341 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/dac_override.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dac_override.sh 2023-10-10T17:44:38.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.332+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c110280 con 0x7f871c1063f0 2023-10-10T17:44:38.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.335+0000 7f870b7fe700 1 -- 172.21.15.78:0/1125374846 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f871401dab0 con 0x7f871c1063f0 2023-10-10T17:44:38.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.335+0000 7f870b7fe700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f87040a12c0 0x7f87040a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.335+0000 7f871a59c700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f87040a12c0 0x7f87040a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.335+0000 7f870b7fe700 1 -- 172.21.15.78:0/1125374846 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f87140ce830 con 0x7f871c1063f0 2023-10-10T17:44:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.336+0000 7f871a59c700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f87040a12c0 0x7f87040a3770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8710005950 tx=0x7f871000b500 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.336+0000 7f872d149b80 1 Processor -- start 2023-10-10T17:44:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.336+0000 7f872d149b80 1 -- start start 2023-10-10T17:44:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.336+0000 7f872d149b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a646df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.336+0000 7f872d149b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6472370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.336+0000 7f872d149b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a64728b0 0x5592a6472d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f872d149b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a6373060 con 0x5592a646db60 2023-10-10T17:44:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f872d149b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a63c58a0 con 0x5592a64728b0 2023-10-10T17:44:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f872d149b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a63bed30 con 0x5592a646e470 2023-10-10T17:44:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a646df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a646df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45680/0 (socket says 172.21.15.78:45680) 2023-10-10T17:44:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 -- 172.21.15.78:0/1926902589 learned_addr learned my addr 172.21.15.78:0/1926902589 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f8719d9b700 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6472370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 -- 172.21.15.78:0/1926902589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 msgr2=0x5592a6472370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6472370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871ad9d700 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a64728b0 0x5592a6472d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 -- 172.21.15.78:0/1926902589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a64728b0 msgr2=0x5592a6472d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a64728b0 0x5592a6472d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 -- 172.21.15.78:0/1926902589 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592a6240630 con 0x5592a646db60 2023-10-10T17:44:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f8719d9b700 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6472370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f871a59c700 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a646df30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8710010f70 tx=0x7f8710013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=563e578fd4d55eb0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f8708ff9700 1 -- 172.21.15.78:0/1926902589 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710004220 con 0x5592a646db60 2023-10-10T17:44:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f8708ff9700 1 -- 172.21.15.78:0/1926902589 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8710015070 con 0x5592a646db60 2023-10-10T17:44:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.337+0000 7f8708ff9700 1 -- 172.21.15.78:0/1926902589 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f871001e680 con 0x5592a646db60 2023-10-10T17:44:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 -- 172.21.15.78:0/1926902589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 msgr2=0x5592a646df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a646df30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8710010f70 tx=0x7f8710013000 comp rx=0 tx=0).stop 2023-10-10T17:44:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 -- 172.21.15.78:0/1926902589 shutdown_connections 2023-10-10T17:44:38.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a64728b0 0x5592a6472d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a646df30 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 --2- 172.21.15.78:0/1926902589 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6472370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 -- 172.21.15.78:0/1926902589 >> 172.21.15.78:0/1926902589 conn(0x5592a6463360 msgr2=0x5592a6468b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 -- 172.21.15.78:0/1926902589 shutdown_connections 2023-10-10T17:44:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 -- 172.21.15.78:0/1926902589 wait complete. 2023-10-10T17:44:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 -- start start 2023-10-10T17:44:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.338+0000 7f872d149b80 1 Processor -- start 2023-10-10T17:44:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6479ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646e470 0x5592a6477270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64778a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a63608e0 con 0x5592a646e470 2023-10-10T17:44:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a6360580 con 0x5592a646db60 2023-10-10T17:44:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a6242920 con 0x5592a64728b0 2023-10-10T17:44:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 10 client.0 _ll_register_callbacks cb 0x5592a637e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646e470 0x5592a6477270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646e470 0x5592a6477270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45688/0 (socket says 172.21.15.78:45688) 2023-10-10T17:44:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 -- 172.21.15.78:0/715053481 learned_addr learned my addr 172.21.15.78:0/715053481 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f871ad9d700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64778a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f871a59c700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6479ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 -- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 msgr2=0x5592a64778a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64778a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 -- 172.21.15.78:0/715053481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 msgr2=0x5592a6479ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6479ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 -- 172.21.15.78:0/715053481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8710010c50 con 0x5592a646e470 2023-10-10T17:44:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f871a59c700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6479ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8719d9b700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646e470 0x5592a6477270 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f870c007fa0 tx=0x7f870c0127e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b3dbb9856fd432 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8701ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8701ffb700 1 -- 172.21.15.78:0/715053481 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870c010930 con 0x5592a646e470 2023-10-10T17:44:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8701ffb700 1 -- 172.21.15.78:0/715053481 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f870c010a90 con 0x5592a646e470 2023-10-10T17:44:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5592a6460b20 con 0x5592a646e470 2023-10-10T17:44:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.339+0000 7f8701ffb700 1 -- 172.21.15.78:0/715053481 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f870c010c50 con 0x5592a646e470 2023-10-10T17:44:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 10 client.16104 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5592a6385190 con 0x5592a646e470 2023-10-10T17:44:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f8701ffb700 1 -- 172.21.15.78:0/715053481 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5592a6460b20 con 0x5592a646e470 2023-10-10T17:44:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f8701ffb700 1 -- 172.21.15.78:0/715053481 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f870c01a3e0 con 0x5592a646e470 2023-10-10T17:44:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 10 client.16104 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 10 client.16104 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:44:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 20 client.16104 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a6487940 0x5592a6489d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 4 client.16104 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5592a63810a0 con 0x5592a6487940 2023-10-10T17:44:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.340+0000 7f871a59c700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a6487940 0x5592a6489d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.341+0000 7f8701ffb700 10 client.16104 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:44:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.341+0000 7f871a59c700 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a6487940 0x5592a6489d50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d836d6008f5f8bea server_cookie=dafc08614527c1cc in_seq=0 out_seq=0 2023-10-10T17:44:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.341+0000 7f8701ffb700 1 -- 172.21.15.78:0/715053481 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5592a63810a0 con 0x5592a6487940 2023-10-10T17:44:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.341+0000 7f8701ffb700 10 client.16104 handle_command_reply: tid=0 2023-10-10T17:44:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.341+0000 7f872d149b80 1 client.16104 shutdown 2023-10-10T17:44:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646e470 msgr2=0x5592a6477270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646e470 0x5592a6477270 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f870c007fa0 tx=0x7f870c0127e0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 shutdown_connections 2023-10-10T17:44:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a6487940 0x5592a6489d50 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64778a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6479ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f8701ffb700 0 client.16104 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:44:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 --2- 172.21.15.78:0/715053481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646e470 0x5592a6477270 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 >> 172.21.15.78:0/715053481 conn(0x5592a6463360 msgr2=0x5592a6473d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 shutdown_connections 2023-10-10T17:44:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 1 -- 172.21.15.78:0/715053481 wait complete. 2023-10-10T17:44:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.342+0000 7f872d149b80 20 client.16104 trim_cache size 0 max 16384 2023-10-10T17:44:38.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 Processor -- start 2023-10-10T17:44:38.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 -- start start 2023-10-10T17:44:38.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6483150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6479b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a64728b0 0x5592a64708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a64757d0 con 0x5592a64728b0 2023-10-10T17:44:38.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a6471420 con 0x5592a646db60 2023-10-10T17:44:38.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f872d149b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a646e900 con 0x5592a646e470 2023-10-10T17:44:38.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a64728b0 0x5592a64708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a64728b0 0x5592a64708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45698/0 (socket says 172.21.15.78:45698) 2023-10-10T17:44:38.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 -- 172.21.15.78:0/2929281821 learned_addr learned my addr 172.21.15.78:0/2929281821 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f8719d9b700 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6479b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871a59c700 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6483150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 -- 172.21.15.78:0/2929281821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 msgr2=0x5592a6479b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6479b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 -- 172.21.15.78:0/2929281821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 msgr2=0x5592a6483150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6483150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.356+0000 7f871ad9d700 1 -- 172.21.15.78:0/2929281821 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592a6384ee0 con 0x5592a64728b0 2023-10-10T17:44:38.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f8719d9b700 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6479b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:38.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f871a59c700 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6483150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:38.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f871ad9d700 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a64728b0 0x5592a64708b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f871401dd60 tx=0x7f8714009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9a6868bc10feee server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f87097fa700 1 -- 172.21.15.78:0/2929281821 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87140bb0c0 con 0x5592a64728b0 2023-10-10T17:44:38.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f87097fa700 1 -- 172.21.15.78:0/2929281821 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f871409b660 con 0x5592a64728b0 2023-10-10T17:44:38.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f87097fa700 1 -- 172.21.15.78:0/2929281821 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87140860e0 con 0x5592a64728b0 2023-10-10T17:44:38.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f872d149b80 1 -- 172.21.15.78:0/2929281821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a64728b0 msgr2=0x5592a64708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.357+0000 7f872d149b80 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a64728b0 0x5592a64708b0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f871401dd60 tx=0x7f8714009040 comp rx=0 tx=0).stop 2023-10-10T17:44:38.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- 172.21.15.78:0/2929281821 shutdown_connections 2023-10-10T17:44:38.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a64728b0 0x5592a64708b0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646db60 0x5592a6483150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 --2- 172.21.15.78:0/2929281821 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a646e470 0x5592a6479b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- 172.21.15.78:0/2929281821 >> 172.21.15.78:0/2929281821 conn(0x5592a6463360 msgr2=0x5592a6474c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- 172.21.15.78:0/2929281821 shutdown_connections 2023-10-10T17:44:38.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- 172.21.15.78:0/2929281821 wait complete. 2023-10-10T17:44:38.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- start start 2023-10-10T17:44:38.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 Processor -- start 2023-10-10T17:44:38.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a647c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646e470 0x5592a647ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64da800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a647d6e0 con 0x5592a646db60 2023-10-10T17:44:38.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a647d850 con 0x5592a646e470 2023-10-10T17:44:38.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592a64dadc0 con 0x5592a64728b0 2023-10-10T17:44:38.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f872d149b80 10 client.0 _ll_register_callbacks cb 0x7f870c007a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f871a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a647c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f871a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a647c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45700/0 (socket says 172.21.15.78:45700) 2023-10-10T17:44:38.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f871a59c700 1 -- 172.21.15.78:0/144039236 learned_addr learned my addr 172.21.15.78:0/144039236 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.358+0000 7f8719d9b700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646e470 0x5592a647ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871ad9d700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64da800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871a59c700 1 -- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 msgr2=0x5592a64da800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871a59c700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64da800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871a59c700 1 -- 172.21.15.78:0/144039236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646e470 msgr2=0x5592a647ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871a59c700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646e470 0x5592a647ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871a59c700 1 -- 172.21.15.78:0/144039236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f871c110280 con 0x5592a646db60 2023-10-10T17:44:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871ad9d700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64da800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:38.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f871a59c700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a647c990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8710005e10 tx=0x7f8710010ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa907639ba445544 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f8701ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f8701ffb700 1 -- 172.21.15.78:0/144039236 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8710003ae0 con 0x5592a646db60 2023-10-10T17:44:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f8701ffb700 1 -- 172.21.15.78:0/144039236 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f8710038020 con 0x5592a646db60 2023-10-10T17:44:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8710015070 con 0x5592a646db60 2023-10-10T17:44:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f8701ffb700 1 -- 172.21.15.78:0/144039236 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8710033610 con 0x5592a646db60 2023-10-10T17:44:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f872d149b80 10 client.16110 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5592a61d5c60 con 0x5592a646db60 2023-10-10T17:44:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.359+0000 7f8701ffb700 1 -- 172.21.15.78:0/144039236 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8710015070 con 0x5592a646db60 2023-10-10T17:44:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f8701ffb700 1 -- 172.21.15.78:0/144039236 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8710024070 con 0x5592a646db60 2023-10-10T17:44:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f872d149b80 10 client.16110 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f872d149b80 10 client.16110 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:44:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f872d149b80 20 client.16110 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f872d149b80 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a64dc4b0 0x5592a64de8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f872d149b80 4 client.16110 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:44:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5592a638d660 con 0x5592a64dc4b0 2023-10-10T17:44:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f8719d9b700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a64dc4b0 0x5592a64de8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f8701ffb700 10 client.16110 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:44:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.360+0000 7f8719d9b700 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a64dc4b0 0x5592a64de8c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c89fec4c5428903f server_cookie=c04bf39e73a42bf in_seq=0 out_seq=0 2023-10-10T17:44:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f8701ffb700 1 -- 172.21.15.78:0/144039236 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5592a638d660 con 0x5592a64dc4b0 2023-10-10T17:44:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f8701ffb700 10 client.16110 handle_command_reply: tid=0 2023-10-10T17:44:38.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 client.16110 shutdown 2023-10-10T17:44:38.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 msgr2=0x5592a647c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a647c990 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8710005e10 tx=0x7f8710010ad0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 shutdown_connections 2023-10-10T17:44:38.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5592a64dc4b0 0x5592a64de8c0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5592a64728b0 0x5592a64da800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5592a646db60 0x5592a647c990 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 --2- 172.21.15.78:0/144039236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5592a646e470 0x5592a647ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f8701ffb700 0 client.16110 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:44:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 >> 172.21.15.78:0/144039236 conn(0x5592a6463360 msgr2=0x5592a6483b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 shutdown_connections 2023-10-10T17:44:38.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 1 -- 172.21.15.78:0/144039236 wait complete. 2023-10-10T17:44:38.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f872d149b80 20 client.16110 trim_cache size 0 max 16384 2023-10-10T17:44:38.391 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:44:38.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f87040a12c0 msgr2=0x7f87040a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f87040a12c0 0x7f87040a3770 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8710005950 tx=0x7f871000b500 comp rx=0 tx=0).stop 2023-10-10T17:44:38.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c1063f0 msgr2=0x7f871c1985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.361+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c1063f0 0x7f871c1985d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8714009fd0 tx=0x7f871400ce50 comp rx=0 tx=0).stop 2023-10-10T17:44:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 shutdown_connections 2023-10-10T17:44:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f871c100a50 0x7f871c19e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f87040a12c0 0x7f87040a3770 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f871c1063f0 0x7f871c1985d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 --2- 172.21.15.78:0/1125374846 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f871c101400 0x7f871c19eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 >> 172.21.15.78:0/1125374846 conn(0x7f871c077590 msgr2=0x7f871c10b710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 shutdown_connections 2023-10-10T17:44:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.362+0000 7f8720fbd700 1 -- 172.21.15.78:0/1125374846 wait complete. 2023-10-10T17:44:38.395 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:44:38.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.554+0000 7f664919f700 1 -- 172.21.15.78:0/1310424999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 msgr2=0x7f66440ff8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.554+0000 7f664919f700 1 --2- 172.21.15.78:0/1310424999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f66440ff8a0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f66380099c0 tx=0x7f6638009cd0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.554+0000 7f664919f700 1 -- 172.21.15.78:0/1310424999 shutdown_connections 2023-10-10T17:44:38.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 --2- 172.21.15.78:0/1310424999 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6644104510 0x7f664410c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 --2- 172.21.15.78:0/1310424999 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f66440ff8a0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 --2- 172.21.15.78:0/1310424999 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f66440ffe70 0x7f6644103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 -- 172.21.15.78:0/1310424999 >> 172.21.15.78:0/1310424999 conn(0x7f6644076070 msgr2=0x7f6644076470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 -- 172.21.15.78:0/1310424999 shutdown_connections 2023-10-10T17:44:38.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 -- 172.21.15.78:0/1310424999 wait complete. 2023-10-10T17:44:38.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 Processor -- start 2023-10-10T17:44:38.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.555+0000 7f664919f700 1 -- start start 2023-10-10T17:44:38.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664919f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f6644100ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664919f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66440ffe70 0x7f6644101410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664919f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644104510 0x7f66441a4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664919f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644101c40 con 0x7f66440ff4d0 2023-10-10T17:44:38.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664919f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66441a4a90 con 0x7f6644104510 2023-10-10T17:44:38.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664919f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66441a4bd0 con 0x7f66440ffe70 2023-10-10T17:44:38.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f6644100ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f6644100ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45714/0 (socket says 172.21.15.78:45714) 2023-10-10T17:44:38.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 -- 172.21.15.78:0/1774284862 learned_addr learned my addr 172.21.15.78:0/1774284862 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664259c700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66440ffe70 0x7f6644101410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664359e700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644104510 0x7f66441a4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 -- 172.21.15.78:0/1774284862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66440ffe70 msgr2=0x7f6644101410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66440ffe70 0x7f6644101410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 -- 172.21.15.78:0/1774284862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644104510 msgr2=0x7f66441a4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644104510 0x7f66441a4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 -- 172.21.15.78:0/1774284862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66380095f0 con 0x7f66440ff4d0 2023-10-10T17:44:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f6642d9d700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f6644100ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f663800b580 tx=0x7f663800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f662bfff700 1 -- 172.21.15.78:0/1774284862 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663801d070 con 0x7f66440ff4d0 2023-10-10T17:44:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f662bfff700 1 -- 172.21.15.78:0/1774284862 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6638005030 con 0x7f66440ff4d0 2023-10-10T17:44:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f662bfff700 1 -- 172.21.15.78:0/1774284862 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6638004ad0 con 0x7f66440ff4d0 2023-10-10T17:44:38.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.556+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664410f2d0 con 0x7f66440ff4d0 2023-10-10T17:44:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.557+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664410f140 con 0x7f66440ff4d0 2023-10-10T17:44:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.559+0000 7f662bfff700 1 -- 172.21.15.78:0/1774284862 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6638005410 con 0x7f66440ff4d0 2023-10-10T17:44:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.559+0000 7f662bfff700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f662c0a12c0 0x7f662c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.559+0000 7f662bfff700 1 -- 172.21.15.78:0/1774284862 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f66380cdd50 con 0x7f66440ff4d0 2023-10-10T17:44:38.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.559+0000 7f664259c700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f662c0a12c0 0x7f662c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f664259c700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f662c0a12c0 0x7f662c0a3770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f66441026c0 tx=0x7f663400a380 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 Processor -- start 2023-10-10T17:44:38.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 -- start start 2023-10-10T17:44:38.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5464f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5469370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d54698b0 0x5622d5469d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d536a060 con 0x5622d5465470 2023-10-10T17:44:38.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d53bc8a0 con 0x5622d5464b60 2023-10-10T17:44:38.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f665532bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d53b5d30 con 0x5622d54698b0 2023-10-10T17:44:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f664259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5469370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f664259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5469370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45724/0 (socket says 172.21.15.78:45724) 2023-10-10T17:44:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.560+0000 7f664259c700 1 -- 172.21.15.78:0/1909427065 learned_addr learned my addr 172.21.15.78:0/1909427065 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f6642d9d700 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5464f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f664359e700 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d54698b0 0x5622d5469d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f664259c700 1 -- 172.21.15.78:0/1909427065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d54698b0 msgr2=0x5622d5469d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f664259c700 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d54698b0 0x5622d5469d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f664259c700 1 -- 172.21.15.78:0/1909427065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 msgr2=0x5622d5464f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f664259c700 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5464f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f664259c700 1 -- 172.21.15.78:0/1909427065 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622d5237630 con 0x5622d5465470 2023-10-10T17:44:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f6642d9d700 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5464f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f664259c700 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5469370 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f663400fe40 tx=0x7f6634012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543592f9c1b3c73e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f66297fa700 1 -- 172.21.15.78:0/1909427065 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6634003a90 con 0x5622d5465470 2023-10-10T17:44:38.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f66297fa700 1 -- 172.21.15.78:0/1909427065 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6634003bf0 con 0x5622d5465470 2023-10-10T17:44:38.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f66297fa700 1 -- 172.21.15.78:0/1909427065 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663401d660 con 0x5622d5465470 2023-10-10T17:44:38.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.561+0000 7f665532bb80 1 -- 172.21.15.78:0/1909427065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 msgr2=0x5622d5469370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5469370 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f663400fe40 tx=0x7f6634012000 comp rx=0 tx=0).stop 2023-10-10T17:44:38.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- 172.21.15.78:0/1909427065 shutdown_connections 2023-10-10T17:44:38.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d54698b0 0x5622d5469d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5464f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 --2- 172.21.15.78:0/1909427065 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5469370 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- 172.21.15.78:0/1909427065 >> 172.21.15.78:0/1909427065 conn(0x5622d545a360 msgr2=0x5622d545fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- 172.21.15.78:0/1909427065 shutdown_connections 2023-10-10T17:44:38.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- 172.21.15.78:0/1909427065 wait complete. 2023-10-10T17:44:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- start start 2023-10-10T17:44:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 Processor -- start 2023-10-10T17:44:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d546e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d546e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d546edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d53578e0 con 0x5622d5465470 2023-10-10T17:44:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d5357580 con 0x5622d54698b0 2023-10-10T17:44:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d5239920 con 0x5622d5464b60 2023-10-10T17:44:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f665532bb80 10 client.0 _ll_register_callbacks cb 0x5622d5375780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f664259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d546e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f664259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d546e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45730/0 (socket says 172.21.15.78:45730) 2023-10-10T17:44:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.562+0000 7f664259c700 1 -- 172.21.15.78:0/2412933509 learned_addr learned my addr 172.21.15.78:0/2412933509 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f6642d9d700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d546e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664359e700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d546edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664259c700 1 -- 172.21.15.78:0/2412933509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 msgr2=0x5622d546e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664259c700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d546e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664259c700 1 -- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 msgr2=0x5622d546edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664259c700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d546edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664259c700 1 -- 172.21.15.78:0/2412933509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663400fb20 con 0x5622d5465470 2023-10-10T17:44:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664359e700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d546edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f664259c700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d546e880 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6634003f70 tx=0x7f6634003940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5da77529c5514e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f6626ff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f6626ff5700 1 -- 172.21.15.78:0/2412933509 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663402b070 con 0x5622d5465470 2023-10-10T17:44:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622d5457b20 con 0x5622d5465470 2023-10-10T17:44:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f6626ff5700 1 -- 172.21.15.78:0/2412933509 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f6634010640 con 0x5622d5465470 2023-10-10T17:44:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f6626ff5700 1 -- 172.21.15.78:0/2412933509 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6634010840 con 0x5622d5465470 2023-10-10T17:44:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f665532bb80 10 client.16122 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5622d537c190 con 0x5622d5465470 2023-10-10T17:44:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f6626ff5700 1 -- 172.21.15.78:0/2412933509 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5622d5457b20 con 0x5622d5465470 2023-10-10T17:44:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f6626ff5700 1 -- 172.21.15.78:0/2412933509 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f663401d5c0 con 0x5622d5465470 2023-10-10T17:44:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f665532bb80 10 client.16122 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f665532bb80 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.563+0000 7f665532bb80 20 client.16122 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.564+0000 7f665532bb80 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d547e7f0 0x5622d5480c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.564+0000 7f665532bb80 4 client.16122 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.564+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622d53780a0 con 0x5622d547e7f0 2023-10-10T17:44:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.564+0000 7f6642d9d700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d547e7f0 0x5622d5480c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.564+0000 7f6626ff5700 10 client.16122 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.564+0000 7f6642d9d700 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d547e7f0 0x5622d5480c00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccf6490146900004 server_cookie=4350aba6bd291d46 in_seq=0 out_seq=0 2023-10-10T17:44:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f6626ff5700 1 -- 172.21.15.78:0/2412933509 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5622d53780a0 con 0x5622d547e7f0 2023-10-10T17:44:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f6626ff5700 10 client.16122 handle_command_reply: tid=0 2023-10-10T17:44:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 client.16122 shutdown 2023-10-10T17:44:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 msgr2=0x5622d546e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d546e880 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f6634003f70 tx=0x7f6634003940 comp rx=0 tx=0).stop 2023-10-10T17:44:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 shutdown_connections 2023-10-10T17:44:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d546e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d547e7f0 0x5622d5480c00 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d546edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f6626ff5700 0 client.16122 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 --2- 172.21.15.78:0/2412933509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d546e880 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 >> 172.21.15.78:0/2412933509 conn(0x5622d545a360 msgr2=0x5622d545ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 shutdown_connections 2023-10-10T17:44:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 1 -- 172.21.15.78:0/2412933509 wait complete. 2023-10-10T17:44:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.565+0000 7f665532bb80 20 client.16122 trim_cache size 0 max 16384 2023-10-10T17:44:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 Processor -- start 2023-10-10T17:44:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 -- start start 2023-10-10T17:44:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5467970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5465470 0x5622d5467eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d54698b0 0x5622d545f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d5465bc0 con 0x5622d54698b0 2023-10-10T17:44:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d54611c0 con 0x5622d5464b60 2023-10-10T17:44:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f665532bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d54684b0 con 0x5622d5465470 2023-10-10T17:44:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d54698b0 0x5622d545f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d54698b0 0x5622d545f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45740/0 (socket says 172.21.15.78:45740) 2023-10-10T17:44:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 -- 172.21.15.78:0/411197090 learned_addr learned my addr 172.21.15.78:0/411197090 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664259c700 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5465470 0x5622d5467eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f6642d9d700 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5467970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 -- 172.21.15.78:0/411197090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5465470 msgr2=0x5622d5467eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5465470 0x5622d5467eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 -- 172.21.15.78:0/411197090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 msgr2=0x5622d5467970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5467970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f664359e700 1 -- 172.21.15.78:0/411197090 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622d5469f50 con 0x5622d54698b0 2023-10-10T17:44:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.578+0000 7f6642d9d700 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5467970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f664359e700 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d54698b0 0x5622d545f560 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f663c009530 tx=0x7f663c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b0ca46fe9e7b00 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f6629ffb700 1 -- 172.21.15.78:0/411197090 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c00e040 con 0x5622d54698b0 2023-10-10T17:44:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f6629ffb700 1 -- 172.21.15.78:0/411197090 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f663c00f040 con 0x5622d54698b0 2023-10-10T17:44:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f6629ffb700 1 -- 172.21.15.78:0/411197090 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c016840 con 0x5622d54698b0 2023-10-10T17:44:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 -- 172.21.15.78:0/411197090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d54698b0 msgr2=0x5622d545f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d54698b0 0x5622d545f560 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f663c009530 tx=0x7f663c00b040 comp rx=0 tx=0).stop 2023-10-10T17:44:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 -- 172.21.15.78:0/411197090 shutdown_connections 2023-10-10T17:44:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d54698b0 0x5622d545f560 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d5464b60 0x5622d5467970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 --2- 172.21.15.78:0/411197090 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5465470 0x5622d5467eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 -- 172.21.15.78:0/411197090 >> 172.21.15.78:0/411197090 conn(0x5622d545a360 msgr2=0x5622d547ac80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 -- 172.21.15.78:0/411197090 shutdown_connections 2023-10-10T17:44:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.579+0000 7f665532bb80 1 -- 172.21.15.78:0/411197090 wait complete. 2023-10-10T17:44:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 -- start start 2023-10-10T17:44:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 Processor -- start 2023-10-10T17:44:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d5474bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5475110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d54d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d5475900 con 0x5622d5465470 2023-10-10T17:44:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d5475a70 con 0x5622d54698b0 2023-10-10T17:44:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622d54d1c00 con 0x5622d5464b60 2023-10-10T17:44:38.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f665532bb80 10 client.0 _ll_register_callbacks cb 0x7f663400fd40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:38.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5475110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5475110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45744/0 (socket says 172.21.15.78:45744) 2023-10-10T17:44:38.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 -- 172.21.15.78:0/828655581 learned_addr learned my addr 172.21.15.78:0/828655581 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:38.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f6642d9d700 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d5474bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664359e700 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d54d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 -- 172.21.15.78:0/828655581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 msgr2=0x5622d5474bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d5474bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 -- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 msgr2=0x5622d54d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:38.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d54d1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 -- 172.21.15.78:0/828655581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c0091e0 con 0x5622d5465470 2023-10-10T17:44:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f664259c700 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5475110 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x5622d546f9f0 tx=0x7f6634005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d52f6fe6e620dfc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.580+0000 7f6626ff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f6626ff5700 1 -- 172.21.15.78:0/828655581 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663402b070 con 0x5622d5465470 2023-10-10T17:44:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f6626ff5700 1 -- 172.21.15.78:0/828655581 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f663401d610 con 0x5622d5465470 2023-10-10T17:44:38.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6634003bf0 con 0x5622d5465470 2023-10-10T17:44:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f6626ff5700 1 -- 172.21.15.78:0/828655581 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6634010640 con 0x5622d5465470 2023-10-10T17:44:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 10 client.16128 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5622d51e3de0 con 0x5622d5465470 2023-10-10T17:44:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f6626ff5700 1 -- 172.21.15.78:0/828655581 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6634003bf0 con 0x5622d5465470 2023-10-10T17:44:38.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f6626ff5700 1 -- 172.21.15.78:0/828655581 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f66340043c0 con 0x5622d5465470 2023-10-10T17:44:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 10 client.16128 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 20 client.16128 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d54d2e40 0x5622d54d5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:38.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 4 client.16128 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:44:38.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5622d5384660 con 0x5622d54d2e40 2023-10-10T17:44:38.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.581+0000 7f6642d9d700 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d54d2e40 0x5622d54d5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:38.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f6642d9d700 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d54d2e40 0x5622d54d5250 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a59d0283b7aebb94 server_cookie=1bbd817cfe0ac33d in_seq=0 out_seq=0 2023-10-10T17:44:38.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f6626ff5700 10 client.16128 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f6626ff5700 1 -- 172.21.15.78:0/828655581 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5622d5384660 con 0x5622d54d2e40 2023-10-10T17:44:38.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f6626ff5700 10 client.16128 handle_command_reply: tid=0 2023-10-10T17:44:38.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 client.16128 shutdown 2023-10-10T17:44:38.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 msgr2=0x5622d5475110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5475110 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x5622d546f9f0 tx=0x7f6634005ea0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 shutdown_connections 2023-10-10T17:44:38.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5622d5464b60 0x5622d5474bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5622d54d2e40 0x5622d54d5250 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5622d54698b0 0x5622d54d1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f6626ff5700 0 client.16128 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:38.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 --2- 172.21.15.78:0/828655581 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5622d5465470 0x5622d5475110 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 >> 172.21.15.78:0/828655581 conn(0x5622d545a360 msgr2=0x5622d5467970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 shutdown_connections 2023-10-10T17:44:38.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 1 -- 172.21.15.78:0/828655581 wait complete. 2023-10-10T17:44:38.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.582+0000 7f665532bb80 20 client.16128 trim_cache size 0 max 16384 2023-10-10T17:44:38.613 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:44:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f662c0a12c0 msgr2=0x7f662c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f662c0a12c0 0x7f662c0a3770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f66441026c0 tx=0x7f663400a380 comp rx=0 tx=0).stop 2023-10-10T17:44:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 msgr2=0x7f6644100ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:38.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f6644100ed0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f663800b580 tx=0x7f663800f690 comp rx=0 tx=0).stop 2023-10-10T17:44:38.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 shutdown_connections 2023-10-10T17:44:38.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f662c0a12c0 0x7f662c0a3770 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6644104510 0x7f66441a4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f66440ff4d0 0x7f6644100ed0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 --2- 172.21.15.78:0/1774284862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f66440ffe70 0x7f6644101410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:38.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 >> 172.21.15.78:0/1774284862 conn(0x7f6644076070 msgr2=0x7f6644069100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:38.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 shutdown_connections 2023-10-10T17:44:38.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:38.583+0000 7f664919f700 1 -- 172.21.15.78:0/1774284862 wait complete. 2023-10-10T17:44:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:38 smithi078 ceph-mon[99494]: pgmap v66: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 15 op/s 2023-10-10T17:44:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:38 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4176982345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:44:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:38 smithi190 ceph-mon[101220]: pgmap v66: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 15 op/s 2023-10-10T17:44:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:38 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4176982345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:44:38.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:38 smithi157 ceph-mon[100896]: pgmap v66: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 15 op/s 2023-10-10T17:44:38.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:38 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4176982345' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:44:39.618 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:44:39.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:44:39.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 -- 172.21.15.78:0/1889050089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c1005d0 msgr2=0x7fb84c103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1889050089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c1005d0 0x7fb84c103e60 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb83c0099c0 tx=0x7fb83c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 -- 172.21.15.78:0/1889050089 shutdown_connections 2023-10-10T17:44:39.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1889050089 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb84c104590 0x7fb84c10c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1889050089 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb84c0ffc30 0x7fb84c100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1889050089 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c1005d0 0x7fb84c103e60 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 -- 172.21.15.78:0/1889050089 >> 172.21.15.78:0/1889050089 conn(0x7fb84c075470 msgr2=0x7fb84c075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 -- 172.21.15.78:0/1889050089 shutdown_connections 2023-10-10T17:44:39.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 -- 172.21.15.78:0/1889050089 wait complete. 2023-10-10T17:44:39.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.763+0000 7fb851cf2700 1 Processor -- start 2023-10-10T17:44:39.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb851cf2700 1 -- start start 2023-10-10T17:44:39.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb851cf2700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb84c0ffc30 0x7fb84c197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb851cf2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb84c1005d0 0x7fb84c197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb851cf2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c104590 0x7fb84c1a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb851cf2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c1a27f0 con 0x7fb84c104590 2023-10-10T17:44:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb851cf2700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c1a2930 con 0x7fb84c0ffc30 2023-10-10T17:44:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb851cf2700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c1a2a70 con 0x7fb84c1005d0 2023-10-10T17:44:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c104590 0x7fb84c1a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c104590 0x7fb84c1a22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45758/0 (socket says 172.21.15.78:45758) 2023-10-10T17:44:39.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 -- 172.21.15.78:0/1257025202 learned_addr learned my addr 172.21.15.78:0/1257025202 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb84c0ffc30 0x7fb84c197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84affd700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb84c1005d0 0x7fb84c197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 -- 172.21.15.78:0/1257025202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb84c1005d0 msgr2=0x7fb84c197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb84c1005d0 0x7fb84c197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 -- 172.21.15.78:0/1257025202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb84c0ffc30 msgr2=0x7fb84c197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb84c0ffc30 0x7fb84c197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.764+0000 7fb84bfff700 1 -- 172.21.15.78:0/1257025202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0095f0 con 0x7fb84c104590 2023-10-10T17:44:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.765+0000 7fb84bfff700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c104590 0x7fb84c1a22b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb844007ee0 tx=0x7fb84400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.765+0000 7fb848ff9700 1 -- 172.21.15.78:0/1257025202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844017070 con 0x7fb84c104590 2023-10-10T17:44:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.765+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c10f2c0 con 0x7fb84c104590 2023-10-10T17:44:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.765+0000 7fb848ff9700 1 -- 172.21.15.78:0/1257025202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb84400f040 con 0x7fb84c104590 2023-10-10T17:44:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.765+0000 7fb848ff9700 1 -- 172.21.15.78:0/1257025202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844005550 con 0x7fb84c104590 2023-10-10T17:44:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.765+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c10f130 con 0x7fb84c104590 2023-10-10T17:44:39.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.767+0000 7fb848ff9700 1 -- 172.21.15.78:0/1257025202 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fb84400db20 con 0x7fb84c104590 2023-10-10T17:44:39.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.768+0000 7fb848ff9700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8340a12c0 0x7fb8340a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.768+0000 7fb848ff9700 1 -- 172.21.15.78:0/1257025202 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fb8440cc6b0 con 0x7fb84c104590 2023-10-10T17:44:39.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.768+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8340a12c0 0x7fb8340a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.768+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8340a12c0 0x7fb8340a3770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb840005950 tx=0x7fb84000b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 Processor -- start 2023-10-10T17:44:39.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 -- start start 2023-10-10T17:44:39.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 0x56466f65d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f65d900 0x56466f661800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f661d40 0x56466f662190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f55e0e0 con 0x56466f661d40 2023-10-10T17:44:39.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f5b0920 con 0x56466f65cff0 2023-10-10T17:44:39.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f5a9db0 con 0x56466f65d900 2023-10-10T17:44:39.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f661d40 0x56466f662190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f661d40 0x56466f662190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45766/0 (socket says 172.21.15.78:45766) 2023-10-10T17:44:39.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 -- 172.21.15.78:0/2978752079 learned_addr learned my addr 172.21.15.78:0/2978752079 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84affd700 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f65d900 0x56466f661800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 0x56466f65d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 -- 172.21.15.78:0/2978752079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f65d900 msgr2=0x56466f661800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f65d900 0x56466f661800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 -- 172.21.15.78:0/2978752079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 msgr2=0x56466f65d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 0x56466f65d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 -- 172.21.15.78:0/2978752079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56466f42b6b0 con 0x56466f661d40 2023-10-10T17:44:39.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84affd700 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f65d900 0x56466f661800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:39.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.769+0000 7fb84bfff700 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f661d40 0x56466f662190 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb8440a1370 tx=0x7fb844004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6054543cffa998e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb831ffb700 1 -- 172.21.15.78:0/2978752079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8440a18a0 con 0x56466f661d40 2023-10-10T17:44:39.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb831ffb700 1 -- 172.21.15.78:0/2978752079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8440840c0 con 0x56466f661d40 2023-10-10T17:44:39.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb831ffb700 1 -- 172.21.15.78:0/2978752079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844095b60 con 0x56466f661d40 2023-10-10T17:44:39.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2978752079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f661d40 msgr2=0x56466f662190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f661d40 0x56466f662190 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb8440a1370 tx=0x7fb844004f10 comp rx=0 tx=0).stop 2023-10-10T17:44:39.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2978752079 shutdown_connections 2023-10-10T17:44:39.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f661d40 0x56466f662190 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 0x56466f65d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2978752079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f65d900 0x56466f661800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2978752079 >> 172.21.15.78:0/2978752079 conn(0x56466f64e3e0 msgr2=0x56466f653bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2978752079 shutdown_connections 2023-10-10T17:44:39.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.770+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2978752079 wait complete. 2023-10-10T17:44:39.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.771+0000 7fb85de7eb80 1 -- start start 2023-10-10T17:44:39.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.771+0000 7fb85de7eb80 1 Processor -- start 2023-10-10T17:44:39.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.771+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f669550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.771+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f6667d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f666ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f54b960 con 0x56466f65cff0 2023-10-10T17:44:39.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f54b600 con 0x56466f65d900 2023-10-10T17:44:39.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f42d9a0 con 0x56466f661d40 2023-10-10T17:44:39.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 10 client.0 _ll_register_callbacks cb 0x56466f569800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:39.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f669550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f669550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45768/0 (socket says 172.21.15.78:45768) 2023-10-10T17:44:39.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2844677543 learned_addr learned my addr 172.21.15.78:0/2844677543 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84affd700 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f6667d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84bfff700 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f666ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2844677543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 msgr2=0x56466f666ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f666ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 msgr2=0x56466f6667d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f6667d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2844677543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c10f130 con 0x56466f65cff0 2023-10-10T17:44:39.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f669550 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb8400116c0 tx=0x7fb8400047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743f75c0139b08fc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb82affd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:39.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb82affd700 1 -- 172.21.15.78:0/2844677543 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84001e830 con 0x56466f65cff0 2023-10-10T17:44:39.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb82affd700 1 -- 172.21.15.78:0/2844677543 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fb840017070 con 0x56466f65cff0 2023-10-10T17:44:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56466f64bba0 con 0x56466f65cff0 2023-10-10T17:44:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb82affd700 1 -- 172.21.15.78:0/2844677543 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb840011e90 con 0x56466f65cff0 2023-10-10T17:44:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 10 client.16140 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.772+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56466f570210 con 0x56466f65cff0 2023-10-10T17:44:39.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb82affd700 1 -- 172.21.15.78:0/2844677543 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56466f64bba0 con 0x56466f65cff0 2023-10-10T17:44:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb82affd700 1 -- 172.21.15.78:0/2844677543 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb840027a70 con 0x56466f65cff0 2023-10-10T17:44:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb85de7eb80 10 client.16140 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb85de7eb80 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb85de7eb80 20 client.16140 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f676e10 0x56466f679220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb85de7eb80 4 client.16140 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56466f56c120 con 0x56466f676e10 2023-10-10T17:44:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.773+0000 7fb84affd700 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f676e10 0x56466f679220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.774+0000 7fb82affd700 10 client.16140 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:39.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.774+0000 7fb84affd700 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f676e10 0x56466f679220 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a4c6f42b707ceff server_cookie=cc4f84de3ffd643b in_seq=0 out_seq=0 2023-10-10T17:44:39.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.774+0000 7fb82affd700 1 -- 172.21.15.78:0/2844677543 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56466f56c120 con 0x56466f676e10 2023-10-10T17:44:39.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.774+0000 7fb82affd700 10 client.16140 handle_command_reply: tid=0 2023-10-10T17:44:39.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.774+0000 7fb85de7eb80 1 client.16140 shutdown 2023-10-10T17:44:39.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.774+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 msgr2=0x56466f669550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.774+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f669550 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb8400116c0 tx=0x7fb8400047c0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 shutdown_connections 2023-10-10T17:44:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f666ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f676e10 0x56466f679220 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f669550 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2844677543 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f6667d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb82affd700 0 client.16140 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 >> 172.21.15.78:0/2844677543 conn(0x56466f64e3e0 msgr2=0x56466f6631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 shutdown_connections 2023-10-10T17:44:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2844677543 wait complete. 2023-10-10T17:44:39.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.775+0000 7fb85de7eb80 20 client.16140 trim_cache size 0 max 16384 2023-10-10T17:44:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 Processor -- start 2023-10-10T17:44:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 -- start start 2023-10-10T17:44:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 0x56466f653d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65d900 0x56466f661590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f65e050 con 0x56466f65d900 2023-10-10T17:44:39.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f676bf0 con 0x56466f65cff0 2023-10-10T17:44:39.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f64df60 con 0x56466f661d40 2023-10-10T17:44:39.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65d900 0x56466f661590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65d900 0x56466f661590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45778/0 (socket says 172.21.15.78:45778) 2023-10-10T17:44:39.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 -- 172.21.15.78:0/1854217411 learned_addr learned my addr 172.21.15.78:0/1854217411 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84bfff700 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 -- 172.21.15.78:0/1854217411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 msgr2=0x56466f66de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 -- 172.21.15.78:0/1854217411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 msgr2=0x56466f653d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 0x56466f653d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84affd700 1 -- 172.21.15.78:0/1854217411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56466f56ff60 con 0x56466f65d900 2023-10-10T17:44:39.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.792+0000 7fb84bfff700 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:39.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb84affd700 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65d900 0x56466f661590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb84c0768b0 tx=0x7fb83c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25cd889a1a50f497 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb8327fc700 1 -- 172.21.15.78:0/1854217411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c01d040 con 0x56466f65d900 2023-10-10T17:44:39.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb8327fc700 1 -- 172.21.15.78:0/1854217411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb83c01e040 con 0x56466f65d900 2023-10-10T17:44:39.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb8327fc700 1 -- 172.21.15.78:0/1854217411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c00baa0 con 0x56466f65d900 2023-10-10T17:44:39.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 -- 172.21.15.78:0/1854217411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65d900 msgr2=0x56466f661590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65d900 0x56466f661590 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb84c0768b0 tx=0x7fb83c000bc0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 -- 172.21.15.78:0/1854217411 shutdown_connections 2023-10-10T17:44:39.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65cff0 0x56466f653d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/1854217411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65d900 0x56466f661590 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 -- 172.21.15.78:0/1854217411 >> 172.21.15.78:0/1854217411 conn(0x56466f64e3e0 msgr2=0x56466f6579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.793+0000 7fb85de7eb80 1 -- 172.21.15.78:0/1854217411 shutdown_connections 2023-10-10T17:44:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 -- 172.21.15.78:0/1854217411 wait complete. 2023-10-10T17:44:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 -- start start 2023-10-10T17:44:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 Processor -- start 2023-10-10T17:44:39.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f66e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f66be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f66ca90 con 0x56466f65cff0 2023-10-10T17:44:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f66cc00 con 0x56466f65d900 2023-10-10T17:44:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466f6cbce0 con 0x56466f661d40 2023-10-10T17:44:39.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb85de7eb80 10 client.0 _ll_register_callbacks cb 0x7fb840010db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:39.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb84b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f66e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb84b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f66e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45782/0 (socket says 172.21.15.78:45782) 2023-10-10T17:44:39.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.794+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2872429544 learned_addr learned my addr 172.21.15.78:0/2872429544 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84bfff700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84affd700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f66be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2872429544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 msgr2=0x56466f66c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 msgr2=0x56466f66be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f66be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84b7fe700 1 -- 172.21.15.78:0/2872429544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c10c950 con 0x56466f65cff0 2023-10-10T17:44:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84affd700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f66be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb84b7fe700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f66e030 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb840025390 tx=0x7fb84000bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf01d32f459bbf0f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb82affd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb82affd700 1 -- 172.21.15.78:0/2872429544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb840005070 con 0x56466f65cff0 2023-10-10T17:44:39.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb82affd700 1 -- 172.21.15.78:0/2872429544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fb840017070 con 0x56466f65cff0 2023-10-10T17:44:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8440840c0 con 0x56466f65cff0 2023-10-10T17:44:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb82affd700 1 -- 172.21.15.78:0/2872429544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8400051d0 con 0x56466f65cff0 2023-10-10T17:44:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb85de7eb80 10 client.16146 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x56466f3c0ce0 con 0x56466f65cff0 2023-10-10T17:44:39.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.795+0000 7fb82affd700 1 -- 172.21.15.78:0/2872429544 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8440840c0 con 0x56466f65cff0 2023-10-10T17:44:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb82affd700 1 -- 172.21.15.78:0/2872429544 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fb8400046d0 con 0x56466f65cff0 2023-10-10T17:44:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb85de7eb80 10 client.16146 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb85de7eb80 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb85de7eb80 20 client.16146 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:39.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f6cd340 0x56466f6cf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb85de7eb80 4 client.16146 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:44:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56466f5786e0 con 0x56466f6cd340 2023-10-10T17:44:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb84affd700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f6cd340 0x56466f6cf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb84affd700 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f6cd340 0x56466f6cf750 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=465e54ba9ca13c30 server_cookie=c2780c12a4e44cd0 in_seq=0 out_seq=0 2023-10-10T17:44:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.796+0000 7fb82affd700 10 client.16146 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:39.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb82affd700 1 -- 172.21.15.78:0/2872429544 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56466f5786e0 con 0x56466f6cd340 2023-10-10T17:44:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb82affd700 10 client.16146 handle_command_reply: tid=0 2023-10-10T17:44:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb85de7eb80 1 client.16146 shutdown 2023-10-10T17:44:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 msgr2=0x56466f66e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f66e030 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb840025390 tx=0x7fb84000bb20 comp rx=0 tx=0).stop 2023-10-10T17:44:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 shutdown_connections 2023-10-10T17:44:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56466f661d40 0x56466f66c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.797+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56466f6cd340 0x56466f6cf750 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56466f65cff0 0x56466f66e030 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb85de7eb80 1 --2- 172.21.15.78:0/2872429544 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56466f65d900 0x56466f66be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb82affd700 0 client.16146 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 >> 172.21.15.78:0/2872429544 conn(0x56466f64e3e0 msgr2=0x56466f661400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 shutdown_connections 2023-10-10T17:44:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb85de7eb80 1 -- 172.21.15.78:0/2872429544 wait complete. 2023-10-10T17:44:39.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb85de7eb80 20 client.16146 trim_cache size 0 max 16384 2023-10-10T17:44:39.821 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:44:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:44:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "5988305e-c7b4-4847-bed6-b60443d09eb5", 2023-10-10T17:44:39.821 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:44:39.822 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:44:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8340a12c0 msgr2=0x7fb8340a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8340a12c0 0x7fb8340a3770 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb840005950 tx=0x7fb84000b410 comp rx=0 tx=0).stop 2023-10-10T17:44:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c104590 msgr2=0x7fb84c1a22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c104590 0x7fb84c1a22b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb844007ee0 tx=0x7fb84400d3b0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 shutdown_connections 2023-10-10T17:44:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8340a12c0 0x7fb8340a3770 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb84c104590 0x7fb84c1a22b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb84c0ffc30 0x7fb84c197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 --2- 172.21.15.78:0/1257025202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb84c1005d0 0x7fb84c197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 >> 172.21.15.78:0/1257025202 conn(0x7fb84c075470 msgr2=0x7fb84c0fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 shutdown_connections 2023-10-10T17:44:39.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.798+0000 7fb851cf2700 1 -- 172.21.15.78:0/1257025202 wait complete. 2023-10-10T17:44:39.825 INFO:tasks.workunit.client.0.smithi078.stderr:+ set -e 2023-10-10T17:44:39.826 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir -p testdir 2023-10-10T17:44:39.826 INFO:tasks.workunit.client.0.smithi078.stderr:+ file=test_chmod.155860 2023-10-10T17:44:39.826 INFO:tasks.workunit.client.0.smithi078.stderr:+ echo foo 2023-10-10T17:44:39.827 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:44:39.828 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo chmod 600 testdir 2023-10-10T17:44:39.828 INFO:tasks.workunit.client.0.smithi078.stderr:+ expect_failure cat testdir/test_chmod.155860 2023-10-10T17:44:39.829 INFO:tasks.workunit.client.0.smithi078.stderr:+ cat testdir/test_chmod.155860 2023-10-10T17:44:39.838 INFO:tasks.workunit.client.0.smithi078.stderr:cat: testdir/test_chmod.155860: Permission denied 2023-10-10T17:44:39.838 INFO:tasks.workunit.client.0.smithi078.stderr:+ return 0 2023-10-10T17:44:39.838 INFO:tasks.workunit.client.0.smithi078.stderr:+ sudo cat testdir/test_chmod.155860 2023-10-10T17:44:39.847 INFO:tasks.workunit.client.0.smithi078.stdout:foo 2023-10-10T17:44:39.849 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T17:44:39.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T17:44:39.937 INFO:tasks.workunit:Running workunit fs/misc/direct_io.py... 2023-10-10T17:44:39.938 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/direct_io.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/direct_io.py 2023-10-10T17:44:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 -- 172.21.15.78:0/4200584761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0106540 msgr2=0x7f49d0106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/4200584761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0106540 0x7f49d0106910 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009990 tx=0x7f49c8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 -- 172.21.15.78:0/4200584761 shutdown_connections 2023-10-10T17:44:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/4200584761 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d0069b30 0x7f49d010d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/4200584761 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0106540 0x7f49d0106910 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/4200584761 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d0069180 0x7f49d00695f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 -- 172.21.15.78:0/4200584761 >> 172.21.15.78:0/4200584761 conn(0x7f49d00763a0 msgr2=0x7f49d00767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 -- 172.21.15.78:0/4200584761 shutdown_connections 2023-10-10T17:44:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.958+0000 7f49d4b9c700 1 -- 172.21.15.78:0/4200584761 wait complete. 2023-10-10T17:44:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 Processor -- start 2023-10-10T17:44:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 -- start start 2023-10-10T17:44:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0069180 0x7f49d01965f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d0069b30 0x7f49d0196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d0106540 0x7f49d019af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d019b5d0 con 0x7f49d0069180 2023-10-10T17:44:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d019b740 con 0x7f49d0069b30 2023-10-10T17:44:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49d4b9c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d019b900 con 0x7f49d0106540 2023-10-10T17:44:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49ce59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0069180 0x7f49d01965f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49ce59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0069180 0x7f49d01965f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45796/0 (socket says 172.21.15.78:45796) 2023-10-10T17:44:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49ce59c700 1 -- 172.21.15.78:0/2223344354 learned_addr learned my addr 172.21.15.78:0/2223344354 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49ced9d700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d0106540 0x7f49d019af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.959+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d0069b30 0x7f49d0196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49ce59c700 1 -- 172.21.15.78:0/2223344354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d0106540 msgr2=0x7f49d019af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49ce59c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d0106540 0x7f49d019af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49ce59c700 1 -- 172.21.15.78:0/2223344354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d0069b30 msgr2=0x7f49d0196b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49ce59c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d0069b30 0x7f49d0196b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49ce59c700 1 -- 172.21.15.78:0/2223344354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80095f0 con 0x7f49d0069180 2023-10-10T17:44:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d0069b30 0x7f49d0196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:44:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49ce59c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0069180 0x7f49d01965f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f49c80095c0 tx=0x7f49c800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49bf7fe700 1 -- 172.21.15.78:0/2223344354 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c801d070 con 0x7f49d0069180 2023-10-10T17:44:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49bf7fe700 1 -- 172.21.15.78:0/2223344354 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49c800fc00 con 0x7f49d0069180 2023-10-10T17:44:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d0110540 con 0x7f49d0069180 2023-10-10T17:44:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49bf7fe700 1 -- 172.21.15.78:0/2223344354 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c80173f0 con 0x7f49d0069180 2023-10-10T17:44:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.960+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d01103b0 con 0x7f49d0069180 2023-10-10T17:44:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.962+0000 7f49bf7fe700 1 -- 172.21.15.78:0/2223344354 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f49c8022470 con 0x7f49d0069180 2023-10-10T17:44:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.963+0000 7f49bf7fe700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f49b80a12c0 0x7f49b80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.963+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f49b80a12c0 0x7f49b80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.963+0000 7f49bf7fe700 1 -- 172.21.15.78:0/2223344354 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f49c80ce280 con 0x7f49d0069180 2023-10-10T17:44:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.963+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f49b80a12c0 0x7f49b80a3770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49c4005fd0 tx=0x7f49c4005ee0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 Processor -- start 2023-10-10T17:44:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 -- start start 2023-10-10T17:44:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359cef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359cf470 0x556b359d3370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359d3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b358d4060 con 0x556b359cf470 2023-10-10T17:44:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b359268a0 con 0x556b359ceb60 2023-10-10T17:44:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b3591fd30 con 0x556b359d38b0 2023-10-10T17:44:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359cf470 0x556b359d3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359cf470 0x556b359d3370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45806/0 (socket says 172.21.15.78:45806) 2023-10-10T17:44:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 -- 172.21.15.78:0/2190503773 learned_addr learned my addr 172.21.15.78:0/2190503773 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49ced9d700 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359d3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49ce59c700 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359cef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 -- 172.21.15.78:0/2190503773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 msgr2=0x556b359d3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359d3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 -- 172.21.15.78:0/2190503773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 msgr2=0x556b359cef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359cef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49cdd9b700 1 -- 172.21.15.78:0/2190503773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b357a1630 con 0x556b359cf470 2023-10-10T17:44:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.964+0000 7f49ce59c700 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.965+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359cf470 0x556b359d3370 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f49c400eda0 tx=0x7f49c4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a99daaa4b82668 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.965+0000 7f49bcff9700 1 -- 172.21.15.78:0/2190503773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c4022070 con 0x556b359cf470 2023-10-10T17:44:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.965+0000 7f49bcff9700 1 -- 172.21.15.78:0/2190503773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49c400edd0 con 0x556b359cf470 2023-10-10T17:44:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.965+0000 7f49bcff9700 1 -- 172.21.15.78:0/2190503773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c40053a0 con 0x556b359cf470 2023-10-10T17:44:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.965+0000 7f49e0d28b80 1 -- 172.21.15.78:0/2190503773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359cf470 msgr2=0x556b359d3370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.965+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359cf470 0x556b359d3370 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f49c400eda0 tx=0x7f49c4009f90 comp rx=0 tx=0).stop 2023-10-10T17:44:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- 172.21.15.78:0/2190503773 shutdown_connections 2023-10-10T17:44:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359d3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359cef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/2190503773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359cf470 0x556b359d3370 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- 172.21.15.78:0/2190503773 >> 172.21.15.78:0/2190503773 conn(0x556b359c4360 msgr2=0x556b359c9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- 172.21.15.78:0/2190503773 shutdown_connections 2023-10-10T17:44:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- 172.21.15.78:0/2190503773 wait complete. 2023-10-10T17:44:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- start start 2023-10-10T17:44:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 Processor -- start 2023-10-10T17:44:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359db040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359d8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359d8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b358c18e0 con 0x556b359d38b0 2023-10-10T17:44:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b358c1580 con 0x556b359ceb60 2023-10-10T17:44:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b357a3920 con 0x556b359cf470 2023-10-10T17:44:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49e0d28b80 10 client.0 _ll_register_callbacks cb 0x556b358df780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49ced9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359d8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49ced9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359d8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45814/0 (socket says 172.21.15.78:45814) 2023-10-10T17:44:39.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49ced9d700 1 -- 172.21.15.78:0/3984532188 learned_addr learned my addr 172.21.15.78:0/3984532188 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.966+0000 7f49ce59c700 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359db040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359d8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49ced9d700 1 -- 172.21.15.78:0/3984532188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 msgr2=0x556b359d8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49ced9d700 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359d8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49ced9d700 1 -- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 msgr2=0x556b359db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49ced9d700 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359db040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49ced9d700 1 -- 172.21.15.78:0/3984532188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c400ea80 con 0x556b359d38b0 2023-10-10T17:44:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49ced9d700 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359d8860 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f49c0007b20 tx=0x7f49c0007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334ce578ada7b640 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3984532188 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0009940 con 0x556b359d38b0 2023-10-10T17:44:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b359c1b20 con 0x556b359d38b0 2023-10-10T17:44:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3984532188 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f49c0012b40 con 0x556b359d38b0 2023-10-10T17:44:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3984532188 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49c0012d00 con 0x556b359d38b0 2023-10-10T17:44:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49e0d28b80 10 client.16158 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:39.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556b358e6190 con 0x556b359d38b0 2023-10-10T17:44:39.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.967+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3984532188 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556b359c1b20 con 0x556b359d38b0 2023-10-10T17:44:39.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3984532188 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f49c0014040 con 0x556b359d38b0 2023-10-10T17:44:39.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49e0d28b80 10 client.16158 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:39.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49e0d28b80 10 client.16158 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49e0d28b80 20 client.16158 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b359e8830 0x556b359eac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49e0d28b80 4 client.16158 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b358e20a0 con 0x556b359e8830 2023-10-10T17:44:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49ce59c700 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b359e8830 0x556b359eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49b5ffb700 10 client.16158 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.968+0000 7f49ce59c700 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b359e8830 0x556b359eac40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d68aa806f8139394 server_cookie=c79fa133e3a59249 in_seq=0 out_seq=0 2023-10-10T17:44:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3984532188 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556b358e20a0 con 0x556b359e8830 2023-10-10T17:44:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49b5ffb700 10 client.16158 handle_command_reply: tid=0 2023-10-10T17:44:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 client.16158 shutdown 2023-10-10T17:44:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 msgr2=0x556b359d8860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359d8860 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f49c0007b20 tx=0x7f49c0007e30 comp rx=0 tx=0).stop 2023-10-10T17:44:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 shutdown_connections 2023-10-10T17:44:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b359e8830 0x556b359eac40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359d8860 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49b5ffb700 0 client.16158 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:39.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359db040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3984532188 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359d8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:39.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 >> 172.21.15.78:0/3984532188 conn(0x556b359c4360 msgr2=0x556b359c8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:39.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 shutdown_connections 2023-10-10T17:44:39.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3984532188 wait complete. 2023-10-10T17:44:39.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.969+0000 7f49e0d28b80 20 client.16158 trim_cache size 0 max 16384 2023-10-10T17:44:39.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 Processor -- start 2023-10-10T17:44:39.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 -- start start 2023-10-10T17:44:39.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359ceb60 0x556b359dab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359cf470 0x556b359d1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359c9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:39.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b359d67d0 con 0x556b359ceb60 2023-10-10T17:44:39.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b359d23c0 con 0x556b359cf470 2023-10-10T17:44:39.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b359ca780 con 0x556b359d38b0 2023-10-10T17:44:39.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359ceb60 0x556b359dab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:39.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359ceb60 0x556b359dab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45818/0 (socket says 172.21.15.78:45818) 2023-10-10T17:44:39.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 -- 172.21.15.78:0/807756390 learned_addr learned my addr 172.21.15.78:0/807756390 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:39.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ced9d700 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:40.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 -- 172.21.15.78:0/807756390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 msgr2=0x556b359c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:40.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359c9560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 -- 172.21.15.78:0/807756390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359cf470 msgr2=0x556b359d1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:40.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359cf470 0x556b359d1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:40.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359cf470 0x556b359d1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 -- 172.21.15.78:0/807756390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b358e5ee0 con 0x556b359ceb60 2023-10-10T17:44:40.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.995+0000 7f49ce59c700 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359ceb60 0x556b359dab70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f49c80975b0 tx=0x7f49c800b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29dc35a049190e7c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:40.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49bd7fa700 1 -- 172.21.15.78:0/807756390 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c80b7070 con 0x556b359ceb60 2023-10-10T17:44:40.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49bd7fa700 1 -- 172.21.15.78:0/807756390 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49c80b2070 con 0x556b359ceb60 2023-10-10T17:44:40.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49bd7fa700 1 -- 172.21.15.78:0/807756390 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c8099540 con 0x556b359ceb60 2023-10-10T17:44:40.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 -- 172.21.15.78:0/807756390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359ceb60 msgr2=0x556b359dab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:40.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359ceb60 0x556b359dab70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f49c80975b0 tx=0x7f49c800b540 comp rx=0 tx=0).stop 2023-10-10T17:44:40.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 -- 172.21.15.78:0/807756390 shutdown_connections 2023-10-10T17:44:40.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359d38b0 0x556b359c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359ceb60 0x556b359dab70 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/807756390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359cf470 0x556b359d1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 -- 172.21.15.78:0/807756390 >> 172.21.15.78:0/807756390 conn(0x556b359c4360 msgr2=0x556b359e4150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:40.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 -- 172.21.15.78:0/807756390 shutdown_connections 2023-10-10T17:44:40.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.996+0000 7f49e0d28b80 1 -- 172.21.15.78:0/807756390 wait complete. 2023-10-10T17:44:40.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 -- start start 2023-10-10T17:44:40.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 Processor -- start 2023-10-10T17:44:40.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359df340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:40.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359df880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:40.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359dd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:40.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b359ddf50 con 0x556b359d38b0 2023-10-10T17:44:40.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b359de0c0 con 0x556b359ceb60 2023-10-10T17:44:40.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b359de280 con 0x556b359cf470 2023-10-10T17:44:40.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49e0d28b80 10 client.0 _ll_register_callbacks cb 0x556b358df780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:40.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359dd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:40.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359dd990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45828/0 (socket says 172.21.15.78:45828) 2023-10-10T17:44:40.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 -- 172.21.15.78:0/3217312184 learned_addr learned my addr 172.21.15.78:0/3217312184 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:40.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49cdd9b700 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359df880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:40.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 -- 172.21.15.78:0/3217312184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 msgr2=0x556b359df880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:40.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359df880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 -- 172.21.15.78:0/3217312184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 msgr2=0x556b359df340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:40.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359df340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.997+0000 7f49ced9d700 1 -- 172.21.15.78:0/3217312184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d01103b0 con 0x556b359d38b0 2023-10-10T17:44:40.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49ced9d700 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359dd990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f49c0015540 tx=0x7f49c0015dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45914f65f7a37371 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:40.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49b5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:40.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3217312184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0015390 con 0x556b359d38b0 2023-10-10T17:44:40.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3217312184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f49c0006040 con 0x556b359d38b0 2023-10-10T17:44:40.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49c400edd0 con 0x556b359d38b0 2023-10-10T17:44:40.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3217312184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49c0004e80 con 0x556b359d38b0 2023-10-10T17:44:40.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 10 client.16164 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:40.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556b35736c60 con 0x556b359d38b0 2023-10-10T17:44:40.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3217312184 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49c400edd0 con 0x556b359d38b0 2023-10-10T17:44:40.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3217312184 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f49c0007b50 con 0x556b359d38b0 2023-10-10T17:44:40.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 10 client.16164 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 20 client.16164 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b35a3d720 0x556b35a3fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 4 client.16164 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:44:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.998+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556b358ee660 con 0x556b35a3d720 2023-10-10T17:44:40.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49ce59c700 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b35a3d720 0x556b35a3fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49b5ffb700 10 client.16164 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49ce59c700 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b35a3d720 0x556b35a3fb30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca479189182c8c94 server_cookie=2d3722054859161f in_seq=0 out_seq=0 2023-10-10T17:44:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49b5ffb700 1 -- 172.21.15.78:0/3217312184 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556b358ee660 con 0x556b35a3d720 2023-10-10T17:44:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49b5ffb700 10 client.16164 handle_command_reply: tid=0 2023-10-10T17:44:40.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 client.16164 shutdown 2023-10-10T17:44:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 msgr2=0x556b359dd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359dd990 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f49c0015540 tx=0x7f49c0015dd0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 shutdown_connections 2023-10-10T17:44:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556b35a3d720 0x556b35a3fb30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556b359d38b0 0x556b359dd990 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49b5ffb700 0 client.16164 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:40.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556b359ceb60 0x556b359df340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 --2- 172.21.15.78:0/3217312184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556b359cf470 0x556b359df880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 >> 172.21.15.78:0/3217312184 conn(0x556b359c4360 msgr2=0x556b359d1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:40.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 shutdown_connections 2023-10-10T17:44:40.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:39.999+0000 7f49e0d28b80 1 -- 172.21.15.78:0/3217312184 wait complete. 2023-10-10T17:44:40.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49e0d28b80 20 client.16164 trim_cache size 0 max 16384 2023-10-10T17:44:40.017 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:44:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:44:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:44:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "5988305e-c7b4-4847-bed6-b60443d09eb5": { 2023-10-10T17:44:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:44:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5988305e-c7b4-4847-bed6-b60443d09eb5", 2023-10-10T17:44:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:44:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:44:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:44:40.019 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:44:40.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f49b80a12c0 msgr2=0x7f49b80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:40.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f49b80a12c0 0x7f49b80a3770 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f49c4005fd0 tx=0x7f49c4005ee0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0069180 msgr2=0x7f49d01965f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:40.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0069180 0x7f49d01965f0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f49c80095c0 tx=0x7f49c800f710 comp rx=0 tx=0).stop 2023-10-10T17:44:40.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 shutdown_connections 2023-10-10T17:44:40.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f49d0106540 0x7f49d019af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f49b80a12c0 0x7f49b80a3770 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49d0069180 0x7f49d01965f0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 --2- 172.21.15.78:0/2223344354 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f49d0069b30 0x7f49d0196b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:40.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 >> 172.21.15.78:0/2223344354 conn(0x7f49d00763a0 msgr2=0x7f49d00fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:40.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 shutdown_connections 2023-10-10T17:44:40.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:44:40.000+0000 7f49d4b9c700 1 -- 172.21.15.78:0/2223344354 wait complete. 2023-10-10T17:44:40.023 INFO:tasks.cephfs.filesystem:scrub status for tag:5988305e-c7b4-4847-bed6-b60443d09eb5 - {'path': '/', 'tag': '5988305e-c7b4-4847-bed6-b60443d09eb5', 'options': 'recursive,force'} 2023-10-10T17:44:40.124 INFO:tasks.workunit.client.0.smithi078.stdout:ok 2023-10-10T17:44:40.128 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T17:44:40.129 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T17:44:40.170 INFO:tasks.workunit:Running workunit fs/misc/dirfrag.sh... 2023-10-10T17:44:40.170 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/dirfrag.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/dirfrag.sh 2023-10-10T17:44:40.248 INFO:tasks.workunit.client.0.smithi078.stdout:creating folder hierarchy 2023-10-10T17:44:40.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:40 smithi190 ceph-mon[101220]: pgmap v67: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-10-10T17:44:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:40 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:44:40.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:40 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:44:40.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:40 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:44:40.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:40 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:44:40.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:40 smithi157 ceph-mon[100896]: pgmap v67: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-10-10T17:44:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:40 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:44:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:40 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:44:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:40 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:44:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:40 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:44:41.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:40 smithi078 ceph-mon[99494]: pgmap v67: 129 pgs: 129 active+clean; 2.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-10-10T17:44:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:40 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:44:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:40 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:44:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:40 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:44:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:40 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:44:41.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:41 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:44:41.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:41 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:44:42.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:41 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:44:42.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:42 smithi157 ceph-mon[100896]: pgmap v68: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 183 KiB/s wr, 18 op/s 2023-10-10T17:44:43.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:42 smithi078 ceph-mon[99494]: pgmap v68: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 183 KiB/s wr, 18 op/s 2023-10-10T17:44:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:42 smithi190 ceph-mon[101220]: pgmap v68: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 183 KiB/s wr, 18 op/s 2023-10-10T17:44:44.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:44 smithi157 ceph-mon[100896]: pgmap v69: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 204 KiB/s wr, 24 op/s 2023-10-10T17:44:45.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:44 smithi078 ceph-mon[99494]: pgmap v69: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 204 KiB/s wr, 24 op/s 2023-10-10T17:44:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:44 smithi190 ceph-mon[101220]: pgmap v69: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 204 KiB/s wr, 24 op/s 2023-10-10T17:44:46.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:46 smithi157 ceph-mon[100896]: pgmap v70: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 196 KiB/s wr, 20 op/s 2023-10-10T17:44:47.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:46 smithi078 ceph-mon[99494]: pgmap v70: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 196 KiB/s wr, 20 op/s 2023-10-10T17:44:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:46 smithi190 ceph-mon[101220]: pgmap v70: 129 pgs: 129 active+clean; 4.0 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 196 KiB/s wr, 20 op/s 2023-10-10T17:44:48.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:48 smithi157 ceph-mon[100896]: pgmap v71: 129 pgs: 129 active+clean; 15 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2023-10-10T17:44:49.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:48 smithi078 ceph-mon[99494]: pgmap v71: 129 pgs: 129 active+clean; 15 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2023-10-10T17:44:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:48 smithi190 ceph-mon[101220]: pgmap v71: 129 pgs: 129 active+clean; 15 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 1.4 MiB/s wr, 76 op/s 2023-10-10T17:44:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:50 smithi157 ceph-mon[100896]: pgmap v72: 129 pgs: 129 active+clean; 15 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2023-10-10T17:44:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:50 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:44:51.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:50 smithi078 ceph-mon[99494]: pgmap v72: 129 pgs: 129 active+clean; 15 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2023-10-10T17:44:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:50 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:44:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:50 smithi190 ceph-mon[101220]: pgmap v72: 129 pgs: 129 active+clean; 15 MiB data, 102 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 1.4 MiB/s wr, 73 op/s 2023-10-10T17:44:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:50 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:44:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 -- 172.21.15.78:0/2841849051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 msgr2=0x7fd97c10a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 --2- 172.21.15.78:0/2841849051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c10a2e0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd970009990 tx=0x7fd970009ca0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 -- 172.21.15.78:0/2841849051 shutdown_connections 2023-10-10T17:44:51.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 --2- 172.21.15.78:0/2841849051 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd97c10a820 0x7fd97c10cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 --2- 172.21.15.78:0/2841849051 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd97c101300 0x7fd97c1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 --2- 172.21.15.78:0/2841849051 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c10a2e0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 -- 172.21.15.78:0/2841849051 >> 172.21.15.78:0/2841849051 conn(0x7fd97c0763a0 msgr2=0x7fd97c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 -- 172.21.15.78:0/2841849051 shutdown_connections 2023-10-10T17:44:51.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.250+0000 7fd9810ea700 1 -- 172.21.15.78:0/2841849051 wait complete. 2023-10-10T17:44:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 Processor -- start 2023-10-10T17:44:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 -- start start 2023-10-10T17:44:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd97c101300 0x7fd97c196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c196880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd97c10a820 0x7fd97c19ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c19b320 con 0x7fd97c101c10 2023-10-10T17:44:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c19b490 con 0x7fd97c101300 2023-10-10T17:44:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd9810ea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c19b650 con 0x7fd97c10a820 2023-10-10T17:44:51.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd97a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c196880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd97ad9d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd97c101300 0x7fd97c196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd97a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c196880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45886/0 (socket says 172.21.15.78:45886) 2023-10-10T17:44:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd97a59c700 1 -- 172.21.15.78:0/594579125 learned_addr learned my addr 172.21.15.78:0/594579125 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.251+0000 7fd97b59e700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd97c10a820 0x7fd97c19ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97a59c700 1 -- 172.21.15.78:0/594579125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd97c10a820 msgr2=0x7fd97c19ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97a59c700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd97c10a820 0x7fd97c19ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97a59c700 1 -- 172.21.15.78:0/594579125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd97c101300 msgr2=0x7fd97c196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97a59c700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd97c101300 0x7fd97c196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97a59c700 1 -- 172.21.15.78:0/594579125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700095f0 con 0x7fd97c101c10 2023-10-10T17:44:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97b59e700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd97c10a820 0x7fd97c19ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd97c101300 0x7fd97c196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:44:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd97a59c700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c196880 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd970000c00 tx=0x7fd970004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:51.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd963fff700 1 -- 172.21.15.78:0/594579125 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97001d070 con 0x7fd97c101c10 2023-10-10T17:44:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c1102b0 con 0x7fd97c101c10 2023-10-10T17:44:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd963fff700 1 -- 172.21.15.78:0/594579125 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd970021410 con 0x7fd97c101c10 2023-10-10T17:44:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd963fff700 1 -- 172.21.15.78:0/594579125 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970017650 con 0x7fd97c101c10 2023-10-10T17:44:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.252+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c110120 con 0x7fd97c101c10 2023-10-10T17:44:51.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.254+0000 7fd963fff700 1 -- 172.21.15.78:0/594579125 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fd970021580 con 0x7fd97c101c10 2023-10-10T17:44:51.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.255+0000 7fd963fff700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd9640a12c0 0x7fd9640a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.255+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd9640a12c0 0x7fd9640a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.255+0000 7fd963fff700 1 -- 172.21.15.78:0/594579125 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd9700ce350 con 0x7fd97c101c10 2023-10-10T17:44:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd9640a12c0 0x7fd9640a3770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd96c011490 tx=0x7fd96c009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:51.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 Processor -- start 2023-10-10T17:44:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 -- start start 2023-10-10T17:44:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d1b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d15970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d15ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91c17000 con 0x562a91d1baf0 2023-10-10T17:44:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91c69840 con 0x562a91b6d6b0 2023-10-10T17:44:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd98d276b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91c62cd0 con 0x562a91d1bf60 2023-10-10T17:44:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd97a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d15970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd97ad9d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d1b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd97a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d15970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45894/0 (socket says 172.21.15.78:45894) 2023-10-10T17:44:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd97a59c700 1 -- 172.21.15.78:0/4282573079 learned_addr learned my addr 172.21.15.78:0/4282573079 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.256+0000 7fd97b59e700 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d15ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd97a59c700 1 -- 172.21.15.78:0/4282573079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 msgr2=0x562a91d15ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd97a59c700 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d15ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.270 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd97a59c700 1 -- 172.21.15.78:0/4282573079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 msgr2=0x562a91d1b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd97a59c700 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d1b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd97a59c700 1 -- 172.21.15.78:0/4282573079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a91ae45d0 con 0x562a91d1baf0 2023-10-10T17:44:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd97a59c700 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d15970 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd970097680 tx=0x7fd97009ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754334cd49742c6e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd9617fa700 1 -- 172.21.15.78:0/4282573079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97009f1b0 con 0x562a91d1baf0 2023-10-10T17:44:51.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd9617fa700 1 -- 172.21.15.78:0/4282573079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9700a3540 con 0x562a91d1baf0 2023-10-10T17:44:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.257+0000 7fd9617fa700 1 -- 172.21.15.78:0/4282573079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97009b5e0 con 0x562a91d1baf0 2023-10-10T17:44:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 -- 172.21.15.78:0/4282573079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 msgr2=0x562a91d15970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d15970 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd970097680 tx=0x7fd97009ad40 comp rx=0 tx=0).stop 2023-10-10T17:44:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 -- 172.21.15.78:0/4282573079 shutdown_connections 2023-10-10T17:44:51.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d15ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d1b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 --2- 172.21.15.78:0/4282573079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d15970 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 -- 172.21.15.78:0/4282573079 >> 172.21.15.78:0/4282573079 conn(0x562a91d06f50 msgr2=0x562a91d0c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 -- 172.21.15.78:0/4282573079 shutdown_connections 2023-10-10T17:44:51.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 -- 172.21.15.78:0/4282573079 wait complete. 2023-10-10T17:44:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 -- start start 2023-10-10T17:44:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.258+0000 7fd98d276b80 1 Processor -- start 2023-10-10T17:44:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d24b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d250b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1bf60 0x562a91d1ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91c04880 con 0x562a91d1bf60 2023-10-10T17:44:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91c04520 con 0x562a91b6d6b0 2023-10-10T17:44:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91ae68c0 con 0x562a91d1baf0 2023-10-10T17:44:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 10 client.0 _ll_register_callbacks cb 0x562a91c22720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:51.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1bf60 0x562a91d1ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97ad9d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d24b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1bf60 0x562a91d1ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45902/0 (socket says 172.21.15.78:45902) 2023-10-10T17:44:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 -- 172.21.15.78:0/1868121684 learned_addr learned my addr 172.21.15.78:0/1868121684 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97a59c700 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d250b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 -- 172.21.15.78:0/1868121684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 msgr2=0x562a91d250b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d250b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 -- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 msgr2=0x562a91d24b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d24b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 -- 172.21.15.78:0/1868121684 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c110120 con 0x562a91d1bf60 2023-10-10T17:44:51.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d24b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:44:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd97b59e700 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1bf60 0x562a91d1ee00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd974007bf0 tx=0x7fd974007c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d73aca014ebc81 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd95eff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd95eff5700 1 -- 172.21.15.78:0/1868121684 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97400e070 con 0x562a91d1bf60 2023-10-10T17:44:51.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a91d04710 con 0x562a91d1bf60 2023-10-10T17:44:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd95eff5700 1 -- 172.21.15.78:0/1868121684 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd9740136e0 con 0x562a91d1bf60 2023-10-10T17:44:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.259+0000 7fd95eff5700 1 -- 172.21.15.78:0/1868121684 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9740138e0 con 0x562a91d1bf60 2023-10-10T17:44:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 10 client.16194 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562a91c29130 con 0x562a91d1bf60 2023-10-10T17:44:51.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd95eff5700 1 -- 172.21.15.78:0/1868121684 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562a91d04710 con 0x562a91d1bf60 2023-10-10T17:44:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd95eff5700 1 -- 172.21.15.78:0/1868121684 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd974016af0 con 0x562a91d1bf60 2023-10-10T17:44:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 10 client.16194 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 10 client.16194 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 20 client.16194 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:51.280 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d2f0c0 0x562a91d314d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 4 client.16194 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:44:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a91c25040 con 0x562a91d2f0c0 2023-10-10T17:44:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.260+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d2f0c0 0x562a91d314d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.261+0000 7fd95eff5700 10 client.16194 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:51.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.261+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d2f0c0 0x562a91d314d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64a99210dec459c4 server_cookie=930c1a77b30d2f6b in_seq=0 out_seq=0 2023-10-10T17:44:51.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.261+0000 7fd95eff5700 1 -- 172.21.15.78:0/1868121684 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562a91c25040 con 0x562a91d2f0c0 2023-10-10T17:44:51.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.261+0000 7fd95eff5700 10 client.16194 handle_command_reply: tid=0 2023-10-10T17:44:51.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.261+0000 7fd98d276b80 1 client.16194 shutdown 2023-10-10T17:44:51.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1bf60 msgr2=0x562a91d1ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:51.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1bf60 0x562a91d1ee00 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd974007bf0 tx=0x7fd974007c20 comp rx=0 tx=0).stop 2023-10-10T17:44:51.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 shutdown_connections 2023-10-10T17:44:51.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d2f0c0 0x562a91d314d0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1bf60 0x562a91d1ee00 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d24b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd95eff5700 0 client.16194 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1868121684 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d250b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 >> 172.21.15.78:0/1868121684 conn(0x562a91d06f50 msgr2=0x562a91d0b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:51.284 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 shutdown_connections 2023-10-10T17:44:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 1 -- 172.21.15.78:0/1868121684 wait complete. 2023-10-10T17:44:51.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.262+0000 7fd98d276b80 20 client.16194 trim_cache size 0 max 16384 2023-10-10T17:44:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.276+0000 7fd98d276b80 1 Processor -- start 2023-10-10T17:44:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.276+0000 7fd98d276b80 1 -- start start 2023-10-10T17:44:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91b6d6b0 0x562a91d0bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d18020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91d1bf60 0x562a91d18560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd98d276b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91d16a70 con 0x562a91b6d6b0 2023-10-10T17:44:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd98d276b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91d2c770 con 0x562a91d1bf60 2023-10-10T17:44:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd98d276b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91d0dba0 con 0x562a91d1baf0 2023-10-10T17:44:51.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91b6d6b0 0x562a91d0bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91b6d6b0 0x562a91d0bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45906/0 (socket says 172.21.15.78:45906) 2023-10-10T17:44:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 -- 172.21.15.78:0/772822547 learned_addr learned my addr 172.21.15.78:0/772822547 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97b59e700 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91d1bf60 0x562a91d18560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97a59c700 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d18020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 -- 172.21.15.78:0/772822547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 msgr2=0x562a91d18020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d18020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 -- 172.21.15.78:0/772822547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91d1bf60 msgr2=0x562a91d18560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91d1bf60 0x562a91d18560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 -- 172.21.15.78:0/772822547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a91c28e80 con 0x562a91b6d6b0 2023-10-10T17:44:51.293 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97b59e700 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91d1bf60 0x562a91d18560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:44:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91b6d6b0 0x562a91d0bf60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x562a91d31d70 tx=0x7fd96c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=215ac8cc18ff147b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd961ffb700 1 -- 172.21.15.78:0/772822547 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c024040 con 0x562a91b6d6b0 2023-10-10T17:44:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.277+0000 7fd961ffb700 1 -- 172.21.15.78:0/772822547 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd96c029e10 con 0x562a91b6d6b0 2023-10-10T17:44:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd961ffb700 1 -- 172.21.15.78:0/772822547 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c01eb70 con 0x562a91b6d6b0 2023-10-10T17:44:51.294 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 -- 172.21.15.78:0/772822547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91b6d6b0 msgr2=0x562a91d0bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91b6d6b0 0x562a91d0bf60 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x562a91d31d70 tx=0x7fd96c014000 comp rx=0 tx=0).stop 2023-10-10T17:44:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 -- 172.21.15.78:0/772822547 shutdown_connections 2023-10-10T17:44:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91d1bf60 0x562a91d18560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91b6d6b0 0x562a91d0bf60 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.295 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 --2- 172.21.15.78:0/772822547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1baf0 0x562a91d18020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 -- 172.21.15.78:0/772822547 >> 172.21.15.78:0/772822547 conn(0x562a91d06f50 msgr2=0x562a91d1f790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 -- 172.21.15.78:0/772822547 shutdown_connections 2023-10-10T17:44:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 -- 172.21.15.78:0/772822547 wait complete. 2023-10-10T17:44:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.278+0000 7fd98d276b80 1 -- start start 2023-10-10T17:44:51.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 1 Processor -- start 2023-10-10T17:44:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d22720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d22c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d231a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91d81f40 con 0x562a91d1baf0 2023-10-10T17:44:51.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91d82080 con 0x562a91b6d6b0 2023-10-10T17:44:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a91d821c0 con 0x562a91d1bf60 2023-10-10T17:44:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd98d276b80 10 client.0 _ll_register_callbacks cb 0x7fd974007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:44:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d22c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d22c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45914/0 (socket says 172.21.15.78:45914) 2023-10-10T17:44:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 -- 172.21.15.78:0/1771831447 learned_addr learned my addr 172.21.15.78:0/1771831447 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:44:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d22720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97b59e700 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d231a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 -- 172.21.15.78:0/1771831447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 msgr2=0x562a91d231a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d231a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 -- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 msgr2=0x562a91d22720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:44:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d22720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 -- 172.21.15.78:0/1771831447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96c017070 con 0x562a91d1baf0 2023-10-10T17:44:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd97a59c700 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d22c60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd970097bf0 tx=0x7fd970005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9860f2c9a845f43d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:44:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.279+0000 7fd95eff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:44:51.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd95eff5700 1 -- 172.21.15.78:0/1771831447 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970021a60 con 0x562a91d1baf0 2023-10-10T17:44:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd95eff5700 1 -- 172.21.15.78:0/1771831447 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd97009f1b0 con 0x562a91d1baf0 2023-10-10T17:44:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9700a3540 con 0x562a91d1baf0 2023-10-10T17:44:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd95eff5700 1 -- 172.21.15.78:0/1771831447 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd97009a670 con 0x562a91d1baf0 2023-10-10T17:44:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 10 client.16200 fetch_fsmap learned FSMap version 24 2023-10-10T17:44:51.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x562a91a79c00 con 0x562a91d1baf0 2023-10-10T17:44:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd95eff5700 1 -- 172.21.15.78:0/1771831447 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9700a3540 con 0x562a91d1baf0 2023-10-10T17:44:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd95eff5700 1 -- 172.21.15.78:0/1771831447 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 24) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd9700b9250 con 0x562a91d1baf0 2023-10-10T17:44:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 10 client.16200 fetch_fsmap finished waiting for FSMap version 24 2023-10-10T17:44:51.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:44:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 20 client.16200 populate_metadata read hostname 'smithi078' 2023-10-10T17:44:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d83480 0x562a91d85890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:44:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 4 client.16200 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:44:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562a91c31600 con 0x562a91d83480 2023-10-10T17:44:51.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.280+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d83480 0x562a91d85890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:44:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.281+0000 7fd95eff5700 10 client.16200 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:51.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:51.281+0000 7fd97ad9d700 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d83480 0x562a91d85890 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6071758deff50ea server_cookie=44162bfd06ec7a92 in_seq=0 out_seq=0 2023-10-10T17:44:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:52 smithi157 ceph-mon[100896]: pgmap v73: 129 pgs: 129 active+clean; 25 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 90 op/s 2023-10-10T17:44:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:52 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:44:52.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:52 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:52.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:52 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:52 smithi078 ceph-mon[99494]: pgmap v73: 129 pgs: 129 active+clean; 25 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 90 op/s 2023-10-10T17:44:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:52 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:44:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:52 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:53.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:52 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:53.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:52 smithi190 ceph-mon[101220]: pgmap v73: 129 pgs: 129 active+clean; 25 MiB data, 113 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 90 op/s 2023-10-10T17:44:53.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:52 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:44:53.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:52 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:53.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:52 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:53.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:53 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:44:53.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:53 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:44:53.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:53 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:54.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:53 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:44:54.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:53 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:44:54.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:53 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:54.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:53 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:44:54.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:53 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:44:54.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:53 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:44:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.933+0000 7fd95eff5700 1 -- 172.21.15.78:0/1771831447 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562a91c31600 con 0x562a91d83480 2023-10-10T17:44:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.933+0000 7fd95eff5700 10 client.16200 handle_command_reply: tid=0 2023-10-10T17:44:54.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.933+0000 7fd98d276b80 1 client.16200 shutdown 2023-10-10T17:44:54.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.933+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 msgr2=0x562a91d22c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d22c60 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd970097bf0 tx=0x7fd970005ef0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 shutdown_connections 2023-10-10T17:44:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562a91d1bf60 0x562a91d231a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562a91d83480 0x562a91d85890 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562a91b6d6b0 0x562a91d22720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 --2- 172.21.15.78:0/1771831447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562a91d1baf0 0x562a91d22c60 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 >> 172.21.15.78:0/1771831447 conn(0x562a91d06f50 msgr2=0x562a91d1c330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd95eff5700 0 client.16200 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:44:54.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.934+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 shutdown_connections 2023-10-10T17:44:54.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.935+0000 7fd98d276b80 1 -- 172.21.15.78:0/1771831447 wait complete. 2023-10-10T17:44:54.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.935+0000 7fd98d276b80 20 client.16200 trim_cache size 0 max 16384 2023-10-10T17:44:54.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:44:54.938 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:44:54.939 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:44:54.939 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:44:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.935+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd9640a12c0 msgr2=0x7fd9640a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.935+0000 7fd9810ea700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd9640a12c0 0x7fd9640a3770 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd96c011490 tx=0x7fd96c009260 comp rx=0 tx=0).stop 2023-10-10T17:44:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.935+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 msgr2=0x7fd97c196880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:44:54.940 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.935+0000 7fd9810ea700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c196880 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd970000c00 tx=0x7fd970004790 comp rx=0 tx=0).stop 2023-10-10T17:44:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 shutdown_connections 2023-10-10T17:44:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd97c10a820 0x7fd97c19ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd9640a12c0 0x7fd9640a3770 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd97c101300 0x7fd97c196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.941 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 --2- 172.21.15.78:0/594579125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd97c101c10 0x7fd97c196880 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:44:54.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 >> 172.21.15.78:0/594579125 conn(0x7fd97c0763a0 msgr2=0x7fd97c0fd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:44:54.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 shutdown_connections 2023-10-10T17:44:54.942 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:44:54.936+0000 7fd9810ea700 1 -- 172.21.15.78:0/594579125 wait complete. 2023-10-10T17:44:54.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:54 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:44:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:54 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:44:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:54 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:44:54.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:54 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:44:54.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:54 smithi157 ceph-mon[100896]: pgmap v74: 129 pgs: 129 active+clean; 27 MiB data, 120 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 87 op/s 2023-10-10T17:44:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:54 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:44:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:54 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:44:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:54 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:44:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:54 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:44:55.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:54 smithi078 ceph-mon[99494]: pgmap v74: 129 pgs: 129 active+clean; 27 MiB data, 120 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 87 op/s 2023-10-10T17:44:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:54 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:44:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:54 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:44:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:54 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:44:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:54 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:44:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:54 smithi190 ceph-mon[101220]: pgmap v74: 129 pgs: 129 active+clean; 27 MiB data, 120 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 87 op/s 2023-10-10T17:44:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:56 smithi157 ceph-mon[100896]: pgmap v75: 129 pgs: 129 active+clean; 27 MiB data, 120 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 81 op/s 2023-10-10T17:44:57.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:56 smithi078 ceph-mon[99494]: pgmap v75: 129 pgs: 129 active+clean; 27 MiB data, 120 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 81 op/s 2023-10-10T17:44:57.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:56 smithi190 ceph-mon[101220]: pgmap v75: 129 pgs: 129 active+clean; 27 MiB data, 120 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 81 op/s 2023-10-10T17:44:57.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:57 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:44:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:57 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:44:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:57 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:44:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:44:58 smithi157 ceph-mon[100896]: pgmap v76: 129 pgs: 129 active+clean; 9.5 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 3.19k op/s 2023-10-10T17:44:59.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:44:58 smithi078 ceph-mon[99494]: pgmap v76: 129 pgs: 129 active+clean; 9.5 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 3.19k op/s 2023-10-10T17:44:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:44:58 smithi190 ceph-mon[101220]: pgmap v76: 129 pgs: 129 active+clean; 9.5 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 3.2 MiB/s wr, 3.19k op/s 2023-10-10T17:45:01.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:00 smithi078 ceph-mon[99494]: pgmap v77: 129 pgs: 129 active+clean; 9.5 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 3.13k op/s 2023-10-10T17:45:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:00 smithi190 ceph-mon[101220]: pgmap v77: 129 pgs: 129 active+clean; 9.5 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 3.13k op/s 2023-10-10T17:45:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:00 smithi157 ceph-mon[100896]: pgmap v77: 129 pgs: 129 active+clean; 9.5 MiB data, 300 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.0 MiB/s wr, 3.13k op/s 2023-10-10T17:45:03.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:02 smithi078 ceph-mon[99494]: pgmap v78: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 3.15k op/s 2023-10-10T17:45:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:02 smithi190 ceph-mon[101220]: pgmap v78: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 3.15k op/s 2023-10-10T17:45:03.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:02 smithi157 ceph-mon[100896]: pgmap v78: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 3.15k op/s 2023-10-10T17:45:05.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:04 smithi078 ceph-mon[99494]: pgmap v79: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 3.14k op/s 2023-10-10T17:45:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:04 smithi190 ceph-mon[101220]: pgmap v79: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 3.14k op/s 2023-10-10T17:45:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:04 smithi157 ceph-mon[100896]: pgmap v79: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.2 MiB/s wr, 3.14k op/s 2023-10-10T17:45:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:06 smithi078 ceph-mon[99494]: pgmap v80: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 3.13k op/s 2023-10-10T17:45:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:06 smithi190 ceph-mon[101220]: pgmap v80: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 3.13k op/s 2023-10-10T17:45:07.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:06 smithi157 ceph-mon[100896]: pgmap v80: 129 pgs: 129 active+clean; 22 MiB data, 308 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.0 MiB/s wr, 3.13k op/s 2023-10-10T17:45:08.058 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/1669838931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f410a820 msgr2=0x7fd0f410cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/1669838931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f410a820 0x7fd0f410cc40 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00b0a0 tx=0x7fd0ec00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/1669838931 shutdown_connections 2023-10-10T17:45:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/1669838931 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f4101300 0x7fd0f41016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.059 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/1669838931 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f410a820 0x7fd0f410cc40 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/1669838931 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0f4101c10 0x7fd0f410a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/1669838931 >> 172.21.15.78:0/1669838931 conn(0x7fd0f40763a0 msgr2=0x7fd0f40767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/1669838931 shutdown_connections 2023-10-10T17:45:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/1669838931 wait complete. 2023-10-10T17:45:08.060 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 Processor -- start 2023-10-10T17:45:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.056+0000 7fd0fc5bb700 1 -- start start 2023-10-10T17:45:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fc5bb700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0f4101300 0x7fd0f40ff130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fc5bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f4101c10 0x7fd0f40ff670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fc5bb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f410a820 0x7fd0f41a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.061 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fc5bb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f40ffea0 con 0x7fd0f4101c10 2023-10-10T17:45:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fc5bb700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f41a4c00 con 0x7fd0f4101300 2023-10-10T17:45:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fc5bb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f41a4d40 con 0x7fd0f410a820 2023-10-10T17:45:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f4101c10 0x7fd0f40ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.062 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f4101c10 0x7fd0f40ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45928/0 (socket says 172.21.15.78:45928) 2023-10-10T17:45:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 -- 172.21.15.78:0/4066454311 learned_addr learned my addr 172.21.15.78:0/4066454311 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fab58700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f410a820 0x7fd0f41a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fa357700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0f4101300 0x7fd0f40ff130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.063 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 -- 172.21.15.78:0/4066454311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f410a820 msgr2=0x7fd0f41a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f410a820 0x7fd0f41a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 -- 172.21.15.78:0/4066454311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0f4101300 msgr2=0x7fd0f40ff130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0f4101300 0x7fd0f40ff130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 -- 172.21.15.78:0/4066454311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ec009d00 con 0x7fd0f4101c10 2023-10-10T17:45:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f4101c10 0x7fd0f40ff670 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400d950 tx=0x7fd0e400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:08.064 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0eb7fe700 1 -- 172.21.15.78:0/4066454311 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400f890 con 0x7fd0f4101c10 2023-10-10T17:45:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0eb7fe700 1 -- 172.21.15.78:0/4066454311 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0e4010460 con 0x7fd0f4101c10 2023-10-10T17:45:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f41102c0 con 0x7fd0f4101c10 2023-10-10T17:45:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.057+0000 7fd0eb7fe700 1 -- 172.21.15.78:0/4066454311 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e400e610 con 0x7fd0f4101c10 2023-10-10T17:45:08.065 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.058+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f4110130 con 0x7fd0f4101c10 2023-10-10T17:45:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.060+0000 7fd0eb7fe700 1 -- 172.21.15.78:0/4066454311 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fd0e40108d0 con 0x7fd0f4101c10 2023-10-10T17:45:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.060+0000 7fd0eb7fe700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0e00a12c0 0x7fd0e00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.061+0000 7fd0fa357700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0e00a12c0 0x7fd0e00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.061+0000 7fd0eb7fe700 1 -- 172.21.15.78:0/4066454311 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd0e40cd0d0 con 0x7fd0f4101c10 2023-10-10T17:45:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.061+0000 7fd0fa357700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0e00a12c0 0x7fd0e00a3770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009c60 tx=0x7fd0f0009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.061+0000 7fd108747b80 1 Processor -- start 2023-10-10T17:45:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd108747b80 1 -- start start 2023-10-10T17:45:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd108747b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2d8966b0 0x560d2da401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd108747b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da3a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd108747b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da3ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd108747b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2d940000 con 0x560d2da40720 2023-10-10T17:45:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd108747b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2d992840 con 0x560d2d8966b0 2023-10-10T17:45:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd108747b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2d98bcd0 con 0x560d2da40b90 2023-10-10T17:45:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da3a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da3a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45936/0 (socket says 172.21.15.78:45936) 2023-10-10T17:45:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1511447851 learned_addr learned my addr 172.21.15.78:0/1511447851 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:08.074 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0fab58700 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da3ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0fa357700 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2d8966b0 0x560d2da401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1511447851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 msgr2=0x560d2da3ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da3ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1511447851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2d8966b0 msgr2=0x560d2da401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.075 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2d8966b0 0x560d2da401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1511447851 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d2d80d5d0 con 0x560d2da40720 2023-10-10T17:45:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.062+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da3a560 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40a1d90 tx=0x7fd0e4099ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d2f87f822871d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd0e8ff9700 1 -- 172.21.15.78:0/1511447851 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e40a22c0 con 0x560d2da40720 2023-10-10T17:45:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd0e8ff9700 1 -- 172.21.15.78:0/1511447851 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0e40ca770 con 0x560d2da40720 2023-10-10T17:45:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd0e8ff9700 1 -- 172.21.15.78:0/1511447851 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4099560 con 0x560d2da40720 2023-10-10T17:45:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 -- 172.21.15.78:0/1511447851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 msgr2=0x560d2da3a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da3a560 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40a1d90 tx=0x7fd0e4099ef0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 -- 172.21.15.78:0/1511447851 shutdown_connections 2023-10-10T17:45:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da3ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2d8966b0 0x560d2da401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 --2- 172.21.15.78:0/1511447851 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da3a560 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 -- 172.21.15.78:0/1511447851 >> 172.21.15.78:0/1511447851 conn(0x560d2da2ff50 msgr2=0x560d2da35270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 -- 172.21.15.78:0/1511447851 shutdown_connections 2023-10-10T17:45:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.063+0000 7fd108747b80 1 -- 172.21.15.78:0/1511447851 wait complete. 2023-10-10T17:45:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 -- start start 2023-10-10T17:45:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 Processor -- start 2023-10-10T17:45:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da499e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da49f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40b90 0x560d2da43c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2d92d880 con 0x560d2da40b90 2023-10-10T17:45:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2d92d520 con 0x560d2da40720 2023-10-10T17:45:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2d80f8c0 con 0x560d2d8966b0 2023-10-10T17:45:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd108747b80 10 client.0 _ll_register_callbacks cb 0x560d2d94b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:08.080 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd0fab58700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40b90 0x560d2da43c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd0fab58700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40b90 0x560d2da43c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45944/0 (socket says 172.21.15.78:45944) 2023-10-10T17:45:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd0fab58700 1 -- 172.21.15.78:0/3988673989 learned_addr learned my addr 172.21.15.78:0/3988673989 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd0fa357700 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.064+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da49f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.081 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0fab58700 1 -- 172.21.15.78:0/3988673989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 msgr2=0x560d2da499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0fab58700 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0fab58700 1 -- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 msgr2=0x560d2da49f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0fab58700 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da49f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0fab58700 1 -- 172.21.15.78:0/3988673989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f4110130 con 0x560d2da40b90 2023-10-10T17:45:08.082 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0fab58700 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40b90 0x560d2da43c70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00b0a0 tx=0x7fd0ec004c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a9ac924c5d3de2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0ddffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3988673989 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec00e040 con 0x560d2da40b90 2023-10-10T17:45:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3988673989 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd0ec0076e0 con 0x560d2da40b90 2023-10-10T17:45:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560d2da2d710 con 0x560d2da40b90 2023-10-10T17:45:08.083 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3988673989 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0ec0078a0 con 0x560d2da40b90 2023-10-10T17:45:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 10 client.16212 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560d2d952130 con 0x560d2da40b90 2023-10-10T17:45:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3988673989 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560d2da2d710 con 0x560d2da40b90 2023-10-10T17:45:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3988673989 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fd0ec025030 con 0x560d2da40b90 2023-10-10T17:45:08.084 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 10 client.16212 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 10 client.16212 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 20 client.16212 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2da53ed0 0x560d2da562e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 4 client.16212 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:08.085 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.065+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d2d94e040 con 0x560d2da53ed0 2023-10-10T17:45:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.066+0000 7fd0fa357700 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2da53ed0 0x560d2da562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.066+0000 7fd0fa357700 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2da53ed0 0x560d2da562e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48126d1543269798 server_cookie=a315f3c5e0193c10 in_seq=0 out_seq=0 2023-10-10T17:45:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.066+0000 7fd0ddffb700 10 client.16212 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3988673989 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560d2d94e040 con 0x560d2da53ed0 2023-10-10T17:45:08.086 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd0ddffb700 10 client.16212 handle_command_reply: tid=0 2023-10-10T17:45:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 client.16212 shutdown 2023-10-10T17:45:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40b90 msgr2=0x560d2da43c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40b90 0x560d2da43c70 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00b0a0 tx=0x7fd0ec004c40 comp rx=0 tx=0).stop 2023-10-10T17:45:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 shutdown_connections 2023-10-10T17:45:08.087 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2da53ed0 0x560d2da562e0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40b90 0x560d2da43c70 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd0ddffb700 0 client.16212 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:08.088 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 --2- 172.21.15.78:0/3988673989 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da49f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 >> 172.21.15.78:0/3988673989 conn(0x560d2da2ff50 msgr2=0x560d2da347e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 shutdown_connections 2023-10-10T17:45:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 1 -- 172.21.15.78:0/3988673989 wait complete. 2023-10-10T17:45:08.089 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.067+0000 7fd108747b80 20 client.16212 trim_cache size 0 max 16384 2023-10-10T17:45:08.093 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.082+0000 7fd108747b80 1 Processor -- start 2023-10-10T17:45:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.082+0000 7fd108747b80 1 -- start start 2023-10-10T17:45:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd108747b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da41670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd108747b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da34f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd108747b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40b90 0x560d2da3a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.094 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd108747b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2da36430 con 0x560d2da40720 2023-10-10T17:45:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd108747b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2da31fd0 con 0x560d2da40b90 2023-10-10T17:45:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd108747b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2da36200 con 0x560d2d8966b0 2023-10-10T17:45:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da34f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da34f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45950/0 (socket says 172.21.15.78:45950) 2023-10-10T17:45:08.095 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1989081160 learned_addr learned my addr 172.21.15.78:0/1989081160 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0fa357700 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da41670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1989081160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 msgr2=0x560d2da41670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da41670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.096 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1989081160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40b90 msgr2=0x560d2da3a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40b90 0x560d2da3a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 -- 172.21.15.78:0/1989081160 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d2d951e80 con 0x560d2da40720 2023-10-10T17:45:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da34f60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40968e0 tx=0x7fd0e4010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a645608f92bfed server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0e97fa700 1 -- 172.21.15.78:0/1989081160 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e4098210 con 0x560d2da40720 2023-10-10T17:45:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0e97fa700 1 -- 172.21.15.78:0/1989081160 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0e40b8170 con 0x560d2da40720 2023-10-10T17:45:08.097 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.083+0000 7fd0e97fa700 1 -- 172.21.15.78:0/1989081160 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0e409a0d0 con 0x560d2da40720 2023-10-10T17:45:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 -- 172.21.15.78:0/1989081160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 msgr2=0x560d2da34f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da34f60 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd0e40968e0 tx=0x7fd0e4010cd0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 -- 172.21.15.78:0/1989081160 shutdown_connections 2023-10-10T17:45:08.098 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2d8966b0 0x560d2da41670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40b90 0x560d2da3a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 --2- 172.21.15.78:0/1989081160 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2da40720 0x560d2da34f60 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 -- 172.21.15.78:0/1989081160 >> 172.21.15.78:0/1989081160 conn(0x560d2da2ff50 msgr2=0x560d2da40f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 -- 172.21.15.78:0/1989081160 shutdown_connections 2023-10-10T17:45:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 -- 172.21.15.78:0/1989081160 wait complete. 2023-10-10T17:45:08.099 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 -- start start 2023-10-10T17:45:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.084+0000 7fd108747b80 1 Processor -- start 2023-10-10T17:45:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2d8966b0 0x560d2da47590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da47ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da48010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.100 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2daa6ec0 con 0x560d2d8966b0 2023-10-10T17:45:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2daa7030 con 0x560d2da40720 2023-10-10T17:45:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d2daa71f0 con 0x560d2da40b90 2023-10-10T17:45:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 10 client.0 _ll_register_callbacks cb 0x7fd0ec00bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2d8966b0 0x560d2da47590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.101 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2d8966b0 0x560d2da47590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45954/0 (socket says 172.21.15.78:45954) 2023-10-10T17:45:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 -- 172.21.15.78:0/3236374839 learned_addr learned my addr 172.21.15.78:0/3236374839 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fab58700 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da48010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da47ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 -- 172.21.15.78:0/3236374839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 msgr2=0x560d2da48010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.102 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da48010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 -- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 msgr2=0x560d2da47ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da47ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 -- 172.21.15.78:0/3236374839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d2d80d5d0 con 0x560d2d8966b0 2023-10-10T17:45:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0fa357700 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2d8966b0 0x560d2da47590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0013010 tx=0x7fd0f00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a414e0c220fb2b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:08.103 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0ddffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3236374839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0023040 con 0x560d2d8966b0 2023-10-10T17:45:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3236374839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fd0f0003680 con 0x560d2d8966b0 2023-10-10T17:45:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd0e40ca770 con 0x560d2d8966b0 2023-10-10T17:45:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3236374839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0f0003840 con 0x560d2d8966b0 2023-10-10T17:45:08.104 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 10 client.16218 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.085+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560d2d7a2c00 con 0x560d2d8966b0 2023-10-10T17:45:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3236374839 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd0e40ca770 con 0x560d2d8966b0 2023-10-10T17:45:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3236374839 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fd0f00308f0 con 0x560d2d8966b0 2023-10-10T17:45:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd108747b80 10 client.16218 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:08.105 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd108747b80 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd108747b80 20 client.16218 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd108747b80 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2daa8960 0x560d2daaad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd108747b80 4 client.16218 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:45:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560d2d95a600 con 0x560d2daa8960 2023-10-10T17:45:08.106 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2daa8960 0x560d2daaad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd0f9b56700 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2daa8960 0x560d2daaad70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1902c099235a942e server_cookie=48df588adadfc8e1 in_seq=0 out_seq=0 2023-10-10T17:45:08.107 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.086+0000 7fd0ddffb700 10 client.16218 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd0ddffb700 1 -- 172.21.15.78:0/3236374839 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560d2d95a600 con 0x560d2daa8960 2023-10-10T17:45:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd0ddffb700 10 client.16218 handle_command_reply: tid=0 2023-10-10T17:45:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 client.16218 shutdown 2023-10-10T17:45:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2d8966b0 msgr2=0x560d2da47590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2d8966b0 0x560d2da47590 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0013010 tx=0x7fd0f00047c0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 shutdown_connections 2023-10-10T17:45:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560d2da40b90 0x560d2da48010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560d2daa8960 0x560d2daaad70 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560d2d8966b0 0x560d2da47590 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 --2- 172.21.15.78:0/3236374839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560d2da40720 0x560d2da47ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd0ddffb700 0 client.16218 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 >> 172.21.15.78:0/3236374839 conn(0x560d2da2ff50 msgr2=0x560d2da417f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 shutdown_connections 2023-10-10T17:45:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 1 -- 172.21.15.78:0/3236374839 wait complete. 2023-10-10T17:45:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.315+0000 7fd108747b80 20 client.16218 trim_cache size 0 max 16384 2023-10-10T17:45:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:45:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:45:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:45:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:45:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0e00a12c0 msgr2=0x7fd0e00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0e00a12c0 0x7fd0e00a3770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009c60 tx=0x7fd0f0009260 comp rx=0 tx=0).stop 2023-10-10T17:45:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f4101c10 msgr2=0x7fd0f40ff670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f4101c10 0x7fd0f40ff670 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400d950 tx=0x7fd0e400dc60 comp rx=0 tx=0).stop 2023-10-10T17:45:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 shutdown_connections 2023-10-10T17:45:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd0f410a820 0x7fd0f41a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0e00a12c0 0x7fd0e00a3770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd0f4101300 0x7fd0f40ff130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 --2- 172.21.15.78:0/4066454311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f4101c10 0x7fd0f40ff670 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 >> 172.21.15.78:0/4066454311 conn(0x7fd0f40763a0 msgr2=0x7fd0f40fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 shutdown_connections 2023-10-10T17:45:08.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:08.316+0000 7fd0fc5bb700 1 -- 172.21.15.78:0/4066454311 wait complete. 2023-10-10T17:45:09.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:08 smithi078 ceph-mon[99494]: pgmap v81: 129 pgs: 129 active+clean; 45 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 3.14k op/s 2023-10-10T17:45:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:08 smithi190 ceph-mon[101220]: pgmap v81: 129 pgs: 129 active+clean; 45 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 3.14k op/s 2023-10-10T17:45:09.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:08 smithi157 ceph-mon[100896]: pgmap v81: 129 pgs: 129 active+clean; 45 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.9 MiB/s wr, 3.14k op/s 2023-10-10T17:45:10.023 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:45:10.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 -- 172.21.15.78:0/832589883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 msgr2=0x7f2c50100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 --2- 172.21.15.78:0/832589883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50100000 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f2c400099c0 tx=0x7f2c40009cd0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 -- 172.21.15.78:0/832589883 shutdown_connections 2023-10-10T17:45:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 --2- 172.21.15.78:0/832589883 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c50104590 0x7f2c5010c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 --2- 172.21.15.78:0/832589883 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50100000 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 --2- 172.21.15.78:0/832589883 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c501005d0 0x7f2c50103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 -- 172.21.15.78:0/832589883 >> 172.21.15.78:0/832589883 conn(0x7f2c50075470 msgr2=0x7f2c50075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 -- 172.21.15.78:0/832589883 shutdown_connections 2023-10-10T17:45:10.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.159+0000 7f2c55b93700 1 -- 172.21.15.78:0/832589883 wait complete. 2023-10-10T17:45:10.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 Processor -- start 2023-10-10T17:45:10.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 -- start start 2023-10-10T17:45:10.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c501005d0 0x7f2c50195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c50104590 0x7f2c50199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5019a300 con 0x7f2c500ffc30 2023-10-10T17:45:10.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5019a470 con 0x7f2c501005d0 2023-10-10T17:45:10.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c55b93700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5019a630 con 0x7f2c50104590 2023-10-10T17:45:10.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50195320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45968/0 (socket says 172.21.15.78:45968) 2023-10-10T17:45:10.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/1049864985 learned_addr learned my addr 172.21.15.78:0/1049864985 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.160+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c50104590 0x7f2c50199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4effd700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c501005d0 0x7f2c50195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/1049864985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c50104590 msgr2=0x7f2c50199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c50104590 0x7f2c50199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/1049864985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c501005d0 msgr2=0x7f2c50195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c501005d0 0x7f2c50195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/1049864985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c400095f0 con 0x7f2c500ffc30 2023-10-10T17:45:10.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50195320 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000b580 tx=0x7f2c4000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4cff9700 1 -- 172.21.15.78:0/1049864985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4001d070 con 0x7f2c500ffc30 2023-10-10T17:45:10.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5010f300 con 0x7f2c500ffc30 2023-10-10T17:45:10.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4cff9700 1 -- 172.21.15.78:0/1049864985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c400054f0 con 0x7f2c500ffc30 2023-10-10T17:45:10.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c4cff9700 1 -- 172.21.15.78:0/1049864985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c40004b30 con 0x7f2c500ffc30 2023-10-10T17:45:10.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.161+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5010f170 con 0x7f2c500ffc30 2023-10-10T17:45:10.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.163+0000 7f2c4cff9700 1 -- 172.21.15.78:0/1049864985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f2c4000fb80 con 0x7f2c500ffc30 2023-10-10T17:45:10.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.164+0000 7f2c4cff9700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f2c380a12c0 0x7f2c380a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.164+0000 7f2c4effd700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f2c380a12c0 0x7f2c380a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.164+0000 7f2c4cff9700 1 -- 172.21.15.78:0/1049864985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f2c400cde00 con 0x7f2c500ffc30 2023-10-10T17:45:10.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.164+0000 7f2c4effd700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f2c380a12c0 0x7f2c380a3770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c50196930 tx=0x7f2c44009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 Processor -- start 2023-10-10T17:45:10.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 -- start start 2023-10-10T17:45:10.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399b970 0x55692399bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x5569239a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x5569239ac910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569238a1060 con 0x55692399b970 2023-10-10T17:45:10.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569238f38a0 con 0x5569239aa530 2023-10-10T17:45:10.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569238ecd30 con 0x55692399c280 2023-10-10T17:45:10.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399b970 0x55692399bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399b970 0x55692399bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45976/0 (socket says 172.21.15.78:45976) 2023-10-10T17:45:10.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/4124340371 learned_addr learned my addr 172.21.15.78:0/4124340371 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.165+0000 7f2c4effd700 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x5569239a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x5569239ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/4124340371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 msgr2=0x5569239a9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x5569239a9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/4124340371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 msgr2=0x5569239ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x5569239ac910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4f7fe700 1 -- 172.21.15.78:0/4124340371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55692376e630 con 0x55692399b970 2023-10-10T17:45:10.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x5569239ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:45:10.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399b970 0x55692399bd40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2c400976a0 tx=0x7f2c40009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaab1e93661f0d64 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c35ffb700 1 -- 172.21.15.78:0/4124340371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4009ac30 con 0x55692399b970 2023-10-10T17:45:10.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c35ffb700 1 -- 172.21.15.78:0/4124340371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c4009ec60 con 0x55692399b970 2023-10-10T17:45:10.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c35ffb700 1 -- 172.21.15.78:0/4124340371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4009a380 con 0x55692399b970 2023-10-10T17:45:10.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/4124340371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399b970 msgr2=0x55692399bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.166+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399b970 0x55692399bd40 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f2c400976a0 tx=0x7f2c40009310 comp rx=0 tx=0).stop 2023-10-10T17:45:10.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/4124340371 shutdown_connections 2023-10-10T17:45:10.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x5569239ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399b970 0x55692399bd40 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/4124340371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x5569239a9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/4124340371 >> 172.21.15.78:0/4124340371 conn(0x556923991360 msgr2=0x556923996b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/4124340371 shutdown_connections 2023-10-10T17:45:10.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/4124340371 wait complete. 2023-10-10T17:45:10.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 -- start start 2023-10-10T17:45:10.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 Processor -- start 2023-10-10T17:45:10.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.167+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x5569239af550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x5569239afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569239aa530 0x5569239a9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55692388e8e0 con 0x55692399c280 2023-10-10T17:45:10.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55692388e580 con 0x55692399b970 2023-10-10T17:45:10.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556923770920 con 0x5569239aa530 2023-10-10T17:45:10.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 10 client.0 _ll_register_callbacks cb 0x5569238ac780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:10.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x5569239afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x5569239afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45984/0 (socket says 172.21.15.78:45984) 2023-10-10T17:45:10.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 -- 172.21.15.78:0/1250266323 learned_addr learned my addr 172.21.15.78:0/1250266323 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569239aa530 0x5569239a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x5569239af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 -- 172.21.15.78:0/1250266323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569239aa530 msgr2=0x5569239a9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569239aa530 0x5569239a9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 -- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 msgr2=0x5569239af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x5569239af550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 -- 172.21.15.78:0/1250266323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c5010f170 con 0x55692399c280 2023-10-10T17:45:10.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x5569239af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:45:10.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c4effd700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x5569239afa90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2c440128b0 tx=0x7f2c440047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b63e487c0d1b2ff8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c2effd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:10.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c2effd700 1 -- 172.21.15.78:0/1250266323 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44020840 con 0x55692399c280 2023-10-10T17:45:10.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c2effd700 1 -- 172.21.15.78:0/1250266323 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f2c44018070 con 0x55692399c280 2023-10-10T17:45:10.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55692398eb20 con 0x55692399c280 2023-10-10T17:45:10.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c2effd700 1 -- 172.21.15.78:0/1250266323 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c44012e20 con 0x55692399c280 2023-10-10T17:45:10.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 10 client.16230 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:10.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.168+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5569238b3190 con 0x55692399c280 2023-10-10T17:45:10.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c2effd700 1 -- 172.21.15.78:0/1250266323 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55692398eb20 con 0x55692399c280 2023-10-10T17:45:10.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c2effd700 1 -- 172.21.15.78:0/1250266323 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f2c44029a90 con 0x55692399c280 2023-10-10T17:45:10.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c61d1fb80 10 client.16230 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:10.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c61d1fb80 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:10.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c61d1fb80 20 client.16230 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:10.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569239b99a0 0x5569239bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c61d1fb80 4 client.16230 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:10.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569238af0a0 con 0x5569239b99a0 2023-10-10T17:45:10.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569239b99a0 0x5569239bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c2effd700 10 client.16230 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:10.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.169+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569239b99a0 0x5569239bbdb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=52dc7678445b43ed server_cookie=850093a5bdba2f1 in_seq=0 out_seq=0 2023-10-10T17:45:10.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c2effd700 1 -- 172.21.15.78:0/1250266323 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5569238af0a0 con 0x5569239b99a0 2023-10-10T17:45:10.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c2effd700 10 client.16230 handle_command_reply: tid=0 2023-10-10T17:45:10.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 client.16230 shutdown 2023-10-10T17:45:10.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 msgr2=0x5569239afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x5569239afa90 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2c440128b0 tx=0x7f2c440047c0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 shutdown_connections 2023-10-10T17:45:10.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569239aa530 0x5569239a9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569239b99a0 0x5569239bbdb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x5569239af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1250266323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x5569239afa90 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c2effd700 0 client.16230 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:10.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 >> 172.21.15.78:0/1250266323 conn(0x556923991360 msgr2=0x556923995de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 shutdown_connections 2023-10-10T17:45:10.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.170+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1250266323 wait complete. 2023-10-10T17:45:10.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.171+0000 7f2c61d1fb80 20 client.16230 trim_cache size 0 max 16384 2023-10-10T17:45:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.190+0000 7f2c61d1fb80 1 Processor -- start 2023-10-10T17:45:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.190+0000 7f2c61d1fb80 1 -- start start 2023-10-10T17:45:10.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399b970 0x5569239a9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x556923996560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x556923996aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556923990f10 con 0x55692399c280 2023-10-10T17:45:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569239978b0 con 0x5569239aa530 2023-10-10T17:45:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55692399d3e0 con 0x55692399b970 2023-10-10T17:45:10.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x556923996560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x556923996560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:45992/0 (socket says 172.21.15.78:45992) 2023-10-10T17:45:10.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 -- 172.21.15.78:0/518703144 learned_addr learned my addr 172.21.15.78:0/518703144 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399b970 0x5569239a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 -- 172.21.15.78:0/518703144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399b970 msgr2=0x5569239a9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399b970 0x5569239a9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 -- 172.21.15.78:0/518703144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 msgr2=0x556923996aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x556923996aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x556923996aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 -- 172.21.15.78:0/518703144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569238b2ee0 con 0x55692399c280 2023-10-10T17:45:10.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x556923996aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:45:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c4effd700 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x556923996560 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2c44000c00 tx=0x7f2c44004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=454d0643f279e957 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c367fc700 1 -- 172.21.15.78:0/518703144 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4401f480 con 0x55692399c280 2023-10-10T17:45:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c367fc700 1 -- 172.21.15.78:0/518703144 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c44016070 con 0x55692399c280 2023-10-10T17:45:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.191+0000 7f2c367fc700 1 -- 172.21.15.78:0/518703144 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4401f810 con 0x55692399c280 2023-10-10T17:45:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/518703144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 msgr2=0x556923996560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x556923996560 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f2c44000c00 tx=0x7f2c44004710 comp rx=0 tx=0).stop 2023-10-10T17:45:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/518703144 shutdown_connections 2023-10-10T17:45:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399b970 0x5569239a9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569239aa530 0x556923996aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/518703144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55692399c280 0x556923996560 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/518703144 >> 172.21.15.78:0/518703144 conn(0x556923991360 msgr2=0x5569239a9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/518703144 shutdown_connections 2023-10-10T17:45:10.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/518703144 wait complete. 2023-10-10T17:45:10.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 -- start start 2023-10-10T17:45:10.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.192+0000 7f2c61d1fb80 1 Processor -- start 2023-10-10T17:45:10.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x556923a019e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x556923a01f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569239aa530 0x556923a0c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556923a0ce00 con 0x5569239aa530 2023-10-10T17:45:10.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556923a0cf70 con 0x55692399b970 2023-10-10T17:45:10.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556923a0d130 con 0x55692399c280 2023-10-10T17:45:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 10 client.0 _ll_register_callbacks cb 0x5569238ac780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569239aa530 0x556923a0c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569239aa530 0x556923a0c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46000/0 (socket says 172.21.15.78:46000) 2023-10-10T17:45:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 -- 172.21.15.78:0/1866771416 learned_addr learned my addr 172.21.15.78:0/1866771416 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4effd700 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x556923a01f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x556923a019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 -- 172.21.15.78:0/1866771416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 msgr2=0x556923a01f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x556923a01f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 -- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 msgr2=0x556923a019e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x556923a019e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 -- 172.21.15.78:0/1866771416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5569238b3190 con 0x5569239aa530 2023-10-10T17:45:10.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c4ffff700 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569239aa530 0x556923a0c870 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f2c48014890 tx=0x7f2c48014ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d36e922eb457c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c2effd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:10.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c2effd700 1 -- 172.21.15.78:0/1866771416 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48004bb0 con 0x5569239aa530 2023-10-10T17:45:10.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c2effd700 1 -- 172.21.15.78:0/1866771416 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f2c48004d10 con 0x5569239aa530 2023-10-10T17:45:10.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.193+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2c4009ec60 con 0x5569239aa530 2023-10-10T17:45:10.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c2effd700 1 -- 172.21.15.78:0/1866771416 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c480127b0 con 0x5569239aa530 2023-10-10T17:45:10.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 10 client.16236 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:10.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556923703c60 con 0x5569239aa530 2023-10-10T17:45:10.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c2effd700 1 -- 172.21.15.78:0/1866771416 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2c4009ec60 con 0x5569239aa530 2023-10-10T17:45:10.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c2effd700 1 -- 172.21.15.78:0/1866771416 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f2c48017960 con 0x5569239aa530 2023-10-10T17:45:10.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 10 client.16236 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:10.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 10 client.16236 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:10.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 20 client.16236 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:10.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556923a0e8f0 0x556923a10d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 4 client.16236 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:45:10.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5569238bb660 con 0x556923a0e8f0 2023-10-10T17:45:10.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.194+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556923a0e8f0 0x556923a10d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c2effd700 10 client.16236 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:10.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c4f7fe700 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556923a0e8f0 0x556923a10d00 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78248a91ed7a87af server_cookie=612495264ce73adf in_seq=0 out_seq=0 2023-10-10T17:45:10.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c2effd700 1 -- 172.21.15.78:0/1866771416 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5569238bb660 con 0x556923a0e8f0 2023-10-10T17:45:10.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c2effd700 10 client.16236 handle_command_reply: tid=0 2023-10-10T17:45:10.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 client.16236 shutdown 2023-10-10T17:45:10.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569239aa530 msgr2=0x556923a0c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569239aa530 0x556923a0c870 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f2c48014890 tx=0x7f2c48014ba0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 shutdown_connections 2023-10-10T17:45:10.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556923a0e8f0 0x556923a10d00 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569239aa530 0x556923a0c870 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c2effd700 0 client.16236 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55692399b970 0x556923a019e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 --2- 172.21.15.78:0/1866771416 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55692399c280 0x556923a01f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 >> 172.21.15.78:0/1866771416 conn(0x556923991360 msgr2=0x55692399b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 shutdown_connections 2023-10-10T17:45:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 1 -- 172.21.15.78:0/1866771416 wait complete. 2023-10-10T17:45:10.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.195+0000 7f2c61d1fb80 20 client.16236 trim_cache size 0 max 16384 2023-10-10T17:45:10.212 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:45:10.212 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:45:10.213 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:45:10.213 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:45:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f2c380a12c0 msgr2=0x7f2c380a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f2c380a12c0 0x7f2c380a3770 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2c50196930 tx=0x7f2c44009260 comp rx=0 tx=0).stop 2023-10-10T17:45:10.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 msgr2=0x7f2c50195320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50195320 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f2c4000b580 tx=0x7f2c4000f690 comp rx=0 tx=0).stop 2023-10-10T17:45:10.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 shutdown_connections 2023-10-10T17:45:10.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2c50104590 0x7f2c50199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f2c380a12c0 0x7f2c380a3770 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c500ffc30 0x7f2c50195320 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 --2- 172.21.15.78:0/1049864985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2c501005d0 0x7f2c50195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 >> 172.21.15.78:0/1049864985 conn(0x7f2c50075470 msgr2=0x7f2c50102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 shutdown_connections 2023-10-10T17:45:10.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.196+0000 7f2c55b93700 1 -- 172.21.15.78:0/1049864985 wait complete. 2023-10-10T17:45:10.217 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:45:10.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:45:10.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 -- 172.21.15.78:0/2117032444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 msgr2=0x7f08641044f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 --2- 172.21.15.78:0/2117032444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f08641044f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f08540099c0 tx=0x7f0854009cd0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 -- 172.21.15.78:0/2117032444 shutdown_connections 2023-10-10T17:45:10.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 --2- 172.21.15.78:0/2117032444 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0864100300 0x7f086410bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 --2- 172.21.15.78:0/2117032444 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f08641044f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 --2- 172.21.15.78:0/2117032444 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08640ff950 0x7f08640ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 -- 172.21.15.78:0/2117032444 >> 172.21.15.78:0/2117032444 conn(0x7f0864075820 msgr2=0x7f0864075c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 -- 172.21.15.78:0/2117032444 shutdown_connections 2023-10-10T17:45:10.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 -- 172.21.15.78:0/2117032444 wait complete. 2023-10-10T17:45:10.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 Processor -- start 2023-10-10T17:45:10.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.339+0000 7f086c5d4700 1 -- start start 2023-10-10T17:45:10.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08640ff950 0x7f0864194fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0864100300 0x7f0864195520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f0864195a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08641a4820 con 0x7f0864104120 2023-10-10T17:45:10.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08641a4990 con 0x7f08640ff950 2023-10-10T17:45:10.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08641a4b50 con 0x7f0864100300 2023-10-10T17:45:10.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f0864195a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f0864195a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46014/0 (socket says 172.21.15.78:46014) 2023-10-10T17:45:10.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 -- 172.21.15.78:0/3605931439 learned_addr learned my addr 172.21.15.78:0/3605931439 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f0869b6f700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0864100300 0x7f0864195520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086a370700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08640ff950 0x7f0864194fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 -- 172.21.15.78:0/3605931439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0864100300 msgr2=0x7f0864195520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0864100300 0x7f0864195520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 -- 172.21.15.78:0/3605931439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08640ff950 msgr2=0x7f0864194fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08640ff950 0x7f0864194fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 -- 172.21.15.78:0/3605931439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08540095f0 con 0x7f0864104120 2023-10-10T17:45:10.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086a370700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08640ff950 0x7f0864194fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:45:10.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086ab71700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f0864195a60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f085c00ef30 tx=0x7f085c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f085b7fe700 1 -- 172.21.15.78:0/3605931439 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c018070 con 0x7f0864104120 2023-10-10T17:45:10.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f086410f300 con 0x7f0864104120 2023-10-10T17:45:10.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f085b7fe700 1 -- 172.21.15.78:0/3605931439 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f085c00f040 con 0x7f0864104120 2023-10-10T17:45:10.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f085b7fe700 1 -- 172.21.15.78:0/3605931439 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f085c00cdb0 con 0x7f0864104120 2023-10-10T17:45:10.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.340+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086410f170 con 0x7f0864104120 2023-10-10T17:45:10.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.341+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f086410f490 con 0x7f0864104120 2023-10-10T17:45:10.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.343+0000 7f085b7fe700 1 -- 172.21.15.78:0/3605931439 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f085c003a00 con 0x7f0864104120 2023-10-10T17:45:10.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.343+0000 7f085b7fe700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08500a12c0 0x7f08500a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.344+0000 7f086a370700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08500a12c0 0x7f08500a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.344+0000 7f085b7fe700 1 -- 172.21.15.78:0/3605931439 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f085c0cd590 con 0x7f0864104120 2023-10-10T17:45:10.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.344+0000 7f086a370700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08500a12c0 0x7f08500a3770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08540098e0 tx=0x7f0854019040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.347+0000 7f085b7fe700 1 -- 172.21.15.78:0/3605931439 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f085c0966c0 con 0x7f0864104120 2023-10-10T17:45:10.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.615+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08640671d0 con 0x7f0864104120 2023-10-10T17:45:10.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.616+0000 7f085b7fe700 1 -- 172.21.15.78:0/3605931439 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+10810 (secure 0 0 0) 0x7f085c099ce0 con 0x7f0864104120 2023-10-10T17:45:10.617 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:45:10.617 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24419,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:44:56.715181+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24467":{"gid":24467,"name":"h","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6837/1666360072","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":1666360072},{"type":"v1","addr":"172.21.15.157:6837","nonce":1666360072}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:45:10.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08500a12c0 msgr2=0x7f08500a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08500a12c0 0x7f08500a3770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f08540098e0 tx=0x7f0854019040 comp rx=0 tx=0).stop 2023-10-10T17:45:10.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 msgr2=0x7f0864195a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f0864195a60 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f085c00ef30 tx=0x7f085c00c3a0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 shutdown_connections 2023-10-10T17:45:10.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08500a12c0 0x7f08500a3770 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0864104120 0x7f0864195a60 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08640ff950 0x7f0864194fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 --2- 172.21.15.78:0/3605931439 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0864100300 0x7f0864195520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 >> 172.21.15.78:0/3605931439 conn(0x7f0864075820 msgr2=0x7f08640fc290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 shutdown_connections 2023-10-10T17:45:10.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.619+0000 7f086c5d4700 1 -- 172.21.15.78:0/3605931439 wait complete. 2023-10-10T17:45:10.622 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 26 2023-10-10T17:45:10.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:45:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 -- 172.21.15.78:0/3145037367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 msgr2=0x7fabc810d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 --2- 172.21.15.78:0/3145037367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc810d0c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fabc000b0d0 tx=0x7fabc000b3e0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 -- 172.21.15.78:0/3145037367 shutdown_connections 2023-10-10T17:45:10.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 --2- 172.21.15.78:0/3145037367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc810d0c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 --2- 172.21.15.78:0/3145037367 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc81004b0 0x7fabc8100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 --2- 172.21.15.78:0/3145037367 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabc8100e50 0x7fabc8104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 -- 172.21.15.78:0/3145037367 >> 172.21.15.78:0/3145037367 conn(0x7fabc80fbd20 msgr2=0x7fabc80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 -- 172.21.15.78:0/3145037367 shutdown_connections 2023-10-10T17:45:10.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.765+0000 7fabcca1c700 1 -- 172.21.15.78:0/3145037367 wait complete. 2023-10-10T17:45:10.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 Processor -- start 2023-10-10T17:45:10.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 -- start start 2023-10-10T17:45:10.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc81004b0 0x7fabc8196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabc8100e50 0x7fabc81969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc819b450 con 0x7fabc81054f0 2023-10-10T17:45:10.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc819b5c0 con 0x7fabc81004b0 2023-10-10T17:45:10.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabcca1c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc819b780 con 0x7fabc8100e50 2023-10-10T17:45:10.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc819ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46028/0 (socket says 172.21.15.78:46028) 2023-10-10T17:45:10.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 -- 172.21.15.78:0/4118377162 learned_addr learned my addr 172.21.15.78:0/4118377162 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabc8100e50 0x7fabc81969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc659c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc81004b0 0x7fabc8196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 -- 172.21.15.78:0/4118377162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabc8100e50 msgr2=0x7fabc81969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabc8100e50 0x7fabc81969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 -- 172.21.15.78:0/4118377162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc81004b0 msgr2=0x7fabc8196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc81004b0 0x7fabc8196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.766+0000 7fabc6d9d700 1 -- 172.21.15.78:0/4118377162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc0009d00 con 0x7fabc81054f0 2023-10-10T17:45:10.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.767+0000 7fabc6d9d700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc819ae10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fabc0007870 tx=0x7fabc00078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.767+0000 7fabb77fe700 1 -- 172.21.15.78:0/4118377162 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc000e040 con 0x7fabc81054f0 2023-10-10T17:45:10.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.767+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc81102c0 con 0x7fabc81054f0 2023-10-10T17:45:10.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.767+0000 7fabb77fe700 1 -- 172.21.15.78:0/4118377162 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabc0005070 con 0x7fabc81054f0 2023-10-10T17:45:10.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.767+0000 7fabb77fe700 1 -- 172.21.15.78:0/4118377162 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc001b6c0 con 0x7fabc81054f0 2023-10-10T17:45:10.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.767+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc8110130 con 0x7fabc81054f0 2023-10-10T17:45:10.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.769+0000 7fabb77fe700 1 -- 172.21.15.78:0/4118377162 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fabc00051e0 con 0x7fabc81054f0 2023-10-10T17:45:10.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.770+0000 7fabb77fe700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fabb00a12c0 0x7fabb00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.770+0000 7fabc659c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fabb00a12c0 0x7fabb00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.770+0000 7fabb77fe700 1 -- 172.21.15.78:0/4118377162 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fabc00ce0d0 con 0x7fabc81054f0 2023-10-10T17:45:10.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.770+0000 7fabc659c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fabb00a12c0 0x7fabb00a3770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009c60 tx=0x7fabb8009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 Processor -- start 2023-10-10T17:45:10.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 -- start start 2023-10-10T17:45:10.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f974f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f979370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f979d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f87a060 con 0x55844f974b60 2023-10-10T17:45:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f8cc8a0 con 0x55844f9798b0 2023-10-10T17:45:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f8c5d30 con 0x55844f975470 2023-10-10T17:45:10.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f974f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f974f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46032/0 (socket says 172.21.15.78:46032) 2023-10-10T17:45:10.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc659c700 1 -- 172.21.15.78:0/2163651233 learned_addr learned my addr 172.21.15.78:0/2163651233 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f979370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc6d9d700 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f979d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc659c700 1 -- 172.21.15.78:0/2163651233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 msgr2=0x55844f979370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc659c700 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f979370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.771+0000 7fabc659c700 1 -- 172.21.15.78:0/2163651233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 msgr2=0x55844f979d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.772+0000 7fabc659c700 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f979d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.772+0000 7fabc659c700 1 -- 172.21.15.78:0/2163651233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55844f747630 con 0x55844f974b60 2023-10-10T17:45:10.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.772+0000 7fabc659c700 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f974f30 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fabb800e4e0 tx=0x7fabb8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeaeaebb61eb3512 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.772+0000 7fabb4ff9700 1 -- 172.21.15.78:0/2163651233 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb800ee00 con 0x55844f974b60 2023-10-10T17:45:10.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.772+0000 7fabb4ff9700 1 -- 172.21.15.78:0/2163651233 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabb8021070 con 0x55844f974b60 2023-10-10T17:45:10.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.772+0000 7fabb4ff9700 1 -- 172.21.15.78:0/2163651233 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb801d7b0 con 0x55844f974b60 2023-10-10T17:45:10.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/2163651233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 msgr2=0x55844f974f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f974f30 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fabb800e4e0 tx=0x7fabb8014000 comp rx=0 tx=0).stop 2023-10-10T17:45:10.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/2163651233 shutdown_connections 2023-10-10T17:45:10.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f979d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f974f30 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/2163651233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f979370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/2163651233 >> 172.21.15.78:0/2163651233 conn(0x55844f96a360 msgr2=0x55844f96fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/2163651233 shutdown_connections 2023-10-10T17:45:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/2163651233 wait complete. 2023-10-10T17:45:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- start start 2023-10-10T17:45:10.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 Processor -- start 2023-10-10T17:45:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f9810c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f975470 0x55844f97e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f9798b0 0x55844f97ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f8678e0 con 0x55844f974b60 2023-10-10T17:45:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f867580 con 0x55844f975470 2023-10-10T17:45:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.773+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f749920 con 0x55844f9798b0 2023-10-10T17:45:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabd8ba8b80 10 client.0 _ll_register_callbacks cb 0x55844f885780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:10.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f9810c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f9810c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46038/0 (socket says 172.21.15.78:46038) 2023-10-10T17:45:10.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 -- 172.21.15.78:0/1587868006 learned_addr learned my addr 172.21.15.78:0/1587868006 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f975470 0x55844f97e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc6d9d700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f9798b0 0x55844f97ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 -- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f9798b0 msgr2=0x55844f97ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f9798b0 0x55844f97ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 -- 172.21.15.78:0/1587868006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f975470 msgr2=0x55844f97e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f975470 0x55844f97e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 -- 172.21.15.78:0/1587868006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb8017070 con 0x55844f974b60 2023-10-10T17:45:10.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc6d9d700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f9798b0 0x55844f97ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:45:10.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabc659c700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f9810c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fabb800e4b0 tx=0x7fabb800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a042997c67d3ab0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabadffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:10.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabadffb700 1 -- 172.21.15.78:0/1587868006 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8005410 con 0x55844f974b60 2023-10-10T17:45:10.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabadffb700 1 -- 172.21.15.78:0/1587868006 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fabb8005570 con 0x55844f974b60 2023-10-10T17:45:10.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55844f967b20 con 0x55844f974b60 2023-10-10T17:45:10.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabadffb700 1 -- 172.21.15.78:0/1587868006 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabb8003680 con 0x55844f974b60 2023-10-10T17:45:10.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabd8ba8b80 10 client.16254 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:10.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.774+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55844f88c190 con 0x55844f974b60 2023-10-10T17:45:10.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabadffb700 1 -- 172.21.15.78:0/1587868006 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55844f967b20 con 0x55844f974b60 2023-10-10T17:45:10.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabadffb700 1 -- 172.21.15.78:0/1587868006 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fabb801c880 con 0x55844f974b60 2023-10-10T17:45:10.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabd8ba8b80 10 client.16254 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:10.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabd8ba8b80 10 client.16254 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:45:10.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabd8ba8b80 20 client.16254 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:10.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f98e800 0x55844f990c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabd8ba8b80 4 client.16254 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:10.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55844f8880a0 con 0x55844f98e800 2023-10-10T17:45:10.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f98e800 0x55844f990c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabadffb700 10 client.16254 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:45:10.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.775+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f98e800 0x55844f990c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60a1e1c6c559196 server_cookie=8b5f01b8675a4356 in_seq=0 out_seq=0 2023-10-10T17:45:10.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.776+0000 7fabadffb700 1 -- 172.21.15.78:0/1587868006 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55844f8880a0 con 0x55844f98e800 2023-10-10T17:45:10.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.776+0000 7fabadffb700 10 client.16254 handle_command_reply: tid=0 2023-10-10T17:45:10.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.776+0000 7fabd8ba8b80 1 client.16254 shutdown 2023-10-10T17:45:10.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.776+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 msgr2=0x55844f9810c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.776+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f9810c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fabb800e4b0 tx=0x7fabb800edf0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 shutdown_connections 2023-10-10T17:45:10.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f98e800 0x55844f990c10 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f9798b0 0x55844f97ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f9810c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/1587868006 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f975470 0x55844f97e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 >> 172.21.15.78:0/1587868006 conn(0x55844f96a360 msgr2=0x55844f97ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabadffb700 0 client.16254 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:45:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 shutdown_connections 2023-10-10T17:45:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/1587868006 wait complete. 2023-10-10T17:45:10.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.777+0000 7fabd8ba8b80 20 client.16254 trim_cache size 0 max 16384 2023-10-10T17:45:10.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.789+0000 7fabd8ba8b80 1 Processor -- start 2023-10-10T17:45:10.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabd8ba8b80 1 -- start start 2023-10-10T17:45:10.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f974b60 0x55844f9771a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f975470 0x55844f980bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f977a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f981770 con 0x55844f975470 2023-10-10T17:45:10.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f98a860 con 0x55844f9798b0 2023-10-10T17:45:10.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f97c8a0 con 0x55844f974b60 2023-10-10T17:45:10.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f975470 0x55844f980bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f975470 0x55844f980bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46048/0 (socket says 172.21.15.78:46048) 2023-10-10T17:45:10.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 -- 172.21.15.78:0/4166722119 learned_addr learned my addr 172.21.15.78:0/4166722119 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc659c700 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f974b60 0x55844f9771a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 -- 172.21.15.78:0/4166722119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f974b60 msgr2=0x55844f9771a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f974b60 0x55844f9771a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 -- 172.21.15.78:0/4166722119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 msgr2=0x55844f977a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f977a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 -- 172.21.15.78:0/4166722119 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55844f96bb00 con 0x55844f975470 2023-10-10T17:45:10.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc659c700 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f974b60 0x55844f9771a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:45:10.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f975470 0x55844f980bf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fabbc000c00 tx=0x7fabbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bca7e2687173d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabb57fa700 1 -- 172.21.15.78:0/4166722119 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc015040 con 0x55844f975470 2023-10-10T17:45:10.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabb57fa700 1 -- 172.21.15.78:0/4166722119 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabbc016040 con 0x55844f975470 2023-10-10T17:45:10.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.790+0000 7fabb57fa700 1 -- 172.21.15.78:0/4166722119 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc004cf0 con 0x55844f975470 2023-10-10T17:45:10.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/4166722119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f975470 msgr2=0x55844f980bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f975470 0x55844f980bf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fabbc000c00 tx=0x7fabbc004750 comp rx=0 tx=0).stop 2023-10-10T17:45:10.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/4166722119 shutdown_connections 2023-10-10T17:45:10.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f974b60 0x55844f9771a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f977a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/4166722119 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f975470 0x55844f980bf0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/4166722119 >> 172.21.15.78:0/4166722119 conn(0x55844f96a360 msgr2=0x55844f98a150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/4166722119 shutdown_connections 2023-10-10T17:45:10.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/4166722119 wait complete. 2023-10-10T17:45:10.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- start start 2023-10-10T17:45:10.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 Processor -- start 2023-10-10T17:45:10.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f983990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f983ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f984410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f9e16c0 con 0x55844f974b60 2023-10-10T17:45:10.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.791+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f9e1800 con 0x55844f9798b0 2023-10-10T17:45:10.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabd8ba8b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55844f9e1970 con 0x55844f975470 2023-10-10T17:45:10.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabd8ba8b80 10 client.0 _ll_register_callbacks cb 0x7fabc00057c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:10.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f983990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f983990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46052/0 (socket says 172.21.15.78:46052) 2023-10-10T17:45:10.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 -- 172.21.15.78:0/288796669 learned_addr learned my addr 172.21.15.78:0/288796669 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f983ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc6d9d700 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f984410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 -- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 msgr2=0x55844f983ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f983ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 -- 172.21.15.78:0/288796669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 msgr2=0x55844f984410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f984410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 -- 172.21.15.78:0/288796669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabbc0227f0 con 0x55844f974b60 2023-10-10T17:45:10.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabc659c700 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f983990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fabb8016040 tx=0x7fabb80264a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f72853b914a5462 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabadffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:10.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabadffb700 1 -- 172.21.15.78:0/288796669 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8005620 con 0x55844f974b60 2023-10-10T17:45:10.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabadffb700 1 -- 172.21.15.78:0/288796669 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fabb801c410 con 0x55844f974b60 2023-10-10T17:45:10.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabb8021070 con 0x55844f974b60 2023-10-10T17:45:10.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabadffb700 1 -- 172.21.15.78:0/288796669 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabb801c5d0 con 0x55844f974b60 2023-10-10T17:45:10.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabd8ba8b80 10 client.16260 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:10.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.792+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55844f6f3de0 con 0x55844f974b60 2023-10-10T17:45:10.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabadffb700 1 -- 172.21.15.78:0/288796669 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fabb8048020 con 0x55844f974b60 2023-10-10T17:45:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabadffb700 1 -- 172.21.15.78:0/288796669 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fabb800e7f0 con 0x55844f974b60 2023-10-10T17:45:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 10 client.16260 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 10 client.16260 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:45:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 20 client.16260 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f9e2d80 0x55844f9e5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 4 client.16260 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:45:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55844f894660 con 0x55844f9e2d80 2023-10-10T17:45:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f9e2d80 0x55844f9e5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabadffb700 10 client.16260 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:45:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabc5d9b700 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f9e2d80 0x55844f9e5190 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7a657f11515e755a server_cookie=1f6607b1496e6fab in_seq=0 out_seq=0 2023-10-10T17:45:10.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabadffb700 1 -- 172.21.15.78:0/288796669 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55844f894660 con 0x55844f9e2d80 2023-10-10T17:45:10.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabadffb700 10 client.16260 handle_command_reply: tid=0 2023-10-10T17:45:10.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 1 client.16260 shutdown 2023-10-10T17:45:10.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 msgr2=0x55844f983990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.793+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f983990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fabb8016040 tx=0x7fabb80264a0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 shutdown_connections 2023-10-10T17:45:10.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55844f9798b0 0x55844f984410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55844f974b60 0x55844f983990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55844f9e2d80 0x55844f9e5190 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 --2- 172.21.15.78:0/288796669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55844f975470 0x55844f983ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabadffb700 0 client.16260 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:45:10.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 >> 172.21.15.78:0/288796669 conn(0x55844f96a360 msgr2=0x55844f977a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 shutdown_connections 2023-10-10T17:45:10.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 1 -- 172.21.15.78:0/288796669 wait complete. 2023-10-10T17:45:10.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabd8ba8b80 20 client.16260 trim_cache size 0 max 16384 2023-10-10T17:45:10.824 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:45:10.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fabb00a12c0 msgr2=0x7fabb00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fabb00a12c0 0x7fabb00a3770 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009c60 tx=0x7fabb8009260 comp rx=0 tx=0).stop 2023-10-10T17:45:10.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 msgr2=0x7fabc819ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc819ae10 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fabc0007870 tx=0x7fabc00078a0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 shutdown_connections 2023-10-10T17:45:10.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fabb00a12c0 0x7fabb00a3770 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fabc81054f0 0x7fabc819ae10 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fabc81004b0 0x7fabc8196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 --2- 172.21.15.78:0/4118377162 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fabc8100e50 0x7fabc81969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 >> 172.21.15.78:0/4118377162 conn(0x7fabc80fbd20 msgr2=0x7fabc80fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 shutdown_connections 2023-10-10T17:45:10.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.794+0000 7fabcca1c700 1 -- 172.21.15.78:0/4118377162 wait complete. 2023-10-10T17:45:10.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:45:10.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 -- 172.21.15.78:0/2916572491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f4106b00 msgr2=0x7fa5f4106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 --2- 172.21.15.78:0/2916572491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f4106b00 0x7fa5f4106f70 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009990 tx=0x7fa5e4009ca0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 -- 172.21.15.78:0/2916572491 shutdown_connections 2023-10-10T17:45:10.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 --2- 172.21.15.78:0/2916572491 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f410c940 0x7fa5f410cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 --2- 172.21.15.78:0/2916572491 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f41074b0 0x7fa5f4113770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 --2- 172.21.15.78:0/2916572491 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f4106b00 0x7fa5f4106f70 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 -- 172.21.15.78:0/2916572491 >> 172.21.15.78:0/2916572491 conn(0x7fa5f4073dd0 msgr2=0x7fa5f40761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 -- 172.21.15.78:0/2916572491 shutdown_connections 2023-10-10T17:45:10.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.971+0000 7fa5fb821700 1 -- 172.21.15.78:0/2916572491 wait complete. 2023-10-10T17:45:10.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 Processor -- start 2023-10-10T17:45:10.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 -- start start 2023-10-10T17:45:10.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f4106b00 0x7fa5f41a2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f41074b0 0x7fa5f41a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f410c940 0x7fa5f419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f419cdf0 con 0x7fa5f410c940 2023-10-10T17:45:10.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f419cf60 con 0x7fa5f41074b0 2023-10-10T17:45:10.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5fb821700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f419d120 con 0x7fa5f4106b00 2023-10-10T17:45:10.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5f9dbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f410c940 0x7fa5f419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5f9dbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f410c940 0x7fa5f419c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46070/0 (socket says 172.21.15.78:46070) 2023-10-10T17:45:10.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5f8dbc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f41074b0 0x7fa5f41a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/3698789487 learned_addr learned my addr 172.21.15.78:0/3698789487 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.972+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f4106b00 0x7fa5f41a2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/3698789487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f4106b00 msgr2=0x7fa5f41a2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f4106b00 0x7fa5f41a2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/3698789487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f41074b0 msgr2=0x7fa5f41a2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f41074b0 0x7fa5f41a2c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/3698789487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009c40 con 0x7fa5f410c940 2023-10-10T17:45:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f41074b0 0x7fa5f41a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f410c940 0x7fa5f419c780 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c370 tx=0x7fa5ec00c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5ea7fc700 1 -- 172.21.15.78:0/3698789487 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec017070 con 0x7fa5f410c940 2023-10-10T17:45:10.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e40095f0 con 0x7fa5f410c940 2023-10-10T17:45:10.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5ea7fc700 1 -- 172.21.15.78:0/3698789487 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5ec00f040 con 0x7fa5f410c940 2023-10-10T17:45:10.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5ea7fc700 1 -- 172.21.15.78:0/3698789487 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00e050 con 0x7fa5f410c940 2023-10-10T17:45:10.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.973+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4116990 con 0x7fa5f410c940 2023-10-10T17:45:10.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.975+0000 7fa5ea7fc700 1 -- 172.21.15.78:0/3698789487 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa5ec00cb60 con 0x7fa5f410c940 2023-10-10T17:45:10.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.976+0000 7fa5ea7fc700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5e00a12c0 0x7fa5e00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.976+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5e00a12c0 0x7fa5e00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa5ea7fc700 1 -- 172.21.15.78:0/3698789487 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa5ec0cc9d0 con 0x7fa5f410c940 2023-10-10T17:45:10.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5e00a12c0 0x7fa5e00a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4077ed0 tx=0x7fa5f0009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 Processor -- start 2023-10-10T17:45:10.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 -- start start 2023-10-10T17:45:10.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c608f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c60dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c50e060 con 0x562e7c60d8b0 2023-10-10T17:45:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c5608a0 con 0x562e7c609470 2023-10-10T17:45:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.977+0000 7fa6079adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c559d30 con 0x562e7c608b60 2023-10-10T17:45:10.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c608f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c60dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c608f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37604/0 (socket says 172.21.15.78:37604) 2023-10-10T17:45:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c60dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46078/0 (socket says 172.21.15.78:46078) 2023-10-10T17:45:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f95bd700 1 -- 172.21.15.78:0/3638280321 learned_addr learned my addr 172.21.15.78:0/3638280321 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/3638280321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 msgr2=0x562e7c608f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c608f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/3638280321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 msgr2=0x562e7c60d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/3638280321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e7c3db630 con 0x562e7c60d8b0 2023-10-10T17:45:10.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c608f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c60dd00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0c9df0 tx=0x7fa5ec0992b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa917cb4e0e27e7a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5df7fe700 1 -- 172.21.15.78:0/3638280321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec099770 con 0x562e7c60d8b0 2023-10-10T17:45:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5df7fe700 1 -- 172.21.15.78:0/3638280321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5ec09d7a0 con 0x562e7c60d8b0 2023-10-10T17:45:10.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.978+0000 7fa5df7fe700 1 -- 172.21.15.78:0/3638280321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0999d0 con 0x562e7c60d8b0 2023-10-10T17:45:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 -- 172.21.15.78:0/3638280321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 msgr2=0x562e7c60dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c60dd00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0c9df0 tx=0x7fa5ec0992b0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 -- 172.21.15.78:0/3638280321 shutdown_connections 2023-10-10T17:45:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c608f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c60dd00 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3638280321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 -- 172.21.15.78:0/3638280321 >> 172.21.15.78:0/3638280321 conn(0x562e7c5fe360 msgr2=0x562e7c603b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 -- 172.21.15.78:0/3638280321 shutdown_connections 2023-10-10T17:45:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 -- 172.21.15.78:0/3638280321 wait complete. 2023-10-10T17:45:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.979+0000 7fa6079adb80 1 -- start start 2023-10-10T17:45:10.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 1 Processor -- start 2023-10-10T17:45:10.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c612360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c6128a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c612de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c4fb8e0 con 0x562e7c608b60 2023-10-10T17:45:10.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c4fb580 con 0x562e7c60d8b0 2023-10-10T17:45:10.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c3dd920 con 0x562e7c609470 2023-10-10T17:45:10.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa6079adb80 10 client.0 _ll_register_callbacks cb 0x562e7c519780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:10.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c612360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c612360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46080/0 (socket says 172.21.15.78:46080) 2023-10-10T17:45:10.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f8dbc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c6128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 -- 172.21.15.78:0/3347329333 learned_addr learned my addr 172.21.15.78:0/3347329333 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c612de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 -- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 msgr2=0x562e7c6128a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c6128a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 -- 172.21.15.78:0/3347329333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 msgr2=0x562e7c612de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c612de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f95bd700 1 -- 172.21.15.78:0/3347329333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4116990 con 0x562e7c608b60 2023-10-10T17:45:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.980+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c6128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c612360 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa5f001dbd0 tx=0x7fa5f00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5e6a3aedaaabac server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:10.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa5dcff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa5dcff9700 1 -- 172.21.15.78:0/3347329333 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0022070 con 0x562e7c608b60 2023-10-10T17:45:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa5dcff9700 1 -- 172.21.15.78:0/3347329333 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa5f0004020 con 0x562e7c608b60 2023-10-10T17:45:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e7c5fbb20 con 0x562e7c608b60 2023-10-10T17:45:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa5dcff9700 1 -- 172.21.15.78:0/3347329333 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5f00041e0 con 0x562e7c608b60 2023-10-10T17:45:10.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa6079adb80 10 client.16272 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562e7c520190 con 0x562e7c608b60 2023-10-10T17:45:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.981+0000 7fa5dcff9700 1 -- 172.21.15.78:0/3347329333 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562e7c5fbb20 con 0x562e7c608b60 2023-10-10T17:45:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa5dcff9700 1 -- 172.21.15.78:0/3347329333 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fa5f0003840 con 0x562e7c608b60 2023-10-10T17:45:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa6079adb80 10 client.16272 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa6079adb80 10 client.16272 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:45:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa6079adb80 20 client.16272 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c622980 0x562e7c624d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa6079adb80 4 client.16272 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e7c51c0a0 con 0x562e7c622980 2023-10-10T17:45:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c622980 0x562e7c624d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:10.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c622980 0x562e7c624d90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=95020d471f658fe9 server_cookie=9364babba6ae4f6e in_seq=0 out_seq=0 2023-10-10T17:45:11.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.982+0000 7fa5dcff9700 10 client.16272 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:45:11.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa5dcff9700 1 -- 172.21.15.78:0/3347329333 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562e7c51c0a0 con 0x562e7c622980 2023-10-10T17:45:11.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa5dcff9700 10 client.16272 handle_command_reply: tid=0 2023-10-10T17:45:11.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 client.16272 shutdown 2023-10-10T17:45:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 msgr2=0x562e7c612360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c612360 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa5f001dbd0 tx=0x7fa5f00047c0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 shutdown_connections 2023-10-10T17:45:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c612de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c612360 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c622980 0x562e7c624d90 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 --2- 172.21.15.78:0/3347329333 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c6128a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa5dcff9700 0 client.16272 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:45:11.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 >> 172.21.15.78:0/3347329333 conn(0x562e7c5fe360 msgr2=0x562e7c60ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 shutdown_connections 2023-10-10T17:45:11.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 1 -- 172.21.15.78:0/3347329333 wait complete. 2023-10-10T17:45:11.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:10.986+0000 7fa6079adb80 20 client.16272 trim_cache size 0 max 16384 2023-10-10T17:45:11.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.000+0000 7fa6079adb80 1 Processor -- start 2023-10-10T17:45:11.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.000+0000 7fa6079adb80 1 -- start start 2023-10-10T17:45:11.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c619940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c60c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c60ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa6079adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c60f3c0 con 0x562e7c608b60 2023-10-10T17:45:11.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa6079adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c602c00 con 0x562e7c60d8b0 2023-10-10T17:45:11.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa6079adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c60e190 con 0x562e7c609470 2023-10-10T17:45:11.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c619940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c619940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46088/0 (socket says 172.21.15.78:46088) 2023-10-10T17:45:11.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 -- 172.21.15.78:0/1761518219 learned_addr learned my addr 172.21.15.78:0/1761518219 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:11.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c60c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c60ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 -- 172.21.15.78:0/1761518219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 msgr2=0x562e7c60c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c60c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 -- 172.21.15.78:0/1761518219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 msgr2=0x562e7c60ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c60ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 -- 172.21.15.78:0/1761518219 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e7c5ffb00 con 0x562e7c608b60 2023-10-10T17:45:11.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c60c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:11.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c619940 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0003c70 tx=0x7fa5f0005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d405d54045da7ff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5dffff700 1 -- 172.21.15.78:0/1761518219 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0022070 con 0x562e7c608b60 2023-10-10T17:45:11.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5dffff700 1 -- 172.21.15.78:0/1761518219 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5f00042d0 con 0x562e7c608b60 2023-10-10T17:45:11.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.001+0000 7fa5dffff700 1 -- 172.21.15.78:0/1761518219 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f001c6e0 con 0x562e7c608b60 2023-10-10T17:45:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 -- 172.21.15.78:0/1761518219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 msgr2=0x562e7c619940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c619940 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0003c70 tx=0x7fa5f0005840 comp rx=0 tx=0).stop 2023-10-10T17:45:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 -- 172.21.15.78:0/1761518219 shutdown_connections 2023-10-10T17:45:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c60d8b0 0x562e7c60ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c608b60 0x562e7c619940 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1761518219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c609470 0x562e7c60c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 -- 172.21.15.78:0/1761518219 >> 172.21.15.78:0/1761518219 conn(0x562e7c5fe360 msgr2=0x562e7c60b970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 -- 172.21.15.78:0/1761518219 shutdown_connections 2023-10-10T17:45:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.002+0000 7fa6079adb80 1 -- 172.21.15.78:0/1761518219 wait complete. 2023-10-10T17:45:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 -- start start 2023-10-10T17:45:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 Processor -- start 2023-10-10T17:45:11.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c619b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c6101e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c6108e0 con 0x562e7c60d8b0 2023-10-10T17:45:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c611ba0 con 0x562e7c609470 2023-10-10T17:45:11.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e7c611d60 con 0x562e7c608b60 2023-10-10T17:45:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa6079adb80 10 client.0 _ll_register_callbacks cb 0x562e7c519780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c619b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa5f9dbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c6101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa5f9dbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c6101e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46098/0 (socket says 172.21.15.78:46098) 2023-10-10T17:45:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa5f95bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c619b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37624/0 (socket says 172.21.15.78:37624) 2023-10-10T17:45:11.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/1565715671 learned_addr learned my addr 172.21.15.78:0/1565715671 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.003+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/1565715671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 msgr2=0x562e7c619b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c619b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/1565715671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 msgr2=0x562e7c60fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5f9dbe700 1 -- 172.21.15.78:0/1565715671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f4114070 con 0x562e7c60d8b0 2023-10-10T17:45:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5f8dbc700 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5f9dbe700 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c6101e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec099470 tx=0x7fa5ec095be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd065742827fa384 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5dcff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:11.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5dcff9700 1 -- 172.21.15.78:0/1565715671 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec099770 con 0x562e7c60d8b0 2023-10-10T17:45:11.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5dcff9700 1 -- 172.21.15.78:0/1565715671 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa5ec0a1bc0 con 0x562e7c60d8b0 2023-10-10T17:45:11.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5ec09d7a0 con 0x562e7c60d8b0 2023-10-10T17:45:11.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5dcff9700 1 -- 172.21.15.78:0/1565715671 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5ec0b9110 con 0x562e7c60d8b0 2023-10-10T17:45:11.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa6079adb80 10 client.16278 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:11.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x562e7c520190 con 0x562e7c60d8b0 2023-10-10T17:45:11.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5dcff9700 1 -- 172.21.15.78:0/1565715671 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5ec09d7a0 con 0x562e7c60d8b0 2023-10-10T17:45:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.004+0000 7fa5dcff9700 1 -- 172.21.15.78:0/1565715671 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fa5ec004bd0 con 0x562e7c60d8b0 2023-10-10T17:45:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa6079adb80 10 client.16278 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa6079adb80 10 client.16278 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:45:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa6079adb80 20 client.16278 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c676c90 0x562e7c679070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa6079adb80 4 client.16278 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:45:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562e7c528660 con 0x562e7c676c90 2023-10-10T17:45:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c676c90 0x562e7c679070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa5dcff9700 10 client.16278 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:45:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.005+0000 7fa5f95bd700 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c676c90 0x562e7c679070 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fa9cd2e2b148d7c server_cookie=ebc2acfc44ceb8cf in_seq=0 out_seq=0 2023-10-10T17:45:11.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa5dcff9700 1 -- 172.21.15.78:0/1565715671 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562e7c528660 con 0x562e7c676c90 2023-10-10T17:45:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa5dcff9700 10 client.16278 handle_command_reply: tid=0 2023-10-10T17:45:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 client.16278 shutdown 2023-10-10T17:45:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 msgr2=0x562e7c6101e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c6101e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec099470 tx=0x7fa5ec095be0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 shutdown_connections 2023-10-10T17:45:11.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x562e7c676c90 0x562e7c679070 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562e7c608b60 0x562e7c619b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562e7c60d8b0 0x562e7c6101e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa5dcff9700 0 client.16278 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:45:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.006+0000 7fa6079adb80 1 --2- 172.21.15.78:0/1565715671 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562e7c609470 0x562e7c60fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 >> 172.21.15.78:0/1565715671 conn(0x562e7c5fe360 msgr2=0x562e7c60c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 shutdown_connections 2023-10-10T17:45:11.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa6079adb80 1 -- 172.21.15.78:0/1565715671 wait complete. 2023-10-10T17:45:11.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa6079adb80 20 client.16278 trim_cache size 0 max 16384 2023-10-10T17:45:11.024 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:45:11.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5e00a12c0 msgr2=0x7fa5e00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa5fb821700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5e00a12c0 0x7fa5e00a3770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4077ed0 tx=0x7fa5f0009260 comp rx=0 tx=0).stop 2023-10-10T17:45:11.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f410c940 msgr2=0x7fa5f419c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.007+0000 7fa5fb821700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f410c940 0x7fa5f419c780 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00c370 tx=0x7fa5ec00c680 comp rx=0 tx=0).stop 2023-10-10T17:45:11.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 shutdown_connections 2023-10-10T17:45:11.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f4106b00 0x7fa5f41a2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5e00a12c0 0x7fa5e00a3770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f410c940 0x7fa5f419c780 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 --2- 172.21.15.78:0/3698789487 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f41074b0 0x7fa5f41a2c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 >> 172.21.15.78:0/3698789487 conn(0x7fa5f4073dd0 msgr2=0x7fa5f410d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 shutdown_connections 2023-10-10T17:45:11.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.008+0000 7fa5fb821700 1 -- 172.21.15.78:0/3698789487 wait complete. 2023-10-10T17:45:11.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:10 smithi078 ceph-mon[99494]: pgmap v82: 129 pgs: 129 active+clean; 45 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2023-10-10T17:45:11.028 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:10 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3605931439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:45:11.029 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:45:11.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:10 smithi190 ceph-mon[101220]: pgmap v82: 129 pgs: 129 active+clean; 45 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2023-10-10T17:45:11.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:10 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3605931439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:45:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 -- 172.21.15.78:0/3036244000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14073720 msgr2=0x7fdb14110e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 --2- 172.21.15.78:0/3036244000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14073720 0x7fdb14110e50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fdb00009990 tx=0x7fdb00009ca0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 -- 172.21.15.78:0/3036244000 shutdown_connections 2023-10-10T17:45:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 --2- 172.21.15.78:0/3036244000 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14111390 0x7fdb14113770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 --2- 172.21.15.78:0/3036244000 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb14072e10 0x7fdb140731e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 --2- 172.21.15.78:0/3036244000 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14073720 0x7fdb14110e50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 -- 172.21.15.78:0/3036244000 >> 172.21.15.78:0/3036244000 conn(0x7fdb140770e0 msgr2=0x7fdb140774e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 -- 172.21.15.78:0/3036244000 shutdown_connections 2023-10-10T17:45:11.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.164+0000 7fdb1acea700 1 -- 172.21.15.78:0/3036244000 wait complete. 2023-10-10T17:45:11.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 Processor -- start 2023-10-10T17:45:11.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 -- start start 2023-10-10T17:45:11.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14072e10 0x7fdb141a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb14073720 0x7fdb141a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14111390 0x7fdb1419c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419cec0 con 0x7fdb14111390 2023-10-10T17:45:11.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419d030 con 0x7fdb14073720 2023-10-10T17:45:11.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb1acea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1419d1f0 con 0x7fdb14072e10 2023-10-10T17:45:11.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb18a86700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14072e10 0x7fdb141a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb19287700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14111390 0x7fdb1419c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb18a86700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14072e10 0x7fdb141a27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37638/0 (socket says 172.21.15.78:37638) 2023-10-10T17:45:11.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb18a86700 1 -- 172.21.15.78:0/2897179932 learned_addr learned my addr 172.21.15.78:0/2897179932 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:11.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb19287700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14111390 0x7fdb1419c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46112/0 (socket says 172.21.15.78:46112) 2023-10-10T17:45:11.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb13fff700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb14073720 0x7fdb141a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb19287700 1 -- 172.21.15.78:0/2897179932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14072e10 msgr2=0x7fdb141a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb19287700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14072e10 0x7fdb141a27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb19287700 1 -- 172.21.15.78:0/2897179932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb14073720 msgr2=0x7fdb141a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb19287700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb14073720 0x7fdb141a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb19287700 1 -- 172.21.15.78:0/2897179932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb000095f0 con 0x7fdb14111390 2023-10-10T17:45:11.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.165+0000 7fdb18a86700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14072e10 0x7fdb141a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:11.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.166+0000 7fdb19287700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14111390 0x7fdb1419c850 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00ef90 tx=0x7fdb0c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.166+0000 7fdb11ffb700 1 -- 172.21.15.78:0/2897179932 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c018070 con 0x7fdb14111390 2023-10-10T17:45:11.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.166+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb141169a0 con 0x7fdb14111390 2023-10-10T17:45:11.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.166+0000 7fdb11ffb700 1 -- 172.21.15.78:0/2897179932 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb0c00f040 con 0x7fdb14111390 2023-10-10T17:45:11.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.166+0000 7fdb11ffb700 1 -- 172.21.15.78:0/2897179932 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c00cc30 con 0x7fdb14111390 2023-10-10T17:45:11.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.166+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14116810 con 0x7fdb14111390 2023-10-10T17:45:11.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.169+0000 7fdb11ffb700 1 -- 172.21.15.78:0/2897179932 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fdb0c003a00 con 0x7fdb14111390 2023-10-10T17:45:11.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.170+0000 7fdb11ffb700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb040a12c0 0x7fdb040a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.170+0000 7fdb18a86700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb040a12c0 0x7fdb040a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.170+0000 7fdb11ffb700 1 -- 172.21.15.78:0/2897179932 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fdb0c0ccb70 con 0x7fdb14111390 2023-10-10T17:45:11.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.170+0000 7fdb18a86700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb040a12c0 0x7fdb040a3770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdb08011550 tx=0x7fdb08009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.170+0000 7fdb26e76b80 1 Processor -- start 2023-10-10T17:45:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.170+0000 7fdb26e76b80 1 -- start start 2023-10-10T17:45:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c5b60 0x560f1e5c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c6470 0x560f1e5ca370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e4cb060 con 0x560f1e5ca8b0 2023-10-10T17:45:11.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e51d8a0 con 0x560f1e5c6470 2023-10-10T17:45:11.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e516d30 con 0x560f1e5c5b60 2023-10-10T17:45:11.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c5b60 0x560f1e5c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb19287700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c5b60 0x560f1e5c5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37646/0 (socket says 172.21.15.78:37646) 2023-10-10T17:45:11.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 -- 172.21.15.78:0/3607325754 learned_addr learned my addr 172.21.15.78:0/3607325754 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:11.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb13fff700 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c6470 0x560f1e5ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 -- 172.21.15.78:0/3607325754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c6470 msgr2=0x560f1e5ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c6470 0x560f1e5ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 -- 172.21.15.78:0/3607325754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 msgr2=0x560f1e5cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 -- 172.21.15.78:0/3607325754 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f1e398630 con 0x560f1e5c5b60 2023-10-10T17:45:11.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb19287700 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:11.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb13fff700 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c6470 0x560f1e5ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:11.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.171+0000 7fdb18a86700 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c5b60 0x560f1e5c5f30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009880 tx=0x7fdb08014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1b631f9da40c7de server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdafeffd700 1 -- 172.21.15.78:0/3607325754 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb08026630 con 0x560f1e5c5b60 2023-10-10T17:45:11.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdafeffd700 1 -- 172.21.15.78:0/3607325754 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb08021070 con 0x560f1e5c5b60 2023-10-10T17:45:11.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdafeffd700 1 -- 172.21.15.78:0/3607325754 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0801d670 con 0x560f1e5c5b60 2023-10-10T17:45:11.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3607325754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c5b60 msgr2=0x560f1e5c5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c5b60 0x560f1e5c5f30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdb08009880 tx=0x7fdb08014000 comp rx=0 tx=0).stop 2023-10-10T17:45:11.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3607325754 shutdown_connections 2023-10-10T17:45:11.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c5b60 0x560f1e5c5f30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3607325754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c6470 0x560f1e5ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.172+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3607325754 >> 172.21.15.78:0/3607325754 conn(0x560f1e5bb360 msgr2=0x560f1e5c0b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.173+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3607325754 shutdown_connections 2023-10-10T17:45:11.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.173+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3607325754 wait complete. 2023-10-10T17:45:11.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.173+0000 7fdb26e76b80 1 -- start start 2023-10-10T17:45:11.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.173+0000 7fdb26e76b80 1 Processor -- start 2023-10-10T17:45:11.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e4b88e0 con 0x560f1e5c6470 2023-10-10T17:45:11.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e4b8580 con 0x560f1e5c5b60 2023-10-10T17:45:11.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e39a920 con 0x560f1e5ca8b0 2023-10-10T17:45:11.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb26e76b80 10 client.0 _ll_register_callbacks cb 0x560f1e4d6780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:11.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46124/0 (socket says 172.21.15.78:46124) 2023-10-10T17:45:11.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 -- 172.21.15.78:0/75455405 learned_addr learned my addr 172.21.15.78:0/75455405 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:11.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb19287700 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb18a86700 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 -- 172.21.15.78:0/75455405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 msgr2=0x560f1e5cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cfac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 -- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 msgr2=0x560f1e5d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 -- 172.21.15.78:0/75455405 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb08017070 con 0x560f1e5c6470 2023-10-10T17:45:11.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdb13fff700 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cf380 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdb00015000 tx=0x7fdb00005f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8510e7341ec1ff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdaf3fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:11.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.174+0000 7fdaf3fff700 1 -- 172.21.15.78:0/75455405 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0001d070 con 0x560f1e5c6470 2023-10-10T17:45:11.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdaf3fff700 1 -- 172.21.15.78:0/75455405 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fdb00004500 con 0x560f1e5c6470 2023-10-10T17:45:11.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560f1e5b8b20 con 0x560f1e5c6470 2023-10-10T17:45:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdaf3fff700 1 -- 172.21.15.78:0/75455405 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb0002d690 con 0x560f1e5c6470 2023-10-10T17:45:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 10 client.16290 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560f1e4dd190 con 0x560f1e5c6470 2023-10-10T17:45:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdaf3fff700 1 -- 172.21.15.78:0/75455405 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x560f1e5b8b20 con 0x560f1e5c6470 2023-10-10T17:45:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdaf3fff700 1 -- 172.21.15.78:0/75455405 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdb00004750 con 0x560f1e5c6470 2023-10-10T17:45:11.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 10 client.16290 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 10 client.16290 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 20 client.16290 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e5df900 0x560f1e5e1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 4 client.16290 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f1e4d90a0 con 0x560f1e5df900 2023-10-10T17:45:11.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.175+0000 7fdb18a86700 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e5df900 0x560f1e5e1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.176+0000 7fdaf3fff700 10 client.16290 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.176+0000 7fdb18a86700 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e5df900 0x560f1e5e1d10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f520ab36933510 server_cookie=593faae560dc7ab in_seq=0 out_seq=0 2023-10-10T17:45:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.177+0000 7fdaf3fff700 1 -- 172.21.15.78:0/75455405 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x560f1e4d90a0 con 0x560f1e5df900 2023-10-10T17:45:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.177+0000 7fdaf3fff700 10 client.16290 handle_command_reply: tid=0 2023-10-10T17:45:11.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.177+0000 7fdb26e76b80 1 client.16290 shutdown 2023-10-10T17:45:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.177+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 msgr2=0x560f1e5cf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.177+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cf380 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fdb00015000 tx=0x7fdb00005f00 comp rx=0 tx=0).stop 2023-10-10T17:45:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.177+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 shutdown_connections 2023-10-10T17:45:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cfac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e5df900 0x560f1e5e1d10 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdaf3fff700 0 client.16290 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/75455405 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cf380 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 >> 172.21.15.78:0/75455405 conn(0x560f1e5bb360 msgr2=0x560f1e5cbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 shutdown_connections 2023-10-10T17:45:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 1 -- 172.21.15.78:0/75455405 wait complete. 2023-10-10T17:45:11.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.178+0000 7fdb26e76b80 20 client.16290 trim_cache size 0 max 16384 2023-10-10T17:45:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 Processor -- start 2023-10-10T17:45:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 -- start start 2023-10-10T17:45:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e5c9510 con 0x560f1e5c6470 2023-10-10T17:45:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e5d0810 con 0x560f1e5c5b60 2023-10-10T17:45:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e5cf6a0 con 0x560f1e5ca8b0 2023-10-10T17:45:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb13fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb13fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46132/0 (socket says 172.21.15.78:46132) 2023-10-10T17:45:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb13fff700 1 -- 172.21.15.78:0/3974512058 learned_addr learned my addr 172.21.15.78:0/3974512058 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb18a86700 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb19287700 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.191+0000 7fdb13fff700 1 -- 172.21.15.78:0/3974512058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 msgr2=0x560f1e5cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdb13fff700 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdb13fff700 1 -- 172.21.15.78:0/3974512058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 msgr2=0x560f1e5d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdb13fff700 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdb13fff700 1 -- 172.21.15.78:0/3974512058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f1e4dcee0 con 0x560f1e5c6470 2023-10-10T17:45:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdb13fff700 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cec20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000b6f0 tx=0x7fdb00009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bf469ca06e3ebc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdaff7fe700 1 -- 172.21.15.78:0/3974512058 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0001d030 con 0x560f1e5c6470 2023-10-10T17:45:11.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdaff7fe700 1 -- 172.21.15.78:0/3974512058 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb0001e040 con 0x560f1e5c6470 2023-10-10T17:45:11.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.192+0000 7fdaff7fe700 1 -- 172.21.15.78:0/3974512058 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0000e7d0 con 0x560f1e5c6470 2023-10-10T17:45:11.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3974512058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 msgr2=0x560f1e5cec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cec20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000b6f0 tx=0x7fdb00009ca0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3974512058 shutdown_connections 2023-10-10T17:45:11.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5ca8b0 0x560f1e5cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/3974512058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5c6470 0x560f1e5cec20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3974512058 >> 172.21.15.78:0/3974512058 conn(0x560f1e5bb360 msgr2=0x560f1e5dbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3974512058 shutdown_connections 2023-10-10T17:45:11.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 -- 172.21.15.78:0/3974512058 wait complete. 2023-10-10T17:45:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 -- start start 2023-10-10T17:45:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 Processor -- start 2023-10-10T17:45:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.193+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c6470 0x560f1e5d6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e6327c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e5d6910 con 0x560f1e5ca8b0 2023-10-10T17:45:11.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e632d00 con 0x560f1e5c5b60 2023-10-10T17:45:11.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f1e632e40 con 0x560f1e5c6470 2023-10-10T17:45:11.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 10 client.0 _ll_register_callbacks cb 0x7fdb0000bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:11.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e6327c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e6327c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46140/0 (socket says 172.21.15.78:46140) 2023-10-10T17:45:11.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 -- 172.21.15.78:0/1855205407 learned_addr learned my addr 172.21.15.78:0/1855205407 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:11.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb13fff700 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c6470 0x560f1e5d6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb18a86700 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 -- 172.21.15.78:0/1855205407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c6470 msgr2=0x560f1e5d6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c6470 0x560f1e5d6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 -- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 msgr2=0x560f1e5d5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 -- 172.21.15.78:0/1855205407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f1e4dd190 con 0x560f1e5ca8b0 2023-10-10T17:45:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb19287700 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e6327c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c095720 tx=0x7fdb0c099300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3704575a9ff54b64 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:11.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdaf3fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:11.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdaf3fff700 1 -- 172.21.15.78:0/1855205407 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0c099e00 con 0x560f1e5ca8b0 2023-10-10T17:45:11.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb08021070 con 0x560f1e5ca8b0 2023-10-10T17:45:11.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdaf3fff700 1 -- 172.21.15.78:0/1855205407 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fdb0c09d9d0 con 0x560f1e5ca8b0 2023-10-10T17:45:11.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdaf3fff700 1 -- 172.21.15.78:0/1855205407 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb0c0a1d60 con 0x560f1e5ca8b0 2023-10-10T17:45:11.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 10 client.16296 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:11.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.194+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560f1e32dc60 con 0x560f1e5ca8b0 2023-10-10T17:45:11.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdaf3fff700 1 -- 172.21.15.78:0/1855205407 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb0c0c5020 con 0x560f1e5ca8b0 2023-10-10T17:45:11.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdaf3fff700 1 -- 172.21.15.78:0/1855205407 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdb0c0b90b0 con 0x560f1e5ca8b0 2023-10-10T17:45:11.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdb26e76b80 10 client.16296 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:11.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdb26e76b80 10 client.16296 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:11.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdb26e76b80 20 client.16296 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:11.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e6343c0 0x560f1e6367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:11.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdb26e76b80 4 client.16296 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:45:11.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560f1e4e5660 con 0x560f1e6343c0 2023-10-10T17:45:11.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.195+0000 7fdb18a86700 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e6343c0 0x560f1e6367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:11.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.196+0000 7fdaf3fff700 10 client.16296 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:11.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.196+0000 7fdb18a86700 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e6343c0 0x560f1e6367d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f8c3b99a109cd00 server_cookie=b5b751e955eaf195 in_seq=0 out_seq=0 2023-10-10T17:45:11.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdaf3fff700 1 -- 172.21.15.78:0/1855205407 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560f1e4e5660 con 0x560f1e6343c0 2023-10-10T17:45:11.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdaf3fff700 10 client.16296 handle_command_reply: tid=0 2023-10-10T17:45:11.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 client.16296 shutdown 2023-10-10T17:45:11.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 msgr2=0x560f1e6327c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e6327c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c095720 tx=0x7fdb0c099300 comp rx=0 tx=0).stop 2023-10-10T17:45:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 shutdown_connections 2023-10-10T17:45:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x560f1e6343c0 0x560f1e6367d0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x560f1e5ca8b0 0x560f1e6327c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x560f1e5c5b60 0x560f1e5d5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 --2- 172.21.15.78:0/1855205407 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x560f1e5c6470 0x560f1e5d6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 >> 172.21.15.78:0/1855205407 conn(0x560f1e5bb360 msgr2=0x560f1e5c9920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdaf3fff700 0 client.16296 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 shutdown_connections 2023-10-10T17:45:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 1 -- 172.21.15.78:0/1855205407 wait complete. 2023-10-10T17:45:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.197+0000 7fdb26e76b80 20 client.16296 trim_cache size 0 max 16384 2023-10-10T17:45:11.217 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:45:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb040a12c0 msgr2=0x7fdb040a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb040a12c0 0x7fdb040a3770 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdb08011550 tx=0x7fdb08009260 comp rx=0 tx=0).stop 2023-10-10T17:45:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14111390 msgr2=0x7fdb1419c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14111390 0x7fdb1419c850 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7fdb0c00ef90 tx=0x7fdb0c00c3a0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 shutdown_connections 2023-10-10T17:45:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb14072e10 0x7fdb141a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb040a12c0 0x7fdb040a3770 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb14111390 0x7fdb1419c850 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 --2- 172.21.15.78:0/2897179932 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb14073720 0x7fdb141a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 >> 172.21.15.78:0/2897179932 conn(0x7fdb140770e0 msgr2=0x7fdb140fee00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 shutdown_connections 2023-10-10T17:45:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:11.198+0000 7fdb1acea700 1 -- 172.21.15.78:0/2897179932 wait complete. 2023-10-10T17:45:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:10 smithi157 ceph-mon[100896]: pgmap v82: 129 pgs: 129 active+clean; 45 MiB data, 322 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 3.1 MiB/s wr, 28 op/s 2023-10-10T17:45:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:10 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3605931439' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:45:12.222 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:45:12.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:45:12.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 -- 172.21.15.78:0/3308084015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb4100f50 msgr2=0x7fdcb4104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 --2- 172.21.15.78:0/3308084015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb4100f50 0x7fdcb4104f10 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdca80099c0 tx=0x7fdca8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 -- 172.21.15.78:0/3308084015 shutdown_connections 2023-10-10T17:45:12.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 --2- 172.21.15.78:0/3308084015 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcb41055f0 0x7fdcb410d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 --2- 172.21.15.78:0/3308084015 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb41005b0 0x7fdcb4100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 --2- 172.21.15.78:0/3308084015 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb4100f50 0x7fdcb4104f10 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 -- 172.21.15.78:0/3308084015 >> 172.21.15.78:0/3308084015 conn(0x7fdcb4077170 msgr2=0x7fdcb4077570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 -- 172.21.15.78:0/3308084015 shutdown_connections 2023-10-10T17:45:12.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 -- 172.21.15.78:0/3308084015 wait complete. 2023-10-10T17:45:12.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.366+0000 7fdcba156700 1 Processor -- start 2023-10-10T17:45:12.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcba156700 1 -- start start 2023-10-10T17:45:12.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcba156700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcb41005b0 0x7fdcb4196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcba156700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb4100f50 0x7fdcb41969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcba156700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb41055f0 0x7fdcb419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcba156700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb419b440 con 0x7fdcb41055f0 2023-10-10T17:45:12.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcba156700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb419b5b0 con 0x7fdcb4100f50 2023-10-10T17:45:12.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcba156700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb419b770 con 0x7fdcb41005b0 2023-10-10T17:45:12.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb41055f0 0x7fdcb419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb4100f50 0x7fdcb41969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcb41005b0 0x7fdcb4196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb4100f50 0x7fdcb41969a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47874/0 (socket says 172.21.15.78:47874) 2023-10-10T17:45:12.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb2ffd700 1 -- 172.21.15.78:0/1006487371 learned_addr learned my addr 172.21.15.78:0/1006487371 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1006487371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcb41005b0 msgr2=0x7fdcb4196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcb41005b0 0x7fdcb4196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1006487371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb4100f50 msgr2=0x7fdcb41969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb4100f50 0x7fdcb41969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1006487371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80095f0 con 0x7fdcb41055f0 2023-10-10T17:45:12.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb2ffd700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb4100f50 0x7fdcb41969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.367+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcb41005b0 0x7fdcb4196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.368+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb41055f0 0x7fdcb419ae00 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00efc0 tx=0x7fdcac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.368+0000 7fdcb0ff9700 1 -- 172.21.15.78:0/1006487371 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac018070 con 0x7fdcb41055f0 2023-10-10T17:45:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.368+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb4110370 con 0x7fdcb41055f0 2023-10-10T17:45:12.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.368+0000 7fdcb0ff9700 1 -- 172.21.15.78:0/1006487371 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcac00f040 con 0x7fdcb41055f0 2023-10-10T17:45:12.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.368+0000 7fdcb0ff9700 1 -- 172.21.15.78:0/1006487371 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac00cd40 con 0x7fdcb41055f0 2023-10-10T17:45:12.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.368+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb41101e0 con 0x7fdcb41055f0 2023-10-10T17:45:12.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.370+0000 7fdcb0ff9700 1 -- 172.21.15.78:0/1006487371 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fdcac003a00 con 0x7fdcb41055f0 2023-10-10T17:45:12.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.371+0000 7fdcb0ff9700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdc9c0a12c0 0x7fdc9c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.371+0000 7fdcb0ff9700 1 -- 172.21.15.78:0/1006487371 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fdcac0ccb80 con 0x7fdcb41055f0 2023-10-10T17:45:12.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.371+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdc9c0a12c0 0x7fdc9c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.371+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdc9c0a12c0 0x7fdc9c0a3770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdca4011490 tx=0x7fdca4009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.372+0000 7fdcc62e2b80 1 Processor -- start 2023-10-10T17:45:12.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.372+0000 7fdcc62e2b80 1 -- start start 2023-10-10T17:45:12.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fbcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95ec20e0 con 0x556f95fbd4f0 2023-10-10T17:45:12.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95f14920 con 0x556f95fbcbe0 2023-10-10T17:45:12.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95f0ddb0 con 0x556f95fc1930 2023-10-10T17:45:12.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37688/0 (socket says 172.21.15.78:37688) 2023-10-10T17:45:12.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1230315021 learned_addr learned my addr 172.21.15.78:0/1230315021 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fbcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1230315021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 msgr2=0x556f95fbcfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fbcfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1230315021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 msgr2=0x556f95fc13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1230315021 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f95d8f6b0 con 0x556f95fc1930 2023-10-10T17:45:12.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc1d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdcac0090a0 tx=0x7fdcac095e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7f437d2852b93d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdc99ffb700 1 -- 172.21.15.78:0/1230315021 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac004ad0 con 0x556f95fc1930 2023-10-10T17:45:12.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdc99ffb700 1 -- 172.21.15.78:0/1230315021 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcac0999b0 con 0x556f95fc1930 2023-10-10T17:45:12.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.373+0000 7fdc99ffb700 1 -- 172.21.15.78:0/1230315021 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac099e70 con 0x556f95fc1930 2023-10-10T17:45:12.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1230315021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 msgr2=0x556f95fc1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc1d80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fdcac0090a0 tx=0x7fdcac095e20 comp rx=0 tx=0).stop 2023-10-10T17:45:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1230315021 shutdown_connections 2023-10-10T17:45:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc1d80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fbcfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1230315021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1230315021 >> 172.21.15.78:0/1230315021 conn(0x556f95fb23e0 msgr2=0x556f95fb7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1230315021 shutdown_connections 2023-10-10T17:45:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.374+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1230315021 wait complete. 2023-10-10T17:45:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 -- start start 2023-10-10T17:45:12.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 Processor -- start 2023-10-10T17:45:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fc9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95eaf960 con 0x556f95fbd4f0 2023-10-10T17:45:12.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95eaf600 con 0x556f95fbcbe0 2023-10-10T17:45:12.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95d919a0 con 0x556f95fc1930 2023-10-10T17:45:12.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcc62e2b80 10 client.0 _ll_register_callbacks cb 0x556f95ecd800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:12.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fc9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46166/0 (socket says 172.21.15.78:46166) 2023-10-10T17:45:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcb2ffd700 1 -- 172.21.15.78:0/3290880241 learned_addr learned my addr 172.21.15.78:0/3290880241 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.375+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcb3fff700 1 -- 172.21.15.78:0/3290880241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 msgr2=0x556f95fc9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fc9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcb3fff700 1 -- 172.21.15.78:0/3290880241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 msgr2=0x556f95fc63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcb3fff700 1 -- 172.21.15.78:0/3290880241 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb41101e0 con 0x556f95fc1930 2023-10-10T17:45:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcb2ffd700 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc6a90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcac003fb0 tx=0x7fdcac004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9825b34b45a7ddef server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdc92ffd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:45:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdc92ffd700 1 -- 172.21.15.78:0/3290880241 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac004ad0 con 0x556f95fc1930 2023-10-10T17:45:12.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdc92ffd700 1 -- 172.21.15.78:0/3290880241 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fdcac0a1d70 con 0x556f95fc1930 2023-10-10T17:45:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556f95fafba0 con 0x556f95fc1930 2023-10-10T17:45:12.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdc92ffd700 1 -- 172.21.15.78:0/3290880241 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcac099c10 con 0x556f95fc1930 2023-10-10T17:45:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcc62e2b80 10 client.25768 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556f95ed4210 con 0x556f95fc1930 2023-10-10T17:45:12.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.376+0000 7fdc92ffd700 1 -- 172.21.15.78:0/3290880241 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556f95fafba0 con 0x556f95fc1930 2023-10-10T17:45:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdc92ffd700 1 -- 172.21.15.78:0/3290880241 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdcac0957a0 con 0x556f95fc1930 2023-10-10T17:45:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcc62e2b80 10 client.25768 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:12.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcc62e2b80 10 client.25768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcc62e2b80 20 client.25768 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f95fda8d0 0x556f95fdccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcc62e2b80 4 client.25768 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f95ed0120 con 0x556f95fda8d0 2023-10-10T17:45:12.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f95fda8d0 0x556f95fdccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdc92ffd700 10 client.25768 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.377+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f95fda8d0 0x556f95fdccb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ec99c2b7132be7d server_cookie=1d84c33d78494835 in_seq=0 out_seq=0 2023-10-10T17:45:12.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.378+0000 7fdc92ffd700 1 -- 172.21.15.78:0/3290880241 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556f95ed0120 con 0x556f95fda8d0 2023-10-10T17:45:12.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.378+0000 7fdc92ffd700 10 client.25768 handle_command_reply: tid=0 2023-10-10T17:45:12.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.378+0000 7fdcc62e2b80 1 client.25768 shutdown 2023-10-10T17:45:12.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 msgr2=0x556f95fc6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc6a90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdcac003fb0 tx=0x7fdcac004d30 comp rx=0 tx=0).stop 2023-10-10T17:45:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 shutdown_connections 2023-10-10T17:45:12.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fc6a90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f95fda8d0 0x556f95fdccb0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fc9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/3290880241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdc92ffd700 0 client.25768 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 >> 172.21.15.78:0/3290880241 conn(0x556f95fb23e0 msgr2=0x556f95fb6e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 shutdown_connections 2023-10-10T17:45:12.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/3290880241 wait complete. 2023-10-10T17:45:12.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.379+0000 7fdcc62e2b80 20 client.25768 trim_cache size 0 max 16384 2023-10-10T17:45:12.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 Processor -- start 2023-10-10T17:45:12.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 -- start start 2023-10-10T17:45:12.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fd1e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc3d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fb75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95fc4900 con 0x556f95fbd4f0 2023-10-10T17:45:12.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95fc04c0 con 0x556f95fbcbe0 2023-10-10T17:45:12.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95fb8800 con 0x556f95fc1930 2023-10-10T17:45:12.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fb75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc3d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fd1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc3d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46174/0 (socket says 172.21.15.78:46174) 2023-10-10T17:45:12.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb3fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fb75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37706/0 (socket says 172.21.15.78:37706) 2023-10-10T17:45:12.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fd1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47894/0 (socket says 172.21.15.78:47894) 2023-10-10T17:45:12.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb2ffd700 1 -- 172.21.15.78:0/1109897596 learned_addr learned my addr 172.21.15.78:0/1109897596 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1109897596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 msgr2=0x556f95fd1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fd1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1109897596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 msgr2=0x556f95fc3d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc3d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb3fff700 1 -- 172.21.15.78:0/1109897596 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f95fc1fd0 con 0x556f95fc1930 2023-10-10T17:45:12.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.397+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fd1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fb75e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcac0963d0 tx=0x7fdcac0956c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=714fc0c090b1f59e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdc9a7fc700 1 -- 172.21.15.78:0/1109897596 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac004ad0 con 0x556f95fc1930 2023-10-10T17:45:12.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdc9a7fc700 1 -- 172.21.15.78:0/1109897596 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcac098bc0 con 0x556f95fc1930 2023-10-10T17:45:12.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdc9a7fc700 1 -- 172.21.15.78:0/1109897596 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcac0a18e0 con 0x556f95fc1930 2023-10-10T17:45:12.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1109897596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 msgr2=0x556f95fb75e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fb75e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdcac0963d0 tx=0x7fdcac0956c0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1109897596 shutdown_connections 2023-10-10T17:45:12.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fc1930 0x556f95fb75e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fbcbe0 0x556f95fd1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.398+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/1109897596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbd4f0 0x556f95fc3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1109897596 >> 172.21.15.78:0/1109897596 conn(0x556f95fb23e0 msgr2=0x556f95fbf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1109897596 shutdown_connections 2023-10-10T17:45:12.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/1109897596 wait complete. 2023-10-10T17:45:12.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 -- start start 2023-10-10T17:45:12.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 Processor -- start 2023-10-10T17:45:12.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbcbe0 0x556f95fc6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fbd4f0 0x556f95fc4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fc1930 0x556f95fc5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95fd2460 con 0x556f95fbcbe0 2023-10-10T17:45:12.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95fc5600 con 0x556f95fc1930 2023-10-10T17:45:12.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.399+0000 7fdcc62e2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f95fc57c0 con 0x556f95fbd4f0 2023-10-10T17:45:12.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcc62e2b80 10 client.0 _ll_register_callbacks cb 0x7fdcac0b7f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:12.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbcbe0 0x556f95fc6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbcbe0 0x556f95fc6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46178/0 (socket says 172.21.15.78:46178) 2023-10-10T17:45:12.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 -- 172.21.15.78:0/2877274983 learned_addr learned my addr 172.21.15.78:0/2877274983 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fc1930 0x556f95fc5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 -- 172.21.15.78:0/2877274983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fbd4f0 msgr2=0x556f95fc4b50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T17:45:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fbd4f0 0x556f95fc4b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 -- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fc1930 msgr2=0x556f95fc5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fc1930 0x556f95fc5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 -- 172.21.15.78:0/2877274983 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca4017070 con 0x556f95fbcbe0 2023-10-10T17:45:12.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb3fff700 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fc1930 0x556f95fc5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcb37fe700 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbcbe0 0x556f95fc6160 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdca4005d90 tx=0x7fdca40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c0fd55df6c48eac server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdc92ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:12.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdc92ffd700 1 -- 172.21.15.78:0/2877274983 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca4024040 con 0x556f95fbcbe0 2023-10-10T17:45:12.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdc92ffd700 1 -- 172.21.15.78:0/2877274983 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fdca4003be0 con 0x556f95fbcbe0 2023-10-10T17:45:12.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.400+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcac0999b0 con 0x556f95fbcbe0 2023-10-10T17:45:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdc92ffd700 1 -- 172.21.15.78:0/2877274983 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdca4003da0 con 0x556f95fbcbe0 2023-10-10T17:45:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 10 client.16311 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x556f95ed4210 con 0x556f95fbcbe0 2023-10-10T17:45:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdc92ffd700 1 -- 172.21.15.78:0/2877274983 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdcac0999b0 con 0x556f95fbcbe0 2023-10-10T17:45:12.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdc92ffd700 1 -- 172.21.15.78:0/2877274983 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fdca4032a30 con 0x556f95fbcbe0 2023-10-10T17:45:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 10 client.16311 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 20 client.16311 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f9602fb50 0x556f96031f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 4 client.16311 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:45:12.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556f95edc6e0 con 0x556f9602fb50 2023-10-10T17:45:12.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.401+0000 7fdcb2ffd700 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f9602fb50 0x556f96031f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.402+0000 7fdc92ffd700 10 client.16311 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.402+0000 7fdcb2ffd700 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f9602fb50 0x556f96031f60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e21bef57a7a3a9e server_cookie=157ec358149ad723 in_seq=0 out_seq=0 2023-10-10T17:45:12.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.402+0000 7fdc92ffd700 1 -- 172.21.15.78:0/2877274983 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556f95edc6e0 con 0x556f9602fb50 2023-10-10T17:45:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.402+0000 7fdc92ffd700 10 client.16311 handle_command_reply: tid=0 2023-10-10T17:45:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.402+0000 7fdcc62e2b80 1 client.16311 shutdown 2023-10-10T17:45:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbcbe0 msgr2=0x556f95fc6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbcbe0 0x556f95fc6160 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdca4005d90 tx=0x7fdca40047c0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 shutdown_connections 2023-10-10T17:45:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556f9602fb50 0x556f96031f60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556f95fc1930 0x556f95fc5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdc92ffd700 0 client.16311 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556f95fbcbe0 0x556f95fc6160 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 --2- 172.21.15.78:0/2877274983 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556f95fbd4f0 0x556f95fc4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 >> 172.21.15.78:0/2877274983 conn(0x556f95fb23e0 msgr2=0x556f95fc7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 shutdown_connections 2023-10-10T17:45:12.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 1 -- 172.21.15.78:0/2877274983 wait complete. 2023-10-10T17:45:12.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.403+0000 7fdcc62e2b80 20 client.16311 trim_cache size 0 max 16384 2023-10-10T17:45:12.434 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:45:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:45:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:45:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:45:12.435 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:45:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdc9c0a12c0 msgr2=0x7fdc9c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdc9c0a12c0 0x7fdc9c0a3770 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdca4011490 tx=0x7fdca4009260 comp rx=0 tx=0).stop 2023-10-10T17:45:12.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb41055f0 msgr2=0x7fdcb419ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb41055f0 0x7fdcb419ae00 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdcac00efc0 tx=0x7fdcac00c3a0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 shutdown_connections 2023-10-10T17:45:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdcb41005b0 0x7fdcb4196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdc9c0a12c0 0x7fdc9c0a3770 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdcb41055f0 0x7fdcb419ae00 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 --2- 172.21.15.78:0/1006487371 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdcb4100f50 0x7fdcb41969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 >> 172.21.15.78:0/1006487371 conn(0x7fdcb4077170 msgr2=0x7fdcb40fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 shutdown_connections 2023-10-10T17:45:12.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.404+0000 7fdcba156700 1 -- 172.21.15.78:0/1006487371 wait complete. 2023-10-10T17:45:12.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:45:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.571+0000 7f77f35ab700 1 -- 172.21.15.78:0/2361216696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 msgr2=0x7f77ec0feec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.571+0000 7f77f35ab700 1 --2- 172.21.15.78:0/2361216696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 0x7f77ec0feec0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009920 tx=0x7f77e8009c30 comp rx=0 tx=0).stop 2023-10-10T17:45:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 -- 172.21.15.78:0/2361216696 shutdown_connections 2023-10-10T17:45:12.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 --2- 172.21.15.78:0/2361216696 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 0x7f77ec0feec0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 --2- 172.21.15.78:0/2361216696 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec1055d0 0x7f77ec10d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 --2- 172.21.15.78:0/2361216696 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77ec0ff490 0x7f77ec104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 -- 172.21.15.78:0/2361216696 >> 172.21.15.78:0/2361216696 conn(0x7f77ec075920 msgr2=0x7f77ec075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 -- 172.21.15.78:0/2361216696 shutdown_connections 2023-10-10T17:45:12.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 -- 172.21.15.78:0/2361216696 wait complete. 2023-10-10T17:45:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 Processor -- start 2023-10-10T17:45:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.572+0000 7f77f35ab700 1 -- start start 2023-10-10T17:45:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 0x7f77ec1965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec0ff490 0x7f77ec196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77ec1055d0 0x7f77ec19af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec19b580 con 0x7f77ec0ff490 2023-10-10T17:45:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec19b6f0 con 0x7f77ec1055d0 2023-10-10T17:45:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec19b8b0 con 0x7f77ec0feaf0 2023-10-10T17:45:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec0ff490 0x7f77ec196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec0ff490 0x7f77ec196ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46194/0 (socket says 172.21.15.78:46194) 2023-10-10T17:45:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 -- 172.21.15.78:0/3718877811 learned_addr learned my addr 172.21.15.78:0/3718877811 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f1347700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 0x7f77ec1965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f1b48700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77ec1055d0 0x7f77ec19af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 -- 172.21.15.78:0/3718877811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 msgr2=0x7f77ec1965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 0x7f77ec1965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 -- 172.21.15.78:0/3718877811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77ec1055d0 msgr2=0x7f77ec19af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77ec1055d0 0x7f77ec19af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 -- 172.21.15.78:0/3718877811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e4009d00 con 0x7f77ec0ff490 2023-10-10T17:45:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f0b46700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec0ff490 0x7f77ec196ae0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e940 tx=0x7f77dc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77e27fc700 1 -- 172.21.15.78:0/3718877811 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc009730 con 0x7f77ec0ff490 2023-10-10T17:45:12.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e8009520 con 0x7f77ec0ff490 2023-10-10T17:45:12.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77e27fc700 1 -- 172.21.15.78:0/3718877811 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77dc010430 con 0x7f77ec0ff490 2023-10-10T17:45:12.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77e27fc700 1 -- 172.21.15.78:0/3718877811 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00cea0 con 0x7f77ec0ff490 2023-10-10T17:45:12.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.573+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec110420 con 0x7f77ec0ff490 2023-10-10T17:45:12.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.576+0000 7f77e27fc700 1 -- 172.21.15.78:0/3718877811 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f77dc010810 con 0x7f77ec0ff490 2023-10-10T17:45:12.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.576+0000 7f77e27fc700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f77d80a12c0 0x7f77d80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.576+0000 7f77f1347700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f77d80a12c0 0x7f77d80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.576+0000 7f77e27fc700 1 -- 172.21.15.78:0/3718877811 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f77dc014070 con 0x7f77ec0ff490 2023-10-10T17:45:12.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.577+0000 7f77f1347700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f77d80a12c0 0x7f77d80a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f77e800c010 tx=0x7f77e8019040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.577+0000 7f77ff737b80 1 Processor -- start 2023-10-10T17:45:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.577+0000 7f77ff737b80 1 -- start start 2023-10-10T17:45:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9bf70 0x559f46b9c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9c880 0x559f46ba0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46ba1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77ff737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46a9d060 con 0x559f46b9c880 2023-10-10T17:45:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77ff737b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46aef8a0 con 0x559f46b9bf70 2023-10-10T17:45:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77ff737b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46ae8d30 con 0x559f46ba0cc0 2023-10-10T17:45:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9c880 0x559f46ba0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f1347700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9bf70 0x559f46b9c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9c880 0x559f46ba0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46202/0 (socket says 172.21.15.78:46202) 2023-10-10T17:45:12.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 -- 172.21.15.78:0/32749677 learned_addr learned my addr 172.21.15.78:0/32749677 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f1b48700 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46ba1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 -- 172.21.15.78:0/32749677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 msgr2=0x559f46ba1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46ba1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 -- 172.21.15.78:0/32749677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9bf70 msgr2=0x559f46b9c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9bf70 0x559f46b9c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 -- 172.21.15.78:0/32749677 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f4696a630 con 0x559f46b9c880 2023-10-10T17:45:12.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f1347700 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9bf70 0x559f46b9c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77f0b46700 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9c880 0x559f46ba0780 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0969c0 tx=0x7f77dc0999a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2aa28ad380455b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77d77fe700 1 -- 172.21.15.78:0/32749677 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc0a23a0 con 0x559f46b9c880 2023-10-10T17:45:12.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77d77fe700 1 -- 172.21.15.78:0/32749677 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77dc0982f0 con 0x559f46b9c880 2023-10-10T17:45:12.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.578+0000 7f77d77fe700 1 -- 172.21.15.78:0/32749677 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc0a1f10 con 0x559f46b9c880 2023-10-10T17:45:12.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 -- 172.21.15.78:0/32749677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9c880 msgr2=0x559f46ba0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9c880 0x559f46ba0780 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0969c0 tx=0x7f77dc0999a0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 -- 172.21.15.78:0/32749677 shutdown_connections 2023-10-10T17:45:12.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46ba1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9bf70 0x559f46b9c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 --2- 172.21.15.78:0/32749677 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9c880 0x559f46ba0780 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 -- 172.21.15.78:0/32749677 >> 172.21.15.78:0/32749677 conn(0x559f46b8d360 msgr2=0x559f46b92b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 -- 172.21.15.78:0/32749677 shutdown_connections 2023-10-10T17:45:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.579+0000 7f77ff737b80 1 -- 172.21.15.78:0/32749677 wait complete. 2023-10-10T17:45:12.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 -- start start 2023-10-10T17:45:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 Processor -- start 2023-10-10T17:45:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46ba8480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba5770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46ba5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46a8a8e0 con 0x559f46ba0cc0 2023-10-10T17:45:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46a8a580 con 0x559f46b9c880 2023-10-10T17:45:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f4696c920 con 0x559f46b9bf70 2023-10-10T17:45:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77ff737b80 10 client.0 _ll_register_callbacks cb 0x559f46aa8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77f1b48700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46ba5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77f0b46700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77f1347700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46ba8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77f1b48700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46ba5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46210/0 (socket says 172.21.15.78:46210) 2023-10-10T17:45:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77f0b46700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47930/0 (socket says 172.21.15.78:47930) 2023-10-10T17:45:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.580+0000 7f77f1b48700 1 -- 172.21.15.78:0/4028456810 learned_addr learned my addr 172.21.15.78:0/4028456810 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77f0b46700 1 -- 172.21.15.78:0/4028456810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 msgr2=0x559f46ba8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77f0b46700 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46ba8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77f0b46700 1 -- 172.21.15.78:0/4028456810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 msgr2=0x559f46ba5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77f0b46700 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46ba5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77f0b46700 1 -- 172.21.15.78:0/4028456810 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ec110420 con 0x559f46b9c880 2023-10-10T17:45:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77f1b48700 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46ba5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77f0b46700 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba5770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e910 tx=0x7f77dc00c3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef1e374a950ae272 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:45:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77d4ff9700 1 -- 172.21.15.78:0/4028456810 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc0b8200 con 0x559f46b9c880 2023-10-10T17:45:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77d4ff9700 1 -- 172.21.15.78:0/4028456810 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f77dc0a1f10 con 0x559f46b9c880 2023-10-10T17:45:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f46b8ab20 con 0x559f46b9c880 2023-10-10T17:45:12.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77d4ff9700 1 -- 172.21.15.78:0/4028456810 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77dc0b8360 con 0x559f46b9c880 2023-10-10T17:45:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77ff737b80 10 client.25685 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x559f46aaf190 con 0x559f46b9c880 2023-10-10T17:45:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.581+0000 7f77d4ff9700 1 -- 172.21.15.78:0/4028456810 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77dc063080 con 0x559f46b9c880 2023-10-10T17:45:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77d4ff9700 1 -- 172.21.15.78:0/4028456810 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f77dc0cad20 con 0x559f46b9c880 2023-10-10T17:45:12.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77ff737b80 10 client.25685 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77ff737b80 10 client.25685 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77ff737b80 20 client.25685 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77ff737b80 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46bb5910 0x559f46bb7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77ff737b80 4 client.25685 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:12.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f46aab0a0 con 0x559f46bb5910 2023-10-10T17:45:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77f1347700 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46bb5910 0x559f46bb7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77d4ff9700 10 client.25685 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.582+0000 7f77f1347700 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46bb5910 0x559f46bb7d20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0871c398a8246f9 server_cookie=83476c8b7f33986 in_seq=0 out_seq=0 2023-10-10T17:45:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77d4ff9700 1 -- 172.21.15.78:0/4028456810 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559f46aab0a0 con 0x559f46bb5910 2023-10-10T17:45:12.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77d4ff9700 10 client.25685 handle_command_reply: tid=0 2023-10-10T17:45:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 client.25685 shutdown 2023-10-10T17:45:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 msgr2=0x559f46ba5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba5770 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e910 tx=0x7f77dc00c3d0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 shutdown_connections 2023-10-10T17:45:12.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46ba8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46bb5910 0x559f46bb7d20 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46ba5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 --2- 172.21.15.78:0/4028456810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba5770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77d4ff9700 0 client.25685 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 >> 172.21.15.78:0/4028456810 conn(0x559f46b8d360 msgr2=0x559f46b91de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 shutdown_connections 2023-10-10T17:45:12.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 1 -- 172.21.15.78:0/4028456810 wait complete. 2023-10-10T17:45:12.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.584+0000 7f77ff737b80 20 client.25685 trim_cache size 0 max 16384 2023-10-10T17:45:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 Processor -- start 2023-10-10T17:45:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 -- start start 2023-10-10T17:45:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46b9e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46b9ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46aab0a0 con 0x559f46ba0cc0 2023-10-10T17:45:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46b8a710 con 0x559f46b9c880 2023-10-10T17:45:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77ff737b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46b9d000 con 0x559f46b9bf70 2023-10-10T17:45:12.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77f1b48700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46b9ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77f1b48700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46b9ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46218/0 (socket says 172.21.15.78:46218) 2023-10-10T17:45:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77f1347700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46b9e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77f1b48700 1 -- 172.21.15.78:0/3965123406 learned_addr learned my addr 172.21.15.78:0/3965123406 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.601+0000 7f77f0b46700 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77f1b48700 1 -- 172.21.15.78:0/3965123406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 msgr2=0x559f46b9e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77f1b48700 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46b9e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77f1b48700 1 -- 172.21.15.78:0/3965123406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 msgr2=0x559f46ba1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77f1b48700 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77f1b48700 1 -- 172.21.15.78:0/3965123406 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f46ba24d0 con 0x559f46ba0cc0 2023-10-10T17:45:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77f0b46700 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77f1b48700 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46b9ee20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009610 tx=0x7f77e40099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888e0088a3545892 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77d7fff700 1 -- 172.21.15.78:0/3965123406 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4004450 con 0x559f46ba0cc0 2023-10-10T17:45:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77d7fff700 1 -- 172.21.15.78:0/3965123406 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77e400f040 con 0x559f46ba0cc0 2023-10-10T17:45:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.602+0000 7f77d7fff700 1 -- 172.21.15.78:0/3965123406 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e4017410 con 0x559f46ba0cc0 2023-10-10T17:45:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 -- 172.21.15.78:0/3965123406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 msgr2=0x559f46b9ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46b9ee20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009610 tx=0x7f77e40099e0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 -- 172.21.15.78:0/3965123406 shutdown_connections 2023-10-10T17:45:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46b9bf70 0x559f46b9e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46ba0cc0 0x559f46b9ee20 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 --2- 172.21.15.78:0/3965123406 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46ba1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 -- 172.21.15.78:0/3965123406 >> 172.21.15.78:0/3965123406 conn(0x559f46b8d360 msgr2=0x559f46bb1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 -- 172.21.15.78:0/3965123406 shutdown_connections 2023-10-10T17:45:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 -- 172.21.15.78:0/3965123406 wait complete. 2023-10-10T17:45:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 -- start start 2023-10-10T17:45:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.603+0000 7f77ff737b80 1 Processor -- start 2023-10-10T17:45:12.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9bf70 0x559f46bac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46bacca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77ff737b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46baad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77ff737b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46bab310 con 0x559f46b9bf70 2023-10-10T17:45:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77ff737b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46bab480 con 0x559f46b9c880 2023-10-10T17:45:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77ff737b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f46bab640 con 0x559f46ba0cc0 2023-10-10T17:45:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77ff737b80 10 client.0 _ll_register_callbacks cb 0x7f77e800f460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9bf70 0x559f46bac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9bf70 0x559f46bac760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46220/0 (socket says 172.21.15.78:46220) 2023-10-10T17:45:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 -- 172.21.15.78:0/46047394 learned_addr learned my addr 172.21.15.78:0/46047394 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:12.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f0b46700 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46bacca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1b48700 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46baad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 -- 172.21.15.78:0/46047394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 msgr2=0x559f46baad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46baad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 -- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 msgr2=0x559f46bacca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:12.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46bacca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 -- 172.21.15.78:0/46047394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f46aaf190 con 0x559f46b9bf70 2023-10-10T17:45:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77f1347700 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9bf70 0x559f46bac760 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009920 tx=0x7f77e8009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f7f1527a2bc131 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.604+0000 7f77d4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77d4ff9700 1 -- 172.21.15.78:0/46047394 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e802c040 con 0x559f46b9bf70 2023-10-10T17:45:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77d4ff9700 1 -- 172.21.15.78:0/46047394 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f77e800b700 con 0x559f46b9bf70 2023-10-10T17:45:12.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77dc0982f0 con 0x559f46b9bf70 2023-10-10T17:45:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77d4ff9700 1 -- 172.21.15.78:0/46047394 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77e800b8c0 con 0x559f46b9bf70 2023-10-10T17:45:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 10 client.16329 fetch_fsmap learned FSMap version 26 2023-10-10T17:45:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x7f77e40092f0 con 0x559f46b9bf70 2023-10-10T17:45:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77d4ff9700 1 -- 172.21.15.78:0/46047394 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f77dc0982f0 con 0x559f46b9bf70 2023-10-10T17:45:12.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77d4ff9700 1 -- 172.21.15.78:0/46047394 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 26) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f77e803a020 con 0x559f46b9bf70 2023-10-10T17:45:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 10 client.16329 fetch_fsmap finished waiting for FSMap version 26 2023-10-10T17:45:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 20 client.16329 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46c0a8f0 0x559f46c0cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:12.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 4 client.16329 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:45:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.605+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559f46ab7660 con 0x559f46c0a8f0 2023-10-10T17:45:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.606+0000 7f77f0b46700 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46c0a8f0 0x559f46c0cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.606+0000 7f77f0b46700 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46c0a8f0 0x559f46c0cd00 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=646ec2bf35adb779 server_cookie=50661002f7b382ef in_seq=0 out_seq=0 2023-10-10T17:45:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.606+0000 7f77d4ff9700 10 client.16329 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.607+0000 7f77d4ff9700 1 -- 172.21.15.78:0/46047394 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x559f46ab7660 con 0x559f46c0a8f0 2023-10-10T17:45:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.607+0000 7f77d4ff9700 10 client.16329 handle_command_reply: tid=0 2023-10-10T17:45:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.607+0000 7f77ff737b80 1 client.16329 shutdown 2023-10-10T17:45:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.607+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9bf70 msgr2=0x559f46bac760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.607+0000 7f77ff737b80 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9bf70 0x559f46bac760 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009920 tx=0x7f77e8009c30 comp rx=0 tx=0).stop 2023-10-10T17:45:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 shutdown_connections 2023-10-10T17:45:12.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f46ba0cc0 0x559f46baad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x559f46c0a8f0 0x559f46c0cd00 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f46b9bf70 0x559f46bac760 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 --2- 172.21.15.78:0/46047394 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f46b9c880 0x559f46bacca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 >> 172.21.15.78:0/46047394 conn(0x559f46b8d360 msgr2=0x559f46ba17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77d4ff9700 0 client.16329 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 shutdown_connections 2023-10-10T17:45:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 1 -- 172.21.15.78:0/46047394 wait complete. 2023-10-10T17:45:12.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77ff737b80 20 client.16329 trim_cache size 0 max 16384 2023-10-10T17:45:12.625 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:45:12.626 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (2931 inodes in the stack)", 2023-10-10T17:45:12.626 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:45:12.626 INFO:teuthology.orchestra.run.smithi078.stdout: "5ecf957f-7c91-40df-8738-d2beae2feba1": { 2023-10-10T17:45:12.626 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:45:12.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:45:12.627 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:45:12.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:45:12.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:45:12.627 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:45:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f77d80a12c0 msgr2=0x7f77d80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f77d80a12c0 0x7f77d80a3770 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f77e800c010 tx=0x7f77e8019040 comp rx=0 tx=0).stop 2023-10-10T17:45:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec0ff490 msgr2=0x7f77ec196ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec0ff490 0x7f77ec196ae0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00e940 tx=0x7f77dc00ec50 comp rx=0 tx=0).stop 2023-10-10T17:45:12.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 shutdown_connections 2023-10-10T17:45:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f77ec0feaf0 0x7f77ec1965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f77d80a12c0 0x7f77d80a3770 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f77ec1055d0 0x7f77ec19af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 --2- 172.21.15.78:0/3718877811 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77ec0ff490 0x7f77ec196ae0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:12.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 >> 172.21.15.78:0/3718877811 conn(0x7f77ec075920 msgr2=0x7f77ec0fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:12.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 shutdown_connections 2023-10-10T17:45:12.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:12.608+0000 7f77f35ab700 1 -- 172.21.15.78:0/3718877811 wait complete. 2023-10-10T17:45:12.631 INFO:tasks.cephfs.filesystem:scrub status for tag:5ecf957f-7c91-40df-8738-d2beae2feba1 - {'path': '/', 'tag': '5ecf957f-7c91-40df-8738-d2beae2feba1', 'options': 'recursive,force'} 2023-10-10T17:45:12.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:12 smithi078 ceph-mon[99494]: pgmap v83: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 155 op/s 2023-10-10T17:45:13.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:12 smithi190 ceph-mon[101220]: pgmap v83: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 155 op/s 2023-10-10T17:45:13.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:12 smithi157 ceph-mon[100896]: pgmap v83: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 5.0 MiB/s wr, 155 op/s 2023-10-10T17:45:14.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:13 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:45:14.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:13 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:14.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:13 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:14.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:13 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:14.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:13 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:14.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:13 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:14.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:13 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:45:14.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:13 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:14.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:13 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:14.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:13 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:14.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:13 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:14.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:13 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:14.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:13 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:45:14.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:13 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:14.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:13 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:14.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:13 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:14.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:13 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:14.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:13 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:15.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:14 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:15.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:14 smithi078 ceph-mon[99494]: pgmap v84: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 135 op/s 2023-10-10T17:45:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:14 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:14 smithi190 ceph-mon[101220]: pgmap v84: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 135 op/s 2023-10-10T17:45:15.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:14 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:14 smithi157 ceph-mon[100896]: pgmap v84: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 135 op/s 2023-10-10T17:45:16.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:15 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:16.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:15 smithi078 ceph-mon[99494]: pgmap v85: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 135 op/s 2023-10-10T17:45:16.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:15 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:16.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:15 smithi190 ceph-mon[101220]: pgmap v85: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 135 op/s 2023-10-10T17:45:16.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:15 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:16.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:15 smithi157 ceph-mon[100896]: pgmap v85: 129 pgs: 129 active+clean; 62 MiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 135 op/s 2023-10-10T17:45:18.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:18 smithi078 ceph-mon[99494]: pgmap v86: 129 pgs: 129 active+clean; 82 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.5 MiB/s wr, 453 op/s 2023-10-10T17:45:18.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:18 smithi190 ceph-mon[101220]: pgmap v86: 129 pgs: 129 active+clean; 82 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.5 MiB/s wr, 453 op/s 2023-10-10T17:45:18.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:18 smithi157 ceph-mon[100896]: pgmap v86: 129 pgs: 129 active+clean; 82 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 5.5 MiB/s wr, 453 op/s 2023-10-10T17:45:20.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:20 smithi190 ceph-mon[101220]: pgmap v87: 129 pgs: 129 active+clean; 82 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 444 op/s 2023-10-10T17:45:20.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:20 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:20.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:20 smithi157 ceph-mon[100896]: pgmap v87: 129 pgs: 129 active+clean; 82 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 444 op/s 2023-10-10T17:45:20.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:20 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:21.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:20 smithi078 ceph-mon[99494]: pgmap v87: 129 pgs: 129 active+clean; 82 MiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.6 MiB/s wr, 444 op/s 2023-10-10T17:45:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:20 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.442+0000 7f0853421700 1 -- 172.21.15.78:0/3261896341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0ff380 msgr2=0x7f084c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:21.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.442+0000 7f0853421700 1 --2- 172.21.15.78:0/3261896341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0ff380 0x7f084c104e50 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f083c0099c0 tx=0x7f083c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.442+0000 7f0853421700 1 -- 172.21.15.78:0/3261896341 shutdown_connections 2023-10-10T17:45:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.442+0000 7f0853421700 1 --2- 172.21.15.78:0/3261896341 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f084c105580 0x7f084c10d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.442+0000 7f0853421700 1 --2- 172.21.15.78:0/3261896341 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f084c0fe9e0 0x7f084c0fedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.442+0000 7f0853421700 1 --2- 172.21.15.78:0/3261896341 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0ff380 0x7f084c104e50 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.444 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.442+0000 7f0853421700 1 -- 172.21.15.78:0/3261896341 >> 172.21.15.78:0/3261896341 conn(0x7f084c075810 msgr2=0x7f084c075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 -- 172.21.15.78:0/3261896341 shutdown_connections 2023-10-10T17:45:21.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 -- 172.21.15.78:0/3261896341 wait complete. 2023-10-10T17:45:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 Processor -- start 2023-10-10T17:45:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 -- start start 2023-10-10T17:45:21.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0fe9e0 0x7f084c06b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f084c0ff380 0x7f084c06b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f084c105580 0x7f084c1aa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1aaea0 con 0x7f084c0fe9e0 2023-10-10T17:45:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1ab010 con 0x7f084c105580 2023-10-10T17:45:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.443+0000 7f0853421700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c1ab1d0 con 0x7f084c0ff380 2023-10-10T17:45:21.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0fe9e0 0x7f084c06b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08509bc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f084c0ff380 0x7f084c06b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0fe9e0 0x7f084c06b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46236/0 (socket says 172.21.15.78:46236) 2023-10-10T17:45:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 -- 172.21.15.78:0/3619978155 learned_addr learned my addr 172.21.15.78:0/3619978155 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08519be700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f084c105580 0x7f084c1aa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 -- 172.21.15.78:0/3619978155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f084c0ff380 msgr2=0x7f084c06b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f084c0ff380 0x7f084c06b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 -- 172.21.15.78:0/3619978155 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f084c105580 msgr2=0x7f084c1aa8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f084c105580 0x7f084c1aa8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 -- 172.21.15.78:0/3619978155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c0095f0 con 0x7f084c0fe9e0 2023-10-10T17:45:21.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08519be700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f084c105580 0x7f084c1aa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08511bd700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0fe9e0 0x7f084c06b050 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f084800e9a0 tx=0x7f084800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f08427fc700 1 -- 172.21.15.78:0/3619978155 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848009790 con 0x7f084c0fe9e0 2023-10-10T17:45:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.444+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c1102d0 con 0x7f084c0fe9e0 2023-10-10T17:45:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.445+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c110140 con 0x7f084c0fe9e0 2023-10-10T17:45:21.451 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.445+0000 7f08427fc700 1 -- 172.21.15.78:0/3619978155 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0848010430 con 0x7f084c0fe9e0 2023-10-10T17:45:21.452 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.445+0000 7f08427fc700 1 -- 172.21.15.78:0/3619978155 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848018680 con 0x7f084c0fe9e0 2023-10-10T17:45:21.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.448+0000 7f08427fc700 1 -- 172.21.15.78:0/3619978155 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f084800ca10 con 0x7f084c0fe9e0 2023-10-10T17:45:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.448+0000 7f08427fc700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08380a5750 0x7f08380a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f08427fc700 1 -- 172.21.15.78:0/3619978155 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0848014070 con 0x7f084c0fe9e0 2023-10-10T17:45:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 Processor -- start 2023-10-10T17:45:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 -- start start 2023-10-10T17:45:21.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f08509bc700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08380a5750 0x7f08380a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55919301a6b0 0x5591931c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931be560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4b90 0x5591931beb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591930c4000 con 0x55919301a6b0 2023-10-10T17:45:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559193116840 con 0x5591931c4b90 2023-10-10T17:45:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f085f5adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919310fcd0 con 0x5591931c4720 2023-10-10T17:45:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f08511bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55919301a6b0 0x5591931c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f08511bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55919301a6b0 0x5591931c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46244/0 (socket says 172.21.15.78:46244) 2023-10-10T17:45:21.458 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.449+0000 7f08511bd700 1 -- 172.21.15.78:0/4038967435 learned_addr learned my addr 172.21.15.78:0/4038967435 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08519be700 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4b90 0x5591931beb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08509bc700 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931be560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08511bd700 1 -- 172.21.15.78:0/4038967435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 msgr2=0x5591931be560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08511bd700 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931be560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08511bd700 1 -- 172.21.15.78:0/4038967435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4b90 msgr2=0x5591931beb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08511bd700 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4b90 0x5591931beb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08509bc700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08380a5750 0x7f08380a7c00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f083c00b580 tx=0x7f083c003680 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:21.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08511bd700 1 -- 172.21.15.78:0/4038967435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559192f915d0 con 0x55919301a6b0 2023-10-10T17:45:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08519be700 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4b90 0x5591931beb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:45:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08511bd700 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55919301a6b0 0x5591931c41e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f08480a1dc0 tx=0x7f0848099fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799ee43e76bbba90 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08377fe700 1 -- 172.21.15.78:0/4038967435 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08480980a0 con 0x55919301a6b0 2023-10-10T17:45:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08377fe700 1 -- 172.21.15.78:0/4038967435 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f084809a100 con 0x55919301a6b0 2023-10-10T17:45:21.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.450+0000 7f08377fe700 1 -- 172.21.15.78:0/4038967435 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08480a20c0 con 0x55919301a6b0 2023-10-10T17:45:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 -- 172.21.15.78:0/4038967435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55919301a6b0 msgr2=0x5591931c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55919301a6b0 0x5591931c41e0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f08480a1dc0 tx=0x7f0848099fa0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 -- 172.21.15.78:0/4038967435 shutdown_connections 2023-10-10T17:45:21.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4b90 0x5591931beb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55919301a6b0 0x5591931c41e0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 --2- 172.21.15.78:0/4038967435 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931be560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 -- 172.21.15.78:0/4038967435 >> 172.21.15.78:0/4038967435 conn(0x5591931b3f50 msgr2=0x5591931b9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 -- 172.21.15.78:0/4038967435 shutdown_connections 2023-10-10T17:45:21.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.451+0000 7f085f5adb80 1 -- 172.21.15.78:0/4038967435 wait complete. 2023-10-10T17:45:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 -- start start 2023-10-10T17:45:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 Processor -- start 2023-10-10T17:45:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55919301a6b0 0x5591931cab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4720 0x5591931c7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591930b1880 con 0x5591931c4b90 2023-10-10T17:45:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591930b1520 con 0x5591931c4720 2023-10-10T17:45:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559192f938c0 con 0x55919301a6b0 2023-10-10T17:45:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.452+0000 7f085f5adb80 10 client.0 _ll_register_callbacks cb 0x5591930cf720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4720 0x5591931c7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08511bd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55919301a6b0 0x5591931cab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4720 0x5591931c7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47974/0 (socket says 172.21.15.78:47974) 2023-10-10T17:45:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 -- 172.21.15.78:0/3274461576 learned_addr learned my addr 172.21.15.78:0/3274461576 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08519be700 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 -- 172.21.15.78:0/3274461576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55919301a6b0 msgr2=0x5591931cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:21.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55919301a6b0 0x5591931cab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 -- 172.21.15.78:0/3274461576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 msgr2=0x5591931c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 -- 172.21.15.78:0/3274461576 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c110140 con 0x5591931c4720 2023-10-10T17:45:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f08509bc700 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4720 0x5591931c7d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f083c0098e0 tx=0x7f083c009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83bd4e7bf3e47685 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:21.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.453+0000 7f0834ff9700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:45:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f0834ff9700 1 -- 172.21.15.78:0/3274461576 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c00fc50 con 0x5591931c4720 2023-10-10T17:45:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f0834ff9700 1 -- 172.21.15.78:0/3274461576 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f083c00be50 con 0x5591931c4720 2023-10-10T17:45:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5591931b1710 con 0x5591931c4720 2023-10-10T17:45:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f0834ff9700 1 -- 172.21.15.78:0/3274461576 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f083c00fdb0 con 0x5591931c4720 2023-10-10T17:45:21.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 10 client.25703 fetch_fsmap learned FSMap version 28 2023-10-10T17:45:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5591930d6130 con 0x5591931c4720 2023-10-10T17:45:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f0834ff9700 1 -- 172.21.15.78:0/3274461576 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5591931b1710 con 0x5591931c4720 2023-10-10T17:45:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f0834ff9700 1 -- 172.21.15.78:0/3274461576 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f083c02d070 con 0x5591931c4720 2023-10-10T17:45:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 10 client.25703 fetch_fsmap finished waiting for FSMap version 28 2023-10-10T17:45:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 10 client.25703 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:21.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 20 client.25703 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591931e0940 0x5591931e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 4 client.25703 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591930d2040 con 0x5591931e0940 2023-10-10T17:45:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.454+0000 7f08511bd700 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591931e0940 0x5591931e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.455+0000 7f0834ff9700 10 client.25703 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:21.471 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.455+0000 7f08511bd700 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591931e0940 0x5591931e2d50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a8f20678b0f6d3 server_cookie=4999dabb01683d26 in_seq=0 out_seq=0 2023-10-10T17:45:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.456+0000 7f0834ff9700 1 -- 172.21.15.78:0/3274461576 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5591930d2040 con 0x5591931e0940 2023-10-10T17:45:21.473 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.456+0000 7f0834ff9700 10 client.25703 handle_command_reply: tid=0 2023-10-10T17:45:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.456+0000 7f085f5adb80 1 client.25703 shutdown 2023-10-10T17:45:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4720 msgr2=0x5591931c7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4720 0x5591931c7d30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f083c0098e0 tx=0x7f083c009310 comp rx=0 tx=0).stop 2023-10-10T17:45:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 shutdown_connections 2023-10-10T17:45:21.474 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55919301a6b0 0x5591931cab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591931e0940 0x5591931e2d50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 --2- 172.21.15.78:0/3274461576 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5591931c4720 0x5591931c7d30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f0834ff9700 0 client.25703 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:21.475 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 >> 172.21.15.78:0/3274461576 conn(0x5591931b3f50 msgr2=0x5591931b87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 shutdown_connections 2023-10-10T17:45:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 1 -- 172.21.15.78:0/3274461576 wait complete. 2023-10-10T17:45:21.476 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.457+0000 7f085f5adb80 20 client.25703 trim_cache size 0 max 16384 2023-10-10T17:45:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.475+0000 7f085f5adb80 1 Processor -- start 2023-10-10T17:45:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f085f5adb80 1 -- start start 2023-10-10T17:45:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d7940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.478 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931ca650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f085f5adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591931c9120 con 0x5591931c4b90 2023-10-10T17:45:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f085f5adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591931c29e0 con 0x55919301a6b0 2023-10-10T17:45:21.479 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f085f5adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591931c3dc0 con 0x5591931c4720 2023-10-10T17:45:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f08519be700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f08519be700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46262/0 (socket says 172.21.15.78:46262) 2023-10-10T17:45:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f08519be700 1 -- 172.21.15.78:0/275838104 learned_addr learned my addr 172.21.15.78:0/275838104 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.476+0000 7f08511bd700 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.480 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f08511bd700 1 -- 172.21.15.78:0/275838104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 msgr2=0x5591931ca650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T17:45:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f08511bd700 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931ca650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f08511bd700 1 -- 172.21.15.78:0/275838104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 msgr2=0x5591931c1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f08511bd700 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.481 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f08511bd700 1 -- 172.21.15.78:0/275838104 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591930d5e80 con 0x55919301a6b0 2023-10-10T17:45:21.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f08511bd700 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d7940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08480a1dc0 tx=0x7f08480961e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fe8fe861398d92d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:21.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f0837fff700 1 -- 172.21.15.78:0/275838104 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08480b8040 con 0x55919301a6b0 2023-10-10T17:45:21.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f0837fff700 1 -- 172.21.15.78:0/275838104 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f084809a270 con 0x55919301a6b0 2023-10-10T17:45:21.482 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.477+0000 7f0837fff700 1 -- 172.21.15.78:0/275838104 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08480a1e60 con 0x55919301a6b0 2023-10-10T17:45:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 -- 172.21.15.78:0/275838104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 msgr2=0x5591931d7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:21.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d7940 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f08480a1dc0 tx=0x7f08480961e0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 -- 172.21.15.78:0/275838104 shutdown_connections 2023-10-10T17:45:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x5591931c1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d7940 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.486 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 --2- 172.21.15.78:0/275838104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931ca650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.487 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 -- 172.21.15.78:0/275838104 >> 172.21.15.78:0/275838104 conn(0x5591931b3f50 msgr2=0x5591931dc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:21.488 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 -- 172.21.15.78:0/275838104 shutdown_connections 2023-10-10T17:45:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 -- 172.21.15.78:0/275838104 wait complete. 2023-10-10T17:45:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 -- start start 2023-10-10T17:45:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.478+0000 7f085f5adb80 1 Processor -- start 2023-10-10T17:45:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931d6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f085f5adb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x55919322d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f085f5adb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919322dcd0 con 0x5591931c4b90 2023-10-10T17:45:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f085f5adb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919322de40 con 0x55919301a6b0 2023-10-10T17:45:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f085f5adb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919322e000 con 0x5591931c4720 2023-10-10T17:45:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f085f5adb80 10 client.0 _ll_register_callbacks cb 0x7f083c00b980 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08511bd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931d6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931d6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37796/0 (socket says 172.21.15.78:37796) 2023-10-10T17:45:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 -- 172.21.15.78:0/1235997429 learned_addr learned my addr 172.21.15.78:0/1235997429 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08511bd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:47986/0 (socket says 172.21.15.78:47986) 2023-10-10T17:45:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08519be700 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x55919322d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 -- 172.21.15.78:0/1235997429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 msgr2=0x5591931d5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 -- 172.21.15.78:0/1235997429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 msgr2=0x55919322d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x55919322d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.479+0000 7f08509bc700 1 -- 172.21.15.78:0/1235997429 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559192f915d0 con 0x5591931c4720 2023-10-10T17:45:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f08509bc700 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931d6050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f083c017530 tx=0x7f083c017d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86935dcfcf9ca210 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f0834ff9700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:45:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f0834ff9700 1 -- 172.21.15.78:0/1235997429 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c00fe60 con 0x5591931c4720 2023-10-10T17:45:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f084809a100 con 0x5591931c4720 2023-10-10T17:45:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f0834ff9700 1 -- 172.21.15.78:0/1235997429 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f083c00be50 con 0x5591931c4720 2023-10-10T17:45:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f0834ff9700 1 -- 172.21.15.78:0/1235997429 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f083c017800 con 0x5591931c4720 2023-10-10T17:45:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f085f5adb80 10 client.25801 fetch_fsmap learned FSMap version 28 2023-10-10T17:45:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559192f26c00 con 0x5591931c4720 2023-10-10T17:45:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.480+0000 7f0834ff9700 1 -- 172.21.15.78:0/1235997429 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f084809a100 con 0x5591931c4720 2023-10-10T17:45:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f0834ff9700 1 -- 172.21.15.78:0/1235997429 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f083c02d070 con 0x5591931c4720 2023-10-10T17:45:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f085f5adb80 10 client.25801 fetch_fsmap finished waiting for FSMap version 28 2023-10-10T17:45:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f085f5adb80 10 client.25801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f085f5adb80 20 client.25801 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f085f5adb80 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591932352b0 0x5591932376c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f085f5adb80 4 client.25801 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:45:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5591930de600 con 0x5591932352b0 2023-10-10T17:45:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f08511bd700 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591932352b0 0x5591932376c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f08511bd700 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591932352b0 0x5591932376c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c6d57f22b9de901 server_cookie=77196799ed5d82ed in_seq=0 out_seq=0 2023-10-10T17:45:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:21.481+0000 7f0834ff9700 10 client.25801 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:22.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:22 smithi190 ceph-mon[101220]: pgmap v88: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.9 MiB/s wr, 598 op/s 2023-10-10T17:45:22.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:22 smithi157 ceph-mon[100896]: pgmap v88: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.9 MiB/s wr, 598 op/s 2023-10-10T17:45:23.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:22 smithi078 ceph-mon[99494]: pgmap v88: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.9 MiB/s wr, 598 op/s 2023-10-10T17:45:24.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:24 smithi157 ceph-mon[100896]: pgmap v89: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 472 op/s 2023-10-10T17:45:25.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:24 smithi078 ceph-mon[99494]: pgmap v89: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 472 op/s 2023-10-10T17:45:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:24 smithi190 ceph-mon[101220]: pgmap v89: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 472 op/s 2023-10-10T17:45:26.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:25 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:26.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:25 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:26.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:25 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:27.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:26 smithi078 ceph-mon[99494]: pgmap v90: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 472 op/s 2023-10-10T17:45:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:26 smithi190 ceph-mon[101220]: pgmap v90: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 472 op/s 2023-10-10T17:45:27.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:26 smithi157 ceph-mon[100896]: pgmap v90: 129 pgs: 129 active+clean; 109 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 4.0 MiB/s wr, 472 op/s 2023-10-10T17:45:28.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:27 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:28.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:27 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:28.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:27 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:29.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:28 smithi078 ceph-mon[99494]: pgmap v91: 129 pgs: 129 active+clean; 124 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 7.0 MiB/s wr, 2.67k op/s 2023-10-10T17:45:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:28 smithi190 ceph-mon[101220]: pgmap v91: 129 pgs: 129 active+clean; 124 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 7.0 MiB/s wr, 2.67k op/s 2023-10-10T17:45:29.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:28 smithi157 ceph-mon[100896]: pgmap v91: 129 pgs: 129 active+clean; 124 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 7.0 MiB/s wr, 2.67k op/s 2023-10-10T17:45:30.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.573+0000 7f0834ff9700 1 -- 172.21.15.78:0/1235997429 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5591930de600 con 0x5591932352b0 2023-10-10T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.573+0000 7f0834ff9700 10 client.25801 handle_command_reply: tid=0 2023-10-10T17:45:30.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.573+0000 7f085f5adb80 1 client.25801 shutdown 2023-10-10T17:45:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.573+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 msgr2=0x5591931d6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:30.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.573+0000 7f085f5adb80 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931d6050 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f083c017530 tx=0x7f083c017d90 comp rx=0 tx=0).stop 2023-10-10T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 shutdown_connections 2023-10-10T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5591932352b0 0x5591932376c0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5591931c4b90 0x55919322d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f0834ff9700 0 client.25801 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:30.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55919301a6b0 0x5591931d5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 --2- 172.21.15.78:0/1235997429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5591931c4720 0x5591931d6050 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 >> 172.21.15.78:0/1235997429 conn(0x5591931b3f50 msgr2=0x5591931c1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 shutdown_connections 2023-10-10T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 1 -- 172.21.15.78:0/1235997429 wait complete. 2023-10-10T17:45:30.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.574+0000 7f085f5adb80 20 client.25801 trim_cache size 0 max 16384 2023-10-10T17:45:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:45:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:45:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:45:30.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:45:30.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08380a5750 msgr2=0x7f08380a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08380a5750 0x7f08380a7c00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f083c00b580 tx=0x7f083c003680 comp rx=0 tx=0).stop 2023-10-10T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0fe9e0 msgr2=0x7f084c06b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0fe9e0 0x7f084c06b050 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f084800e9a0 tx=0x7f084800ecb0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 shutdown_connections 2023-10-10T17:45:30.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08380a5750 0x7f08380a7c00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f084c105580 0x7f084c1aa8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f084c0fe9e0 0x7f084c06b050 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 --2- 172.21.15.78:0/3619978155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f084c0ff380 0x7f084c06b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 >> 172.21.15.78:0/3619978155 conn(0x7f084c075810 msgr2=0x7f084c0fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:30.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 shutdown_connections 2023-10-10T17:45:30.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:30.575+0000 7f0853421700 1 -- 172.21.15.78:0/3619978155 wait complete. 2023-10-10T17:45:31.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:30 smithi078 ceph-mon[99494]: pgmap v92: 129 pgs: 129 active+clean; 124 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.3 MiB/s wr, 2.35k op/s 2023-10-10T17:45:31.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:30 smithi190 ceph-mon[101220]: pgmap v92: 129 pgs: 129 active+clean; 124 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.3 MiB/s wr, 2.35k op/s 2023-10-10T17:45:31.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:30 smithi157 ceph-mon[100896]: pgmap v92: 129 pgs: 129 active+clean; 124 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 5.3 MiB/s wr, 2.35k op/s 2023-10-10T17:45:33.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:32 smithi078 ceph-mon[99494]: pgmap v93: 129 pgs: 129 active+clean; 125 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.5 MiB/s wr, 2.50k op/s 2023-10-10T17:45:33.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:32 smithi190 ceph-mon[101220]: pgmap v93: 129 pgs: 129 active+clean; 125 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.5 MiB/s wr, 2.50k op/s 2023-10-10T17:45:33.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:32 smithi157 ceph-mon[100896]: pgmap v93: 129 pgs: 129 active+clean; 125 MiB data, 583 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 6.5 MiB/s wr, 2.50k op/s 2023-10-10T17:45:34.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:34 smithi190 ceph-mon[101220]: pgmap v94: 129 pgs: 129 active+clean; 135 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.7 MiB/s wr, 2.66k op/s 2023-10-10T17:45:34.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:34 smithi157 ceph-mon[100896]: pgmap v94: 129 pgs: 129 active+clean; 135 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.7 MiB/s wr, 2.66k op/s 2023-10-10T17:45:35.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:34 smithi078 ceph-mon[99494]: pgmap v94: 129 pgs: 129 active+clean; 135 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.7 MiB/s wr, 2.66k op/s 2023-10-10T17:45:36.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:36 smithi190 ceph-mon[101220]: pgmap v95: 129 pgs: 129 active+clean; 135 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.7 MiB/s wr, 2.66k op/s 2023-10-10T17:45:36.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:36 smithi157 ceph-mon[100896]: pgmap v95: 129 pgs: 129 active+clean; 135 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.7 MiB/s wr, 2.66k op/s 2023-10-10T17:45:37.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:36 smithi078 ceph-mon[99494]: pgmap v95: 129 pgs: 129 active+clean; 135 MiB data, 654 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 6.7 MiB/s wr, 2.66k op/s 2023-10-10T17:45:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:38 smithi190 ceph-mon[101220]: pgmap v96: 129 pgs: 129 active+clean; 170 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.7 MiB/s wr, 2.70k op/s 2023-10-10T17:45:38.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:38 smithi157 ceph-mon[100896]: pgmap v96: 129 pgs: 129 active+clean; 170 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.7 MiB/s wr, 2.70k op/s 2023-10-10T17:45:39.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:38 smithi078 ceph-mon[99494]: pgmap v96: 129 pgs: 129 active+clean; 170 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.7 MiB/s wr, 2.70k op/s 2023-10-10T17:45:40.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:40 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:40 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:40 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:40 smithi190 ceph-mon[101220]: pgmap v97: 129 pgs: 129 active+clean; 170 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.7 MiB/s wr, 502 op/s 2023-10-10T17:45:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:40 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:40.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:40 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:40 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:40 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:40 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:40.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:40 smithi157 ceph-mon[100896]: pgmap v97: 129 pgs: 129 active+clean; 170 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.7 MiB/s wr, 502 op/s 2023-10-10T17:45:40.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:40 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:40.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:40 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:41.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:40 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:40 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:40 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:40 smithi078 ceph-mon[99494]: pgmap v97: 129 pgs: 129 active+clean; 170 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.7 MiB/s wr, 502 op/s 2023-10-10T17:45:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:40 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:41.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:40 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:42.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:45:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 -- 172.21.15.78:0/2614397967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da8100f10 msgr2=0x7f3da8104ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 --2- 172.21.15.78:0/2614397967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da8100f10 0x7f3da8104ed0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0099c0 tx=0x7f3d9c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 -- 172.21.15.78:0/2614397967 shutdown_connections 2023-10-10T17:45:42.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 --2- 172.21.15.78:0/2614397967 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 0x7f3da8100940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 --2- 172.21.15.78:0/2614397967 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3da81055b0 0x7f3da810d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 --2- 172.21.15.78:0/2614397967 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da8100f10 0x7f3da8104ed0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 -- 172.21.15.78:0/2614397967 >> 172.21.15.78:0/2614397967 conn(0x7f3da80770e0 msgr2=0x7f3da80774e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:42.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 -- 172.21.15.78:0/2614397967 shutdown_connections 2023-10-10T17:45:42.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.761+0000 7f3dadffc700 1 -- 172.21.15.78:0/2614397967 wait complete. 2023-10-10T17:45:42.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.762+0000 7f3dadffc700 1 Processor -- start 2023-10-10T17:45:42.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.762+0000 7f3dadffc700 1 -- start start 2023-10-10T17:45:42.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3dadffc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 0x7f3da8196490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3dadffc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3da8100f10 0x7f3da81969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3dadffc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da81055b0 0x7f3da819ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3dadffc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da819b470 con 0x7f3da81055b0 2023-10-10T17:45:42.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3dadffc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da819b5e0 con 0x7f3da8100f10 2023-10-10T17:45:42.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3dadffc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da819b7a0 con 0x7f3da8100570 2023-10-10T17:45:42.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da81055b0 0x7f3da819ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 0x7f3da8196490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da81055b0 0x7f3da819ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46282/0 (socket says 172.21.15.78:46282) 2023-10-10T17:45:42.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 0x7f3da8196490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37808/0 (socket says 172.21.15.78:37808) 2023-10-10T17:45:42.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da7fff700 1 -- 172.21.15.78:0/4057490616 learned_addr learned my addr 172.21.15.78:0/4057490616 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:42.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3da8100f10 0x7f3da81969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 -- 172.21.15.78:0/4057490616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3da8100f10 msgr2=0x7f3da81969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3da8100f10 0x7f3da81969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 -- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da81055b0 msgr2=0x7f3da819ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da81055b0 0x7f3da819ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 -- 172.21.15.78:0/4057490616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0095f0 con 0x7f3da8100570 2023-10-10T17:45:42.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da7fff700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da81055b0 0x7f3da819ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:42.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3da8100f10 0x7f3da81969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:42.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.763+0000 7f3da77fe700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 0x7f3da8196490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800e9f0 tx=0x7f3d9800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:42.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.764+0000 7f3da4ff9700 1 -- 172.21.15.78:0/4057490616 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9800c9c0 con 0x7f3da8100570 2023-10-10T17:45:42.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.764+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da8110330 con 0x7f3da8100570 2023-10-10T17:45:42.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.764+0000 7f3da4ff9700 1 -- 172.21.15.78:0/4057490616 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d98010430 con 0x7f3da8100570 2023-10-10T17:45:42.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.764+0000 7f3da4ff9700 1 -- 172.21.15.78:0/4057490616 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d98018680 con 0x7f3da8100570 2023-10-10T17:45:42.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.764+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da81101a0 con 0x7f3da8100570 2023-10-10T17:45:42.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.766+0000 7f3da4ff9700 1 -- 172.21.15.78:0/4057490616 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3d98010860 con 0x7f3da8100570 2023-10-10T17:45:42.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.767+0000 7f3da4ff9700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d900a12c0 0x7f3d900a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.767+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d900a12c0 0x7f3d900a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.767+0000 7f3da4ff9700 1 -- 172.21.15.78:0/4057490616 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3d98014070 con 0x7f3da8100570 2023-10-10T17:45:42.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.767+0000 7f3dba188b80 1 Processor -- start 2023-10-10T17:45:42.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.767+0000 7f3dba188b80 1 -- start start 2023-10-10T17:45:42.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.767+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d900a12c0 0x7f3d900a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0098e0 tx=0x7f3d9c01a040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:42.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b23780 0x564ec5b23b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b26570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b26ab0 0x564ec5b26f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3dba188b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5a23060 con 0x564ec5b24090 2023-10-10T17:45:42.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3dba188b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5a758a0 con 0x564ec5b23780 2023-10-10T17:45:42.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3dba188b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5a6ed30 con 0x564ec5b26ab0 2023-10-10T17:45:42.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b26570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b26570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46288/0 (socket says 172.21.15.78:46288) 2023-10-10T17:45:42.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 -- 172.21.15.78:0/186282729 learned_addr learned my addr 172.21.15.78:0/186282729 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:42.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da7fff700 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b26ab0 0x564ec5b26f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da77fe700 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b23780 0x564ec5b23b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 -- 172.21.15.78:0/186282729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b26ab0 msgr2=0x564ec5b26f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b26ab0 0x564ec5b26f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 -- 172.21.15.78:0/186282729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b23780 msgr2=0x564ec5b23b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b23780 0x564ec5b23b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 -- 172.21.15.78:0/186282729 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ec58f0630 con 0x564ec5b24090 2023-10-10T17:45:42.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da77fe700 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b23780 0x564ec5b23b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:45:42.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b26570 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0095c0 tx=0x7f3d9c009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d72c8013786a6d57 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:42.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3d8dffb700 1 -- 172.21.15.78:0/186282729 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c04c070 con 0x564ec5b24090 2023-10-10T17:45:42.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3d8dffb700 1 -- 172.21.15.78:0/186282729 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d9c01e040 con 0x564ec5b24090 2023-10-10T17:45:42.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.768+0000 7f3d8dffb700 1 -- 172.21.15.78:0/186282729 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c00f920 con 0x564ec5b24090 2023-10-10T17:45:42.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 -- 172.21.15.78:0/186282729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 msgr2=0x564ec5b26570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b26570 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0095c0 tx=0x7f3d9c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 -- 172.21.15.78:0/186282729 shutdown_connections 2023-10-10T17:45:42.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b26ab0 0x564ec5b26f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b23780 0x564ec5b23b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 --2- 172.21.15.78:0/186282729 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b26570 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 -- 172.21.15.78:0/186282729 >> 172.21.15.78:0/186282729 conn(0x564ec5b13360 msgr2=0x564ec5b187c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:42.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 -- 172.21.15.78:0/186282729 shutdown_connections 2023-10-10T17:45:42.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.769+0000 7f3dba188b80 1 -- 172.21.15.78:0/186282729 wait complete. 2023-10-10T17:45:42.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 -- start start 2023-10-10T17:45:42.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 Processor -- start 2023-10-10T17:45:42.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b2d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b2da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b277f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5a108e0 con 0x564ec5b26ab0 2023-10-10T17:45:42.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5a10580 con 0x564ec5b24090 2023-10-10T17:45:42.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec58f2920 con 0x564ec5b23780 2023-10-10T17:45:42.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3dba188b80 10 client.0 _ll_register_callbacks cb 0x564ec5a2e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:42.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b277f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b2d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b277f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46296/0 (socket says 172.21.15.78:46296) 2023-10-10T17:45:42.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b2d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37822/0 (socket says 172.21.15.78:37822) 2023-10-10T17:45:42.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.770+0000 7f3da7fff700 1 -- 172.21.15.78:0/962477387 learned_addr learned my addr 172.21.15.78:0/962477387 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:42.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b2da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da7fff700 1 -- 172.21.15.78:0/962477387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 msgr2=0x564ec5b2d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da7fff700 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b2d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da7fff700 1 -- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 msgr2=0x564ec5b2da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da7fff700 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b2da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da7fff700 1 -- 172.21.15.78:0/962477387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da810d9c0 con 0x564ec5b26ab0 2023-10-10T17:45:42.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b2da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:45:42.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3da7fff700 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b277f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3da0007fb0 tx=0x7f3da00132b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8034c7711152c8b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:42.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3d86ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:42.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3d86ffd700 1 -- 172.21.15.78:0/962477387 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da0004bb0 con 0x564ec5b26ab0 2023-10-10T17:45:42.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3d86ffd700 1 -- 172.21.15.78:0/962477387 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3da0004d10 con 0x564ec5b26ab0 2023-10-10T17:45:42.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ec5b10b20 con 0x564ec5b26ab0 2023-10-10T17:45:42.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3d86ffd700 1 -- 172.21.15.78:0/962477387 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3da0013bf0 con 0x564ec5b26ab0 2023-10-10T17:45:42.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3dba188b80 10 client.16359 fetch_fsmap learned FSMap version 28 2023-10-10T17:45:42.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.771+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564ec5a35190 con 0x564ec5b26ab0 2023-10-10T17:45:42.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3d86ffd700 1 -- 172.21.15.78:0/962477387 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564ec5b10b20 con 0x564ec5b26ab0 2023-10-10T17:45:42.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3d86ffd700 1 -- 172.21.15.78:0/962477387 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f3da0005020 con 0x564ec5b26ab0 2023-10-10T17:45:42.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3dba188b80 10 client.16359 fetch_fsmap finished waiting for FSMap version 28 2023-10-10T17:45:42.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3dba188b80 10 client.16359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:42.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3dba188b80 20 client.16359 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:42.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3dba188b80 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b37a10 0x564ec5b39e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3dba188b80 4 client.16359 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:42.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ec5a310a0 con 0x564ec5b37a10 2023-10-10T17:45:42.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.772+0000 7f3da77fe700 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b37a10 0x564ec5b39e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3d86ffd700 10 client.16359 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:42.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3da77fe700 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b37a10 0x564ec5b39e20 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d72da0408c4da14 server_cookie=61036e81cc8be1b0 in_seq=0 out_seq=0 2023-10-10T17:45:42.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3d86ffd700 1 -- 172.21.15.78:0/962477387 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564ec5a310a0 con 0x564ec5b37a10 2023-10-10T17:45:42.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3d86ffd700 10 client.16359 handle_command_reply: tid=0 2023-10-10T17:45:42.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 client.16359 shutdown 2023-10-10T17:45:42.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 msgr2=0x564ec5b277f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b277f0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f3da0007fb0 tx=0x7f3da00132b0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 shutdown_connections 2023-10-10T17:45:42.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b2d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b37a10 0x564ec5b39e20 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b277f0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 --2- 172.21.15.78:0/962477387 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b2da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3d86ffd700 0 client.16359 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:42.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.773+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 >> 172.21.15.78:0/962477387 conn(0x564ec5b13360 msgr2=0x564ec5b17de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:42.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.774+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 shutdown_connections 2023-10-10T17:45:42.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.774+0000 7f3dba188b80 1 -- 172.21.15.78:0/962477387 wait complete. 2023-10-10T17:45:42.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.774+0000 7f3dba188b80 20 client.16359 trim_cache size 0 max 16384 2023-10-10T17:45:42.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 Processor -- start 2023-10-10T17:45:42.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 -- start start 2023-10-10T17:45:42.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b18560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b27090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b26ab0 0x564ec5b275d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5b26ec0 con 0x564ec5b24090 2023-10-10T17:45:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5b34fe0 con 0x564ec5b26ab0 2023-10-10T17:45:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.791+0000 7f3dba188b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5b22190 con 0x564ec5b23780 2023-10-10T17:45:42.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b27090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b18560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b27090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46302/0 (socket says 172.21.15.78:46302) 2023-10-10T17:45:42.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b18560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37830/0 (socket says 172.21.15.78:37830) 2023-10-10T17:45:42.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 -- 172.21.15.78:0/3384298410 learned_addr learned my addr 172.21.15.78:0/3384298410 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:42.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da7fff700 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b26ab0 0x564ec5b275d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 -- 172.21.15.78:0/3384298410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 msgr2=0x564ec5b18560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b18560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 -- 172.21.15.78:0/3384298410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b26ab0 msgr2=0x564ec5b275d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b26ab0 0x564ec5b275d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 -- 172.21.15.78:0/3384298410 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ec5a34ee0 con 0x564ec5b24090 2023-10-10T17:45:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da77fe700 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b18560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da7fff700 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b26ab0 0x564ec5b275d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b27090 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0036c0 tx=0x7f3d9c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d893204be426afd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3d8e7fc700 1 -- 172.21.15.78:0/3384298410 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c04c040 con 0x564ec5b24090 2023-10-10T17:45:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3d8e7fc700 1 -- 172.21.15.78:0/3384298410 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d9c00fd50 con 0x564ec5b24090 2023-10-10T17:45:42.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.792+0000 7f3d8e7fc700 1 -- 172.21.15.78:0/3384298410 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9c005710 con 0x564ec5b24090 2023-10-10T17:45:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 -- 172.21.15.78:0/3384298410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 msgr2=0x564ec5b27090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b27090 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0036c0 tx=0x7f3d9c00f4a0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 -- 172.21.15.78:0/3384298410 shutdown_connections 2023-10-10T17:45:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b18560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b26ab0 0x564ec5b275d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 --2- 172.21.15.78:0/3384298410 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b24090 0x564ec5b27090 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 -- 172.21.15.78:0/3384298410 >> 172.21.15.78:0/3384298410 conn(0x564ec5b13360 msgr2=0x564ec5b1f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 -- 172.21.15.78:0/3384298410 shutdown_connections 2023-10-10T17:45:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.793+0000 7f3dba188b80 1 -- 172.21.15.78:0/3384298410 wait complete. 2023-10-10T17:45:42.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 -- start start 2023-10-10T17:45:42.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 Processor -- start 2023-10-10T17:45:42.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b7fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b7ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b8a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5b80720 con 0x564ec5b26ab0 2023-10-10T17:45:42.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5b8ae50 con 0x564ec5b24090 2023-10-10T17:45:42.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ec5b8b010 con 0x564ec5b23780 2023-10-10T17:45:42.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3dba188b80 10 client.0 _ll_register_callbacks cb 0x564ec5a2e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:42.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b7fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b8a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3da77fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b7fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37836/0 (socket says 172.21.15.78:37836) 2023-10-10T17:45:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b8a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46310/0 (socket says 172.21.15.78:46310) 2023-10-10T17:45:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3da77fe700 1 -- 172.21.15.78:0/2583318474 learned_addr learned my addr 172.21.15.78:0/2583318474 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.794+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b7ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3da77fe700 1 -- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 msgr2=0x564ec5b7ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3da77fe700 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b7ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3da77fe700 1 -- 172.21.15.78:0/2583318474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 msgr2=0x564ec5b8a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3da77fe700 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b8a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3da77fe700 1 -- 172.21.15.78:0/2583318474 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da81101a0 con 0x564ec5b23780 2023-10-10T17:45:42.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3da77fe700 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b7fa50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800c5b0 tx=0x7f3d9809e070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=637942094dd95fe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3d86ffd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:45:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3d86ffd700 1 -- 172.21.15.78:0/2583318474 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d9809a120 con 0x564ec5b23780 2023-10-10T17:45:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3d86ffd700 1 -- 172.21.15.78:0/2583318474 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3d9809a280 con 0x564ec5b23780 2023-10-10T17:45:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d9c01e040 con 0x564ec5b23780 2023-10-10T17:45:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3d86ffd700 1 -- 172.21.15.78:0/2583318474 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d980b8260 con 0x564ec5b23780 2023-10-10T17:45:42.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3dba188b80 10 client.25819 fetch_fsmap learned FSMap version 28 2023-10-10T17:45:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564ec5a35190 con 0x564ec5b23780 2023-10-10T17:45:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3d86ffd700 1 -- 172.21.15.78:0/2583318474 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d980ca020 con 0x564ec5b23780 2023-10-10T17:45:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.795+0000 7f3d86ffd700 1 -- 172.21.15.78:0/2583318474 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f3d980a1e80 con 0x564ec5b23780 2023-10-10T17:45:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3dba188b80 10 client.25819 fetch_fsmap finished waiting for FSMap version 28 2023-10-10T17:45:42.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3dba188b80 10 client.25819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:42.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3dba188b80 20 client.25819 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:42.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3dba188b80 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b8c750 0x564ec5b8eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:42.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3dba188b80 4 client.25819 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:45:42.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564ec5a3d660 con 0x564ec5b8c750 2023-10-10T17:45:42.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b8c750 0x564ec5b8eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:42.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3da6ffd700 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b8c750 0x564ec5b8eb60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faa46e4817facdb1 server_cookie=13d326afc7d594dd in_seq=0 out_seq=0 2023-10-10T17:45:42.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3d86ffd700 10 client.25819 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:42.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3d86ffd700 1 -- 172.21.15.78:0/2583318474 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564ec5a3d660 con 0x564ec5b8c750 2023-10-10T17:45:42.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3d86ffd700 10 client.25819 handle_command_reply: tid=0 2023-10-10T17:45:42.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.796+0000 7f3dba188b80 1 client.25819 shutdown 2023-10-10T17:45:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 msgr2=0x564ec5b7fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b7fa50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800c5b0 tx=0x7f3d9809e070 comp rx=0 tx=0).stop 2023-10-10T17:45:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 shutdown_connections 2023-10-10T17:45:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564ec5b23780 0x564ec5b7fa50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564ec5b8c750 0x564ec5b8eb60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564ec5b26ab0 0x564ec5b8a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3d86ffd700 0 client.25819 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:42.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 --2- 172.21.15.78:0/2583318474 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564ec5b24090 0x564ec5b7ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 >> 172.21.15.78:0/2583318474 conn(0x564ec5b13360 msgr2=0x564ec5b27090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:42.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 shutdown_connections 2023-10-10T17:45:42.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 1 -- 172.21.15.78:0/2583318474 wait complete. 2023-10-10T17:45:42.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dba188b80 20 client.25819 trim_cache size 0 max 16384 2023-10-10T17:45:42.821 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:45:42.821 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:45:42.822 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:45:42.822 INFO:teuthology.orchestra.run.smithi078.stdout: "5ecf957f-7c91-40df-8738-d2beae2feba1": { 2023-10-10T17:45:42.822 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:45:42.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:45:42.822 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:45:42.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:45:42.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:45:42.823 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:45:42.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.797+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d900a12c0 msgr2=0x7f3d900a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d900a12c0 0x7f3d900a3770 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d9c0098e0 tx=0x7f3d9c01a040 comp rx=0 tx=0).stop 2023-10-10T17:45:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 msgr2=0x7f3da8196490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 0x7f3da8196490 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d9800e9f0 tx=0x7f3d9800ed00 comp rx=0 tx=0).stop 2023-10-10T17:45:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 shutdown_connections 2023-10-10T17:45:42.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3da8100570 0x7f3da8196490 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d900a12c0 0x7f3d900a3770 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3da81055b0 0x7f3da819ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 --2- 172.21.15.78:0/4057490616 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3da8100f10 0x7f3da81969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:42.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 >> 172.21.15.78:0/4057490616 conn(0x7f3da80770e0 msgr2=0x7f3da80fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:42.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 shutdown_connections 2023-10-10T17:45:42.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:45:42.798+0000 7f3dadffc700 1 -- 172.21.15.78:0/4057490616 wait complete. 2023-10-10T17:45:42.827 INFO:tasks.cephfs.filesystem:scrub status for tag:5ecf957f-7c91-40df-8738-d2beae2feba1 - {'path': '/', 'tag': '5ecf957f-7c91-40df-8738-d2beae2feba1', 'options': 'recursive,force'} 2023-10-10T17:45:42.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:42 smithi190 ceph-mon[101220]: pgmap v98: 129 pgs: 129 active+clean; 177 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.4 MiB/s wr, 643 op/s 2023-10-10T17:45:42.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:42 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:42.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:42 smithi157 ceph-mon[100896]: pgmap v98: 129 pgs: 129 active+clean; 177 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.4 MiB/s wr, 643 op/s 2023-10-10T17:45:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:42 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:43.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:42 smithi078 ceph-mon[99494]: pgmap v98: 129 pgs: 129 active+clean; 177 MiB data, 685 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.4 MiB/s wr, 643 op/s 2023-10-10T17:45:43.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:42 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:43.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/483729146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d8410a860 msgr2=0x7f8d8410cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:43.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/483729146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d8410a860 0x7f8d8410cca0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00b0a0 tx=0x7f8d7c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/483729146 shutdown_connections 2023-10-10T17:45:43.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/483729146 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d84101490 0x7f8d84101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/483729146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d8410a860 0x7f8d8410cca0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/483729146 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d84101da0 0x7f8d8410a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/483729146 >> 172.21.15.78:0/483729146 conn(0x7f8d84076370 msgr2=0x7f8d84076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:43.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/483729146 shutdown_connections 2023-10-10T17:45:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.687+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/483729146 wait complete. 2023-10-10T17:45:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 Processor -- start 2023-10-10T17:45:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 -- start start 2023-10-10T17:45:43.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d84101490 0x7f8d840ff0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d84101da0 0x7f8d840ff620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d8410a860 0x7f8d841a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d840ffe50 con 0x7f8d84101da0 2023-10-10T17:45:43.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d841a4b80 con 0x7f8d84101490 2023-10-10T17:45:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8c0c0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d841a4cc0 con 0x7f8d8410a860 2023-10-10T17:45:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d84101da0 0x7f8d840ff620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d89e5c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d84101490 0x7f8d840ff0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d84101da0 0x7f8d840ff620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46322/0 (socket says 172.21.15.78:46322) 2023-10-10T17:45:43.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d89e5c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d84101490 0x7f8d840ff0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48042/0 (socket says 172.21.15.78:48042) 2023-10-10T17:45:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 -- 172.21.15.78:0/4101110215 learned_addr learned my addr 172.21.15.78:0/4101110215 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8a65d700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d8410a860 0x7f8d841a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 -- 172.21.15.78:0/4101110215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d8410a860 msgr2=0x7f8d841a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d8410a860 0x7f8d841a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 -- 172.21.15.78:0/4101110215 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d84101490 msgr2=0x7f8d840ff0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d84101490 0x7f8d840ff0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 -- 172.21.15.78:0/4101110215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c009d00 con 0x7f8d84101da0 2023-10-10T17:45:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d8965b700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d84101da0 0x7f8d840ff620 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000d980 tx=0x7f8d8000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.688+0000 7f8d7affd700 1 -- 172.21.15.78:0/4101110215 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80004220 con 0x7f8d84101da0 2023-10-10T17:45:43.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.689+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d84110320 con 0x7f8d84101da0 2023-10-10T17:45:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.689+0000 7f8d7affd700 1 -- 172.21.15.78:0/4101110215 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d80007640 con 0x7f8d84101da0 2023-10-10T17:45:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.689+0000 7f8d7affd700 1 -- 172.21.15.78:0/4101110215 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80005690 con 0x7f8d84101da0 2023-10-10T17:45:43.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.689+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d84110190 con 0x7f8d84101da0 2023-10-10T17:45:43.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.691+0000 7f8d7affd700 1 -- 172.21.15.78:0/4101110215 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f8d80004750 con 0x7f8d84101da0 2023-10-10T17:45:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.691+0000 7f8d7affd700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8d700a12c0 0x7f8d700a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.691+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8d700a12c0 0x7f8d700a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.691+0000 7f8d7affd700 1 -- 172.21.15.78:0/4101110215 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f8d800ccd60 con 0x7f8d84101da0 2023-10-10T17:45:43.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.692+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8d700a12c0 0x7f8d700a3770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009c60 tx=0x7f8d74009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.692+0000 7f8d9824cb80 1 Processor -- start 2023-10-10T17:45:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.692+0000 7f8d9824cb80 1 -- start start 2023-10-10T17:45:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x5617364641e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x56173645e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.707 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x56173645eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561736364000 con 0x561736464720 2023-10-10T17:45:43.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617363b6840 con 0x561736464b90 2023-10-10T17:45:43.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617363afcd0 con 0x5617362ba6b0 2023-10-10T17:45:43.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x56173645e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.708 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d89e5c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x5617364641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x56173645e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46330/0 (socket says 172.21.15.78:46330) 2023-10-10T17:45:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 -- 172.21.15.78:0/3138165621 learned_addr learned my addr 172.21.15.78:0/3138165621 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d89e5c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x5617364641e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37858/0 (socket says 172.21.15.78:37858) 2023-10-10T17:45:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8a65d700 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x56173645eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.709 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 -- 172.21.15.78:0/3138165621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 msgr2=0x5617364641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x5617364641e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 -- 172.21.15.78:0/3138165621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 msgr2=0x56173645eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x56173645eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 -- 172.21.15.78:0/3138165621 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617362315d0 con 0x561736464720 2023-10-10T17:45:43.710 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x56173645e560 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8d80096570 tx=0x7f8d800223d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c385855dd7fe63c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d6ffff700 1 -- 172.21.15.78:0/3138165621 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d800b9050 con 0x561736464720 2023-10-10T17:45:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d6ffff700 1 -- 172.21.15.78:0/3138165621 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d8009db30 con 0x561736464720 2023-10-10T17:45:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.693+0000 7f8d6ffff700 1 -- 172.21.15.78:0/3138165621 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80099250 con 0x561736464720 2023-10-10T17:45:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3138165621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 msgr2=0x56173645e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:43.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x56173645e560 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8d80096570 tx=0x7f8d800223d0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3138165621 shutdown_connections 2023-10-10T17:45:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x5617364641e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x56173645eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3138165621 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x56173645e560 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3138165621 >> 172.21.15.78:0/3138165621 conn(0x561736453f50 msgr2=0x561736459270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3138165621 shutdown_connections 2023-10-10T17:45:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3138165621 wait complete. 2023-10-10T17:45:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 -- start start 2023-10-10T17:45:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.694+0000 7f8d9824cb80 1 Processor -- start 2023-10-10T17:45:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x56173646ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736467d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464b90 0x561736468460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561736351880 con 0x561736464720 2023-10-10T17:45:43.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561736351520 con 0x5617362ba6b0 2023-10-10T17:45:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617362338c0 con 0x561736464b90 2023-10-10T17:45:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d9824cb80 10 client.0 _ll_register_callbacks cb 0x56173636f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736467d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d8a65d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464b90 0x561736468460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d89e5c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x56173646ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736467d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46336/0 (socket says 172.21.15.78:46336) 2023-10-10T17:45:43.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d8965b700 1 -- 172.21.15.78:0/3914866345 learned_addr learned my addr 172.21.15.78:0/3914866345 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:43.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d8a65d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464b90 0x561736468460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37868/0 (socket says 172.21.15.78:37868) 2023-10-10T17:45:43.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d89e5c700 1 -- 172.21.15.78:0/3914866345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464b90 msgr2=0x561736468460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464b90 0x561736468460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d89e5c700 1 -- 172.21.15.78:0/3914866345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 msgr2=0x561736467d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736467d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d89e5c700 1 -- 172.21.15.78:0/3914866345 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d84110190 con 0x5617362ba6b0 2023-10-10T17:45:43.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d8a65d700 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464b90 0x561736468460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:43.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.695+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736467d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:45:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x56173646ab10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d74011ab0 tx=0x7f8d740047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4034d20796bfd520 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d6d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:45:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/3914866345 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d74022070 con 0x5617362ba6b0 2023-10-10T17:45:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/3914866345 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f8d74027420 con 0x5617362ba6b0 2023-10-10T17:45:43.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561736451710 con 0x5617362ba6b0 2023-10-10T17:45:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/3914866345 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d74004560 con 0x5617362ba6b0 2023-10-10T17:45:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 10 client.25736 fetch_fsmap learned FSMap version 28 2023-10-10T17:45:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561736376130 con 0x5617362ba6b0 2023-10-10T17:45:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/3914866345 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561736451710 con 0x5617362ba6b0 2023-10-10T17:45:43.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/3914866345 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f8d7401c670 con 0x5617362ba6b0 2023-10-10T17:45:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 10 client.25736 fetch_fsmap finished waiting for FSMap version 28 2023-10-10T17:45:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 10 client.25736 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 20 client.25736 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561736478050 0x56173647a460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 4 client.25736 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:45:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.696+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561736372040 con 0x561736478050 2023-10-10T17:45:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.697+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561736478050 0x56173647a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.697+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561736478050 0x56173647a460 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3431163634c5a74 server_cookie=9320b142c22ae8d4 in_seq=0 out_seq=0 2023-10-10T17:45:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.697+0000 7f8d6d7fa700 10 client.25736 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:43.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/3914866345 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561736372040 con 0x561736478050 2023-10-10T17:45:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d6d7fa700 10 client.25736 handle_command_reply: tid=0 2023-10-10T17:45:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 client.25736 shutdown 2023-10-10T17:45:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 msgr2=0x56173646ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x56173646ab10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d74011ab0 tx=0x7f8d740047c0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 shutdown_connections 2023-10-10T17:45:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464b90 0x561736468460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561736478050 0x56173647a460 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x56173646ab10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d6d7fa700 0 client.25736 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3914866345 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736467d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 >> 172.21.15.78:0/3914866345 conn(0x561736453f50 msgr2=0x5617364587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 shutdown_connections 2023-10-10T17:45:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3914866345 wait complete. 2023-10-10T17:45:43.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.699+0000 7f8d9824cb80 20 client.25736 trim_cache size 0 max 16384 2023-10-10T17:45:43.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.713+0000 7f8d9824cb80 1 Processor -- start 2023-10-10T17:45:43.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.713+0000 7f8d9824cb80 1 -- start start 2023-10-10T17:45:43.729 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x56173646d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736461ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x561736462420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56173646dba0 con 0x561736464720 2023-10-10T17:45:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56173647a430 con 0x561736464b90 2023-10-10T17:45:43.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56173645e560 con 0x5617362ba6b0 2023-10-10T17:45:43.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736461ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736461ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46344/0 (socket says 172.21.15.78:46344) 2023-10-10T17:45:43.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 -- 172.21.15.78:0/3442602187 learned_addr learned my addr 172.21.15.78:0/3442602187 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:43.731 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x56173646d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8a65d700 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x561736462420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 -- 172.21.15.78:0/3442602187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 msgr2=0x56173646d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x56173646d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 -- 172.21.15.78:0/3442602187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 msgr2=0x561736462420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.732 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x561736462420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 -- 172.21.15.78:0/3442602187 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561736375e80 con 0x561736464720 2023-10-10T17:45:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d8965b700 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736461ee0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8d80099cc0 tx=0x7f8d800050b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ded92aed8bca5ff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d78ff9700 1 -- 172.21.15.78:0/3442602187 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d800b9040 con 0x561736464720 2023-10-10T17:45:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d78ff9700 1 -- 172.21.15.78:0/3442602187 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d800b22b0 con 0x561736464720 2023-10-10T17:45:43.733 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.714+0000 7f8d78ff9700 1 -- 172.21.15.78:0/3442602187 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d80099d60 con 0x561736464720 2023-10-10T17:45:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3442602187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 msgr2=0x561736461ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736461ee0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8d80099cc0 tx=0x7f8d800050b0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3442602187 shutdown_connections 2023-10-10T17:45:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5617362ba6b0 0x56173646d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561736464b90 0x561736462420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/3442602187 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464720 0x561736461ee0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3442602187 >> 172.21.15.78:0/3442602187 conn(0x561736453f50 msgr2=0x56173645aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3442602187 shutdown_connections 2023-10-10T17:45:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 -- 172.21.15.78:0/3442602187 wait complete. 2023-10-10T17:45:43.735 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 -- start start 2023-10-10T17:45:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.715+0000 7f8d9824cb80 1 Processor -- start 2023-10-10T17:45:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x5617364c4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464720 0x5617364c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464b90 0x56173646ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.736 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56173646d110 con 0x561736464b90 2023-10-10T17:45:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56173646d280 con 0x5617362ba6b0 2023-10-10T17:45:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56173646d440 con 0x561736464720 2023-10-10T17:45:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 10 client.0 _ll_register_callbacks cb 0x56173636f720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:45:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464b90 0x56173646ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.737 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464b90 0x56173646ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46352/0 (socket says 172.21.15.78:46352) 2023-10-10T17:45:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 -- 172.21.15.78:0/876653879 learned_addr learned my addr 172.21.15.78:0/876653879 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:45:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8965b700 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464720 0x5617364c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x5617364c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 -- 172.21.15.78:0/876653879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464720 msgr2=0x5617364c4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464720 0x5617364c4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 -- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 msgr2=0x5617364c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:45:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x5617364c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 -- 172.21.15.78:0/876653879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617362315d0 con 0x561736464b90 2023-10-10T17:45:43.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d8a65d700 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464b90 0x56173646ca00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00b070 tx=0x7f8d7c003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c10fbce8de8a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:45:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d6d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:45:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/876653879 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c00e040 con 0x561736464b90 2023-10-10T17:45:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/876653879 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f8d7c005040 con 0x561736464b90 2023-10-10T17:45:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.716+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d8009db30 con 0x561736464b90 2023-10-10T17:45:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/876653879 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d7c005200 con 0x561736464b90 2023-10-10T17:45:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 10 client.16383 fetch_fsmap learned FSMap version 28 2023-10-10T17:45:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5617361c6c00 con 0x561736464b90 2023-10-10T17:45:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/876653879 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d7c030020 con 0x561736464b90 2023-10-10T17:45:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/876653879 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 28) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f8d7c00b0d0 con 0x561736464b90 2023-10-10T17:45:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 10 client.16383 fetch_fsmap finished waiting for FSMap version 28 2023-10-10T17:45:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 10 client.16383 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:45:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 20 client.16383 populate_metadata read hostname 'smithi078' 2023-10-10T17:45:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5617364cc7c0 0x5617364cebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:45:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 4 client.16383 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:45:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56173637e600 con 0x5617364cc7c0 2023-10-10T17:45:43.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5617364cc7c0 0x5617364cebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:45:43.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d6d7fa700 10 client.16383 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:43.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:43.717+0000 7f8d89e5c700 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5617364cc7c0 0x5617364cebd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7302e7a73b636b23 server_cookie=4662c720e4747fb0 in_seq=0 out_seq=0 2023-10-10T17:45:44.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:44 smithi190 ceph-mon[101220]: pgmap v99: 129 pgs: 129 active+clean; 192 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.4 MiB/s wr, 849 op/s 2023-10-10T17:45:44.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:44 smithi157 ceph-mon[100896]: pgmap v99: 129 pgs: 129 active+clean; 192 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.4 MiB/s wr, 849 op/s 2023-10-10T17:45:45.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:44 smithi078 ceph-mon[99494]: pgmap v99: 129 pgs: 129 active+clean; 192 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.4 MiB/s wr, 849 op/s 2023-10-10T17:45:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:46 smithi157 ceph-mon[100896]: pgmap v100: 129 pgs: 129 active+clean; 192 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2023-10-10T17:45:47.137 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:46 smithi078 ceph-mon[99494]: pgmap v100: 129 pgs: 129 active+clean; 192 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2023-10-10T17:45:47.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:46 smithi190 ceph-mon[101220]: pgmap v100: 129 pgs: 129 active+clean; 192 MiB data, 695 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 4.9 MiB/s wr, 539 op/s 2023-10-10T17:45:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.554+0000 7f8d6d7fa700 1 -- 172.21.15.78:0/876653879 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56173637e600 con 0x5617364cc7c0 2023-10-10T17:45:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.554+0000 7f8d6d7fa700 10 client.16383 handle_command_reply: tid=0 2023-10-10T17:45:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.554+0000 7f8d9824cb80 1 client.16383 shutdown 2023-10-10T17:45:47.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.554+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464b90 msgr2=0x56173646ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.554+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464b90 0x56173646ca00 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00b070 tx=0x7f8d7c003f90 comp rx=0 tx=0).stop 2023-10-10T17:45:47.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.555+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 shutdown_connections 2023-10-10T17:45:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.555+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5617364cc7c0 0x5617364cebd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.555+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561736464b90 0x56173646ca00 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.555+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5617362ba6b0 0x5617364c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.555+0000 7f8d9824cb80 1 --2- 172.21.15.78:0/876653879 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561736464720 0x5617364c4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.555+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 >> 172.21.15.78:0/876653879 conn(0x561736453f50 msgr2=0x561736461ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.555+0000 7f8d6d7fa700 0 client.16383 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:45:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.556+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 shutdown_connections 2023-10-10T17:45:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.556+0000 7f8d9824cb80 1 -- 172.21.15.78:0/876653879 wait complete. 2023-10-10T17:45:47.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.556+0000 7f8d9824cb80 20 client.16383 trim_cache size 0 max 16384 2023-10-10T17:45:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:45:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:45:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:45:47.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:45:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.556+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8d700a12c0 msgr2=0x7f8d700a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.556+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8d700a12c0 0x7f8d700a3770 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d74009c60 tx=0x7f8d74009260 comp rx=0 tx=0).stop 2023-10-10T17:45:47.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.556+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d84101da0 msgr2=0x7f8d840ff620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:45:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.556+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d84101da0 0x7f8d840ff620 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8d8000d980 tx=0x7f8d8000dc90 comp rx=0 tx=0).stop 2023-10-10T17:45:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 shutdown_connections 2023-10-10T17:45:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8d8410a860 0x7f8d841a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8d700a12c0 0x7f8d700a3770 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8d84101490 0x7f8d840ff0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 --2- 172.21.15.78:0/4101110215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8d84101da0 0x7f8d840ff620 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:45:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 >> 172.21.15.78:0/4101110215 conn(0x7f8d84076370 msgr2=0x7f8d840fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:45:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 shutdown_connections 2023-10-10T17:45:47.563 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:45:47.557+0000 7f8d8c0c0700 1 -- 172.21.15.78:0/4101110215 wait complete. 2023-10-10T17:45:48.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:48 smithi157 ceph-mon[100896]: pgmap v101: 129 pgs: 129 active+clean; 216 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.6 MiB/s wr, 1.81k op/s 2023-10-10T17:45:49.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:48 smithi078 ceph-mon[99494]: pgmap v101: 129 pgs: 129 active+clean; 216 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.6 MiB/s wr, 1.81k op/s 2023-10-10T17:45:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:48 smithi190 ceph-mon[101220]: pgmap v101: 129 pgs: 129 active+clean; 216 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.6 MiB/s wr, 1.81k op/s 2023-10-10T17:45:49.250 INFO:tasks.workunit.client.0.smithi078.stdout:created hierarchy, now cleaning up 2023-10-10T17:45:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:50 smithi157 ceph-mon[100896]: pgmap v102: 129 pgs: 129 active+clean; 216 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.6 MiB/s wr, 1.77k op/s 2023-10-10T17:45:51.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:50 smithi078 ceph-mon[99494]: pgmap v102: 129 pgs: 129 active+clean; 216 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.6 MiB/s wr, 1.77k op/s 2023-10-10T17:45:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:50 smithi190 ceph-mon[101220]: pgmap v102: 129 pgs: 129 active+clean; 216 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.6 MiB/s wr, 1.77k op/s 2023-10-10T17:45:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:52 smithi157 ceph-mon[100896]: pgmap v103: 129 pgs: 129 active+clean; 223 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.1 MiB/s wr, 1.77k op/s 2023-10-10T17:45:53.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:52 smithi078 ceph-mon[99494]: pgmap v103: 129 pgs: 129 active+clean; 223 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.1 MiB/s wr, 1.77k op/s 2023-10-10T17:45:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:52 smithi190 ceph-mon[101220]: pgmap v103: 129 pgs: 129 active+clean; 223 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 5.1 MiB/s wr, 1.77k op/s 2023-10-10T17:45:53.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:53 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:45:54.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:53 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:45:54.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:53 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:45:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:54 smithi157 ceph-mon[100896]: pgmap v104: 129 pgs: 129 active+clean; 202 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 2.39k op/s 2023-10-10T17:45:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:54 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:54 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:54.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:54 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:54.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:54 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:54 smithi078 ceph-mon[99494]: pgmap v104: 129 pgs: 129 active+clean; 202 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 2.39k op/s 2023-10-10T17:45:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:54 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:54 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:54 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:54 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:54 smithi190 ceph-mon[101220]: pgmap v104: 129 pgs: 129 active+clean; 202 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 2.39k op/s 2023-10-10T17:45:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:54 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:54 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:54 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:54 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:55.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:55 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:45:55.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:55 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:45:55.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:55 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:56.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:55 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:45:56.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:55 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:45:56.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:55 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:56.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:55 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:45:56.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:55 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:45:56.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:55 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:45:56.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:56 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:45:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:56 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:45:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:56 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:45:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:56 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:45:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:56 smithi157 ceph-mon[100896]: pgmap v105: 129 pgs: 129 active+clean; 202 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 2.03k op/s 2023-10-10T17:45:56.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:56 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:57.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:56 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:45:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:56 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:45:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:56 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:45:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:56 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:45:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:56 smithi078 ceph-mon[99494]: pgmap v105: 129 pgs: 129 active+clean; 202 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 2.03k op/s 2023-10-10T17:45:57.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:56 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:57.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:56 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:45:57.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:56 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:45:57.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:56 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:45:57.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:56 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:45:57.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:56 smithi190 ceph-mon[101220]: pgmap v105: 129 pgs: 129 active+clean; 202 MiB data, 700 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 4.6 MiB/s wr, 2.03k op/s 2023-10-10T17:45:57.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:56 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:45:57.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:57 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:45:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:57 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:57 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:45:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:57 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:57 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:45:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:57 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:45:58.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:45:58 smithi157 ceph-mon[100896]: pgmap v106: 129 pgs: 129 active+clean; 226 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.6 MiB/s wr, 2.16k op/s 2023-10-10T17:45:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:45:58 smithi078 ceph-mon[99494]: pgmap v106: 129 pgs: 129 active+clean; 226 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.6 MiB/s wr, 2.16k op/s 2023-10-10T17:45:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:45:58 smithi190 ceph-mon[101220]: pgmap v106: 129 pgs: 129 active+clean; 226 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.6 MiB/s wr, 2.16k op/s 2023-10-10T17:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.672+0000 7f328191b700 1 -- 172.21.15.78:0/2126302854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0ff3c0 msgr2=0x7f327c0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.672+0000 7f328191b700 1 --2- 172.21.15.78:0/2126302854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0ff3c0 0x7f327c0ff790 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f327400b0d0 tx=0x7f327400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 -- 172.21.15.78:0/2126302854 shutdown_connections 2023-10-10T17:46:00.674 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 --2- 172.21.15.78:0/2126302854 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f327c104400 0x7f327c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 --2- 172.21.15.78:0/2126302854 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c0ff3c0 0x7f327c0ff790 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 --2- 172.21.15.78:0/2126302854 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f327c0ffd60 0x7f327c103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 -- 172.21.15.78:0/2126302854 >> 172.21.15.78:0/2126302854 conn(0x7f327c0763a0 msgr2=0x7f327c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 -- 172.21.15.78:0/2126302854 shutdown_connections 2023-10-10T17:46:00.675 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 -- 172.21.15.78:0/2126302854 wait complete. 2023-10-10T17:46:00.676 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.673+0000 7f328191b700 1 Processor -- start 2023-10-10T17:46:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f328191b700 1 -- start start 2023-10-10T17:46:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f328191b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f327c0ff3c0 0x7f327c195410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f328191b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f327c0ffd60 0x7f327c195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f328191b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c104400 0x7f327c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.678 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f328191b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c19a3f0 con 0x7f327c104400 2023-10-10T17:46:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f328191b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c19a560 con 0x7f327c0ffd60 2023-10-10T17:46:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f328191b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f327c19a720 con 0x7f327c0ff3c0 2023-10-10T17:46:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c104400 0x7f327c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c104400 0x7f327c199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46366/0 (socket says 172.21.15.78:46366) 2023-10-10T17:46:00.679 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327affd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f327c0ff3c0 0x7f327c195410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 -- 172.21.15.78:0/3847542173 learned_addr learned my addr 172.21.15.78:0/3847542173 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f327c0ffd60 0x7f327c195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 -- 172.21.15.78:0/3847542173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f327c0ff3c0 msgr2=0x7f327c195410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f327c0ff3c0 0x7f327c195410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.680 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 -- 172.21.15.78:0/3847542173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f327c0ffd60 msgr2=0x7f327c195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:00.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f327c0ffd60 0x7f327c195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 -- 172.21.15.78:0/3847542173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274009d00 con 0x7f327c104400 2023-10-10T17:46:00.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.674+0000 7f327b7fe700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c104400 0x7f327c199db0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f326c00c900 tx=0x7f326c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:00.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.675+0000 7f3280919700 1 -- 172.21.15.78:0/3847542173 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0078d0 con 0x7f327c104400 2023-10-10T17:46:00.681 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.675+0000 7f3280919700 1 -- 172.21.15.78:0/3847542173 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f326c00f450 con 0x7f327c104400 2023-10-10T17:46:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.675+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327c10f210 con 0x7f327c104400 2023-10-10T17:46:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.675+0000 7f3280919700 1 -- 172.21.15.78:0/3847542173 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c018660 con 0x7f327c104400 2023-10-10T17:46:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.675+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c10f080 con 0x7f327c104400 2023-10-10T17:46:00.682 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.677+0000 7f3280919700 1 -- 172.21.15.78:0/3847542173 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f326c00f5c0 con 0x7f327c104400 2023-10-10T17:46:00.687 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f3280919700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f32680a12c0 0x7f32680a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f327affd700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f32680a12c0 0x7f32680a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f3280919700 1 -- 172.21.15.78:0/3847542173 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f326c0cce40 con 0x7f327c104400 2023-10-10T17:46:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f328daa7b80 1 Processor -- start 2023-10-10T17:46:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f327affd700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f32680a12c0 0x7f32680a3770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3274009fd0 tx=0x7f327400b040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:00.688 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f328daa7b80 1 -- start start 2023-10-10T17:46:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.678+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638ab90 0x564b06384b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f328daa7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b0628a000 con 0x564b0638a720 2023-10-10T17:46:00.689 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f328daa7b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b062dc840 con 0x564b061e06b0 2023-10-10T17:46:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f328daa7b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b062d5cd0 con 0x564b0638ab90 2023-10-10T17:46:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327affd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46372/0 (socket says 172.21.15.78:46372) 2023-10-10T17:46:00.690 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327affd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48092/0 (socket says 172.21.15.78:48092) 2023-10-10T17:46:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 -- 172.21.15.78:0/3446369485 learned_addr learned my addr 172.21.15.78:0/3446369485 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327b7fe700 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638ab90 0x564b06384b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 -- 172.21.15.78:0/3446369485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638ab90 msgr2=0x564b06384b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638ab90 0x564b06384b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.691 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 -- 172.21.15.78:0/3446369485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 msgr2=0x564b0638a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 -- 172.21.15.78:0/3446369485 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b061575d0 con 0x564b0638a720 2023-10-10T17:46:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638a720 0x564b06384560 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3264009cb0 tx=0x7f3264011440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a34fa0aaf479a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f3271ffb700 1 -- 172.21.15.78:0/3446369485 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264011d80 con 0x564b0638a720 2023-10-10T17:46:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f3271ffb700 1 -- 172.21.15.78:0/3446369485 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3264004510 con 0x564b0638a720 2023-10-10T17:46:00.692 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.679+0000 7f3271ffb700 1 -- 172.21.15.78:0/3446369485 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264005280 con 0x564b0638a720 2023-10-10T17:46:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 -- 172.21.15.78:0/3446369485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638a720 msgr2=0x564b06384560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638a720 0x564b06384560 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f3264009cb0 tx=0x7f3264011440 comp rx=0 tx=0).stop 2023-10-10T17:46:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 -- 172.21.15.78:0/3446369485 shutdown_connections 2023-10-10T17:46:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638ab90 0x564b06384b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.693 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3446369485 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 -- 172.21.15.78:0/3446369485 >> 172.21.15.78:0/3446369485 conn(0x564b06379f50 msgr2=0x564b0637f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 -- 172.21.15.78:0/3446369485 shutdown_connections 2023-10-10T17:46:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 -- 172.21.15.78:0/3446369485 wait complete. 2023-10-10T17:46:00.694 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.680+0000 7f328daa7b80 1 -- start start 2023-10-10T17:46:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 1 Processor -- start 2023-10-10T17:46:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06390b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b0638dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b0638e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.695 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b06277880 con 0x564b061e06b0 2023-10-10T17:46:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b06277520 con 0x564b0638ab90 2023-10-10T17:46:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b061598c0 con 0x564b0638a720 2023-10-10T17:46:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f328daa7b80 10 client.0 _ll_register_callbacks cb 0x564b06295720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06390b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.696 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06390b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46376/0 (socket says 172.21.15.78:46376) 2023-10-10T17:46:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 -- 172.21.15.78:0/3351153627 learned_addr learned my addr 172.21.15.78:0/3351153627 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b0638dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327b7fe700 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b0638e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 -- 172.21.15.78:0/3351153627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 msgr2=0x564b0638dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b0638dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 -- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 msgr2=0x564b0638e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b0638e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.681+0000 7f327affd700 1 -- 172.21.15.78:0/3351153627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32640098e0 con 0x564b061e06b0 2023-10-10T17:46:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f327affd700 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06390b10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3274007590 tx=0x7f32740099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca9bf35a5eb4d42 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:00.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f325affd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f325affd700 1 -- 172.21.15.78:0/3351153627 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400e040 con 0x564b061e06b0 2023-10-10T17:46:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f325affd700 1 -- 172.21.15.78:0/3351153627 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3274029430 con 0x564b061e06b0 2023-10-10T17:46:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564b06377710 con 0x564b061e06b0 2023-10-10T17:46:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f325affd700 1 -- 172.21.15.78:0/3351153627 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3274032630 con 0x564b061e06b0 2023-10-10T17:46:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 10 client.16395 fetch_fsmap learned FSMap version 29 2023-10-10T17:46:00.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564b0629c130 con 0x564b061e06b0 2023-10-10T17:46:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f325affd700 1 -- 172.21.15.78:0/3351153627 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564b06377710 con 0x564b061e06b0 2023-10-10T17:46:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f325affd700 1 -- 172.21.15.78:0/3351153627 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 29) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f3274029960 con 0x564b061e06b0 2023-10-10T17:46:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 10 client.16395 fetch_fsmap finished waiting for FSMap version 29 2023-10-10T17:46:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 10 client.16395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:00.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 20 client.16395 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b0639e050 0x564b063a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 4 client.16395 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:46:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b06298040 con 0x564b0639e050 2023-10-10T17:46:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.682+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b0639e050 0x564b063a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.683+0000 7f325affd700 10 client.16395 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:00.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.683+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b0639e050 0x564b063a0460 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ad3f3c837057539 server_cookie=65e8fdd3af3f511 in_seq=0 out_seq=0 2023-10-10T17:46:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f325affd700 1 -- 172.21.15.78:0/3351153627 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x564b06298040 con 0x564b0639e050 2023-10-10T17:46:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f325affd700 10 client.16395 handle_command_reply: tid=0 2023-10-10T17:46:00.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 client.16395 shutdown 2023-10-10T17:46:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 msgr2=0x564b06390b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06390b10 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f3274007590 tx=0x7f32740099e0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 shutdown_connections 2023-10-10T17:46:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b0639e050 0x564b063a0460 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b0638e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06390b10 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f325affd700 0 client.16395 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3351153627 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b0638dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 >> 172.21.15.78:0/3351153627 conn(0x564b06379f50 msgr2=0x564b0637e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:00.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 shutdown_connections 2023-10-10T17:46:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 1 -- 172.21.15.78:0/3351153627 wait complete. 2023-10-10T17:46:00.706 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.688+0000 7f328daa7b80 20 client.16395 trim_cache size 0 max 16384 2023-10-10T17:46:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 Processor -- start 2023-10-10T17:46:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 -- start start 2023-10-10T17:46:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.711 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638ab90 0x564b0637ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b0639b660 con 0x564b0638ab90 2023-10-10T17:46:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b0637bfd0 con 0x564b061e06b0 2023-10-10T17:46:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.703+0000 7f328daa7b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b06385110 con 0x564b0638a720 2023-10-10T17:46:00.712 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638ab90 0x564b0637ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638ab90 0x564b0637ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46388/0 (socket says 172.21.15.78:46388) 2023-10-10T17:46:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 -- 172.21.15.78:0/68650577 learned_addr learned my addr 172.21.15.78:0/68650577 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327affd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327a7fc700 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.713 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 -- 172.21.15.78:0/68650577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 msgr2=0x564b06384560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 -- 172.21.15.78:0/68650577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 msgr2=0x564b0638dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 -- 172.21.15.78:0/68650577 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b0629be80 con 0x564b0638ab90 2023-10-10T17:46:00.714 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f327b7fe700 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638ab90 0x564b0637ef60 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f326c096610 tx=0x7f326c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ef5da50a1feb9a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f32727fc700 1 -- 172.21.15.78:0/68650577 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c099ba0 con 0x564b0638ab90 2023-10-10T17:46:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f32727fc700 1 -- 172.21.15.78:0/68650577 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f326c0b1070 con 0x564b0638ab90 2023-10-10T17:46:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.704+0000 7f32727fc700 1 -- 172.21.15.78:0/68650577 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0ccb80 con 0x564b0638ab90 2023-10-10T17:46:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 -- 172.21.15.78:0/68650577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638ab90 msgr2=0x564b0637ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:00.715 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638ab90 0x564b0637ef60 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f326c096610 tx=0x7f326c00a5c0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 -- 172.21.15.78:0/68650577 shutdown_connections 2023-10-10T17:46:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b0638ab90 0x564b0637ef60 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b061e06b0 0x564b0638dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 --2- 172.21.15.78:0/68650577 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b06384560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.716 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 -- 172.21.15.78:0/68650577 >> 172.21.15.78:0/68650577 conn(0x564b06379f50 msgr2=0x564b0638d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 -- 172.21.15.78:0/68650577 shutdown_connections 2023-10-10T17:46:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 -- 172.21.15.78:0/68650577 wait complete. 2023-10-10T17:46:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 -- start start 2023-10-10T17:46:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.705+0000 7f328daa7b80 1 Processor -- start 2023-10-10T17:46:00.717 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06393170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b063936b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b06391640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b06391d20 con 0x564b061e06b0 2023-10-10T17:46:00.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b06391e90 con 0x564b0638ab90 2023-10-10T17:46:00.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b06392050 con 0x564b0638a720 2023-10-10T17:46:00.718 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 10 client.0 _ll_register_callbacks cb 0x564b06295720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06393170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06393170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46390/0 (socket says 172.21.15.78:46390) 2023-10-10T17:46:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 -- 172.21.15.78:0/3135618758 learned_addr learned my addr 172.21.15.78:0/3135618758 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:00.719 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b063936b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327b7fe700 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b06391640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 -- 172.21.15.78:0/3135618758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 msgr2=0x564b063936b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b063936b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 -- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 msgr2=0x564b06391640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:00.720 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b06391640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 -- 172.21.15.78:0/3135618758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f327c10f080 con 0x564b061e06b0 2023-10-10T17:46:00.721 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f327affd700 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06393170 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3274029d90 tx=0x7f32740296e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba26c985a862e0d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f325affd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f325affd700 1 -- 172.21.15.78:0/3135618758 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327400e040 con 0x564b061e06b0 2023-10-10T17:46:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f325affd700 1 -- 172.21.15.78:0/3135618758 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3274040680 con 0x564b061e06b0 2023-10-10T17:46:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.706+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3264004510 con 0x564b061e06b0 2023-10-10T17:46:00.722 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f325affd700 1 -- 172.21.15.78:0/3135618758 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3274029ae0 con 0x564b061e06b0 2023-10-10T17:46:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 10 client.16401 fetch_fsmap learned FSMap version 29 2023-10-10T17:46:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564b060ecc00 con 0x564b061e06b0 2023-10-10T17:46:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f325affd700 1 -- 172.21.15.78:0/3135618758 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3264004510 con 0x564b061e06b0 2023-10-10T17:46:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f325affd700 1 -- 172.21.15.78:0/3135618758 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 29) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f3274032950 con 0x564b061e06b0 2023-10-10T17:46:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 10 client.16401 fetch_fsmap finished waiting for FSMap version 29 2023-10-10T17:46:00.723 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 10 client.16401 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 20 client.16401 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b063f2da0 0x564b063f5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 4 client.16401 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:46:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564b062a4600 con 0x564b063f2da0 2023-10-10T17:46:00.724 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.707+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b063f2da0 0x564b063f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.708+0000 7f325affd700 10 client.16401 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:00.725 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:00.708+0000 7f327a7fc700 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b063f2da0 0x564b063f5180 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd25732f16c80d90 server_cookie=1af754725bb55de9 in_seq=0 out_seq=0 2023-10-10T17:46:00.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:00 smithi157 ceph-mon[100896]: pgmap v107: 129 pgs: 129 active+clean; 226 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 884 op/s 2023-10-10T17:46:01.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:00 smithi078 ceph-mon[99494]: pgmap v107: 129 pgs: 129 active+clean; 226 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 884 op/s 2023-10-10T17:46:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:00 smithi190 ceph-mon[101220]: pgmap v107: 129 pgs: 129 active+clean; 226 MiB data, 712 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 884 op/s 2023-10-10T17:46:02.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:01 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:46:02.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:01 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:46:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:01 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:46:02.413 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f325affd700 1 -- 172.21.15.78:0/3135618758 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564b062a4600 con 0x564b063f2da0 2023-10-10T17:46:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f325affd700 10 client.16401 handle_command_reply: tid=0 2023-10-10T17:46:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 client.16401 shutdown 2023-10-10T17:46:02.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 msgr2=0x564b06393170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06393170 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3274029d90 tx=0x7f32740296e0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 shutdown_connections 2023-10-10T17:46:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x564b063f2da0 0x564b063f5180 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x564b0638ab90 0x564b06391640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x564b061e06b0 0x564b06393170 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f325affd700 0 client.16401 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 --2- 172.21.15.78:0/3135618758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x564b0638a720 0x564b063936b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 >> 172.21.15.78:0/3135618758 conn(0x564b06379f50 msgr2=0x564b06388e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:02.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 shutdown_connections 2023-10-10T17:46:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.412+0000 7f328daa7b80 1 -- 172.21.15.78:0/3135618758 wait complete. 2023-10-10T17:46:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328daa7b80 20 client.16401 trim_cache size 0 max 16384 2023-10-10T17:46:02.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:46:02.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:46:02.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:46:02.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:46:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f32680a12c0 msgr2=0x7f32680a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f32680a12c0 0x7f32680a3770 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3274009fd0 tx=0x7f327400b040 comp rx=0 tx=0).stop 2023-10-10T17:46:02.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c104400 msgr2=0x7f327c199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c104400 0x7f327c199db0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f326c00c900 tx=0x7f326c00cc10 comp rx=0 tx=0).stop 2023-10-10T17:46:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 shutdown_connections 2023-10-10T17:46:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f327c0ff3c0 0x7f327c195410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f32680a12c0 0x7f32680a3770 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.420 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f327c104400 0x7f327c199db0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 --2- 172.21.15.78:0/3847542173 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f327c0ffd60 0x7f327c195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:02.421 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 >> 172.21.15.78:0/3847542173 conn(0x7f327c0763a0 msgr2=0x7f327c0fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 shutdown_connections 2023-10-10T17:46:02.422 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:02.413+0000 7f328191b700 1 -- 172.21.15.78:0/3847542173 wait complete. 2023-10-10T17:46:03.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:02 smithi078 ceph-mon[99494]: pgmap v108: 129 pgs: 129 active+clean; 226 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 892 op/s 2023-10-10T17:46:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:02 smithi190 ceph-mon[101220]: pgmap v108: 129 pgs: 129 active+clean; 226 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 892 op/s 2023-10-10T17:46:03.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:02 smithi157 ceph-mon[100896]: pgmap v108: 129 pgs: 129 active+clean; 226 MiB data, 718 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 3.9 MiB/s wr, 892 op/s 2023-10-10T17:46:05.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:04 smithi078 ceph-mon[99494]: pgmap v109: 129 pgs: 129 active+clean; 247 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2023-10-10T17:46:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:04 smithi190 ceph-mon[101220]: pgmap v109: 129 pgs: 129 active+clean; 247 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2023-10-10T17:46:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:04 smithi157 ceph-mon[100896]: pgmap v109: 129 pgs: 129 active+clean; 247 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.1 MiB/s wr, 1.09k op/s 2023-10-10T17:46:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:06 smithi078 ceph-mon[99494]: pgmap v110: 129 pgs: 129 active+clean; 247 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 337 op/s 2023-10-10T17:46:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:06 smithi190 ceph-mon[101220]: pgmap v110: 129 pgs: 129 active+clean; 247 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 337 op/s 2023-10-10T17:46:07.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:06 smithi157 ceph-mon[100896]: pgmap v110: 129 pgs: 129 active+clean; 247 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.8 MiB/s wr, 337 op/s 2023-10-10T17:46:09.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:08 smithi078 ceph-mon[99494]: pgmap v111: 129 pgs: 129 active+clean; 268 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.2 MiB/s wr, 691 op/s 2023-10-10T17:46:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:08 smithi190 ceph-mon[101220]: pgmap v111: 129 pgs: 129 active+clean; 268 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.2 MiB/s wr, 691 op/s 2023-10-10T17:46:09.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:08 smithi157 ceph-mon[100896]: pgmap v111: 129 pgs: 129 active+clean; 268 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.2 MiB/s wr, 691 op/s 2023-10-10T17:46:09.228 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-h[125866]: *** Caught signal (Segmentation fault) ** 2023-10-10T17:46:09.228 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-h[125866]: in thread 7fcb3884a700 thread_name:msgr-worker-0 2023-10-10T17:46:11.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:10 smithi078 ceph-mon[99494]: pgmap v112: 129 pgs: 129 active+clean; 268 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.2 MiB/s wr, 563 op/s 2023-10-10T17:46:11.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:10 smithi190 ceph-mon[101220]: pgmap v112: 129 pgs: 129 active+clean; 268 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.2 MiB/s wr, 563 op/s 2023-10-10T17:46:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:10 smithi157 ceph-mon[100896]: pgmap v112: 129 pgs: 129 active+clean; 268 MiB data, 827 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 6.2 MiB/s wr, 563 op/s 2023-10-10T17:46:11.811 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:11 smithi157 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.h.service: Main process exited, code=exited, status=139/n/a 2023-10-10T17:46:12.227 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:11 smithi157 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.h.service: Failed with result 'exit-code'. 2023-10-10T17:46:12.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:46:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 -- 172.21.15.78:0/3072759731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c069a80 msgr2=0x7fed8c104f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:12.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 --2- 172.21.15.78:0/3072759731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c069a80 0x7fed8c104f20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fed800099c0 tx=0x7fed80009cd0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 -- 172.21.15.78:0/3072759731 shutdown_connections 2023-10-10T17:46:12.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 --2- 172.21.15.78:0/3072759731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c0690e0 0x7fed8c0694b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 --2- 172.21.15.78:0/3072759731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed8c105600 0x7fed8c10d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 --2- 172.21.15.78:0/3072759731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c069a80 0x7fed8c104f20 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 -- 172.21.15.78:0/3072759731 >> 172.21.15.78:0/3072759731 conn(0x7fed8c075470 msgr2=0x7fed8c075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:12.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 -- 172.21.15.78:0/3072759731 shutdown_connections 2023-10-10T17:46:12.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.962+0000 7fed92765700 1 -- 172.21.15.78:0/3072759731 wait complete. 2023-10-10T17:46:12.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 Processor -- start 2023-10-10T17:46:12.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 -- start start 2023-10-10T17:46:12.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed8c0690e0 0x7fed8c1963e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c069a80 0x7fed8c196920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c105600 0x7fed8c19ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c19b3c0 con 0x7fed8c105600 2023-10-10T17:46:12.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c19b530 con 0x7fed8c0690e0 2023-10-10T17:46:12.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed92765700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c19b6f0 con 0x7fed8c069a80 2023-10-10T17:46:12.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed90d02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c105600 0x7fed8c19ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed8b7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c069a80 0x7fed8c196920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed90d02700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c105600 0x7fed8c19ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46408/0 (socket says 172.21.15.78:46408) 2023-10-10T17:46:12.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed8c0690e0 0x7fed8c1963e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed90d02700 1 -- 172.21.15.78:0/507018305 learned_addr learned my addr 172.21.15.78:0/507018305 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:12.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed90d02700 1 -- 172.21.15.78:0/507018305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c069a80 msgr2=0x7fed8c196920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:12.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed90d02700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c069a80 0x7fed8c196920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.963+0000 7fed90d02700 1 -- 172.21.15.78:0/507018305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed8c0690e0 msgr2=0x7fed8c1963e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:12.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed90d02700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed8c0690e0 0x7fed8c1963e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed90d02700 1 -- 172.21.15.78:0/507018305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed800095f0 con 0x7fed8c105600 2023-10-10T17:46:12.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed8bfff700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed8c0690e0 0x7fed8c1963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:12.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed90d02700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c105600 0x7fed8c19ad80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fed8400ef30 tx=0x7fed8400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:12.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed897fa700 1 -- 172.21.15.78:0/507018305 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84018070 con 0x7fed8c105600 2023-10-10T17:46:12.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c110380 con 0x7fed8c105600 2023-10-10T17:46:12.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed897fa700 1 -- 172.21.15.78:0/507018305 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed8400f040 con 0x7fed8c105600 2023-10-10T17:46:12.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed897fa700 1 -- 172.21.15.78:0/507018305 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8400cd00 con 0x7fed8c105600 2023-10-10T17:46:12.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.964+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c1101f0 con 0x7fed8c105600 2023-10-10T17:46:12.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.967+0000 7fed897fa700 1 -- 172.21.15.78:0/507018305 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fed84003a00 con 0x7fed8c105600 2023-10-10T17:46:12.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.968+0000 7fed897fa700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fed740a12c0 0x7fed740a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.968+0000 7fed9e8f1b80 1 Processor -- start 2023-10-10T17:46:12.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.968+0000 7fed9e8f1b80 1 -- start start 2023-10-10T17:46:12.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.968+0000 7fed897fa700 1 -- 172.21.15.78:0/507018305 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fed840ccb50 con 0x7fed8c105600 2023-10-10T17:46:12.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.968+0000 7fed8bfff700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fed740a12c0 0x7fed740a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac5392f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac5397370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac5397d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac5298060 con 0x55cac5393470 2023-10-10T17:46:12.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac52ea8a0 con 0x55cac53978b0 2023-10-10T17:46:12.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac52e3d30 con 0x55cac5392b60 2023-10-10T17:46:12.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac5397370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac5397370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46412/0 (socket says 172.21.15.78:46412) 2023-10-10T17:46:12.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 -- 172.21.15.78:0/772700942 learned_addr learned my addr 172.21.15.78:0/772700942 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:12.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed90d02700 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac5397d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8bfff700 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac5392f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8bfff700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fed740a12c0 0x7fed740a3770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fed7c011550 tx=0x7fed7c009f90 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:12.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 -- 172.21.15.78:0/772700942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 msgr2=0x55cac5392f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:12.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac5392f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 -- 172.21.15.78:0/772700942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 msgr2=0x55cac5397d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:12.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac5397d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 -- 172.21.15.78:0/772700942 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cac5165630 con 0x55cac5393470 2023-10-10T17:46:12.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed90d02700 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac5397d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:12.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8bfff700 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac5392f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:46:12.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.969+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac5397370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fed800036c0 tx=0x7fed80005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e37ac6a19536b78 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:12.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed727fc700 1 -- 172.21.15.78:0/772700942 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80026070 con 0x55cac5393470 2023-10-10T17:46:12.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed727fc700 1 -- 172.21.15.78:0/772700942 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed8001e040 con 0x55cac5393470 2023-10-10T17:46:12.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed727fc700 1 -- 172.21.15.78:0/772700942 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8000faa0 con 0x55cac5393470 2023-10-10T17:46:12.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/772700942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 msgr2=0x55cac5397370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:12.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac5397370 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fed800036c0 tx=0x7fed80005c00 comp rx=0 tx=0).stop 2023-10-10T17:46:12.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/772700942 shutdown_connections 2023-10-10T17:46:12.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac5392f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac5397d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/772700942 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac5397370 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:12.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.970+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/772700942 >> 172.21.15.78:0/772700942 conn(0x55cac5388360 msgr2=0x55cac538db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:12.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/772700942 shutdown_connections 2023-10-10T17:46:12.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/772700942 wait complete. 2023-10-10T17:46:12.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 -- start start 2023-10-10T17:46:12.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 Processor -- start 2023-10-10T17:46:12.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac539eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac5393470 0x55cac539c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac53978b0 0x55cac539c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:12.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac52858e0 con 0x55cac53978b0 2023-10-10T17:46:12.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac5285580 con 0x55cac5393470 2023-10-10T17:46:12.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac5167920 con 0x55cac5392b60 2023-10-10T17:46:12.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.971+0000 7fed9e8f1b80 10 client.0 _ll_register_callbacks cb 0x55cac52a3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:12.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac539eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac539eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:37948/0 (socket says 172.21.15.78:37948) 2023-10-10T17:46:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8bfff700 1 -- 172.21.15.78:0/896459525 learned_addr learned my addr 172.21.15.78:0/896459525 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:12.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed90d02700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac53978b0 0x55cac539c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac5393470 0x55cac539c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8b7fe700 1 -- 172.21.15.78:0/896459525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 msgr2=0x55cac539eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:13.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac539eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8b7fe700 1 -- 172.21.15.78:0/896459525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac53978b0 msgr2=0x55cac539c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac53978b0 0x55cac539c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8b7fe700 1 -- 172.21.15.78:0/896459525 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c10da10 con 0x55cac5393470 2023-10-10T17:46:13.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed90d02700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac53978b0 0x55cac539c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:13.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac5393470 0x55cac539c270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fed80000c00 tx=0x7fed80009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=affb0a5f54e29fcf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:13.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:46:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed6b7fe700 1 -- 172.21.15.78:0/896459525 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8000b990 con 0x55cac5393470 2023-10-10T17:46:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.972+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cac5385b20 con 0x55cac5393470 2023-10-10T17:46:13.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed6b7fe700 1 -- 172.21.15.78:0/896459525 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fed8000baf0 con 0x55cac5393470 2023-10-10T17:46:13.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed6b7fe700 1 -- 172.21.15.78:0/896459525 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed8000bcb0 con 0x55cac5393470 2023-10-10T17:46:13.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 10 client.25772 fetch_fsmap learned FSMap version 30 2023-10-10T17:46:13.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cac52aa190 con 0x55cac5393470 2023-10-10T17:46:13.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed6b7fe700 1 -- 172.21.15.78:0/896459525 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cac5385b20 con 0x55cac5393470 2023-10-10T17:46:13.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed6b7fe700 1 -- 172.21.15.78:0/896459525 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 30) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fed80031030 con 0x55cac5393470 2023-10-10T17:46:13.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 10 client.25772 fetch_fsmap finished waiting for FSMap version 30 2023-10-10T17:46:13.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:13.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 20 client.25772 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:13.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac53bd390 0x55cac53bf770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 4 client.25772 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:46:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cac52a60a0 con 0x55cac53bd390 2023-10-10T17:46:13.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.973+0000 7fed8bfff700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac53bd390 0x55cac53bf770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.974+0000 7fed6b7fe700 10 client.25772 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.974+0000 7fed8bfff700 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac53bd390 0x55cac53bf770 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cf04d3f14ff9b42 server_cookie=4357f597fa1b119d in_seq=0 out_seq=0 2023-10-10T17:46:13.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed6b7fe700 1 -- 172.21.15.78:0/896459525 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cac52a60a0 con 0x55cac53bd390 2023-10-10T17:46:13.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed6b7fe700 10 client.25772 handle_command_reply: tid=0 2023-10-10T17:46:13.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 client.25772 shutdown 2023-10-10T17:46:13.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac5393470 msgr2=0x55cac539c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:13.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac5393470 0x55cac539c270 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fed80000c00 tx=0x7fed80009fd0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 shutdown_connections 2023-10-10T17:46:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac539eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac53bd390 0x55cac53bf770 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac53978b0 0x55cac539c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed6b7fe700 0 client.25772 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/896459525 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac5393470 0x55cac539c270 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 >> 172.21.15.78:0/896459525 conn(0x55cac5388360 msgr2=0x55cac5398d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:13.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 shutdown_connections 2023-10-10T17:46:13.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/896459525 wait complete. 2023-10-10T17:46:13.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.977+0000 7fed9e8f1b80 20 client.25772 trim_cache size 0 max 16384 2023-10-10T17:46:13.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.999+0000 7fed9e8f1b80 1 Processor -- start 2023-10-10T17:46:13.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:12.999+0000 7fed9e8f1b80 1 -- start start 2023-10-10T17:46:13.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac538dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53958f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac5393bc0 con 0x55cac5393470 2023-10-10T17:46:13.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac53b9810 con 0x55cac53978b0 2023-10-10T17:46:13.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac53bfdd0 con 0x55cac5392b60 2023-10-10T17:46:13.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46428/0 (socket says 172.21.15.78:46428) 2023-10-10T17:46:13.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3626669336 learned_addr learned my addr 172.21.15.78:0/3626669336 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:13.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed90d02700 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8bfff700 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac538dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3626669336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 msgr2=0x55cac538dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:13.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac538dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3626669336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 msgr2=0x55cac53958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:13.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3626669336 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cac52a9ee0 con 0x55cac5393470 2023-10-10T17:46:13.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed90d02700 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:46:13.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b4950 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fed800098e0 tx=0x7fed800336b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89d62f8c7240fd0b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:13.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed72ffd700 1 -- 172.21.15.78:0/3626669336 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80032c50 con 0x55cac5393470 2023-10-10T17:46:13.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed72ffd700 1 -- 172.21.15.78:0/3626669336 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed8002d070 con 0x55cac5393470 2023-10-10T17:46:13.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.000+0000 7fed72ffd700 1 -- 172.21.15.78:0/3626669336 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed800050e0 con 0x55cac5393470 2023-10-10T17:46:13.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3626669336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 msgr2=0x55cac53b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:13.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b4950 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fed800098e0 tx=0x7fed800336b0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3626669336 shutdown_connections 2023-10-10T17:46:13.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac538dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3626669336 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b4950 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3626669336 >> 172.21.15.78:0/3626669336 conn(0x55cac5388360 msgr2=0x55cac53b9100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:13.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3626669336 shutdown_connections 2023-10-10T17:46:13.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3626669336 wait complete. 2023-10-10T17:46:13.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.001+0000 7fed9e8f1b80 1 -- start start 2023-10-10T17:46:13.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 1 Processor -- start 2023-10-10T17:46:13.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac53b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac53b3750 con 0x55cac5393470 2023-10-10T17:46:13.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac54101e0 con 0x55cac53978b0 2023-10-10T17:46:13.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cac5410320 con 0x55cac5392b60 2023-10-10T17:46:13.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed9e8f1b80 10 client.0 _ll_register_callbacks cb 0x7fed84004cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:13.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed8b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed8b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46434/0 (socket says 172.21.15.78:46434) 2023-10-10T17:46:13.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3986398069 learned_addr learned my addr 172.21.15.78:0/3986398069 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:13.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed90d02700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.002+0000 7fed8bfff700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac53b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3986398069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 msgr2=0x55cac53b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:13.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac53b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 msgr2=0x55cac53b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:13.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed8b7fe700 1 -- 172.21.15.78:0/3986398069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cac52aa190 con 0x55cac5393470 2023-10-10T17:46:13.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed90d02700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:13.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed8b7fe700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b2940 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fed80033490 tx=0x7fed8000ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d391ccc243f0213b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:13.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed6b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:13.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed6b7fe700 1 -- 172.21.15.78:0/3986398069 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80005310 con 0x55cac5393470 2023-10-10T17:46:13.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed6b7fe700 1 -- 172.21.15.78:0/3986398069 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fed80000e40 con 0x55cac5393470 2023-10-10T17:46:13.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed8001e040 con 0x55cac5393470 2023-10-10T17:46:13.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed6b7fe700 1 -- 172.21.15.78:0/3986398069 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed80005470 con 0x55cac5393470 2023-10-10T17:46:13.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed9e8f1b80 10 client.16419 fetch_fsmap learned FSMap version 30 2023-10-10T17:46:13.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55cac50fac60 con 0x55cac5393470 2023-10-10T17:46:13.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.003+0000 7fed6b7fe700 1 -- 172.21.15.78:0/3986398069 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fed8001e040 con 0x55cac5393470 2023-10-10T17:46:13.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed6b7fe700 1 -- 172.21.15.78:0/3986398069 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fed80031070 con 0x55cac5393470 2023-10-10T17:46:13.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed9e8f1b80 10 client.16419 fetch_fsmap finished waiting for FSMap version 30 2023-10-10T17:46:13.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed9e8f1b80 10 client.16419 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:13.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed9e8f1b80 20 client.16419 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:13.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac5411690 0x55cac5413aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:13.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed9e8f1b80 4 client.16419 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:46:13.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cac52b2660 con 0x55cac5411690 2023-10-10T17:46:13.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed8bfff700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac5411690 0x55cac5413aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:13.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed6b7fe700 10 client.16419 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.004+0000 7fed8bfff700 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac5411690 0x55cac5413aa0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=606dd14391d5b32b server_cookie=328b426ddf4034f1 in_seq=0 out_seq=0 2023-10-10T17:46:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed6b7fe700 1 -- 172.21.15.78:0/3986398069 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55cac52b2660 con 0x55cac5411690 2023-10-10T17:46:13.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed6b7fe700 10 client.16419 handle_command_reply: tid=0 2023-10-10T17:46:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 client.16419 shutdown 2023-10-10T17:46:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 msgr2=0x55cac53b2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:13.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b2940 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fed80033490 tx=0x7fed8000ba40 comp rx=0 tx=0).stop 2023-10-10T17:46:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 shutdown_connections 2023-10-10T17:46:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cac5392b60 0x55cac53b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cac5411690 0x55cac5413aa0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cac53978b0 0x55cac53b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed6b7fe700 0 client.16419 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:13.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 --2- 172.21.15.78:0/3986398069 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cac5393470 0x55cac53b2940 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 >> 172.21.15.78:0/3986398069 conn(0x55cac5388360 msgr2=0x55cac53958f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:13.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 shutdown_connections 2023-10-10T17:46:13.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 1 -- 172.21.15.78:0/3986398069 wait complete. 2023-10-10T17:46:13.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.007+0000 7fed9e8f1b80 20 client.16419 trim_cache size 0 max 16384 2023-10-10T17:46:13.047 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:46:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:46:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:46:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "5ecf957f-7c91-40df-8738-d2beae2feba1": { 2023-10-10T17:46:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:46:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:46:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:46:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:46:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:46:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:46:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fed740a12c0 msgr2=0x7fed740a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:13.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fed740a12c0 0x7fed740a3770 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fed7c011550 tx=0x7fed7c009f90 comp rx=0 tx=0).stop 2023-10-10T17:46:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c105600 msgr2=0x7fed8c19ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c105600 0x7fed8c19ad80 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fed8400ef30 tx=0x7fed8400c3a0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 shutdown_connections 2023-10-10T17:46:13.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fed740a12c0 0x7fed740a3770 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed8c105600 0x7fed8c19ad80 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fed8c0690e0 0x7fed8c1963e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 --2- 172.21.15.78:0/507018305 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fed8c069a80 0x7fed8c196920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 >> 172.21.15.78:0/507018305 conn(0x7fed8c075470 msgr2=0x7fed8c0fd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:13.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 shutdown_connections 2023-10-10T17:46:13.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:13.008+0000 7fed92765700 1 -- 172.21.15.78:0/507018305 wait complete. 2023-10-10T17:46:13.054 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:12 smithi078 ceph-mon[99494]: pgmap v113: 129 pgs: 129 active+clean; 276 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 568 op/s 2023-10-10T17:46:13.056 INFO:tasks.cephfs.filesystem:scrub status for tag:5ecf957f-7c91-40df-8738-d2beae2feba1 - {'path': '/', 'tag': '5ecf957f-7c91-40df-8738-d2beae2feba1', 'options': 'recursive,force'} 2023-10-10T17:46:13.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:12 smithi190 ceph-mon[101220]: pgmap v113: 129 pgs: 129 active+clean; 276 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 568 op/s 2023-10-10T17:46:13.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:12 smithi157 ceph-mon[100896]: pgmap v113: 129 pgs: 129 active+clean; 276 MiB data, 833 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.9 MiB/s wr, 568 op/s 2023-10-10T17:46:15.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:14 smithi078 ceph-mon[99494]: pgmap v114: 129 pgs: 129 active+clean; 314 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 10 MiB/s wr, 585 op/s 2023-10-10T17:46:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:14 smithi190 ceph-mon[101220]: pgmap v114: 129 pgs: 129 active+clean; 314 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 10 MiB/s wr, 585 op/s 2023-10-10T17:46:15.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:14 smithi157 ceph-mon[100896]: pgmap v114: 129 pgs: 129 active+clean; 314 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 10 MiB/s wr, 585 op/s 2023-10-10T17:46:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 -- 172.21.15.78:0/636243100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc104470 msgr2=0x7f86fc10c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:15.537 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 --2- 172.21.15.78:0/636243100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc104470 0x7f86fc10c040 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f86f400b0d0 tx=0x7f86f400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 -- 172.21.15.78:0/636243100 shutdown_connections 2023-10-10T17:46:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 --2- 172.21.15.78:0/636243100 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc0ff430 0x7f86fc0ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 --2- 172.21.15.78:0/636243100 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc104470 0x7f86fc10c040 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 --2- 172.21.15.78:0/636243100 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86fc0ffdd0 0x7f86fc103d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.538 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 -- 172.21.15.78:0/636243100 >> 172.21.15.78:0/636243100 conn(0x7f86fc076370 msgr2=0x7f86fc076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 -- 172.21.15.78:0/636243100 shutdown_connections 2023-10-10T17:46:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.524+0000 7f8702450700 1 -- 172.21.15.78:0/636243100 wait complete. 2023-10-10T17:46:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 Processor -- start 2023-10-10T17:46:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 -- start start 2023-10-10T17:46:15.539 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86fc0ff430 0x7f86fc197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc0ffdd0 0x7f86fc197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc104470 0x7f86fc1a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc1981b0 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc1a28f0 con 0x7f86fc0ff430 2023-10-10T17:46:15.540 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f8702450700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc1a2ab0 con 0x7f86fc104470 2023-10-10T17:46:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc0ffdd0 0x7f86fc197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc0ffdd0 0x7f86fc197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46448/0 (socket says 172.21.15.78:46448) 2023-10-10T17:46:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 -- 172.21.15.78:0/1383022532 learned_addr learned my addr 172.21.15.78:0/1383022532 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fbfff700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86fc0ff430 0x7f86fc197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f87009ed700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc104470 0x7f86fc1a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.541 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 -- 172.21.15.78:0/1383022532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc104470 msgr2=0x7f86fc1a23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc104470 0x7f86fc1a23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 -- 172.21.15.78:0/1383022532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86fc0ff430 msgr2=0x7f86fc197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86fc0ff430 0x7f86fc197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.525+0000 7f86fb7fe700 1 -- 172.21.15.78:0/1383022532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f4009d00 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.542 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.526+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc0ffdd0 0x7f86fc197870 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e940 tx=0x7f86ec00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.526+0000 7f86f97fa700 1 -- 172.21.15.78:0/1383022532 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec009730 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.526+0000 7f86f97fa700 1 -- 172.21.15.78:0/1383022532 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86ec010430 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.526+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86fc10f290 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.526+0000 7f86f97fa700 1 -- 172.21.15.78:0/1383022532 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00cea0 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.543 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.526+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86fc10f100 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.528+0000 7f86f97fa700 1 -- 172.21.15.78:0/1383022532 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f86ec010810 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.528+0000 7f86f97fa700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f86e40a1340 0x7f86e40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f86fbfff700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f86e40a1340 0x7f86e40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f86f97fa700 1 -- 172.21.15.78:0/1383022532 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f86ec014070 con 0x7f86fc0ffdd0 2023-10-10T17:46:15.544 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f86fbfff700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f86e40a1340 0x7f86e40a37f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009c60 tx=0x7f86f0009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 Processor -- start 2023-10-10T17:46:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 -- start start 2023-10-10T17:46:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b04366b0 0x55b5b05e45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4af0 0x55b5b05de970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.545 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4f60 0x55b5b05deef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b04e0000 con 0x55b5b05e4af0 2023-10-10T17:46:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b0532840 con 0x55b5b04366b0 2023-10-10T17:46:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.529+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b052bcd0 con 0x55b5b05e4f60 2023-10-10T17:46:15.546 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4af0 0x55b5b05de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4af0 0x55b5b05de970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46456/0 (socket says 172.21.15.78:46456) 2023-10-10T17:46:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 -- 172.21.15.78:0/4257225630 learned_addr learned my addr 172.21.15.78:0/4257225630 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fbfff700 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b04366b0 0x55b5b05e45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f87009ed700 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4f60 0x55b5b05deef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.547 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 -- 172.21.15.78:0/4257225630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4f60 msgr2=0x55b5b05deef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4f60 0x55b5b05deef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 -- 172.21.15.78:0/4257225630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b04366b0 msgr2=0x55b5b05e45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b04366b0 0x55b5b05e45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.548 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 -- 172.21.15.78:0/4257225630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5b03ad5d0 con 0x55b5b05e4af0 2023-10-10T17:46:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4af0 0x55b5b05de970 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f86ec009230 tx=0x7f86ec0997c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905cac9fa721e43a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86e27fc700 1 -- 172.21.15.78:0/4257225630 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec0a22b0 con 0x55b5b05e4af0 2023-10-10T17:46:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86e27fc700 1 -- 172.21.15.78:0/4257225630 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86ec098200 con 0x55b5b05e4af0 2023-10-10T17:46:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.530+0000 7f86e27fc700 1 -- 172.21.15.78:0/4257225630 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec0a1e20 con 0x55b5b05e4af0 2023-10-10T17:46:15.549 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 -- 172.21.15.78:0/4257225630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4af0 msgr2=0x55b5b05de970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4af0 0x55b5b05de970 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f86ec009230 tx=0x7f86ec0997c0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 -- 172.21.15.78:0/4257225630 shutdown_connections 2023-10-10T17:46:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4f60 0x55b5b05deef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b04366b0 0x55b5b05e45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.550 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/4257225630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4af0 0x55b5b05de970 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 -- 172.21.15.78:0/4257225630 >> 172.21.15.78:0/4257225630 conn(0x55b5b05cff50 msgr2=0x55b5b05d5270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 -- 172.21.15.78:0/4257225630 shutdown_connections 2023-10-10T17:46:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 -- 172.21.15.78:0/4257225630 wait complete. 2023-10-10T17:46:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 -- start start 2023-10-10T17:46:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.531+0000 7f870e5dcb80 1 Processor -- start 2023-10-10T17:46:15.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05e8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05e8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05e8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b04cd880 con 0x55b5b04366b0 2023-10-10T17:46:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b04cd520 con 0x55b5b05e4f60 2023-10-10T17:46:15.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b03af8c0 con 0x55b5b05e4af0 2023-10-10T17:46:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f870e5dcb80 10 client.0 _ll_register_callbacks cb 0x55b5b04eb720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05e8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fb7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05e8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05e8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46460/0 (socket says 172.21.15.78:46460) 2023-10-10T17:46:15.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fbfff700 1 -- 172.21.15.78:0/3797311025 learned_addr learned my addr 172.21.15.78:0/3797311025 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f87009ed700 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05e8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fb7fe700 1 -- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 msgr2=0x55b5b05e8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05e8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fb7fe700 1 -- 172.21.15.78:0/3797311025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 msgr2=0x55b5b05e8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05e8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fb7fe700 1 -- 172.21.15.78:0/3797311025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86fc10f100 con 0x55b5b05e4af0 2023-10-10T17:46:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f87009ed700 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05e8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:46:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05e8690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0968d0 tx=0x7f86ec0961a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfd499c35e26e2af server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:46:15.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.532+0000 7f86db7fe700 1 -- 172.21.15.78:0/3797311025 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ec0b8180 con 0x55b5b05e4af0 2023-10-10T17:46:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b5b05cd710 con 0x55b5b05e4af0 2023-10-10T17:46:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f86db7fe700 1 -- 172.21.15.78:0/3797311025 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f86ec0a1e20 con 0x55b5b05e4af0 2023-10-10T17:46:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f86db7fe700 1 -- 172.21.15.78:0/3797311025 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86ec0a1fe0 con 0x55b5b05e4af0 2023-10-10T17:46:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 10 client.25882 fetch_fsmap learned FSMap version 30 2023-10-10T17:46:15.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x55b5b04f2130 con 0x55b5b05e4af0 2023-10-10T17:46:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f86db7fe700 1 -- 172.21.15.78:0/3797311025 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b5b05cd710 con 0x55b5b05e4af0 2023-10-10T17:46:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f86db7fe700 1 -- 172.21.15.78:0/3797311025 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 30) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f86ec0adcf0 con 0x55b5b05e4af0 2023-10-10T17:46:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 10 client.25882 fetch_fsmap finished waiting for FSMap version 30 2023-10-10T17:46:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 10 client.25882 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:15.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 20 client.25882 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b05fa600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 4 client.25882 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:46:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5b04ee040 con 0x55b5b05f81f0 2023-10-10T17:46:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.533+0000 7f86fbfff700 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b05fa600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.558 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f86db7fe700 10 client.25882 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f86fbfff700 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b05fa600 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa2d88f71e4410cd server_cookie=ac37dd3ede5fef0e in_seq=0 out_seq=0 2023-10-10T17:46:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f86db7fe700 1 -- 172.21.15.78:0/3797311025 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b5b04ee040 con 0x55b5b05f81f0 2023-10-10T17:46:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f86db7fe700 10 client.25882 handle_command_reply: tid=0 2023-10-10T17:46:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 client.25882 shutdown 2023-10-10T17:46:15.559 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 msgr2=0x55b5b05e8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05e8690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f86ec0968d0 tx=0x7f86ec0961a0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 shutdown_connections 2023-10-10T17:46:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b05fa600 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05e8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.560 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f86db7fe700 0 client.25882 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05e8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/3797311025 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05e8690 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.534+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 >> 172.21.15.78:0/3797311025 conn(0x55b5b05cff50 msgr2=0x55b5b05d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.535+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 shutdown_connections 2023-10-10T17:46:15.561 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.535+0000 7f870e5dcb80 1 -- 172.21.15.78:0/3797311025 wait complete. 2023-10-10T17:46:15.562 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.535+0000 7f870e5dcb80 20 client.25882 trim_cache size 0 max 16384 2023-10-10T17:46:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 Processor -- start 2023-10-10T17:46:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 -- start start 2023-10-10T17:46:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b04366b0 0x55b5b05e7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4af0 0x55b5b05d4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4f60 0x55b5b05de970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b05e2e20 con 0x55b5b05e4f60 2023-10-10T17:46:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b05e48d0 con 0x55b5b05e4af0 2023-10-10T17:46:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b05d5f40 con 0x55b5b04366b0 2023-10-10T17:46:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f87009ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4f60 0x55b5b05de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f86fbfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b04366b0 0x55b5b05e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f87009ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4f60 0x55b5b05de970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46472/0 (socket says 172.21.15.78:46472) 2023-10-10T17:46:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f87009ed700 1 -- 172.21.15.78:0/2594629111 learned_addr learned my addr 172.21.15.78:0/2594629111 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.549+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4af0 0x55b5b05d4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f87009ed700 1 -- 172.21.15.78:0/2594629111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b04366b0 msgr2=0x55b5b05e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f87009ed700 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b04366b0 0x55b5b05e7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f87009ed700 1 -- 172.21.15.78:0/2594629111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4af0 msgr2=0x55b5b05d4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f87009ed700 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4af0 0x55b5b05d4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f87009ed700 1 -- 172.21.15.78:0/2594629111 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5b05e7cc0 con 0x55b5b05e4f60 2023-10-10T17:46:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4af0 0x55b5b05d4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:15.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f86fbfff700 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b04366b0 0x55b5b05e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f87009ed700 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4f60 0x55b5b05de970 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009cb0 tx=0x7f86f40092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=178ec8365fba1d12 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f86e2ffd700 1 -- 172.21.15.78:0/2594629111 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f4021070 con 0x55b5b05e4f60 2023-10-10T17:46:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f86e2ffd700 1 -- 172.21.15.78:0/2594629111 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86f4009400 con 0x55b5b05e4f60 2023-10-10T17:46:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f86e2ffd700 1 -- 172.21.15.78:0/2594629111 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f400e040 con 0x55b5b05e4f60 2023-10-10T17:46:15.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f870e5dcb80 1 -- 172.21.15.78:0/2594629111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4f60 msgr2=0x55b5b05de970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.550+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4f60 0x55b5b05de970 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f86f4009cb0 tx=0x7f86f40092e0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- 172.21.15.78:0/2594629111 shutdown_connections 2023-10-10T17:46:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b04366b0 0x55b5b05e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b05e4f60 0x55b5b05de970 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/2594629111 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4af0 0x55b5b05d4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- 172.21.15.78:0/2594629111 >> 172.21.15.78:0/2594629111 conn(0x55b5b05cff50 msgr2=0x55b5b05e22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- 172.21.15.78:0/2594629111 shutdown_connections 2023-10-10T17:46:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- 172.21.15.78:0/2594629111 wait complete. 2023-10-10T17:46:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- start start 2023-10-10T17:46:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 Processor -- start 2023-10-10T17:46:15.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05ed500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05eda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05edf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b064b070 con 0x55b5b04366b0 2023-10-10T17:46:15.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b064b1b0 con 0x55b5b05e4f60 2023-10-10T17:46:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5b064b320 con 0x55b5b05e4af0 2023-10-10T17:46:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f870e5dcb80 10 client.0 _ll_register_callbacks cb 0x7f86f00096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f86fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05ed500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f86fb7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05eda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f86fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05ed500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46474/0 (socket says 172.21.15.78:46474) 2023-10-10T17:46:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.551+0000 7f86fbfff700 1 -- 172.21.15.78:0/1847127360 learned_addr learned my addr 172.21.15.78:0/1847127360 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f87009ed700 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05edf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86fbfff700 1 -- 172.21.15.78:0/1847127360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 msgr2=0x55b5b05eda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86fbfff700 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05eda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86fbfff700 1 -- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 msgr2=0x55b5b05edf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:15.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86fbfff700 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05edf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86fbfff700 1 -- 172.21.15.78:0/1847127360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5b04f2130 con 0x55b5b04366b0 2023-10-10T17:46:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05eda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86fbfff700 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05ed500 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x55b5b05faea0 tx=0x7f86f00047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a3413e0d9501c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86db7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:15.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86db7fe700 1 -- 172.21.15.78:0/1847127360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f0022040 con 0x55b5b04366b0 2023-10-10T17:46:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86db7fe700 1 -- 172.21.15.78:0/1847127360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f86f002b250 con 0x55b5b04366b0 2023-10-10T17:46:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86ec098200 con 0x55b5b04366b0 2023-10-10T17:46:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86db7fe700 1 -- 172.21.15.78:0/1847127360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86f002bea0 con 0x55b5b04366b0 2023-10-10T17:46:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f870e5dcb80 10 client.16437 fetch_fsmap learned FSMap version 30 2023-10-10T17:46:15.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7f86fc10c880 con 0x55b5b04366b0 2023-10-10T17:46:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.552+0000 7f86db7fe700 1 -- 172.21.15.78:0/1847127360 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86ec098200 con 0x55b5b04366b0 2023-10-10T17:46:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f86db7fe700 1 -- 172.21.15.78:0/1847127360 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 30) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f86f00346f0 con 0x55b5b04366b0 2023-10-10T17:46:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f870e5dcb80 10 client.16437 fetch_fsmap finished waiting for FSMap version 30 2023-10-10T17:46:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f870e5dcb80 10 client.16437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:15.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f870e5dcb80 20 client.16437 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:15.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b064eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:15.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f870e5dcb80 4 client.16437 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:46:15.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b5b04fa600 con 0x55b5b05f81f0 2023-10-10T17:46:15.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b064eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:15.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f86db7fe700 10 client.16437 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:15.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:15.553+0000 7f86fb7fe700 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b064eb50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce5f409f659db0c8 server_cookie=1109982ad5a4508f in_seq=0 out_seq=0 2023-10-10T17:46:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f86db7fe700 1 -- 172.21.15.78:0/1847127360 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b5b04fa600 con 0x55b5b05f81f0 2023-10-10T17:46:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f86db7fe700 10 client.16437 handle_command_reply: tid=0 2023-10-10T17:46:16.224 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 client.16437 shutdown 2023-10-10T17:46:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 msgr2=0x55b5b05ed500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05ed500 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x55b5b05faea0 tx=0x7f86f00047c0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 shutdown_connections 2023-10-10T17:46:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b5b05f81f0 0x55b5b064eb50 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b5b05e4f60 0x55b5b05edf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b5b04366b0 0x55b5b05ed500 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f86db7fe700 0 client.16437 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 --2- 172.21.15.78:0/1847127360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b5b05e4af0 0x55b5b05eda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 >> 172.21.15.78:0/1847127360 conn(0x55b5b05cff50 msgr2=0x55b5b05de970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 shutdown_connections 2023-10-10T17:46:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.222+0000 7f870e5dcb80 1 -- 172.21.15.78:0/1847127360 wait complete. 2023-10-10T17:46:16.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f870e5dcb80 20 client.16437 trim_cache size 0 max 16384 2023-10-10T17:46:16.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:46:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:46:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:46:16.231 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:46:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f86e40a1340 msgr2=0x7f86e40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f86e40a1340 0x7f86e40a37f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f86f0009c60 tx=0x7f86f0009260 comp rx=0 tx=0).stop 2023-10-10T17:46:16.233 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc0ffdd0 msgr2=0x7f86fc197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc0ffdd0 0x7f86fc197870 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e940 tx=0x7f86ec00ec50 comp rx=0 tx=0).stop 2023-10-10T17:46:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 shutdown_connections 2023-10-10T17:46:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f86fc104470 0x7f86fc1a23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f86e40a1340 0x7f86e40a37f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f86fc0ff430 0x7f86fc197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 --2- 172.21.15.78:0/1383022532 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86fc0ffdd0 0x7f86fc197870 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:16.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 >> 172.21.15.78:0/1383022532 conn(0x7f86fc076370 msgr2=0x7f86fc0fca30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:16.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 shutdown_connections 2023-10-10T17:46:16.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:16.223+0000 7f8702450700 1 -- 172.21.15.78:0/1383022532 wait complete. 2023-10-10T17:46:17.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:16 smithi078 ceph-mon[99494]: pgmap v115: 129 pgs: 129 active+clean; 314 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.3 MiB/s wr, 382 op/s 2023-10-10T17:46:17.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:16 smithi190 ceph-mon[101220]: pgmap v115: 129 pgs: 129 active+clean; 314 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.3 MiB/s wr, 382 op/s 2023-10-10T17:46:17.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:16 smithi157 ceph-mon[100896]: pgmap v115: 129 pgs: 129 active+clean; 314 MiB data, 867 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.3 MiB/s wr, 382 op/s 2023-10-10T17:46:19.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:18 smithi078 ceph-mon[99494]: pgmap v116: 129 pgs: 129 active+clean; 337 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 632 op/s 2023-10-10T17:46:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:18 smithi190 ceph-mon[101220]: pgmap v116: 129 pgs: 129 active+clean; 337 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 632 op/s 2023-10-10T17:46:19.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:18 smithi157 ceph-mon[100896]: pgmap v116: 129 pgs: 129 active+clean; 337 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 632 op/s 2023-10-10T17:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:20 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:20 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:20 smithi078 ceph-mon[99494]: pgmap v117: 129 pgs: 129 active+clean; 337 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.8 MiB/s wr, 279 op/s 2023-10-10T17:46:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:20 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:46:21.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:20 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:21.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:20 smithi190 ceph-mon[101220]: pgmap v117: 129 pgs: 129 active+clean; 337 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.8 MiB/s wr, 279 op/s 2023-10-10T17:46:21.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:20 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:46:21.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:20 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:21.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:20 smithi157 ceph-mon[100896]: pgmap v117: 129 pgs: 129 active+clean; 337 MiB data, 924 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 9.8 MiB/s wr, 279 op/s 2023-10-10T17:46:22.477 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:21 smithi157 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.h.service: Service RestartSec=10s expired, scheduling restart. 2023-10-10T17:46:22.478 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:21 smithi157 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.h.service: Scheduled restart job, restart counter is at 1. 2023-10-10T17:46:22.478 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:21 smithi157 systemd[1]: Stopped Ceph mds.h for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:46:22.478 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:21 smithi157 systemd[1]: Starting Ceph mds.h for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:46:23.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:22 smithi078 ceph-mon[99494]: pgmap v118: 129 pgs: 129 active+clean; 345 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 293 op/s 2023-10-10T17:46:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:22 smithi190 ceph-mon[101220]: pgmap v118: 129 pgs: 129 active+clean; 345 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 293 op/s 2023-10-10T17:46:23.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:22 smithi157 ceph-mon[100896]: pgmap v118: 129 pgs: 129 active+clean; 345 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 293 op/s 2023-10-10T17:46:23.727 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:23 smithi157 podman[129930]: 2023-10-10T17:46:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:23 smithi078 ceph-mon[99494]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2023-10-10T17:46:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:23 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 5 up:standby 2023-10-10T17:46:24.133 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:23 smithi157 ceph-mon[100896]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2023-10-10T17:46:24.134 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:23 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 5 up:standby 2023-10-10T17:46:24.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:23 smithi190 ceph-mon[101220]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2023-10-10T17:46:24.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:23 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 5 up:standby 2023-10-10T17:46:24.477 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:24 smithi157 bash[129930]: 77de51e9bb6ebd2254ff6dee5d08f19e4f7e1469b42fe9d2e023d0ccb6140296 2023-10-10T17:46:24.478 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:24 smithi157 ceph-mds[129984]: -- 172.21.15.157:0/2491489615 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55629fc35520 con 0x55629fc0a800 2023-10-10T17:46:24.478 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:24 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-h[129959]: starting mds.h at 2023-10-10T17:46:24.479 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:46:24 smithi157 systemd[1]: Started Ceph mds.h for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:46:25.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:24 smithi078 ceph-mon[99494]: pgmap v119: 129 pgs: 129 active+clean; 389 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 371 op/s 2023-10-10T17:46:25.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:24 smithi190 ceph-mon[101220]: pgmap v119: 129 pgs: 129 active+clean; 389 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 371 op/s 2023-10-10T17:46:25.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:24 smithi157 ceph-mon[100896]: pgmap v119: 129 pgs: 129 active+clean; 389 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 14 MiB/s wr, 371 op/s 2023-10-10T17:46:26.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:25 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:46:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:25 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:25 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6836/3534679839,v1:172.21.15.157:6837/3534679839] up:boot 2023-10-10T17:46:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:25 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:46:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:25 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:46:26.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:25 smithi078 ceph-mon[99494]: pgmap v120: 129 pgs: 129 active+clean; 389 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 10 MiB/s wr, 346 op/s 2023-10-10T17:46:26.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:25 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:46:26.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:25 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:26.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:25 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6836/3534679839,v1:172.21.15.157:6837/3534679839] up:boot 2023-10-10T17:46:26.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:25 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:46:26.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:25 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:46:26.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:25 smithi190 ceph-mon[101220]: pgmap v120: 129 pgs: 129 active+clean; 389 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 10 MiB/s wr, 346 op/s 2023-10-10T17:46:26.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:25 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:46:26.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:25 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:26.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:25 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6836/3534679839,v1:172.21.15.157:6837/3534679839] up:boot 2023-10-10T17:46:26.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:25 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:46:26.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:25 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-10-10T17:46:26.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:25 smithi157 ceph-mon[100896]: pgmap v120: 129 pgs: 129 active+clean; 389 MiB data, 951 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 10 MiB/s wr, 346 op/s 2023-10-10T17:46:28.627 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:28 smithi078 ceph-mon[99494]: pgmap v121: 129 pgs: 129 active+clean; 435 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 502 op/s 2023-10-10T17:46:28.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:28 smithi190 ceph-mon[101220]: pgmap v121: 129 pgs: 129 active+clean; 435 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 502 op/s 2023-10-10T17:46:28.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:28 smithi157 ceph-mon[100896]: pgmap v121: 129 pgs: 129 active+clean; 435 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 14 MiB/s wr, 502 op/s 2023-10-10T17:46:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 -- 172.21.15.78:0/1358183750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 msgr2=0x7f882c0ff910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:29.339 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 --2- 172.21.15.78:0/1358183750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c0ff910 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8824009990 tx=0x7f8824009ca0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 -- 172.21.15.78:0/1358183750 shutdown_connections 2023-10-10T17:46:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 --2- 172.21.15.78:0/1358183750 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f882c0ffe50 0x7f882c10b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 --2- 172.21.15.78:0/1358183750 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882c104990 0x7f882c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.340 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 --2- 172.21.15.78:0/1358183750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c0ff910 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 -- 172.21.15.78:0/1358183750 >> 172.21.15.78:0/1358183750 conn(0x7f882c074a50 msgr2=0x7f882c074e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 -- 172.21.15.78:0/1358183750 shutdown_connections 2023-10-10T17:46:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.337+0000 7f88312b3700 1 -- 172.21.15.78:0/1358183750 wait complete. 2023-10-10T17:46:29.341 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 Processor -- start 2023-10-10T17:46:29.344 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 -- start start 2023-10-10T17:46:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882c0ffe50 0x7f882c196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.345 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f882c104990 0x7f882c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c19b3f0 con 0x7f882c0ff4a0 2023-10-10T17:46:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c19b560 con 0x7f882c0ffe50 2023-10-10T17:46:29.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f88312b3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c19b720 con 0x7f882c104990 2023-10-10T17:46:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46488/0 (socket says 172.21.15.78:46488) 2023-10-10T17:46:29.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882a7fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882c0ffe50 0x7f882c196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 -- 172.21.15.78:0/1090058017 learned_addr learned my addr 172.21.15.78:0/1090058017 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882b7fe700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f882c104990 0x7f882c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.348 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 -- 172.21.15.78:0/1090058017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f882c104990 msgr2=0x7f882c19adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:29.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f882c104990 0x7f882c19adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 -- 172.21.15.78:0/1090058017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882c0ffe50 msgr2=0x7f882c196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:29.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882c0ffe50 0x7f882c196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882affd700 1 -- 172.21.15.78:0/1090058017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88240095f0 con 0x7f882c0ff4a0 2023-10-10T17:46:29.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.338+0000 7f882a7fc700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882c0ffe50 0x7f882c196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:29.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.339+0000 7f882affd700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c196410 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f881c00ef10 tx=0x7f881c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:29.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.339+0000 7f8823fff700 1 -- 172.21.15.78:0/1090058017 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c00c9c0 con 0x7f882c0ff4a0 2023-10-10T17:46:29.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.339+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882c1102f0 con 0x7f882c0ff4a0 2023-10-10T17:46:29.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.339+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882c110160 con 0x7f882c0ff4a0 2023-10-10T17:46:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.339+0000 7f8823fff700 1 -- 172.21.15.78:0/1090058017 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f881c00f040 con 0x7f882c0ff4a0 2023-10-10T17:46:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.339+0000 7f8823fff700 1 -- 172.21.15.78:0/1090058017 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c014480 con 0x7f882c0ff4a0 2023-10-10T17:46:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.341+0000 7f8823fff700 1 -- 172.21.15.78:0/1090058017 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f881c007690 con 0x7f882c0ff4a0 2023-10-10T17:46:29.351 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.341+0000 7f8823fff700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88180a1340 0x7f88180a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.341+0000 7f882a7fc700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88180a1340 0x7f88180a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.342+0000 7f8823fff700 1 -- 172.21.15.78:0/1090058017 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f881c0cd1e0 con 0x7f882c0ff4a0 2023-10-10T17:46:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.342+0000 7f882a7fc700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88180a1340 0x7f88180a37f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f882c197a20 tx=0x7f8824005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.343+0000 7f883d43fb80 1 Processor -- start 2023-10-10T17:46:29.352 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.343+0000 7f883d43fb80 1 -- start start 2023-10-10T17:46:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.343+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e35de6b0 0x55d0e37881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.343+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e3788720 0x55d0e3782560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.358 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f883d43fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e3688000 con 0x55d0e3788b90 2023-10-10T17:46:29.359 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f883d43fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e36da840 con 0x55d0e35de6b0 2023-10-10T17:46:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f883d43fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e36d3cd0 con 0x55d0e3788720 2023-10-10T17:46:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.360 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46498/0 (socket says 172.21.15.78:46498) 2023-10-10T17:46:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 -- 172.21.15.78:0/265300352 learned_addr learned my addr 172.21.15.78:0/265300352 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882a7fc700 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e3788720 0x55d0e3782560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.361 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882affd700 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e35de6b0 0x55d0e37881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 -- 172.21.15.78:0/265300352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e3788720 msgr2=0x55d0e3782560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e3788720 0x55d0e3782560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.362 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 -- 172.21.15.78:0/265300352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e35de6b0 msgr2=0x55d0e37881e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e35de6b0 0x55d0e37881e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 -- 172.21.15.78:0/265300352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0e35555d0 con 0x55d0e3788b90 2023-10-10T17:46:29.363 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f882b7fe700 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782b00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8814009d30 tx=0x7f8814011720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72728a0331bd2b81 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f88217fa700 1 -- 172.21.15.78:0/265300352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814019630 con 0x55d0e3788b90 2023-10-10T17:46:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f88217fa700 1 -- 172.21.15.78:0/265300352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8814009e80 con 0x55d0e3788b90 2023-10-10T17:46:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.344+0000 7f88217fa700 1 -- 172.21.15.78:0/265300352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881400ec00 con 0x55d0e3788b90 2023-10-10T17:46:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 -- 172.21.15.78:0/265300352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 msgr2=0x55d0e3782b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:29.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782b00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f8814009d30 tx=0x7f8814011720 comp rx=0 tx=0).stop 2023-10-10T17:46:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 -- 172.21.15.78:0/265300352 shutdown_connections 2023-10-10T17:46:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782b00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e35de6b0 0x55d0e37881e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 --2- 172.21.15.78:0/265300352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e3788720 0x55d0e3782560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 -- 172.21.15.78:0/265300352 >> 172.21.15.78:0/265300352 conn(0x55d0e3777f50 msgr2=0x55d0e377d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 -- 172.21.15.78:0/265300352 shutdown_connections 2023-10-10T17:46:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.345+0000 7f883d43fb80 1 -- 172.21.15.78:0/265300352 wait complete. 2023-10-10T17:46:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 -- start start 2023-10-10T17:46:29.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 Processor -- start 2023-10-10T17:46:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3791ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3792020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e378bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e3675880 con 0x55d0e3788b90 2023-10-10T17:46:29.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e3675520 con 0x55d0e3788720 2023-10-10T17:46:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e35578c0 con 0x55d0e35de6b0 2023-10-10T17:46:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f883d43fb80 10 client.0 _ll_register_callbacks cb 0x55d0e3693720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f882b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e378bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f882affd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3791ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f882b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e378bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46506/0 (socket says 172.21.15.78:46506) 2023-10-10T17:46:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f882b7fe700 1 -- 172.21.15.78:0/800885175 learned_addr learned my addr 172.21.15.78:0/800885175 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.346+0000 7f882a7fc700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3792020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882b7fe700 1 -- 172.21.15.78:0/800885175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 msgr2=0x55d0e3791ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.369 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882b7fe700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3791ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882b7fe700 1 -- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 msgr2=0x55d0e3792020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882b7fe700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3792020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882b7fe700 1 -- 172.21.15.78:0/800885175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88140099e0 con 0x55d0e3788b90 2023-10-10T17:46:29.370 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882affd700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3791ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882a7fc700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3792020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f882b7fe700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e378bd30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8814009230 tx=0x7f881400e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f21c6184021e86a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f88127fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f88127fc700 1 -- 172.21.15.78:0/800885175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8814005440 con 0x55d0e3788b90 2023-10-10T17:46:29.371 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f88127fc700 1 -- 172.21.15.78:0/800885175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f88140055a0 con 0x55d0e3788b90 2023-10-10T17:46:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0e3696040 con 0x55d0e3788b90 2023-10-10T17:46:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f88127fc700 1 -- 172.21.15.78:0/800885175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f881400e7e0 con 0x55d0e3788b90 2023-10-10T17:46:29.372 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f883d43fb80 10 client.16452 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55d0e369a130 con 0x55d0e3788b90 2023-10-10T17:46:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f88127fc700 1 -- 172.21.15.78:0/800885175 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d0e3696040 con 0x55d0e3788b90 2023-10-10T17:46:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.347+0000 7f88127fc700 1 -- 172.21.15.78:0/800885175 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8814004ba0 con 0x55d0e3788b90 2023-10-10T17:46:29.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f883d43fb80 10 client.16452 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f883d43fb80 10 client.16452 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f883d43fb80 20 client.16452 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f883d43fb80 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e379be70 0x55d0e379e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.374 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f883d43fb80 4 client.16452 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:46:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d0e36a2600 con 0x55d0e379be70 2023-10-10T17:46:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f882affd700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e379be70 0x55d0e379e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f88127fc700 10 client.16452 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:29.375 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.348+0000 7f882affd700 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e379be70 0x55d0e379e280 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1334532fd3b1f13a server_cookie=87575bb5533077b0 in_seq=0 out_seq=0 2023-10-10T17:46:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f88127fc700 1 -- 172.21.15.78:0/800885175 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d0e36a2600 con 0x55d0e379be70 2023-10-10T17:46:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f88127fc700 10 client.16452 handle_command_reply: tid=0 2023-10-10T17:46:29.382 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f883d43fb80 1 client.16452 shutdown 2023-10-10T17:46:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 msgr2=0x55d0e378bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f883d43fb80 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e378bd30 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f8814009230 tx=0x7f881400e640 comp rx=0 tx=0).stop 2023-10-10T17:46:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 shutdown_connections 2023-10-10T17:46:29.383 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f883d43fb80 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3791ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.358+0000 7f883d43fb80 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e379be70 0x55d0e379e280 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.384 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.359+0000 7f883d43fb80 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e378bd30 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.359+0000 7f88127fc700 0 client.16452 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.359+0000 7f883d43fb80 1 --2- 172.21.15.78:0/800885175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3792020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.359+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 >> 172.21.15.78:0/800885175 conn(0x55d0e3777f50 msgr2=0x55d0e377c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.359+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 shutdown_connections 2023-10-10T17:46:29.385 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.359+0000 7f883d43fb80 1 -- 172.21.15.78:0/800885175 wait complete. 2023-10-10T17:46:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.359+0000 7f883d43fb80 20 client.16452 trim_cache size 0 max 16384 2023-10-10T17:46:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 Processor -- start 2023-10-10T17:46:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 -- start start 2023-10-10T17:46:29.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e377cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3782560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.387 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e378d340 con 0x55d0e3788b90 2023-10-10T17:46:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e3777b70 con 0x55d0e3788720 2023-10-10T17:46:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f883d43fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e377ec50 con 0x55d0e35de6b0 2023-10-10T17:46:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f882b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.388 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f882a7fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3782560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f882affd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e377cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f882b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46514/0 (socket says 172.21.15.78:46514) 2023-10-10T17:46:29.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f882b7fe700 1 -- 172.21.15.78:0/2450380536 learned_addr learned my addr 172.21.15.78:0/2450380536 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:29.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.374+0000 7f882a7fc700 1 -- 172.21.15.78:0/2450380536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 msgr2=0x55d0e377cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.389 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f882a7fc700 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e377cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f882a7fc700 1 -- 172.21.15.78:0/2450380536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 msgr2=0x55d0e3782aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f882a7fc700 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f882a7fc700 1 -- 172.21.15.78:0/2450380536 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d0e3699e80 con 0x55d0e3788720 2023-10-10T17:46:29.390 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f882b7fe700 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:29.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f882affd700 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e377cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:29.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f882a7fc700 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3782560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8824000c00 tx=0x7f8824020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c315264036b32718 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:29.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f8821ffb700 1 -- 172.21.15.78:0/2450380536 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882402b040 con 0x55d0e3788720 2023-10-10T17:46:29.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f8821ffb700 1 -- 172.21.15.78:0/2450380536 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f882402c040 con 0x55d0e3788720 2023-10-10T17:46:29.391 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.375+0000 7f8821ffb700 1 -- 172.21.15.78:0/2450380536 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8824032770 con 0x55d0e3788720 2023-10-10T17:46:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- 172.21.15.78:0/2450380536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 msgr2=0x55d0e3782560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3782560 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8824000c00 tx=0x7f8824020000 comp rx=0 tx=0).stop 2023-10-10T17:46:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- 172.21.15.78:0/2450380536 shutdown_connections 2023-10-10T17:46:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e377cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.392 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788b90 0x55d0e3782aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2450380536 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788720 0x55d0e3782560 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- 172.21.15.78:0/2450380536 >> 172.21.15.78:0/2450380536 conn(0x55d0e3777f50 msgr2=0x55d0e378c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:29.393 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- 172.21.15.78:0/2450380536 shutdown_connections 2023-10-10T17:46:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- 172.21.15.78:0/2450380536 wait complete. 2023-10-10T17:46:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- start start 2023-10-10T17:46:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 Processor -- start 2023-10-10T17:46:29.394 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3794a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788720 0x55d0e3794f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788b90 0x55d0e37eed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e3795690 con 0x55d0e3788720 2023-10-10T17:46:29.395 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e37ef260 con 0x55d0e3788b90 2023-10-10T17:46:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.376+0000 7f883d43fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0e37ef420 con 0x55d0e35de6b0 2023-10-10T17:46:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f883d43fb80 10 client.0 _ll_register_callbacks cb 0x7f8824005e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882affd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3794a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788720 0x55d0e3794f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.396 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882affd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3794a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38046/0 (socket says 172.21.15.78:38046) 2023-10-10T17:46:29.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882affd700 1 -- 172.21.15.78:0/2299178430 learned_addr learned my addr 172.21.15.78:0/2299178430 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:29.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882b7fe700 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788b90 0x55d0e37eed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882a7fc700 1 -- 172.21.15.78:0/2299178430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 msgr2=0x55d0e3794a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.397 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882a7fc700 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3794a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882a7fc700 1 -- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788b90 msgr2=0x55d0e37eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882a7fc700 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788b90 0x55d0e37eed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882a7fc700 1 -- 172.21.15.78:0/2299178430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882c110160 con 0x55d0e3788720 2023-10-10T17:46:29.398 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f882a7fc700 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788720 0x55d0e3794f40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8824005f50 tx=0x7f8824005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3be210e3fe3c5d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f88127fc700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f88127fc700 1 -- 172.21.15.78:0/2299178430 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882403e070 con 0x55d0e3788720 2023-10-10T17:46:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f88127fc700 1 -- 172.21.15.78:0/2299178430 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f8824004b00 con 0x55d0e3788720 2023-10-10T17:46:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.377+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0e3775710 con 0x55d0e3788720 2023-10-10T17:46:29.399 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f88127fc700 1 -- 172.21.15.78:0/2299178430 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f882402b040 con 0x55d0e3788720 2023-10-10T17:46:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 10 client.16458 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55d0e369a130 con 0x55d0e3788720 2023-10-10T17:46:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f88127fc700 1 -- 172.21.15.78:0/2299178430 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d0e3775710 con 0x55d0e3788720 2023-10-10T17:46:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f88127fc700 1 -- 172.21.15.78:0/2299178430 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8824041030 con 0x55d0e3788720 2023-10-10T17:46:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 10 client.16458 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 10 client.16458 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 20 client.16458 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e37f0650 0x55d0e37f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 4 client.16458 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:46:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d0e3522f10 con 0x55d0e37f0650 2023-10-10T17:46:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.378+0000 7f882affd700 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e37f0650 0x55d0e37f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.379+0000 7f882affd700 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e37f0650 0x55d0e37f2a60 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f10cd17bbceacb3d server_cookie=c558f15e8f3c7bbc in_seq=0 out_seq=0 2023-10-10T17:46:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:29.379+0000 7f88127fc700 10 client.16458 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:30.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:30 smithi190 ceph-mon[101220]: pgmap v122: 129 pgs: 129 active+clean; 435 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.2 MiB/s wr, 251 op/s 2023-10-10T17:46:30.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:30 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:46:30.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:30 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:30.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:30 smithi157 ceph-mon[100896]: pgmap v122: 129 pgs: 129 active+clean; 435 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.2 MiB/s wr, 251 op/s 2023-10-10T17:46:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:30 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:46:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:30 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:30 smithi078 ceph-mon[99494]: pgmap v122: 129 pgs: 129 active+clean; 435 MiB data, 956 MiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 8.2 MiB/s wr, 251 op/s 2023-10-10T17:46:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:30 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:46:31.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:30 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:31.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f88127fc700 1 -- 172.21.15.78:0/2299178430 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d0e3522f10 con 0x55d0e37f0650 2023-10-10T17:46:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f88127fc700 10 client.16458 handle_command_reply: tid=0 2023-10-10T17:46:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 client.16458 shutdown 2023-10-10T17:46:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788720 msgr2=0x55d0e3794f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788720 0x55d0e3794f40 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8824005f50 tx=0x7f8824005280 comp rx=0 tx=0).stop 2023-10-10T17:46:31.657 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 shutdown_connections 2023-10-10T17:46:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d0e35de6b0 0x55d0e3794a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d0e37f0650 0x55d0e37f2a60 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d0e3788b90 0x55d0e37eed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f88127fc700 0 client.16458 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:31.658 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 --2- 172.21.15.78:0/2299178430 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d0e3788720 0x55d0e3794f40 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 >> 172.21.15.78:0/2299178430 conn(0x55d0e3777f50 msgr2=0x55d0e3786de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 shutdown_connections 2023-10-10T17:46:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 1 -- 172.21.15.78:0/2299178430 wait complete. 2023-10-10T17:46:31.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.654+0000 7f883d43fb80 20 client.16458 trim_cache size 0 max 16384 2023-10-10T17:46:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:46:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:46:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:46:31.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:46:31.661 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88180a1340 msgr2=0x7f88180a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88180a1340 0x7f88180a37f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f882c197a20 tx=0x7f8824005fd0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 msgr2=0x7f882c196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c196410 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f881c00ef10 tx=0x7f881c00c3a0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.662 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 shutdown_connections 2023-10-10T17:46:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f882c104990 0x7f882c19adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f88180a1340 0x7f88180a37f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f882c0ff4a0 0x7f882c196410 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 --2- 172.21.15.78:0/1090058017 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f882c0ffe50 0x7f882c196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:31.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 >> 172.21.15.78:0/1090058017 conn(0x7f882c074a50 msgr2=0x7f882c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 shutdown_connections 2023-10-10T17:46:31.665 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:31.655+0000 7f88312b3700 1 -- 172.21.15.78:0/1090058017 wait complete. 2023-10-10T17:46:32.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:32 smithi190 ceph-mon[101220]: pgmap v123: 129 pgs: 129 active+clean; 440 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.6 MiB/s wr, 281 op/s 2023-10-10T17:46:32.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:32 smithi157 ceph-mon[100896]: pgmap v123: 129 pgs: 129 active+clean; 440 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.6 MiB/s wr, 281 op/s 2023-10-10T17:46:33.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:32 smithi078 ceph-mon[99494]: pgmap v123: 129 pgs: 129 active+clean; 440 MiB data, 948 MiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.6 MiB/s wr, 281 op/s 2023-10-10T17:46:34.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:34 smithi157 ceph-mon[100896]: pgmap v124: 129 pgs: 129 active+clean; 466 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 628 op/s 2023-10-10T17:46:35.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:34 smithi078 ceph-mon[99494]: pgmap v124: 129 pgs: 129 active+clean; 466 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 628 op/s 2023-10-10T17:46:35.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:34 smithi190 ceph-mon[101220]: pgmap v124: 129 pgs: 129 active+clean; 466 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 10 MiB/s wr, 628 op/s 2023-10-10T17:46:36.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:36 smithi157 ceph-mon[100896]: pgmap v125: 129 pgs: 129 active+clean; 466 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 546 op/s 2023-10-10T17:46:37.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:36 smithi078 ceph-mon[99494]: pgmap v125: 129 pgs: 129 active+clean; 466 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 546 op/s 2023-10-10T17:46:37.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:36 smithi190 ceph-mon[101220]: pgmap v125: 129 pgs: 129 active+clean; 466 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.5 MiB/s wr, 546 op/s 2023-10-10T17:46:38.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:38 smithi157 ceph-mon[100896]: pgmap v126: 129 pgs: 129 active+clean; 473 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.8 MiB/s wr, 771 op/s 2023-10-10T17:46:39.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:38 smithi078 ceph-mon[99494]: pgmap v126: 129 pgs: 129 active+clean; 473 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.8 MiB/s wr, 771 op/s 2023-10-10T17:46:39.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:38 smithi190 ceph-mon[101220]: pgmap v126: 129 pgs: 129 active+clean; 473 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 8.8 MiB/s wr, 771 op/s 2023-10-10T17:46:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:39 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:46:39.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:39 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:40.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:39 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:46:40.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:39 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:40.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:39 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:46:40.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:39 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:40.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:40 smithi157 ceph-mon[100896]: pgmap v127: 129 pgs: 129 active+clean; 473 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 615 op/s 2023-10-10T17:46:41.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:40 smithi078 ceph-mon[99494]: pgmap v127: 129 pgs: 129 active+clean; 473 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 615 op/s 2023-10-10T17:46:41.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:40 smithi190 ceph-mon[101220]: pgmap v127: 129 pgs: 129 active+clean; 473 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.0 MiB/s wr, 615 op/s 2023-10-10T17:46:42.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:42 smithi157 ceph-mon[100896]: pgmap v128: 129 pgs: 129 active+clean; 480 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 636 op/s 2023-10-10T17:46:43.057 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:46:43.097 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:42 smithi078 ceph-mon[99494]: pgmap v128: 129 pgs: 129 active+clean; 480 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 636 op/s 2023-10-10T17:46:43.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:42 smithi190 ceph-mon[101220]: pgmap v128: 129 pgs: 129 active+clean; 480 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 636 op/s 2023-10-10T17:46:43.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 -- 172.21.15.78:0/1740796953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc100f50 msgr2=0x7f17dc104f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 --2- 172.21.15.78:0/1740796953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc100f50 0x7f17dc104f10 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f17d00099c0 tx=0x7f17d0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 -- 172.21.15.78:0/1740796953 shutdown_connections 2023-10-10T17:46:43.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 --2- 172.21.15.78:0/1740796953 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17dc1055f0 0x7f17dc10d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 --2- 172.21.15.78:0/1740796953 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17dc1005b0 0x7f17dc100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 --2- 172.21.15.78:0/1740796953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc100f50 0x7f17dc104f10 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 -- 172.21.15.78:0/1740796953 >> 172.21.15.78:0/1740796953 conn(0x7f17dc077170 msgr2=0x7f17dc077570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:43.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 -- 172.21.15.78:0/1740796953 shutdown_connections 2023-10-10T17:46:43.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.242+0000 7f17e0eca700 1 -- 172.21.15.78:0/1740796953 wait complete. 2023-10-10T17:46:43.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.243+0000 7f17e0eca700 1 Processor -- start 2023-10-10T17:46:43.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.243+0000 7f17e0eca700 1 -- start start 2023-10-10T17:46:43.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17e0eca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc1005b0 0x7f17dc196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17e0eca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17dc100f50 0x7f17dc1969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17e0eca700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17dc1055f0 0x7f17dc19ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17e0eca700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc19b440 con 0x7f17dc1005b0 2023-10-10T17:46:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17e0eca700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc19b5b0 con 0x7f17dc1055f0 2023-10-10T17:46:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17e0eca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc19b770 con 0x7f17dc100f50 2023-10-10T17:46:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc1005b0 0x7f17dc196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc1005b0 0x7f17dc196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46530/0 (socket says 172.21.15.78:46530) 2023-10-10T17:46:43.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 -- 172.21.15.78:0/506888978 learned_addr learned my addr 172.21.15.78:0/506888978 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17dad9d700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17dc1055f0 0x7f17dc19ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17dc100f50 0x7f17dc1969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 -- 172.21.15.78:0/506888978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17dc100f50 msgr2=0x7f17dc1969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17dc100f50 0x7f17dc1969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 -- 172.21.15.78:0/506888978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17dc1055f0 msgr2=0x7f17dc19ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17dc1055f0 0x7f17dc19ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 -- 172.21.15.78:0/506888978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d00095f0 con 0x7f17dc1005b0 2023-10-10T17:46:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.244+0000 7f17da59c700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc1005b0 0x7f17dc196460 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00e9a0 tx=0x7f17cc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.245+0000 7f17cb7fe700 1 -- 172.21.15.78:0/506888978 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc009790 con 0x7f17dc1005b0 2023-10-10T17:46:43.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.245+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc110370 con 0x7f17dc1005b0 2023-10-10T17:46:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.245+0000 7f17cb7fe700 1 -- 172.21.15.78:0/506888978 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17cc010430 con 0x7f17dc1005b0 2023-10-10T17:46:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.245+0000 7f17cb7fe700 1 -- 172.21.15.78:0/506888978 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc018680 con 0x7f17dc1005b0 2023-10-10T17:46:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.245+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc1101e0 con 0x7f17dc1005b0 2023-10-10T17:46:43.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.247+0000 7f17cb7fe700 1 -- 172.21.15.78:0/506888978 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f17cc0105a0 con 0x7f17dc1005b0 2023-10-10T17:46:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.247+0000 7f17cb7fe700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17c40a12c0 0x7f17c40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.248+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17c40a12c0 0x7f17c40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.248+0000 7f17cb7fe700 1 -- 172.21.15.78:0/506888978 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f17cc014070 con 0x7f17dc1005b0 2023-10-10T17:46:43.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.248+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17c40a12c0 0x7f17c40a3770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f17d00098e0 tx=0x7f17d0000bc0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:43.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 Processor -- start 2023-10-10T17:46:43.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 -- start start 2023-10-10T17:46:43.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd98b60 0x55912cd98f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd99470 0x55912cd9d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd9d8b0 0x55912cd9dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cc9e060 con 0x55912cd98b60 2023-10-10T17:46:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912ccf08a0 con 0x55912cd99470 2023-10-10T17:46:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17ed056b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cce9d30 con 0x55912cd9d8b0 2023-10-10T17:46:43.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17da59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd98b60 0x55912cd98f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17da59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd98b60 0x55912cd98f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46538/0 (socket says 172.21.15.78:46538) 2023-10-10T17:46:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17da59c700 1 -- 172.21.15.78:0/4037002339 learned_addr learned my addr 172.21.15.78:0/4037002339 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd99470 0x55912cd9d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.249+0000 7f17dad9d700 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd9d8b0 0x55912cd9dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17da59c700 1 -- 172.21.15.78:0/4037002339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd9d8b0 msgr2=0x55912cd9dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17da59c700 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd9d8b0 0x55912cd9dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17da59c700 1 -- 172.21.15.78:0/4037002339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd99470 msgr2=0x55912cd9d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17da59c700 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd99470 0x55912cd9d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17da59c700 1 -- 172.21.15.78:0/4037002339 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55912cb6b630 con 0x55912cd98b60 2023-10-10T17:46:43.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17da59c700 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd98b60 0x55912cd98f30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f17cc096a20 tx=0x7f17cc099c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6172d0e56c173d65 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17c8ff9700 1 -- 172.21.15.78:0/4037002339 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0a2400 con 0x55912cd98b60 2023-10-10T17:46:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17c8ff9700 1 -- 172.21.15.78:0/4037002339 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17cc010880 con 0x55912cd98b60 2023-10-10T17:46:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.250+0000 7f17c8ff9700 1 -- 172.21.15.78:0/4037002339 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0a1f70 con 0x55912cd98b60 2023-10-10T17:46:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- 172.21.15.78:0/4037002339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd98b60 msgr2=0x55912cd98f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd98b60 0x55912cd98f30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f17cc096a20 tx=0x7f17cc099c20 comp rx=0 tx=0).stop 2023-10-10T17:46:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- 172.21.15.78:0/4037002339 shutdown_connections 2023-10-10T17:46:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd9d8b0 0x55912cd9dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd98b60 0x55912cd98f30 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 --2- 172.21.15.78:0/4037002339 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd99470 0x55912cd9d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- 172.21.15.78:0/4037002339 >> 172.21.15.78:0/4037002339 conn(0x55912cd8e360 msgr2=0x55912cd93b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- 172.21.15.78:0/4037002339 shutdown_connections 2023-10-10T17:46:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- 172.21.15.78:0/4037002339 wait complete. 2023-10-10T17:46:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- start start 2023-10-10T17:46:43.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 Processor -- start 2023-10-10T17:46:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd98b60 0x55912cda5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cc8b8e0 con 0x55912cd9d8b0 2023-10-10T17:46:43.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cc8b580 con 0x55912cd98b60 2023-10-10T17:46:43.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cb6d920 con 0x55912cd99470 2023-10-10T17:46:43.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.251+0000 7f17ed056b80 10 client.0 _ll_register_callbacks cb 0x55912cca9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:43.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd98b60 0x55912cda5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17dad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd98b60 0x55912cda5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48266/0 (socket says 172.21.15.78:48266) 2023-10-10T17:46:43.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 -- 172.21.15.78:0/3807403680 learned_addr learned my addr 172.21.15.78:0/3807403680 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:43.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 -- 172.21.15.78:0/3807403680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 msgr2=0x55912cda2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 -- 172.21.15.78:0/3807403680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 msgr2=0x55912cda2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 -- 172.21.15.78:0/3807403680 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc1101e0 con 0x55912cd98b60 2023-10-10T17:46:43.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17dad9d700 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:43.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17da59c700 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd98b60 0x55912cda5090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0cd2b0 tx=0x7f17cc099750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df806c8b52ae3892 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:43.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17c1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:46:43.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17c1ffb700 1 -- 172.21.15.78:0/3807403680 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17cc0a2400 con 0x55912cd98b60 2023-10-10T17:46:43.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17c1ffb700 1 -- 172.21.15.78:0/3807403680 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f17cc0a2180 con 0x55912cd98b60 2023-10-10T17:46:43.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55912cd8bb20 con 0x55912cd98b60 2023-10-10T17:46:43.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.252+0000 7f17c1ffb700 1 -- 172.21.15.78:0/3807403680 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17cc0b8170 con 0x55912cd98b60 2023-10-10T17:46:43.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 10 client.25832 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:43.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55912ccb0190 con 0x55912cd98b60 2023-10-10T17:46:43.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17c1ffb700 1 -- 172.21.15.78:0/3807403680 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17cc0c8080 con 0x55912cd98b60 2023-10-10T17:46:43.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17c1ffb700 1 -- 172.21.15.78:0/3807403680 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f17cc098fa0 con 0x55912cd98b60 2023-10-10T17:46:43.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 10 client.25832 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:43.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 10 client.25832 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:43.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 20 client.25832 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:43.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912cdb24f0 0x55912cdb4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 4 client.25832 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:46:43.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55912ccac0a0 con 0x55912cdb24f0 2023-10-10T17:46:43.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.253+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912cdb24f0 0x55912cdb4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.254+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912cdb24f0 0x55912cdb4900 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5f37dc311c0e160 server_cookie=3dd1dc08c242f850 in_seq=0 out_seq=0 2023-10-10T17:46:43.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.254+0000 7f17c1ffb700 10 client.25832 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:43.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17c1ffb700 1 -- 172.21.15.78:0/3807403680 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55912ccac0a0 con 0x55912cdb24f0 2023-10-10T17:46:43.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17c1ffb700 10 client.25832 handle_command_reply: tid=0 2023-10-10T17:46:43.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 client.25832 shutdown 2023-10-10T17:46:43.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd98b60 msgr2=0x55912cda5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd98b60 0x55912cda5090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0cd2b0 tx=0x7f17cc099750 comp rx=0 tx=0).stop 2023-10-10T17:46:43.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 shutdown_connections 2023-10-10T17:46:43.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912cdb24f0 0x55912cdb4900 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17c1ffb700 0 client.25832 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:43.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cd98b60 0x55912cda5090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 --2- 172.21.15.78:0/3807403680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 >> 172.21.15.78:0/3807403680 conn(0x55912cd8e360 msgr2=0x55912cd9ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:43.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 shutdown_connections 2023-10-10T17:46:43.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 1 -- 172.21.15.78:0/3807403680 wait complete. 2023-10-10T17:46:43.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.255+0000 7f17ed056b80 20 client.25832 trim_cache size 0 max 16384 2023-10-10T17:46:43.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.273+0000 7f17ed056b80 1 Processor -- start 2023-10-10T17:46:43.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.273+0000 7f17ed056b80 1 -- start start 2023-10-10T17:46:43.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.273+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cdaec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.273+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17ed056b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cda38f0 con 0x55912cd9d8b0 2023-10-10T17:46:43.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17ed056b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cd8dee0 con 0x55912cda5170 2023-10-10T17:46:43.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17ed056b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cda0870 con 0x55912cd99470 2023-10-10T17:46:43.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17da59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cdaec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17da59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cdaec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38082/0 (socket says 172.21.15.78:38082) 2023-10-10T17:46:43.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17da59c700 1 -- 172.21.15.78:0/2383542231 learned_addr learned my addr 172.21.15.78:0/2383542231 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:43.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46554/0 (socket says 172.21.15.78:46554) 2023-10-10T17:46:43.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17dad9d700 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 -- 172.21.15.78:0/2383542231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 msgr2=0x55912cdaec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cdaec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 -- 172.21.15.78:0/2383542231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 msgr2=0x55912cda1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 -- 172.21.15.78:0/2383542231 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55912ccafee0 con 0x55912cd9d8b0 2023-10-10T17:46:43.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.274+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda4c30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f17dc06a370 tx=0x7f17d0022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92639597465825ff server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:43.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.275+0000 7f17c97fa700 1 -- 172.21.15.78:0/2383542231 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002c030 con 0x55912cd9d8b0 2023-10-10T17:46:43.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.275+0000 7f17c97fa700 1 -- 172.21.15.78:0/2383542231 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17d002d040 con 0x55912cd9d8b0 2023-10-10T17:46:43.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.275+0000 7f17c97fa700 1 -- 172.21.15.78:0/2383542231 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d0005070 con 0x55912cd9d8b0 2023-10-10T17:46:43.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.275+0000 7f17ed056b80 1 -- 172.21.15.78:0/2383542231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 msgr2=0x55912cda4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.275+0000 7f17ed056b80 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda4c30 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f17dc06a370 tx=0x7f17d0022000 comp rx=0 tx=0).stop 2023-10-10T17:46:43.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- 172.21.15.78:0/2383542231 shutdown_connections 2023-10-10T17:46:43.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cdaec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 --2- 172.21.15.78:0/2383542231 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda4c30 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- 172.21.15.78:0/2383542231 >> 172.21.15.78:0/2383542231 conn(0x55912cd8e360 msgr2=0x55912cd92550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:43.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- 172.21.15.78:0/2383542231 shutdown_connections 2023-10-10T17:46:43.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- 172.21.15.78:0/2383542231 wait complete. 2023-10-10T17:46:43.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- start start 2023-10-10T17:46:43.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 Processor -- start 2023-10-10T17:46:43.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda7b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda80d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cda86a0 con 0x55912cd9d8b0 2023-10-10T17:46:43.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cda87e0 con 0x55912cda5170 2023-10-10T17:46:43.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55912cda8970 con 0x55912cd99470 2023-10-10T17:46:43.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.276+0000 7f17ed056b80 10 client.0 _ll_register_callbacks cb 0x7f17d0003850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:43.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17da59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46560/0 (socket says 172.21.15.78:46560) 2023-10-10T17:46:43.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 -- 172.21.15.78:0/143466465 learned_addr learned my addr 172.21.15.78:0/143466465 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:43.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17dad9d700 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 -- 172.21.15.78:0/143466465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 msgr2=0x55912cda98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 -- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 msgr2=0x55912cda80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:43.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda80d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 -- 172.21.15.78:0/143466465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc10da00 con 0x55912cd9d8b0 2023-10-10T17:46:43.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17da59c700 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:43.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17d9d9b700 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda7b90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f17d0003fd0 tx=0x7f17d0003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e27eaf284e7b55 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:43.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17c1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17c1ffb700 1 -- 172.21.15.78:0/143466465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002c070 con 0x55912cd9d8b0 2023-10-10T17:46:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17c1ffb700 1 -- 172.21.15.78:0/143466465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f17d000bd80 con 0x55912cd9d8b0 2023-10-10T17:46:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17cc010880 con 0x55912cd9d8b0 2023-10-10T17:46:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17c1ffb700 1 -- 172.21.15.78:0/143466465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17d0033030 con 0x55912cd9d8b0 2023-10-10T17:46:43.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17ed056b80 10 client.16476 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55912cb00c60 con 0x55912cd9d8b0 2023-10-10T17:46:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.277+0000 7f17c1ffb700 1 -- 172.21.15.78:0/143466465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f17cc010880 con 0x55912cd9d8b0 2023-10-10T17:46:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17c1ffb700 1 -- 172.21.15.78:0/143466465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f17d000f460 con 0x55912cd9d8b0 2023-10-10T17:46:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17ed056b80 10 client.16476 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:43.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17ed056b80 10 client.16476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:43.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17ed056b80 20 client.16476 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:43.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17ed056b80 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912ce074e0 0x55912ce098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:43.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17ed056b80 4 client.16476 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:46:43.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55912ccb8660 con 0x55912ce074e0 2023-10-10T17:46:43.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17da59c700 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912ce074e0 0x55912ce098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17c1ffb700 10 client.16476 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:43.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.278+0000 7f17da59c700 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912ce074e0 0x55912ce098f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2875a977e7bfa8a server_cookie=6096570259fcd7e5 in_seq=0 out_seq=0 2023-10-10T17:46:43.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17c1ffb700 1 -- 172.21.15.78:0/143466465 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55912ccb8660 con 0x55912ce074e0 2023-10-10T17:46:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17c1ffb700 10 client.16476 handle_command_reply: tid=0 2023-10-10T17:46:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 client.16476 shutdown 2023-10-10T17:46:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 msgr2=0x55912cda7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda7b90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f17d0003fd0 tx=0x7f17d0003c90 comp rx=0 tx=0).stop 2023-10-10T17:46:43.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 shutdown_connections 2023-10-10T17:46:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55912cd99470 0x55912cda98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55912ce074e0 0x55912ce098f0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55912cda5170 0x55912cda80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17c1ffb700 0 client.16476 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:43.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 --2- 172.21.15.78:0/143466465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55912cd9d8b0 0x55912cda7b90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 >> 172.21.15.78:0/143466465 conn(0x55912cd8e360 msgr2=0x55912cdaec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:43.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 shutdown_connections 2023-10-10T17:46:43.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 1 -- 172.21.15.78:0/143466465 wait complete. 2023-10-10T17:46:43.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.286+0000 7f17ed056b80 20 client.16476 trim_cache size 0 max 16384 2023-10-10T17:46:43.301 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:46:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:46:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:46:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "5ecf957f-7c91-40df-8738-d2beae2feba1": { 2023-10-10T17:46:43.301 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:46:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:46:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:46:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:46:43.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:46:43.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:46:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17c40a12c0 msgr2=0x7f17c40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17c40a12c0 0x7f17c40a3770 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f17d00098e0 tx=0x7f17d0000bc0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc1005b0 msgr2=0x7f17dc196460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:43.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc1005b0 0x7f17dc196460 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f17cc00e9a0 tx=0x7f17cc00ecb0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 shutdown_connections 2023-10-10T17:46:43.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17c40a12c0 0x7f17c40a3770 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17dc1055f0 0x7f17dc19ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17dc1005b0 0x7f17dc196460 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 --2- 172.21.15.78:0/506888978 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f17dc100f50 0x7f17dc1969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:43.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 >> 172.21.15.78:0/506888978 conn(0x7f17dc077170 msgr2=0x7f17dc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:43.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 shutdown_connections 2023-10-10T17:46:43.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:46:43.287+0000 7f17e0eca700 1 -- 172.21.15.78:0/506888978 wait complete. 2023-10-10T17:46:43.307 INFO:tasks.cephfs.filesystem:scrub status for tag:5ecf957f-7c91-40df-8738-d2beae2feba1 - {'path': '/', 'tag': '5ecf957f-7c91-40df-8738-d2beae2feba1', 'options': 'recursive,force'} 2023-10-10T17:46:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 -- 172.21.15.78:0/2782548689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d78104400 msgr2=0x7f3d7810bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 --2- 172.21.15.78:0/2782548689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d78104400 0x7f3d7810bfd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f3d740099c0 tx=0x7f3d74009cd0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.792 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 -- 172.21.15.78:0/2782548689 shutdown_connections 2023-10-10T17:46:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 --2- 172.21.15.78:0/2782548689 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d780ff3c0 0x7f3d780ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 --2- 172.21.15.78:0/2782548689 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d78104400 0x7f3d7810bfd0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 --2- 172.21.15.78:0/2782548689 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d780ffd60 0x7f3d78103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.793 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 -- 172.21.15.78:0/2782548689 >> 172.21.15.78:0/2782548689 conn(0x7f3d780763a0 msgr2=0x7f3d780767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 -- 172.21.15.78:0/2782548689 shutdown_connections 2023-10-10T17:46:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.788+0000 7f3d7f297700 1 -- 172.21.15.78:0/2782548689 wait complete. 2023-10-10T17:46:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 Processor -- start 2023-10-10T17:46:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 -- start start 2023-10-10T17:46:44.794 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d780ff3c0 0x7f3d78100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d780ffd60 0x7f3d78101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d78104400 0x7f3d781a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d78101b30 con 0x7f3d780ff3c0 2023-10-10T17:46:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781a4af0 con 0x7f3d780ffd60 2023-10-10T17:46:44.795 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781a4c30 con 0x7f3d78104400 2023-10-10T17:46:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d780ff3c0 0x7f3d78100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d780ff3c0 0x7f3d78100dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46572/0 (socket says 172.21.15.78:46572) 2023-10-10T17:46:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 -- 172.21.15.78:0/304041547 learned_addr learned my addr 172.21.15.78:0/304041547 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d834700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d78104400 0x7f3d781a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7c832700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d780ffd60 0x7f3d78101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 -- 172.21.15.78:0/304041547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d78104400 msgr2=0x7f3d781a45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d78104400 0x7f3d781a45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 -- 172.21.15.78:0/304041547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d780ffd60 msgr2=0x7f3d78101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d780ffd60 0x7f3d78101300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 -- 172.21.15.78:0/304041547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740095f0 con 0x7f3d780ff3c0 2023-10-10T17:46:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7c832700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d780ffd60 0x7f3d78101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:46:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7d033700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d780ff3c0 0x7f3d78100dc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3d70007f10 tx=0x7f3d7000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d6e7fc700 1 -- 172.21.15.78:0/304041547 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7000ddd0 con 0x7f3d780ff3c0 2023-10-10T17:46:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d6e7fc700 1 -- 172.21.15.78:0/304041547 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d7000f040 con 0x7f3d780ff3c0 2023-10-10T17:46:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7810f1d0 con 0x7f3d780ff3c0 2023-10-10T17:46:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.789+0000 7f3d6e7fc700 1 -- 172.21.15.78:0/304041547 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d70004530 con 0x7f3d780ff3c0 2023-10-10T17:46:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.790+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7810f040 con 0x7f3d780ff3c0 2023-10-10T17:46:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.791+0000 7f3d6e7fc700 1 -- 172.21.15.78:0/304041547 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3d70004ad0 con 0x7f3d780ff3c0 2023-10-10T17:46:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.792+0000 7f3d6e7fc700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d640a12c0 0x7f3d640a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.792+0000 7f3d7c832700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d640a12c0 0x7f3d640a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.792+0000 7f3d6e7fc700 1 -- 172.21.15.78:0/304041547 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3d700ccad0 con 0x7f3d780ff3c0 2023-10-10T17:46:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.793+0000 7f3d7c832700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d640a12c0 0x7f3d640a3770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d781025b0 tx=0x7f3d68005d80 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.793+0000 7f3d8b423b80 1 Processor -- start 2023-10-10T17:46:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d8b423b80 1 -- start start 2023-10-10T17:46:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x55779436d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.806 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x557794367560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794367b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55779426d000 con 0x55779436d720 2023-10-10T17:46:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577942bf840 con 0x55779436db90 2023-10-10T17:46:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577942b8cd0 con 0x5577941c36b0 2023-10-10T17:46:44.807 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x557794367560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x557794367560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46582/0 (socket says 172.21.15.78:46582) 2023-10-10T17:46:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7d033700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x55779436d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 -- 172.21.15.78:0/1340495653 learned_addr learned my addr 172.21.15.78:0/1340495653 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7d834700 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794367b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.808 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 -- 172.21.15.78:0/1340495653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 msgr2=0x55779436d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x55779436d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 -- 172.21.15.78:0/1340495653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 msgr2=0x557794367b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794367b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 -- 172.21.15.78:0/1340495653 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55779413a5d0 con 0x55779436d720 2023-10-10T17:46:44.809 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7d033700 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x55779436d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.794+0000 7f3d7c832700 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x557794367560 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800fc40 tx=0x7f3d68012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448e45681ea594fb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d637fe700 1 -- 172.21.15.78:0/1340495653 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68011ad0 con 0x55779436d720 2023-10-10T17:46:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d637fe700 1 -- 172.21.15.78:0/1340495653 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d68011c30 con 0x55779436d720 2023-10-10T17:46:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d637fe700 1 -- 172.21.15.78:0/1340495653 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6801d610 con 0x55779436d720 2023-10-10T17:46:44.810 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 -- 172.21.15.78:0/1340495653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 msgr2=0x557794367560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x557794367560 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800fc40 tx=0x7f3d68012000 comp rx=0 tx=0).stop 2023-10-10T17:46:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 -- 172.21.15.78:0/1340495653 shutdown_connections 2023-10-10T17:46:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x55779436d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794367b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.811 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/1340495653 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x557794367560 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 -- 172.21.15.78:0/1340495653 >> 172.21.15.78:0/1340495653 conn(0x55779435cf50 msgr2=0x557794362270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 -- 172.21.15.78:0/1340495653 shutdown_connections 2023-10-10T17:46:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.795+0000 7f3d8b423b80 1 -- 172.21.15.78:0/1340495653 wait complete. 2023-10-10T17:46:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 -- start start 2023-10-10T17:46:44.812 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 Processor -- start 2023-10-10T17:46:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577941c36b0 0x557794373af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55779436d720 0x557794370d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794371460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55779425a880 con 0x5577941c36b0 2023-10-10T17:46:44.813 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55779425a520 con 0x55779436db90 2023-10-10T17:46:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55779413c8c0 con 0x55779436d720 2023-10-10T17:46:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d8b423b80 10 client.0 _ll_register_callbacks cb 0x557794278720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d7d033700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577941c36b0 0x557794373af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d7d033700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577941c36b0 0x557794373af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46586/0 (socket says 172.21.15.78:46586) 2023-10-10T17:46:44.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d7d033700 1 -- 172.21.15.78:0/317818185 learned_addr learned my addr 172.21.15.78:0/317818185 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d7d834700 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794371460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.796+0000 7f3d7c832700 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55779436d720 0x557794370d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d7d033700 1 -- 172.21.15.78:0/317818185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55779436d720 msgr2=0x557794370d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d7d033700 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55779436d720 0x557794370d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d7d033700 1 -- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 msgr2=0x557794371460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d7d033700 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794371460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d7d033700 1 -- 172.21.15.78:0/317818185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6800f920 con 0x5577941c36b0 2023-10-10T17:46:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d7d033700 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577941c36b0 0x557794373af0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3d70095d70 tx=0x7f3d70099960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c073959ac71bb8c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d60ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d60ff9700 1 -- 172.21.15.78:0/317818185 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d700b7ea0 con 0x5577941c36b0 2023-10-10T17:46:44.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d60ff9700 1 -- 172.21.15.78:0/317818185 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3d70099cd0 con 0x5577941c36b0 2023-10-10T17:46:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55779435a710 con 0x5577941c36b0 2023-10-10T17:46:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d60ff9700 1 -- 172.21.15.78:0/317818185 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d700048a0 con 0x5577941c36b0 2023-10-10T17:46:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 10 client.16488 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55779427f130 con 0x5577941c36b0 2023-10-10T17:46:44.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d60ff9700 1 -- 172.21.15.78:0/317818185 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55779435a710 con 0x5577941c36b0 2023-10-10T17:46:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d60ff9700 1 -- 172.21.15.78:0/317818185 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3d700b90c0 con 0x5577941c36b0 2023-10-10T17:46:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 10 client.16488 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 10 client.16488 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 20 client.16488 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:44.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557794380fa0 0x5577943833b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 4 client.16488 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:46:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.797+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55779427b040 con 0x557794380fa0 2023-10-10T17:46:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.798+0000 7f3d7c832700 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557794380fa0 0x5577943833b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.798+0000 7f3d60ff9700 10 client.16488 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:44.819 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.798+0000 7f3d7c832700 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557794380fa0 0x5577943833b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7f5dbacdc7d7ed6 server_cookie=4f19a36ffc0ce61c in_seq=0 out_seq=0 2023-10-10T17:46:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d60ff9700 1 -- 172.21.15.78:0/317818185 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55779427b040 con 0x557794380fa0 2023-10-10T17:46:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d60ff9700 10 client.16488 handle_command_reply: tid=0 2023-10-10T17:46:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 client.16488 shutdown 2023-10-10T17:46:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577941c36b0 msgr2=0x557794373af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:44.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577941c36b0 0x557794373af0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3d70095d70 tx=0x7f3d70099960 comp rx=0 tx=0).stop 2023-10-10T17:46:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 shutdown_connections 2023-10-10T17:46:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557794380fa0 0x5577943833b0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x557794371460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577941c36b0 0x557794373af0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d60ff9700 0 client.16488 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/317818185 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55779436d720 0x557794370d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 >> 172.21.15.78:0/317818185 conn(0x55779435cf50 msgr2=0x5577943617e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 shutdown_connections 2023-10-10T17:46:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 1 -- 172.21.15.78:0/317818185 wait complete. 2023-10-10T17:46:44.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.799+0000 7f3d8b423b80 20 client.16488 trim_cache size 0 max 16384 2023-10-10T17:46:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 Processor -- start 2023-10-10T17:46:44.827 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 -- start start 2023-10-10T17:46:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x557794370d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436d720 0x557794376640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436db90 0x557794361f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.828 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55779436d580 con 0x55779436db90 2023-10-10T17:46:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55779437e630 con 0x55779436d720 2023-10-10T17:46:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55779435a5e0 con 0x5577941c36b0 2023-10-10T17:46:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d7d834700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436db90 0x557794361f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d7c832700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436d720 0x557794376640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.829 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d7d033700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x557794370d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d7d834700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436db90 0x557794361f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46598/0 (socket says 172.21.15.78:46598) 2023-10-10T17:46:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.814+0000 7f3d7d834700 1 -- 172.21.15.78:0/3312995488 learned_addr learned my addr 172.21.15.78:0/3312995488 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d7c832700 1 -- 172.21.15.78:0/3312995488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 msgr2=0x557794370d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d7c832700 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x557794370d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.830 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d7c832700 1 -- 172.21.15.78:0/3312995488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436db90 msgr2=0x557794361f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d7c832700 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436db90 0x557794361f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d7c832700 1 -- 172.21.15.78:0/3312995488 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55779427ee80 con 0x55779436d720 2023-10-10T17:46:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d7d033700 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x557794370d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d7c832700 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436d720 0x557794376640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800bfd0 tx=0x7f3d6800b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e11b58ae1065454 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:44.831 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d63fff700 1 -- 172.21.15.78:0/3312995488 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6802d040 con 0x55779436d720 2023-10-10T17:46:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d63fff700 1 -- 172.21.15.78:0/3312995488 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d6802e040 con 0x55779436d720 2023-10-10T17:46:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.815+0000 7f3d63fff700 1 -- 172.21.15.78:0/3312995488 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68005280 con 0x55779436d720 2023-10-10T17:46:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 -- 172.21.15.78:0/3312995488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436d720 msgr2=0x557794376640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436d720 0x557794376640 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d6800bfd0 tx=0x7f3d6800b410 comp rx=0 tx=0).stop 2023-10-10T17:46:44.832 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 -- 172.21.15.78:0/3312995488 shutdown_connections 2023-10-10T17:46:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x557794370d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436db90 0x557794361f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/3312995488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436d720 0x557794376640 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 -- 172.21.15.78:0/3312995488 >> 172.21.15.78:0/3312995488 conn(0x55779435cf50 msgr2=0x557794370610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:44.833 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 -- 172.21.15.78:0/3312995488 shutdown_connections 2023-10-10T17:46:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 -- 172.21.15.78:0/3312995488 wait complete. 2023-10-10T17:46:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 -- start start 2023-10-10T17:46:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 Processor -- start 2023-10-10T17:46:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x5577943c8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.834 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.816+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x5577943c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x5577943d3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577943c9bc0 con 0x55779436d720 2023-10-10T17:46:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577943d4370 con 0x55779436db90 2023-10-10T17:46:44.835 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577943d44b0 con 0x5577941c36b0 2023-10-10T17:46:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 10 client.0 _ll_register_callbacks cb 0x7f3d70099430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x5577943c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x5577943c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46602/0 (socket says 172.21.15.78:46602) 2023-10-10T17:46:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 -- 172.21.15.78:0/4024632115 learned_addr learned my addr 172.21.15.78:0/4024632115 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:44.836 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7d033700 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x5577943c8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7d834700 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x5577943d3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 -- 172.21.15.78:0/4024632115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 msgr2=0x5577943c8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x5577943c8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 -- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 msgr2=0x5577943d3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:44.837 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x5577943d3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 -- 172.21.15.78:0/4024632115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55779427f130 con 0x55779436d720 2023-10-10T17:46:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d7c832700 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x5577943c92f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3d68034460 tx=0x7f3d680057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5db1a348fc127fe server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d60ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:46:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d60ff9700 1 -- 172.21.15.78:0/4024632115 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68011a50 con 0x55779436d720 2023-10-10T17:46:44.838 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d60ff9700 1 -- 172.21.15.78:0/4024632115 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f3d680349d0 con 0x55779436d720 2023-10-10T17:46:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d68011c30 con 0x55779436d720 2023-10-10T17:46:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d60ff9700 1 -- 172.21.15.78:0/4024632115 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d68034b90 con 0x55779436d720 2023-10-10T17:46:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 10 client.16494 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.817+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55779413a5d0 con 0x55779436d720 2023-10-10T17:46:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d60ff9700 1 -- 172.21.15.78:0/4024632115 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d68011c30 con 0x55779436d720 2023-10-10T17:46:44.839 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d60ff9700 1 -- 172.21.15.78:0/4024632115 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3d68045080 con 0x55779436d720 2023-10-10T17:46:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d8b423b80 10 client.16494 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d8b423b80 10 client.16494 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d8b423b80 20 client.16494 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:44.840 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5577943d5760 0x5577943d7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d8b423b80 4 client.16494 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:46:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557794287600 con 0x5577943d5760 2023-10-10T17:46:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d7d033700 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5577943d5760 0x5577943d7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d60ff9700 10 client.16494 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:44.841 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:44.818+0000 7f3d7d033700 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5577943d5760 0x5577943d7b70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9934934630bc1cf server_cookie=58cedfad8843d258 in_seq=0 out_seq=0 2023-10-10T17:46:44.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:44 smithi157 ceph-mon[100896]: pgmap v129: 129 pgs: 129 active+clean; 500 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.9 MiB/s wr, 777 op/s 2023-10-10T17:46:45.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:44 smithi078 ceph-mon[99494]: pgmap v129: 129 pgs: 129 active+clean; 500 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.9 MiB/s wr, 777 op/s 2023-10-10T17:46:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:44 smithi190 ceph-mon[101220]: pgmap v129: 129 pgs: 129 active+clean; 500 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.9 MiB/s wr, 777 op/s 2023-10-10T17:46:45.271 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.269+0000 7f3d60ff9700 1 -- 172.21.15.78:0/4024632115 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557794287600 con 0x5577943d5760 2023-10-10T17:46:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.269+0000 7f3d60ff9700 10 client.16494 handle_command_reply: tid=0 2023-10-10T17:46:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 client.16494 shutdown 2023-10-10T17:46:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 msgr2=0x5577943c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x5577943c92f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3d68034460 tx=0x7f3d680057b0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.272 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 shutdown_connections 2023-10-10T17:46:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577941c36b0 0x5577943c8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5577943d5760 0x5577943d7b70 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55779436db90 0x5577943d3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 --2- 172.21.15.78:0/4024632115 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55779436d720 0x5577943c92f0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 >> 172.21.15.78:0/4024632115 conn(0x55779435cf50 msgr2=0x55779436df60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:45.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d60ff9700 0 client.16494 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 shutdown_connections 2023-10-10T17:46:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.270+0000 7f3d8b423b80 1 -- 172.21.15.78:0/4024632115 wait complete. 2023-10-10T17:46:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d8b423b80 20 client.16494 trim_cache size 0 max 16384 2023-10-10T17:46:45.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:46:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:46:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:46:45.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:46:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d640a12c0 msgr2=0x7f3d640a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d640a12c0 0x7f3d640a3770 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f3d781025b0 tx=0x7f3d68005d80 comp rx=0 tx=0).stop 2023-10-10T17:46:45.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d780ff3c0 msgr2=0x7f3d78100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d780ff3c0 0x7f3d78100dc0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f3d70007f10 tx=0x7f3d7000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 shutdown_connections 2023-10-10T17:46:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d78104400 0x7f3d781a45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3d640a12c0 0x7f3d640a3770 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.278 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d780ff3c0 0x7f3d78100dc0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 --2- 172.21.15.78:0/304041547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d780ffd60 0x7f3d78101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 >> 172.21.15.78:0/304041547 conn(0x7f3d780763a0 msgr2=0x7f3d78068060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 shutdown_connections 2023-10-10T17:46:45.279 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:45.271+0000 7f3d7f297700 1 -- 172.21.15.78:0/304041547 wait complete. 2023-10-10T17:46:45.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:45 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:46:45.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:45 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:46.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:45 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:46:46.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:45 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:46.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:45 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:46:46.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:45 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:46.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:46 smithi157 ceph-mon[100896]: pgmap v130: 129 pgs: 129 active+clean; 500 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 415 op/s 2023-10-10T17:46:47.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:46 smithi078 ceph-mon[99494]: pgmap v130: 129 pgs: 129 active+clean; 500 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 415 op/s 2023-10-10T17:46:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:46 smithi190 ceph-mon[101220]: pgmap v130: 129 pgs: 129 active+clean; 500 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.6 MiB/s wr, 415 op/s 2023-10-10T17:46:48.981 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:48 smithi157 ceph-mon[100896]: pgmap v131: 129 pgs: 129 active+clean; 512 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.0 MiB/s wr, 743 op/s 2023-10-10T17:46:49.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:48 smithi078 ceph-mon[99494]: pgmap v131: 129 pgs: 129 active+clean; 512 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.0 MiB/s wr, 743 op/s 2023-10-10T17:46:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:48 smithi190 ceph-mon[101220]: pgmap v131: 129 pgs: 129 active+clean; 512 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 7.0 MiB/s wr, 743 op/s 2023-10-10T17:46:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:50 smithi157 ceph-mon[100896]: pgmap v132: 129 pgs: 129 active+clean; 512 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 518 op/s 2023-10-10T17:46:51.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:50 smithi078 ceph-mon[99494]: pgmap v132: 129 pgs: 129 active+clean; 512 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 518 op/s 2023-10-10T17:46:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:50 smithi190 ceph-mon[101220]: pgmap v132: 129 pgs: 129 active+clean; 512 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.7 MiB/s wr, 518 op/s 2023-10-10T17:46:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:52 smithi157 ceph-mon[100896]: pgmap v133: 129 pgs: 129 active+clean; 523 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 543 op/s 2023-10-10T17:46:53.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:52 smithi078 ceph-mon[99494]: pgmap v133: 129 pgs: 129 active+clean; 523 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 543 op/s 2023-10-10T17:46:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:52 smithi190 ceph-mon[101220]: pgmap v133: 129 pgs: 129 active+clean; 523 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.6 MiB/s wr, 543 op/s 2023-10-10T17:46:54.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:53 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:46:54.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:53 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:54.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:53 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:46:54.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:53 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:54.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:53 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:46:54.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:53 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:55.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:54 smithi078 ceph-mon[99494]: pgmap v134: 129 pgs: 129 active+clean; 539 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.3 MiB/s wr, 703 op/s 2023-10-10T17:46:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:54 smithi190 ceph-mon[101220]: pgmap v134: 129 pgs: 129 active+clean; 539 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.3 MiB/s wr, 703 op/s 2023-10-10T17:46:55.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:54 smithi157 ceph-mon[100896]: pgmap v134: 129 pgs: 129 active+clean; 539 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 6.3 MiB/s wr, 703 op/s 2023-10-10T17:46:56.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:55 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:46:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:55 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:46:56.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:55 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:46:57.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:56 smithi078 ceph-mon[99494]: pgmap v135: 129 pgs: 129 active+clean; 539 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.7 MiB/s wr, 533 op/s 2023-10-10T17:46:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:56 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:46:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:56 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:46:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:56 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:46:57.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:56 smithi190 ceph-mon[101220]: pgmap v135: 129 pgs: 129 active+clean; 539 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.7 MiB/s wr, 533 op/s 2023-10-10T17:46:57.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:56 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:46:57.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:56 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:46:57.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:56 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:46:57.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:56 smithi157 ceph-mon[100896]: pgmap v135: 129 pgs: 129 active+clean; 539 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.7 MiB/s wr, 533 op/s 2023-10-10T17:46:57.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:56 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:46:57.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:56 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:46:57.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:56 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:46:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:57 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:46:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:57 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:46:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:57 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:46:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:57 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:46:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:57 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:46:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:57 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:46:58.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:57 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:46:58.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:57 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:46:58.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:57 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:46:58.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:57 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:46:58.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:57 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:46:58.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:57 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:46:58.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.429+0000 7fa80ad0f700 1 -- 172.21.15.78:0/1389082040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8040ff230 msgr2=0x7fa8040ff600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.429+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/1389082040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8040ff230 0x7fa8040ff600 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80099c0 tx=0x7fa7f8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.430+0000 7fa80ad0f700 1 -- 172.21.15.78:0/1389082040 shutdown_connections 2023-10-10T17:46:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.430+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/1389082040 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8040ff230 0x7fa8040ff600 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.430+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/1389082040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 0x7fa804104ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.431 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.430+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/1389082040 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8040ffb40 0x7fa804104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.430+0000 7fa80ad0f700 1 -- 172.21.15.78:0/1389082040 >> 172.21.15.78:0/1389082040 conn(0x7fa8040763a0 msgr2=0x7fa8040767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.430+0000 7fa80ad0f700 1 -- 172.21.15.78:0/1389082040 shutdown_connections 2023-10-10T17:46:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.430+0000 7fa80ad0f700 1 -- 172.21.15.78:0/1389082040 wait complete. 2023-10-10T17:46:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 Processor -- start 2023-10-10T17:46:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 -- start start 2023-10-10T17:46:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8040ff230 0x7fa8041a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8040ffb40 0x7fa8041a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 0x7fa80419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80419ced0 con 0x7fa804104630 2023-10-10T17:46:58.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80419d040 con 0x7fa8040ff230 2023-10-10T17:46:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.431+0000 7fa80ad0f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80419d200 con 0x7fa8040ffb40 2023-10-10T17:46:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa8092ac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 0x7fa80419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa808aab700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8040ff230 0x7fa8041a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa8092ac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 0x7fa80419c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46612/0 (socket says 172.21.15.78:46612) 2023-10-10T17:46:58.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa8092ac700 1 -- 172.21.15.78:0/487505392 learned_addr learned my addr 172.21.15.78:0/487505392 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa808aab700 1 -- 172.21.15.78:0/487505392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8040ffb40 msgr2=0x7fa8041a2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa808aab700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8040ffb40 0x7fa8041a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa808aab700 1 -- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 msgr2=0x7fa80419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa808aab700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 0x7fa80419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa808aab700 1 -- 172.21.15.78:0/487505392 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f80095f0 con 0x7fa8040ff230 2023-10-10T17:46:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.432+0000 7fa8092ac700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 0x7fa80419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.436+0000 7fa808aab700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8040ff230 0x7fa8041a27f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8000c00 tx=0x7fa7f8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.438+0000 7fa801ffb700 1 -- 172.21.15.78:0/487505392 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f801d070 con 0x7fa8040ff230 2023-10-10T17:46:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.438+0000 7fa801ffb700 1 -- 172.21.15.78:0/487505392 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7f8003f90 con 0x7fa8040ff230 2023-10-10T17:46:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.438+0000 7fa801ffb700 1 -- 172.21.15.78:0/487505392 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f800fc30 con 0x7fa8040ff230 2023-10-10T17:46:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.438+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8041169b0 con 0x7fa8040ff230 2023-10-10T17:46:58.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.438+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa804116820 con 0x7fa8040ff230 2023-10-10T17:46:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.443+0000 7fa801ffb700 1 -- 172.21.15.78:0/487505392 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa7f800f460 con 0x7fa8040ff230 2023-10-10T17:46:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.444+0000 7fa801ffb700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa7ec0a14d0 0x7fa7ec0a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.444+0000 7fa801ffb700 1 -- 172.21.15.78:0/487505392 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa7f80cdd10 con 0x7fa8040ff230 2023-10-10T17:46:58.445 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.444+0000 7fa803fff700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa7ec0a14d0 0x7fa7ec0a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 Processor -- start 2023-10-10T17:46:58.446 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 -- start start 2023-10-10T17:46:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477ad560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.447 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3b90 0x55ff477adb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff476b3000 con 0x55ff477b3720 2023-10-10T17:46:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff47705840 con 0x55ff477b3b90 2023-10-10T17:46:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff476fecd0 con 0x55ff476096b0 2023-10-10T17:46:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa808aab700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.448 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa808aab700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38146/0 (socket says 172.21.15.78:38146) 2023-10-10T17:46:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa808aab700 1 -- 172.21.15.78:0/4163977026 learned_addr learned my addr 172.21.15.78:0/4163977026 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.445+0000 7fa803fff700 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477ad560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa808aab700 1 -- 172.21.15.78:0/4163977026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3b90 msgr2=0x55ff477adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa808aab700 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3b90 0x55ff477adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa808aab700 1 -- 172.21.15.78:0/4163977026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 msgr2=0x55ff477ad560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:58.449 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa808aab700 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477ad560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa808aab700 1 -- 172.21.15.78:0/4163977026 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff475805d0 con 0x55ff476096b0 2023-10-10T17:46:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa803fff700 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:46:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa808aab700 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b31e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8097000 tx=0x7fa7f809a9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b25af03397368d86 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:58.450 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.446+0000 7fa803fff700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa7ec0a14d0 0x7fa7ec0a3980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0060b0 tx=0x7fa7fc006040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:58.451 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.447+0000 7fa7eaffd700 1 -- 172.21.15.78:0/4163977026 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f809ab00 con 0x55ff476096b0 2023-10-10T17:46:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.447+0000 7fa7eaffd700 1 -- 172.21.15.78:0/4163977026 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7f809eb30 con 0x55ff476096b0 2023-10-10T17:46:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.447+0000 7fa7eaffd700 1 -- 172.21.15.78:0/4163977026 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f809a1f0 con 0x55ff476096b0 2023-10-10T17:46:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.448+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4163977026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 msgr2=0x55ff477b31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:58.452 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.448+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b31e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8097000 tx=0x7fa7f809a9e0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.449+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4163977026 shutdown_connections 2023-10-10T17:46:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.449+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b31e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.449+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3b90 0x55ff477adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.449+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4163977026 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477ad560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.449+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4163977026 >> 172.21.15.78:0/4163977026 conn(0x55ff477a2f50 msgr2=0x55ff477a8270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:58.453 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.449+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4163977026 shutdown_connections 2023-10-10T17:46:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.449+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4163977026 wait complete. 2023-10-10T17:46:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 -- start start 2023-10-10T17:46:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 Processor -- start 2023-10-10T17:46:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff476096b0 0x55ff477bc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.454 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477bcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff477b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff476a0880 con 0x55ff477b3720 2023-10-10T17:46:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff476a0520 con 0x55ff476096b0 2023-10-10T17:46:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff475828c0 con 0x55ff477b3b90 2023-10-10T17:46:58.455 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.450+0000 7fa816e9bb80 10 client.0 _ll_register_callbacks cb 0x55ff476be720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff477b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff477b6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38152/0 (socket says 172.21.15.78:38152) 2023-10-10T17:46:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 -- 172.21.15.78:0/1995701047 learned_addr learned my addr 172.21.15.78:0/1995701047 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 -- 172.21.15.78:0/1995701047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff476096b0 msgr2=0x55ff477bc7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T17:46:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff476096b0 0x55ff477bc7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 -- 172.21.15.78:0/1995701047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 msgr2=0x55ff477bcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477bcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.451+0000 7fa8092ac700 1 -- 172.21.15.78:0/1995701047 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa804116820 con 0x55ff477b3b90 2023-10-10T17:46:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.453+0000 7fa8092ac700 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff477b6a50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400e750 tx=0x7fa7f400eb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ba0f9099bf9e1f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.453+0000 7fa7e3fff700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:46:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.453+0000 7fa7e3fff700 1 -- 172.21.15.78:0/1995701047 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4013680 con 0x55ff477b3b90 2023-10-10T17:46:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.453+0000 7fa7e3fff700 1 -- 172.21.15.78:0/1995701047 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa7f40137e0 con 0x55ff477b3b90 2023-10-10T17:46:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.453+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ff477a0710 con 0x55ff477b3b90 2023-10-10T17:46:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.455+0000 7fa7e3fff700 1 -- 172.21.15.78:0/1995701047 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7f40139a0 con 0x55ff477b3b90 2023-10-10T17:46:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.455+0000 7fa816e9bb80 10 client.25957 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.455+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ff476c5130 con 0x55ff477b3b90 2023-10-10T17:46:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa7e3fff700 1 -- 172.21.15.78:0/1995701047 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ff477a0710 con 0x55ff477b3b90 2023-10-10T17:46:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa7e3fff700 1 -- 172.21.15.78:0/1995701047 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa7f4011070 con 0x55ff477b3b90 2023-10-10T17:46:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa816e9bb80 10 client.25957 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa816e9bb80 10 client.25957 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa816e9bb80 20 client.25957 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff477d7e60 0x55ff477da270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa816e9bb80 4 client.25957 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:46:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff476c1040 con 0x55ff477d7e60 2023-10-10T17:46:58.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.456+0000 7fa808aab700 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff477d7e60 0x55ff477da270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.457+0000 7fa7e3fff700 10 client.25957 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.457+0000 7fa808aab700 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff477d7e60 0x55ff477da270 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96eb8c0d28c3cac1 server_cookie=58dbf0f53be2a936 in_seq=0 out_seq=0 2023-10-10T17:46:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.458+0000 7fa7e3fff700 1 -- 172.21.15.78:0/1995701047 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ff476c1040 con 0x55ff477d7e60 2023-10-10T17:46:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.458+0000 7fa7e3fff700 10 client.25957 handle_command_reply: tid=0 2023-10-10T17:46:58.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.458+0000 7fa816e9bb80 1 client.25957 shutdown 2023-10-10T17:46:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 msgr2=0x55ff477b6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff477b6a50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400e750 tx=0x7fa7f400eb10 comp rx=0 tx=0).stop 2023-10-10T17:46:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 shutdown_connections 2023-10-10T17:46:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff477b6a50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff477d7e60 0x55ff477da270 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff476096b0 0x55ff477bc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/1995701047 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3720 0x55ff477bcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa7e3fff700 0 client.25957 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 >> 172.21.15.78:0/1995701047 conn(0x55ff477a2f50 msgr2=0x55ff477a77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 shutdown_connections 2023-10-10T17:46:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 1 -- 172.21.15.78:0/1995701047 wait complete. 2023-10-10T17:46:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.459+0000 7fa816e9bb80 20 client.25957 trim_cache size 0 max 16384 2023-10-10T17:46:58.485 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 Processor -- start 2023-10-10T17:46:58.490 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 -- start start 2023-10-10T17:46:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477a7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3b90 0x55ff477b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.491 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff477b4fb0 con 0x55ff477b3b90 2023-10-10T17:46:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff477b1720 con 0x55ff477b3720 2023-10-10T17:46:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.485+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff477d4610 con 0x55ff476096b0 2023-10-10T17:46:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.486+0000 7fa803fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477a7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.486+0000 7fa803fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477a7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48352/0 (socket says 172.21.15.78:48352) 2023-10-10T17:46:58.492 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.486+0000 7fa803fff700 1 -- 172.21.15.78:0/717609356 learned_addr learned my addr 172.21.15.78:0/717609356 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.487+0000 7fa803fff700 1 -- 172.21.15.78:0/717609356 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 msgr2=0x55ff477b0e60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T17:46:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.487+0000 7fa803fff700 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b0e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.487+0000 7fa803fff700 1 -- 172.21.15.78:0/717609356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3b90 msgr2=0x55ff477b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.490+0000 7fa8092ac700 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3b90 0x55ff477b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.490+0000 7fa803fff700 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3b90 0x55ff477b3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.493 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.490+0000 7fa803fff700 1 -- 172.21.15.78:0/717609356 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff476c4e80 con 0x55ff477b3720 2023-10-10T17:46:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.492+0000 7fa803fff700 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477a7f60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc007f00 tx=0x7fa7fc00b050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b7bd20ece48c32e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.492+0000 7fa7eb7fe700 1 -- 172.21.15.78:0/717609356 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc022400 con 0x55ff477b3720 2023-10-10T17:46:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.492+0000 7fa7eb7fe700 1 -- 172.21.15.78:0/717609356 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7fc00f040 con 0x55ff477b3720 2023-10-10T17:46:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.492+0000 7fa7eb7fe700 1 -- 172.21.15.78:0/717609356 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc021900 con 0x55ff477b3720 2023-10-10T17:46:58.495 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.493+0000 7fa816e9bb80 1 -- 172.21.15.78:0/717609356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 msgr2=0x55ff477a7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.493+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477a7f60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc007f00 tx=0x7fa7fc00b050 comp rx=0 tx=0).stop 2023-10-10T17:46:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.493+0000 7fa816e9bb80 1 -- 172.21.15.78:0/717609356 shutdown_connections 2023-10-10T17:46:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff476096b0 0x55ff477b0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff477b3b90 0x55ff477b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.496 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/717609356 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477a7f60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 -- 172.21.15.78:0/717609356 >> 172.21.15.78:0/717609356 conn(0x55ff477a2f50 msgr2=0x55ff477b7380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 -- 172.21.15.78:0/717609356 shutdown_connections 2023-10-10T17:46:58.497 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 -- 172.21.15.78:0/717609356 wait complete. 2023-10-10T17:46:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 -- start start 2023-10-10T17:46:58.498 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.494+0000 7fa816e9bb80 1 Processor -- start 2023-10-10T17:46:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff476096b0 0x55ff477ba550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477baa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa816e9bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff4782ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.499 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff477bb330 con 0x55ff476096b0 2023-10-10T17:46:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff4782b260 con 0x55ff477b3720 2023-10-10T17:46:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa816e9bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff4782b3a0 con 0x55ff477b3b90 2023-10-10T17:46:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa816e9bb80 10 client.0 _ll_register_callbacks cb 0x55ff476be720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:46:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa8092ac700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff4782ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa8092ac700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff4782ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38168/0 (socket says 172.21.15.78:38168) 2023-10-10T17:46:58.500 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa8092ac700 1 -- 172.21.15.78:0/4021214014 learned_addr learned my addr 172.21.15.78:0/4021214014 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:46:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.495+0000 7fa803fff700 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477baa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.496+0000 7fa8092ac700 1 -- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 msgr2=0x55ff477baa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.496+0000 7fa8092ac700 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477baa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.496+0000 7fa8092ac700 1 -- 172.21.15.78:0/4021214014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff476096b0 msgr2=0x55ff477ba550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:46:58.501 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.496+0000 7fa8092ac700 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff476096b0 0x55ff477ba550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.496+0000 7fa8092ac700 1 -- 172.21.15.78:0/4021214014 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7fc007c70 con 0x55ff477b3b90 2023-10-10T17:46:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.496+0000 7fa8092ac700 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff4782ad20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4000c00 tx=0x7fa7f400f140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79340c162b3cc693 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:46:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa7e3fff700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:46:58.502 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa7e3fff700 1 -- 172.21.15.78:0/4021214014 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4005e30 con 0x55ff477b3b90 2023-10-10T17:46:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa7e3fff700 1 -- 172.21.15.78:0/4021214014 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa7f40123d0 con 0x55ff477b3b90 2023-10-10T17:46:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7f809eb30 con 0x55ff477b3b90 2023-10-10T17:46:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa7e3fff700 1 -- 172.21.15.78:0/4021214014 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7f4012590 con 0x55ff477b3b90 2023-10-10T17:46:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa816e9bb80 10 client.25960 fetch_fsmap learned FSMap version 32 2023-10-10T17:46:58.503 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ff47515c00 con 0x55ff477b3b90 2023-10-10T17:46:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa7e3fff700 1 -- 172.21.15.78:0/4021214014 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7f809eb30 con 0x55ff477b3b90 2023-10-10T17:46:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.497+0000 7fa7e3fff700 1 -- 172.21.15.78:0/4021214014 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa7f4013ca0 con 0x55ff477b3b90 2023-10-10T17:46:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.498+0000 7fa816e9bb80 10 client.25960 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:46:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.498+0000 7fa816e9bb80 10 client.25960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:46:58.504 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.498+0000 7fa816e9bb80 20 client.25960 populate_metadata read hostname 'smithi078' 2023-10-10T17:46:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.498+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff4782ca80 0x55ff4782ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:46:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.498+0000 7fa816e9bb80 4 client.25960 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:46:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.498+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ff476cd600 con 0x55ff4782ca80 2023-10-10T17:46:58.505 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.501+0000 7fa808aab700 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff4782ca80 0x55ff4782ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:46:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.503+0000 7fa7e3fff700 10 client.25960 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:58.506 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:58.503+0000 7fa808aab700 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff4782ca80 0x55ff4782ee90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1f55ffe61107355 server_cookie=ec6682ef4aa5eab5 in_seq=0 out_seq=0 2023-10-10T17:46:59.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:58 smithi078 ceph-mon[99494]: pgmap v136: 129 pgs: 129 active+clean; 571 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.3 MiB/s wr, 762 op/s 2023-10-10T17:46:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:58 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:46:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:46:58 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:58 smithi190 ceph-mon[101220]: pgmap v136: 129 pgs: 129 active+clean; 571 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.3 MiB/s wr, 762 op/s 2023-10-10T17:46:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:58 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:46:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:46:58 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:59.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:58 smithi157 ceph-mon[100896]: pgmap v136: 129 pgs: 129 active+clean; 571 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.3 MiB/s wr, 762 op/s 2023-10-10T17:46:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:58 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:46:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:46:58 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:46:59.814 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa7e3fff700 1 -- 172.21.15.78:0/4021214014 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ff476cd600 con 0x55ff4782ca80 2023-10-10T17:46:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa7e3fff700 10 client.25960 handle_command_reply: tid=0 2023-10-10T17:46:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 client.25960 shutdown 2023-10-10T17:46:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 msgr2=0x55ff4782ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff4782ad20 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4000c00 tx=0x7fa7f400f140 comp rx=0 tx=0).stop 2023-10-10T17:46:59.815 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 shutdown_connections 2023-10-10T17:46:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ff477b3b90 0x55ff4782ad20 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ff4782ca80 0x55ff4782ee90 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ff476096b0 0x55ff477ba550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 --2- 172.21.15.78:0/4021214014 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ff477b3720 0x55ff477baa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.816 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 >> 172.21.15.78:0/4021214014 conn(0x55ff477a2f50 msgr2=0x55ff477b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:59.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa7e3fff700 0 client.25960 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:46:59.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 shutdown_connections 2023-10-10T17:46:59.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.813+0000 7fa816e9bb80 1 -- 172.21.15.78:0/4021214014 wait complete. 2023-10-10T17:46:59.817 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa816e9bb80 20 client.25960 trim_cache size 0 max 16384 2023-10-10T17:46:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:46:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:46:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:46:59.818 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:46:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa7ec0a14d0 msgr2=0x7fa7ec0a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa7ec0a14d0 0x7fa7ec0a3980 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0060b0 tx=0x7fa7fc006040 comp rx=0 tx=0).stop 2023-10-10T17:46:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8040ff230 msgr2=0x7fa8041a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:46:59.820 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8040ff230 0x7fa8041a27f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8000c00 tx=0x7fa7f8004730 comp rx=0 tx=0).stop 2023-10-10T17:46:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 shutdown_connections 2023-10-10T17:46:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa7ec0a14d0 0x7fa7ec0a3980 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa804104630 0x7fa80419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8040ff230 0x7fa8041a27f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.821 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.814+0000 7fa80ad0f700 1 --2- 172.21.15.78:0/487505392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8040ffb40 0x7fa8041a2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:46:59.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.815+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 >> 172.21.15.78:0/487505392 conn(0x7fa8040763a0 msgr2=0x7fa804101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:46:59.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.815+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 shutdown_connections 2023-10-10T17:46:59.822 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:46:59.815+0000 7fa80ad0f700 1 -- 172.21.15.78:0/487505392 wait complete. 2023-10-10T17:47:01.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:00 smithi078 ceph-mon[99494]: pgmap v137: 129 pgs: 129 active+clean; 571 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.9 MiB/s wr, 435 op/s 2023-10-10T17:47:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:00 smithi190 ceph-mon[101220]: pgmap v137: 129 pgs: 129 active+clean; 571 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.9 MiB/s wr, 435 op/s 2023-10-10T17:47:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:00 smithi157 ceph-mon[100896]: pgmap v137: 129 pgs: 129 active+clean; 571 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.9 MiB/s wr, 435 op/s 2023-10-10T17:47:03.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:02 smithi078 ceph-mon[99494]: pgmap v138: 129 pgs: 129 active+clean; 578 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 477 op/s 2023-10-10T17:47:03.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:02 smithi190 ceph-mon[101220]: pgmap v138: 129 pgs: 129 active+clean; 578 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 477 op/s 2023-10-10T17:47:03.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:02 smithi157 ceph-mon[100896]: pgmap v138: 129 pgs: 129 active+clean; 578 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 477 op/s 2023-10-10T17:47:05.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:04 smithi078 ceph-mon[99494]: pgmap v139: 129 pgs: 129 active+clean; 579 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 696 op/s 2023-10-10T17:47:05.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:04 smithi190 ceph-mon[101220]: pgmap v139: 129 pgs: 129 active+clean; 579 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 696 op/s 2023-10-10T17:47:05.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:04 smithi157 ceph-mon[100896]: pgmap v139: 129 pgs: 129 active+clean; 579 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 6.1 MiB/s wr, 696 op/s 2023-10-10T17:47:07.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:06 smithi078 ceph-mon[99494]: pgmap v140: 129 pgs: 129 active+clean; 579 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 515 op/s 2023-10-10T17:47:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:06 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:06 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:06 smithi190 ceph-mon[101220]: pgmap v140: 129 pgs: 129 active+clean; 579 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 515 op/s 2023-10-10T17:47:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:06 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:07.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:06 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:07.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:06 smithi157 ceph-mon[100896]: pgmap v140: 129 pgs: 129 active+clean; 579 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 515 op/s 2023-10-10T17:47:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:06 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:06 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:09.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:08 smithi078 ceph-mon[99494]: pgmap v141: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.2 MiB/s wr, 741 op/s 2023-10-10T17:47:09.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:08 smithi190 ceph-mon[101220]: pgmap v141: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.2 MiB/s wr, 741 op/s 2023-10-10T17:47:09.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:08 smithi157 ceph-mon[100896]: pgmap v141: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.2 MiB/s wr, 741 op/s 2023-10-10T17:47:11.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:10 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:11.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:10 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:11.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:10 smithi078 ceph-mon[99494]: pgmap v142: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.5 MiB/s wr, 512 op/s 2023-10-10T17:47:11.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:10 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:11.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:10 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:11.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:10 smithi190 ceph-mon[101220]: pgmap v142: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.5 MiB/s wr, 512 op/s 2023-10-10T17:47:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:10 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:11.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:10 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:11.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:10 smithi157 ceph-mon[100896]: pgmap v142: 129 pgs: 129 active+clean; 607 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.5 MiB/s wr, 512 op/s 2023-10-10T17:47:12.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 -- 172.21.15.78:0/1165769242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 msgr2=0x7f557011bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:12.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 --2- 172.21.15.78:0/1165769242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f557011bd20 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f556800b0a0 tx=0x7f556800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 -- 172.21.15.78:0/1165769242 shutdown_connections 2023-10-10T17:47:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 --2- 172.21.15.78:0/1165769242 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f557011bd20 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 --2- 172.21.15.78:0/1165769242 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5570081420 0x7f5570079340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 --2- 172.21.15.78:0/1165769242 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5570079880 0x7f5570079cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 -- 172.21.15.78:0/1165769242 >> 172.21.15.78:0/1165769242 conn(0x7f5570074840 msgr2=0x7f5570076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 -- 172.21.15.78:0/1165769242 shutdown_connections 2023-10-10T17:47:12.976 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.973+0000 7f5574bfc700 1 -- 172.21.15.78:0/1165769242 wait complete. 2023-10-10T17:47:12.977 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 Processor -- start 2023-10-10T17:47:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 -- start start 2023-10-10T17:47:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5570079880 0x7f55701a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5570081420 0x7f55701a5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f55701a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.978 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55701aa2d0 con 0x7f5570119940 2023-10-10T17:47:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55701aa440 con 0x7f5570081420 2023-10-10T17:47:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.974+0000 7f5574bfc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55701aa600 con 0x7f5570079880 2023-10-10T17:47:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f55701a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f5567fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5570081420 0x7f55701a5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.979 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f55701a9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46654/0 (socket says 172.21.15.78:46654) 2023-10-10T17:47:12.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 -- 172.21.15.78:0/2566799196 learned_addr learned my addr 172.21.15.78:0/2566799196 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:12.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 -- 172.21.15.78:0/2566799196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5570079880 msgr2=0x7f55701a52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:12.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5570079880 0x7f55701a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.980 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 -- 172.21.15.78:0/2566799196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5570081420 msgr2=0x7f55701a5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5570081420 0x7f55701a5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 -- 172.21.15.78:0/2566799196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5568009d00 con 0x7f5570119940 2023-10-10T17:47:12.981 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f556ed9d700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f55701a9c90 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f556800b780 tx=0x7f556800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f55677fe700 1 -- 172.21.15.78:0/2566799196 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556800e040 con 0x7f5570119940 2023-10-10T17:47:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f55677fe700 1 -- 172.21.15.78:0/2566799196 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5568004e60 con 0x7f5570119940 2023-10-10T17:47:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.975+0000 7f55677fe700 1 -- 172.21.15.78:0/2566799196 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680052a0 con 0x7f5570119940 2023-10-10T17:47:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.976+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557011f110 con 0x7f5570119940 2023-10-10T17:47:12.982 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.976+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557011ef80 con 0x7f5570119940 2023-10-10T17:47:12.983 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.978+0000 7f55677fe700 1 -- 172.21.15.78:0/2566799196 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5568005400 con 0x7f5570119940 2023-10-10T17:47:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.979+0000 7f55677fe700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f555c0a1340 0x7f555c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.979+0000 7f556e59c700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f555c0a1340 0x7f555c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.979+0000 7f55677fe700 1 -- 172.21.15.78:0/2566799196 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f55680ce500 con 0x7f5570119940 2023-10-10T17:47:12.984 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.979+0000 7f556e59c700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f555c0a1340 0x7f555c0a37f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5560005950 tx=0x7f55600058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 Processor -- start 2023-10-10T17:47:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 -- start start 2023-10-10T17:47:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee5696b0 0x556bee7131e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee713720 0x556bee70d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.985 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713b90 0x556bee70db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee613000 con 0x556bee5696b0 2023-10-10T17:47:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee665840 con 0x556bee713b90 2023-10-10T17:47:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f5580d88b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee65ecd0 con 0x556bee713720 2023-10-10T17:47:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.980+0000 7f556e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee5696b0 0x556bee7131e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.986 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee5696b0 0x556bee7131e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46658/0 (socket says 172.21.15.78:46658) 2023-10-10T17:47:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556e59c700 1 -- 172.21.15.78:0/1464681031 learned_addr learned my addr 172.21.15.78:0/1464681031 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556ed9d700 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713b90 0x556bee70db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f5567fff700 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee713720 0x556bee70d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556ed9d700 1 -- 172.21.15.78:0/1464681031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee713720 msgr2=0x556bee70d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:12.987 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556ed9d700 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee713720 0x556bee70d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556ed9d700 1 -- 172.21.15.78:0/1464681031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee5696b0 msgr2=0x556bee7131e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556ed9d700 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee5696b0 0x556bee7131e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556ed9d700 1 -- 172.21.15.78:0/1464681031 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bee4e05d0 con 0x556bee713b90 2023-10-10T17:47:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556e59c700 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee5696b0 0x556bee7131e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:47:12.988 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f556ed9d700 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713b90 0x556bee70db00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55680cb920 tx=0x7f55680099e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=331eccaf89a989c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f5564ff9700 1 -- 172.21.15.78:0/1464681031 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5568011e60 con 0x556bee713b90 2023-10-10T17:47:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f5564ff9700 1 -- 172.21.15.78:0/1464681031 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f556809b330 con 0x556bee713b90 2023-10-10T17:47:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.981+0000 7f5564ff9700 1 -- 172.21.15.78:0/1464681031 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680ce170 con 0x556bee713b90 2023-10-10T17:47:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.982+0000 7f5580d88b80 1 -- 172.21.15.78:0/1464681031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713b90 msgr2=0x556bee70db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:12.989 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.982+0000 7f5580d88b80 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713b90 0x556bee70db00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55680cb920 tx=0x7f55680099e0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.983+0000 7f5580d88b80 1 -- 172.21.15.78:0/1464681031 shutdown_connections 2023-10-10T17:47:12.992 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.983+0000 7f5580d88b80 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713b90 0x556bee70db00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.983+0000 7f5580d88b80 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee5696b0 0x556bee7131e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.983+0000 7f5580d88b80 1 --2- 172.21.15.78:0/1464681031 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee713720 0x556bee70d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.983+0000 7f5580d88b80 1 -- 172.21.15.78:0/1464681031 >> 172.21.15.78:0/1464681031 conn(0x556bee702f50 msgr2=0x556bee708270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.984+0000 7f5580d88b80 1 -- 172.21.15.78:0/1464681031 shutdown_connections 2023-10-10T17:47:12.993 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.984+0000 7f5580d88b80 1 -- 172.21.15.78:0/1464681031 wait complete. 2023-10-10T17:47:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.984+0000 7f5580d88b80 1 -- start start 2023-10-10T17:47:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.984+0000 7f5580d88b80 1 Processor -- start 2023-10-10T17:47:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee719910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.994 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713720 0x556bee716b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713b90 0x556bee7170d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5580d88b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee600880 con 0x556bee713b90 2023-10-10T17:47:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5580d88b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee600520 con 0x556bee713720 2023-10-10T17:47:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5580d88b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee4e28c0 con 0x556bee5696b0 2023-10-10T17:47:12.995 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5580d88b80 10 client.0 _ll_register_callbacks cb 0x556bee61e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713b90 0x556bee7170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713b90 0x556bee7170d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46666/0 (socket says 172.21.15.78:46666) 2023-10-10T17:47:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 -- 172.21.15.78:0/2163865985 learned_addr learned my addr 172.21.15.78:0/2163865985 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556e59c700 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee719910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.996 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f5567fff700 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713720 0x556bee716b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 -- 172.21.15.78:0/2163865985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 msgr2=0x556bee719910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee719910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 -- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713720 msgr2=0x556bee716b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713720 0x556bee716b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:12.997 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.985+0000 7f556ed9d700 1 -- 172.21.15.78:0/2163865985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557011ef80 con 0x556bee713b90 2023-10-10T17:47:12.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.986+0000 7f556ed9d700 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713b90 0x556bee7170d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5568009fd0 tx=0x7f55680a1340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e10dc12517d7557 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:12.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.986+0000 7f5555ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:47:12.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.986+0000 7f5555ffb700 1 -- 172.21.15.78:0/2163865985 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680ce380 con 0x556bee713b90 2023-10-10T17:47:12.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.986+0000 7f5555ffb700 1 -- 172.21.15.78:0/2163865985 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f556809f130 con 0x556bee713b90 2023-10-10T17:47:12.998 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.986+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556bee700710 con 0x556bee713b90 2023-10-10T17:47:12.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.987+0000 7f5555ffb700 1 -- 172.21.15.78:0/2163865985 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f556809f360 con 0x556bee713b90 2023-10-10T17:47:12.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.988+0000 7f5580d88b80 10 client.16512 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:12.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.988+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x556bee625130 con 0x556bee713b90 2023-10-10T17:47:12.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5555ffb700 1 -- 172.21.15.78:0/2163865985 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556bee700710 con 0x556bee713b90 2023-10-10T17:47:12.999 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5555ffb700 1 -- 172.21.15.78:0/2163865985 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f55680a3260 con 0x556bee713b90 2023-10-10T17:47:13.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5580d88b80 10 client.16512 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:13.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5580d88b80 10 client.16512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:13.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5580d88b80 20 client.16512 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:13.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5580d88b80 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee7378e0 0x556bee739cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5580d88b80 4 client.16512 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:47:13.000 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.990+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556bee621040 con 0x556bee7378e0 2023-10-10T17:47:13.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.991+0000 7f556e59c700 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee7378e0 0x556bee739cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.992+0000 7f556e59c700 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee7378e0 0x556bee739cc0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73c5271eda03eaf5 server_cookie=8248e33ef55cfcbf in_seq=0 out_seq=0 2023-10-10T17:47:13.001 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.992+0000 7f5555ffb700 10 client.16512 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:13.002 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.996+0000 7f5555ffb700 1 -- 172.21.15.78:0/2163865985 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556bee621040 con 0x556bee7378e0 2023-10-10T17:47:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.996+0000 7f5555ffb700 10 client.16512 handle_command_reply: tid=0 2023-10-10T17:47:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.996+0000 7f5580d88b80 1 client.16512 shutdown 2023-10-10T17:47:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713b90 msgr2=0x556bee7170d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713b90 0x556bee7170d0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f5568009fd0 tx=0x7f55680a1340 comp rx=0 tx=0).stop 2023-10-10T17:47:13.003 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 shutdown_connections 2023-10-10T17:47:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee719910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee7378e0 0x556bee739cc0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713b90 0x556bee7170d0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5555ffb700 0 client.16512 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:13.004 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 --2- 172.21.15.78:0/2163865985 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee713720 0x556bee716b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 >> 172.21.15.78:0/2163865985 conn(0x556bee702f50 msgr2=0x556bee7077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 shutdown_connections 2023-10-10T17:47:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 1 -- 172.21.15.78:0/2163865985 wait complete. 2023-10-10T17:47:13.005 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:12.997+0000 7f5580d88b80 20 client.16512 trim_cache size 0 max 16384 2023-10-10T17:47:13.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 Processor -- start 2023-10-10T17:47:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 -- start start 2023-10-10T17:47:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee702f50 0x556bee72f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee716350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee718d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee700600 con 0x556bee713720 2023-10-10T17:47:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee704fd0 con 0x556bee702f50 2023-10-10T17:47:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5580d88b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee708e40 con 0x556bee5696b0 2023-10-10T17:47:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f556ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee718d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f556ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee718d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46674/0 (socket says 172.21.15.78:46674) 2023-10-10T17:47:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f556e59c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee702f50 0x556bee72f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f556ed9d700 1 -- 172.21.15.78:0/492292889 learned_addr learned my addr 172.21.15.78:0/492292889 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.019+0000 7f5567fff700 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee716350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f556ed9d700 1 -- 172.21.15.78:0/492292889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 msgr2=0x556bee716350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f556ed9d700 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee716350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f556ed9d700 1 -- 172.21.15.78:0/492292889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee702f50 msgr2=0x556bee72f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f556ed9d700 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee702f50 0x556bee72f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f556ed9d700 1 -- 172.21.15.78:0/492292889 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bee624e80 con 0x556bee713720 2023-10-10T17:47:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f556ed9d700 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee718d30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f5568099680 tx=0x7f55680099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9b692781bd007b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:13.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f55657fa700 1 -- 172.21.15.78:0/492292889 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680ce380 con 0x556bee713720 2023-10-10T17:47:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f55657fa700 1 -- 172.21.15.78:0/492292889 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f556809f130 con 0x556bee713720 2023-10-10T17:47:13.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.020+0000 7f55657fa700 1 -- 172.21.15.78:0/492292889 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55680a3260 con 0x556bee713720 2023-10-10T17:47:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 -- 172.21.15.78:0/492292889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 msgr2=0x556bee718d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee718d30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f5568099680 tx=0x7f55680099e0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 -- 172.21.15.78:0/492292889 shutdown_connections 2023-10-10T17:47:13.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee718d30 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee702f50 0x556bee72f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 --2- 172.21.15.78:0/492292889 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee5696b0 0x556bee716350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 -- 172.21.15.78:0/492292889 >> 172.21.15.78:0/492292889 conn(0x556bee716eb0 msgr2=0x556bee7145e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:13.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 -- 172.21.15.78:0/492292889 shutdown_connections 2023-10-10T17:47:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 -- 172.21.15.78:0/492292889 wait complete. 2023-10-10T17:47:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 -- start start 2023-10-10T17:47:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.021+0000 7f5580d88b80 1 Processor -- start 2023-10-10T17:47:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee5696b0 0x556bee72f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee702f50 0x556bee70ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f5580d88b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee70f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f5580d88b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee70f880 con 0x556bee713720 2023-10-10T17:47:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f5580d88b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee70f9f0 con 0x556bee5696b0 2023-10-10T17:47:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f5580d88b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bee70fbb0 con 0x556bee702f50 2023-10-10T17:47:13.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f5580d88b80 10 client.0 _ll_register_callbacks cb 0x556bee61e720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee5696b0 0x556bee72f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee70f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee5696b0 0x556bee72f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48398/0 (socket says 172.21.15.78:48398) 2023-10-10T17:47:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 -- 172.21.15.78:0/3212545531 learned_addr learned my addr 172.21.15.78:0/3212545531 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:13.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 -- 172.21.15.78:0/3212545531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee702f50 msgr2=0x556bee70ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee702f50 0x556bee70ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 -- 172.21.15.78:0/3212545531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 msgr2=0x556bee70f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee70f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.022+0000 7f556e59c700 1 -- 172.21.15.78:0/3212545531 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bee625130 con 0x556bee5696b0 2023-10-10T17:47:13.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.023+0000 7f556e59c700 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee5696b0 0x556bee72f0b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f556000e950 tx=0x7f5560004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3394d91a144a596 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.023+0000 7f5555ffb700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:47:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.024+0000 7f5555ffb700 1 -- 172.21.15.78:0/3212545531 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560022040 con 0x556bee5696b0 2023-10-10T17:47:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.024+0000 7f5555ffb700 1 -- 172.21.15.78:0/3212545531 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5560027b50 con 0x556bee5696b0 2023-10-10T17:47:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.024+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f556809b330 con 0x556bee5696b0 2023-10-10T17:47:13.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.024+0000 7f5555ffb700 1 -- 172.21.15.78:0/3212545531 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5560027d10 con 0x556bee5696b0 2023-10-10T17:47:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.024+0000 7f5580d88b80 10 client.25886 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.024+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x556bee475c00 con 0x556bee5696b0 2023-10-10T17:47:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5555ffb700 1 -- 172.21.15.78:0/3212545531 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f556809b330 con 0x556bee5696b0 2023-10-10T17:47:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5555ffb700 1 -- 172.21.15.78:0/3212545531 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5560005390 con 0x556bee5696b0 2023-10-10T17:47:13.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5580d88b80 10 client.25886 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5580d88b80 10 client.25886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5580d88b80 20 client.25886 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5580d88b80 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee713b90 0x556bee78ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5580d88b80 4 client.25886 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:47:13.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556bee62d600 con 0x556bee713b90 2023-10-10T17:47:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.025+0000 7f5567fff700 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee713b90 0x556bee78ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.026+0000 7f5555ffb700 10 client.25886 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:13.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:13.026+0000 7f5567fff700 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee713b90 0x556bee78ed40 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c26650550cb20f77 server_cookie=7f18998453acf86a in_seq=0 out_seq=0 2023-10-10T17:47:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:12 smithi078 ceph-mon[99494]: pgmap v143: 129 pgs: 129 active+clean; 622 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.7 MiB/s wr, 532 op/s 2023-10-10T17:47:13.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:12 smithi190 ceph-mon[101220]: pgmap v143: 129 pgs: 129 active+clean; 622 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.7 MiB/s wr, 532 op/s 2023-10-10T17:47:13.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:12 smithi157 ceph-mon[100896]: pgmap v143: 129 pgs: 129 active+clean; 622 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.7 MiB/s wr, 532 op/s 2023-10-10T17:47:13.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:47:13.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.516+0000 7f553fc98700 1 -- 172.21.15.78:0/605815526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5538107af0 msgr2=0x7f553810eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:13.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.516+0000 7f553fc98700 1 --2- 172.21.15.78:0/605815526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5538107af0 0x7f553810eb50 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f55340099c0 tx=0x7f5534009cd0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.516+0000 7f553fc98700 1 -- 172.21.15.78:0/605815526 shutdown_connections 2023-10-10T17:47:13.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.516+0000 7f553fc98700 1 --2- 172.21.15.78:0/605815526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5538119a40 0x7f553811be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.516+0000 7f553fc98700 1 --2- 172.21.15.78:0/605815526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5538107af0 0x7f553810eb50 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.516+0000 7f553fc98700 1 --2- 172.21.15.78:0/605815526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5538107ec0 0x7f5538108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.516+0000 7f553fc98700 1 -- 172.21.15.78:0/605815526 >> 172.21.15.78:0/605815526 conn(0x7f553806c7b0 msgr2=0x7f5538071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:13.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.520+0000 7f553fc98700 1 -- 172.21.15.78:0/605815526 shutdown_connections 2023-10-10T17:47:13.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.521+0000 7f553fc98700 1 -- 172.21.15.78:0/605815526 wait complete. 2023-10-10T17:47:13.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.522+0000 7f553fc98700 1 Processor -- start 2023-10-10T17:47:13.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.523+0000 7f553fc98700 1 -- start start 2023-10-10T17:47:13.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553fc98700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5538107af0 0x7f55381a51c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553fc98700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5538107ec0 0x7f55381a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553fc98700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5538119a40 0x7f55381a9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553fc98700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381aa1a0 con 0x7f5538107ec0 2023-10-10T17:47:13.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553fc98700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381aa310 con 0x7f5538119a40 2023-10-10T17:47:13.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553fc98700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55381aa4d0 con 0x7f5538107af0 2023-10-10T17:47:13.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553da34700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5538107af0 0x7f55381a51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553da34700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5538107af0 0x7f55381a51c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38218/0 (socket says 172.21.15.78:38218) 2023-10-10T17:47:13.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553da34700 1 -- 172.21.15.78:0/339152763 learned_addr learned my addr 172.21.15.78:0/339152763 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:13.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.524+0000 7f553e235700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5538119a40 0x7f55381a9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553e235700 1 -- 172.21.15.78:0/339152763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5538107af0 msgr2=0x7f55381a51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:13.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553e235700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5538107af0 0x7f55381a51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553e235700 1 -- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5538107ec0 msgr2=0x7f55381a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:13.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553e235700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5538107ec0 0x7f55381a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553e235700 1 -- 172.21.15.78:0/339152763 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55340095f0 con 0x7f5538119a40 2023-10-10T17:47:13.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553e235700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5538119a40 0x7f55381a9b60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f553000ef90 tx=0x7f553000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:13.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f552effd700 1 -- 172.21.15.78:0/339152763 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5530018070 con 0x7f5538119a40 2023-10-10T17:47:13.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f552effd700 1 -- 172.21.15.78:0/339152763 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f553000f040 con 0x7f5538119a40 2023-10-10T17:47:13.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f552effd700 1 -- 172.21.15.78:0/339152763 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553000cce0 con 0x7f5538119a40 2023-10-10T17:47:13.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553811f190 con 0x7f5538119a40 2023-10-10T17:47:13.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.525+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553811f000 con 0x7f5538119a40 2023-10-10T17:47:13.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.532+0000 7f554be24b80 1 Processor -- start 2023-10-10T17:47:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.533+0000 7f552effd700 1 -- 172.21.15.78:0/339152763 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5530003a00 con 0x7f5538119a40 2023-10-10T17:47:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.534+0000 7f552effd700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f55240a14d0 0x7f55240a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.534+0000 7f552effd700 1 -- 172.21.15.78:0/339152763 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f55300cc9e0 con 0x7f5538119a40 2023-10-10T17:47:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.534+0000 7f554be24b80 1 -- start start 2023-10-10T17:47:13.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.535+0000 7f554be24b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b47f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.535+0000 7f554be24b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b4c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.535+0000 7f554be24b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b4c8b0 0x556503b4cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.535+0000 7f554be24b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503a4d060 con 0x556503b48470 2023-10-10T17:47:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.535+0000 7f554be24b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503a9f8a0 con 0x556503b47b60 2023-10-10T17:47:13.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.535+0000 7f554be24b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503a98d30 con 0x556503b4c8b0 2023-10-10T17:47:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.536+0000 7f553e235700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b4c8b0 0x556503b4cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.537+0000 7f553d233700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f55240a14d0 0x7f55240a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.537+0000 7f553d233700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b4c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.537+0000 7f553d233700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b4c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46698/0 (socket says 172.21.15.78:46698) 2023-10-10T17:47:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.537+0000 7f553d233700 1 -- 172.21.15.78:0/2564924620 learned_addr learned my addr 172.21.15.78:0/2564924620 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.538+0000 7f553d233700 1 -- 172.21.15.78:0/2564924620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b4c8b0 msgr2=0x556503b4cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.538+0000 7f553d233700 1 --2- 172.21.15.78:0/2564924620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b4c8b0 0x556503b4cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.538+0000 7f553d233700 1 -- 172.21.15.78:0/2564924620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 msgr2=0x556503b47f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.538+0000 7f553d233700 1 --2- 172.21.15.78:0/2564924620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b47f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.538+0000 7f553d233700 1 -- 172.21.15.78:0/2564924620 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55650391a630 con 0x556503b48470 2023-10-10T17:47:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.538+0000 7f553d233700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f55240a14d0 0x7f55240a3980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f55280106b0 tx=0x7f5528015000 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.538+0000 7f553d233700 1 --2- 172.21.15.78:0/2564924620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b4c370 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f5528013b70 tx=0x7f5528013e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8c3e79ae859ef3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.539+0000 7f5523fff700 1 -- 172.21.15.78:0/2564924620 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528023070 con 0x556503b48470 2023-10-10T17:47:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.539+0000 7f5523fff700 1 -- 172.21.15.78:0/2564924620 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5528019040 con 0x556503b48470 2023-10-10T17:47:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.539+0000 7f5523fff700 1 -- 172.21.15.78:0/2564924620 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528004830 con 0x556503b48470 2023-10-10T17:47:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 -- 172.21.15.78:0/2564924620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 msgr2=0x556503b4c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 --2- 172.21.15.78:0/2564924620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b4c370 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f5528013b70 tx=0x7f5528013e80 comp rx=0 tx=0).stop 2023-10-10T17:47:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 -- 172.21.15.78:0/2564924620 shutdown_connections 2023-10-10T17:47:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 --2- 172.21.15.78:0/2564924620 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b4c8b0 0x556503b4cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 --2- 172.21.15.78:0/2564924620 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b47f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 --2- 172.21.15.78:0/2564924620 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b4c370 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 -- 172.21.15.78:0/2564924620 >> 172.21.15.78:0/2564924620 conn(0x556503b3d360 msgr2=0x556503b42b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 -- 172.21.15.78:0/2564924620 shutdown_connections 2023-10-10T17:47:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.540+0000 7f554be24b80 1 -- 172.21.15.78:0/2564924620 wait complete. 2023-10-10T17:47:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 -- start start 2023-10-10T17:47:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 Processor -- start 2023-10-10T17:47:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b48470 0x556503b51340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b51a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503a3a8e0 con 0x556503b4c8b0 2023-10-10T17:47:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503a3a580 con 0x556503b47b60 2023-10-10T17:47:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55650391c920 con 0x556503b48470 2023-10-10T17:47:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.541+0000 7f554be24b80 10 client.0 _ll_register_callbacks cb 0x556503a58780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553e235700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b51a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553d233700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b48470 0x556503b51340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553e235700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b51a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46704/0 (socket says 172.21.15.78:46704) 2023-10-10T17:47:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553e235700 1 -- 172.21.15.78:0/1362056977 learned_addr learned my addr 172.21.15.78:0/1362056977 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553da34700 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553d233700 1 -- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 msgr2=0x556503b540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553d233700 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553d233700 1 -- 172.21.15.78:0/1362056977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 msgr2=0x556503b51a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553d233700 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b51a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553d233700 1 -- 172.21.15.78:0/1362056977 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5528013720 con 0x556503b48470 2023-10-10T17:47:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.542+0000 7f553da34700 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:47:13.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.545+0000 7f553d233700 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b48470 0x556503b51340 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5528013b40 tx=0x7f5528005950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcc13b617140dd7d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:13.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.545+0000 7f55217fa700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:47:13.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.547+0000 7f55217fa700 1 -- 172.21.15.78:0/1362056977 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528023040 con 0x556503b48470 2023-10-10T17:47:13.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.547+0000 7f55217fa700 1 -- 172.21.15.78:0/1362056977 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5528009640 con 0x556503b48470 2023-10-10T17:47:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.547+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556503b3ab20 con 0x556503b48470 2023-10-10T17:47:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.547+0000 7f55217fa700 1 -- 172.21.15.78:0/1362056977 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5528013580 con 0x556503b48470 2023-10-10T17:47:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.547+0000 7f554be24b80 10 client.25975 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:13.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.547+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x556503a5f190 con 0x556503b48470 2023-10-10T17:47:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.549+0000 7f55217fa700 1 -- 172.21.15.78:0/1362056977 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556503b3ab20 con 0x556503b48470 2023-10-10T17:47:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.549+0000 7f55217fa700 1 -- 172.21.15.78:0/1362056977 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5528004fe0 con 0x556503b48470 2023-10-10T17:47:13.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.550+0000 7f554be24b80 10 client.25975 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.550+0000 7f554be24b80 10 client.25975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.550+0000 7f554be24b80 20 client.25975 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.550+0000 7f554be24b80 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503b618e0 0x556503b63cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.550+0000 7f554be24b80 4 client.25975 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:47:13.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.550+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556503a5b0a0 con 0x556503b618e0 2023-10-10T17:47:13.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.550+0000 7f553da34700 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503b618e0 0x556503b63cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:13.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.551+0000 7f553da34700 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503b618e0 0x556503b63cf0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1f51cb8de8fbc35 server_cookie=428aab5eb1c82fab in_seq=0 out_seq=0 2023-10-10T17:47:13.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:13.551+0000 7f55217fa700 10 client.25975 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:14 smithi157 ceph-mon[100896]: pgmap v144: 129 pgs: 129 active+clean; 636 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 648 op/s 2023-10-10T17:47:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:14 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:14.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:14 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:15.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5555ffb700 1 -- 172.21.15.78:0/3212545531 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556bee62d600 con 0x556bee713b90 2023-10-10T17:47:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5555ffb700 10 client.25886 handle_command_reply: tid=0 2023-10-10T17:47:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 client.25886 shutdown 2023-10-10T17:47:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee5696b0 msgr2=0x556bee72f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee5696b0 0x556bee72f0b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f556000e950 tx=0x7f5560004750 comp rx=0 tx=0).stop 2023-10-10T17:47:15.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 shutdown_connections 2023-10-10T17:47:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556bee713b90 0x556bee78ed40 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556bee713720 0x556bee70f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5555ffb700 0 client.25886 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:15.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556bee5696b0 0x556bee72f0b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 --2- 172.21.15.78:0/3212545531 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556bee702f50 0x556bee70ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.022+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 >> 172.21.15.78:0/3212545531 conn(0x556bee716eb0 msgr2=0x556bee7340e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 shutdown_connections 2023-10-10T17:47:15.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5580d88b80 1 -- 172.21.15.78:0/3212545531 wait complete. 2023-10-10T17:47:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f55217fa700 1 -- 172.21.15.78:0/1362056977 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556503a5b0a0 con 0x556503b618e0 2023-10-10T17:47:15.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f55217fa700 10 client.25975 handle_command_reply: tid=0 2023-10-10T17:47:15.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5580d88b80 20 client.25886 trim_cache size 0 max 16384 2023-10-10T17:47:15.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 client.25975 shutdown 2023-10-10T17:47:15.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:47:15.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:47:15.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:47:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:47:15.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b48470 msgr2=0x556503b51340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b48470 0x556503b51340 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5528013b40 tx=0x7f5528005950 comp rx=0 tx=0).stop 2023-10-10T17:47:15.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 shutdown_connections 2023-10-10T17:47:15.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503b618e0 0x556503b63cf0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b51a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b47b60 0x556503b540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 --2- 172.21.15.78:0/1362056977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b48470 0x556503b51340 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 >> 172.21.15.78:0/1362056977 conn(0x556503b3d360 msgr2=0x556503b4dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:15.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f55217fa700 0 client.25975 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f555c0a1340 msgr2=0x7f555c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5574bfc700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f555c0a1340 0x7f555c0a37f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5560005950 tx=0x7f55600058e0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 msgr2=0x7f55701a9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5574bfc700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f55701a9c90 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f556800b780 tx=0x7f556800bd60 comp rx=0 tx=0).stop 2023-10-10T17:47:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.023+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 shutdown_connections 2023-10-10T17:47:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f5574bfc700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5570079880 0x7f55701a52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f5574bfc700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f555c0a1340 0x7f555c0a37f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f5574bfc700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5570119940 0x7f55701a9c90 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f5574bfc700 1 --2- 172.21.15.78:0/2566799196 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5570081420 0x7f55701a5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 >> 172.21.15.78:0/2566799196 conn(0x7f5570074840 msgr2=0x7f5570119160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:15.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 shutdown_connections 2023-10-10T17:47:15.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f554be24b80 1 -- 172.21.15.78:0/1362056977 wait complete. 2023-10-10T17:47:15.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.024+0000 7f554be24b80 20 client.25975 trim_cache size 0 max 16384 2023-10-10T17:47:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.025+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 shutdown_connections 2023-10-10T17:47:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:15.025+0000 7f5574bfc700 1 -- 172.21.15.78:0/2566799196 wait complete. 2023-10-10T17:47:15.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 Processor -- start 2023-10-10T17:47:15.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 -- start start 2023-10-10T17:47:15.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b4a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:15.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b58940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:15.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b4c8b0 0x556503b5d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:15.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503b5efa0 con 0x556503b48470 2023-10-10T17:47:15.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503b3cee0 con 0x556503b4c8b0 2023-10-10T17:47:15.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f554be24b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503b4e350 con 0x556503b47b60 2023-10-10T17:47:15.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b58940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:15.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b58940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46710/0 (socket says 172.21.15.78:46710) 2023-10-10T17:47:15.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 -- 172.21.15.78:0/2429252357 learned_addr learned my addr 172.21.15.78:0/2429252357 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:15.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553da34700 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b4a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:15.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553e235700 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b4c8b0 0x556503b5d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:15.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 -- 172.21.15.78:0/2429252357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 msgr2=0x556503b4a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:15.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b4a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 -- 172.21.15.78:0/2429252357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b4c8b0 msgr2=0x556503b5d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:15.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b4c8b0 0x556503b5d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 -- 172.21.15.78:0/2429252357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556503b3eb00 con 0x556503b48470 2023-10-10T17:47:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553e235700 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b4c8b0 0x556503b5d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:47:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f553d233700 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b58940 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f55280092a0 tx=0x7f5528009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe948dff1beacc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.043+0000 7f552cdac700 1 -- 172.21.15.78:0/2429252357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528023040 con 0x556503b48470 2023-10-10T17:47:15.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f552cdac700 1 -- 172.21.15.78:0/2429252357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5528013580 con 0x556503b48470 2023-10-10T17:47:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f552cdac700 1 -- 172.21.15.78:0/2429252357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552800e890 con 0x556503b48470 2023-10-10T17:47:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 -- 172.21.15.78:0/2429252357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 msgr2=0x556503b58940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b58940 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f55280092a0 tx=0x7f5528009fd0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 -- 172.21.15.78:0/2429252357 shutdown_connections 2023-10-10T17:47:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b4a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b4c8b0 0x556503b5d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 --2- 172.21.15.78:0/2429252357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b48470 0x556503b58940 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 -- 172.21.15.78:0/2429252357 >> 172.21.15.78:0/2429252357 conn(0x556503b3d360 msgr2=0x556503b638f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 -- 172.21.15.78:0/2429252357 shutdown_connections 2023-10-10T17:47:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.044+0000 7f554be24b80 1 -- 172.21.15.78:0/2429252357 wait complete. 2023-10-10T17:47:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 -- start start 2023-10-10T17:47:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 Processor -- start 2023-10-10T17:47:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b50ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b48470 0x556503b531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b53730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503b51610 con 0x556503b4c8b0 2023-10-10T17:47:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503b53c70 con 0x556503b48470 2023-10-10T17:47:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556503b53e30 con 0x556503b47b60 2023-10-10T17:47:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f554be24b80 10 client.0 _ll_register_callbacks cb 0x7f55280046f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b53730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b53730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46718/0 (socket says 172.21.15.78:46718) 2023-10-10T17:47:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 -- 172.21.15.78:0/1223956263 learned_addr learned my addr 172.21.15.78:0/1223956263 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553da34700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b50ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553d233700 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b48470 0x556503b531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 -- 172.21.15.78:0/1223956263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 msgr2=0x556503b50ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b50ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 -- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b48470 msgr2=0x556503b531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b48470 0x556503b531f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.045+0000 7f553e235700 1 -- 172.21.15.78:0/1223956263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556503a5f190 con 0x556503b4c8b0 2023-10-10T17:47:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f553e235700 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b53730 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f55300090a0 tx=0x7f5530004ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813e63f0492f037d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f55217fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:47:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f55217fa700 1 -- 172.21.15.78:0/1223956263 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553000ce40 con 0x556503b4c8b0 2023-10-10T17:47:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f55217fa700 1 -- 172.21.15.78:0/1223956263 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f55300997d0 con 0x556503b4c8b0 2023-10-10T17:47:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5528019040 con 0x556503b4c8b0 2023-10-10T17:47:15.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f55217fa700 1 -- 172.21.15.78:0/1223956263 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f553009d800 con 0x556503b4c8b0 2023-10-10T17:47:15.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 10 client.16533 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:15.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5565038c6de0 con 0x556503b4c8b0 2023-10-10T17:47:15.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f55217fa700 1 -- 172.21.15.78:0/1223956263 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5528019040 con 0x556503b4c8b0 2023-10-10T17:47:15.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f55217fa700 1 -- 172.21.15.78:0/1223956263 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5530098cc0 con 0x556503b4c8b0 2023-10-10T17:47:15.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 10 client.16533 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:15.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:15.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 20 client.16533 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:15.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503bb59b0 0x556503bb7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:15.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 4 client.16533 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:47:15.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.046+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556503a67660 con 0x556503bb59b0 2023-10-10T17:47:15.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f553da34700 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503bb59b0 0x556503bb7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:15.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f553da34700 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503bb59b0 0x556503bb7dc0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c16877f25d9c7d2 server_cookie=b6f3176bc743d212 in_seq=0 out_seq=0 2023-10-10T17:47:15.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f55217fa700 10 client.16533 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:15.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f55217fa700 1 -- 172.21.15.78:0/1223956263 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x556503a67660 con 0x556503bb59b0 2023-10-10T17:47:15.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f55217fa700 10 client.16533 handle_command_reply: tid=0 2023-10-10T17:47:15.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f554be24b80 1 client.16533 shutdown 2023-10-10T17:47:15.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 msgr2=0x556503b53730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f554be24b80 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b53730 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f55300090a0 tx=0x7f5530004ba0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 shutdown_connections 2023-10-10T17:47:15.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f554be24b80 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556503b47b60 0x556503b50ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.047+0000 7f554be24b80 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556503bb59b0 0x556503bb7dc0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f554be24b80 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556503b4c8b0 0x556503b53730 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f554be24b80 1 --2- 172.21.15.78:0/1223956263 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556503b48470 0x556503b531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 >> 172.21.15.78:0/1223956263 conn(0x556503b3d360 msgr2=0x556503b4aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:15.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f55217fa700 0 client.16533 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:15.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 shutdown_connections 2023-10-10T17:47:15.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f554be24b80 1 -- 172.21.15.78:0/1223956263 wait complete. 2023-10-10T17:47:15.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f554be24b80 20 client.16533 trim_cache size 0 max 16384 2023-10-10T17:47:15.069 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:47:15.069 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:47:15.069 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:47:15.070 INFO:teuthology.orchestra.run.smithi078.stdout: "5ecf957f-7c91-40df-8738-d2beae2feba1": { 2023-10-10T17:47:15.070 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:47:15.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:47:15.070 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:47:15.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:47:15.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:47:15.071 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:47:15.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f55240a14d0 msgr2=0x7f55240a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f55240a14d0 0x7f55240a3980 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f55280106b0 tx=0x7f5528015000 comp rx=0 tx=0).stop 2023-10-10T17:47:15.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5538119a40 msgr2=0x7f55381a9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:15.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5538119a40 0x7f55381a9b60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f553000ef90 tx=0x7f553000c3a0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 shutdown_connections 2023-10-10T17:47:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5538107af0 0x7f55381a51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f55240a14d0 0x7f55240a3980 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5538119a40 0x7f55381a9b60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 --2- 172.21.15.78:0/339152763 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5538107ec0 0x7f55381a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 >> 172.21.15.78:0/339152763 conn(0x7f553806c7b0 msgr2=0x7f553811a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 shutdown_connections 2023-10-10T17:47:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:15.048+0000 7f553fc98700 1 -- 172.21.15.78:0/339152763 wait complete. 2023-10-10T17:47:15.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:14 smithi078 ceph-mon[99494]: pgmap v144: 129 pgs: 129 active+clean; 636 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 648 op/s 2023-10-10T17:47:15.075 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:14 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:15.076 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:14 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:15.077 INFO:tasks.cephfs.filesystem:scrub status for tag:5ecf957f-7c91-40df-8738-d2beae2feba1 - {'path': '/', 'tag': '5ecf957f-7c91-40df-8738-d2beae2feba1', 'options': 'recursive,force'} 2023-10-10T17:47:15.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:14 smithi190 ceph-mon[101220]: pgmap v144: 129 pgs: 129 active+clean; 636 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.3 MiB/s wr, 648 op/s 2023-10-10T17:47:15.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:14 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:15.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:14 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:16.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:16 smithi157 ceph-mon[100896]: pgmap v145: 129 pgs: 129 active+clean; 636 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 403 op/s 2023-10-10T17:47:17.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:16 smithi078 ceph-mon[99494]: pgmap v145: 129 pgs: 129 active+clean; 636 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 403 op/s 2023-10-10T17:47:17.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:16 smithi190 ceph-mon[101220]: pgmap v145: 129 pgs: 129 active+clean; 636 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.8 MiB/s wr, 403 op/s 2023-10-10T17:47:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:18 smithi157 ceph-mon[100896]: pgmap v146: 129 pgs: 129 active+clean; 644 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.3 MiB/s wr, 719 op/s 2023-10-10T17:47:18.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:18 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:18.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:18 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:19.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:18 smithi078 ceph-mon[99494]: pgmap v146: 129 pgs: 129 active+clean; 644 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.3 MiB/s wr, 719 op/s 2023-10-10T17:47:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:18 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:18 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:19.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:18 smithi190 ceph-mon[101220]: pgmap v146: 129 pgs: 129 active+clean; 644 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.3 MiB/s wr, 719 op/s 2023-10-10T17:47:19.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:18 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:18 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:20.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:20 smithi157 ceph-mon[100896]: pgmap v147: 129 pgs: 129 active+clean; 644 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 493 op/s 2023-10-10T17:47:21.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:20 smithi078 ceph-mon[99494]: pgmap v147: 129 pgs: 129 active+clean; 644 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 493 op/s 2023-10-10T17:47:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:20 smithi190 ceph-mon[101220]: pgmap v147: 129 pgs: 129 active+clean; 644 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 493 op/s 2023-10-10T17:47:21.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:21 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:21.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:21 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:21.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:21 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:21.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:21 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:21.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:21 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:21.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:21 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:22.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:21 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:22.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:21 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:22.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:21 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:22.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:21 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:22.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:21 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:22.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:21 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:22.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:21 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:21 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:21 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:21 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:21 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:21 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:22.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:22 smithi157 ceph-mon[100896]: pgmap v148: 129 pgs: 129 active+clean; 652 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 519 op/s 2023-10-10T17:47:23.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:22 smithi078 ceph-mon[99494]: pgmap v148: 129 pgs: 129 active+clean; 652 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 519 op/s 2023-10-10T17:47:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:22 smithi190 ceph-mon[101220]: pgmap v148: 129 pgs: 129 active+clean; 652 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.5 MiB/s wr, 519 op/s 2023-10-10T17:47:23.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:23 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:23.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:23 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:24.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:23 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:23 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:24.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:23 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:24.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:23 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:24.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:24 smithi157 ceph-mon[100896]: pgmap v149: 129 pgs: 129 active+clean; 671 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.9 MiB/s wr, 710 op/s 2023-10-10T17:47:25.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:24 smithi078 ceph-mon[99494]: pgmap v149: 129 pgs: 129 active+clean; 671 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.9 MiB/s wr, 710 op/s 2023-10-10T17:47:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:24 smithi190 ceph-mon[101220]: pgmap v149: 129 pgs: 129 active+clean; 671 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.9 MiB/s wr, 710 op/s 2023-10-10T17:47:25.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:25 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:25.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:25 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:25 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:26.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:25 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:26.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:25 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:26.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:25 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:26.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:26 smithi157 ceph-mon[100896]: pgmap v150: 129 pgs: 129 active+clean; 671 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.7 MiB/s wr, 553 op/s 2023-10-10T17:47:27.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:26 smithi078 ceph-mon[99494]: pgmap v150: 129 pgs: 129 active+clean; 671 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.7 MiB/s wr, 553 op/s 2023-10-10T17:47:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:26 smithi190 ceph-mon[101220]: pgmap v150: 129 pgs: 129 active+clean; 671 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.7 MiB/s wr, 553 op/s 2023-10-10T17:47:28.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:27 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:28.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:27 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:27 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:28.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:27 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:28.209 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.208+0000 7f963d186700 1 -- 172.21.15.78:0/1211551910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9638107d60 msgr2=0x7f9638108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:28.210 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.208+0000 7f963d186700 1 --2- 172.21.15.78:0/1211551910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9638107d60 0x7f9638108130 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f96280074e0 tx=0x7f962800af90 comp rx=0 tx=0).stop 2023-10-10T17:47:28.213 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.209+0000 7f963d186700 1 -- 172.21.15.78:0/1211551910 shutdown_connections 2023-10-10T17:47:28.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.209+0000 7f963d186700 1 --2- 172.21.15.78:0/1211551910 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96381111d0 0x7f9638113660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.209+0000 7f963d186700 1 --2- 172.21.15.78:0/1211551910 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9638107d60 0x7f9638108130 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.209+0000 7f963d186700 1 --2- 172.21.15.78:0/1211551910 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 0x7f9638110b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.214 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.209+0000 7f963d186700 1 -- 172.21.15.78:0/1211551910 >> 172.21.15.78:0/1211551910 conn(0x7f963806ca10 msgr2=0x7f96380716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:28.215 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.211+0000 7f963d186700 1 -- 172.21.15.78:0/1211551910 shutdown_connections 2023-10-10T17:47:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.213+0000 7f963d186700 1 -- 172.21.15.78:0/1211551910 wait complete. 2023-10-10T17:47:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.214+0000 7f963d186700 1 Processor -- start 2023-10-10T17:47:28.216 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963d186700 1 -- start start 2023-10-10T17:47:28.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963d186700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9638107d60 0x7f96381a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.217 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963d186700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 0x7f96381a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963d186700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96381111d0 0x7f963819c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963d186700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963819d070 con 0x7f9638107d60 2023-10-10T17:47:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963d186700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963819d1e0 con 0x7f9638108670 2023-10-10T17:47:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963d186700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963819d3a0 con 0x7f96381111d0 2023-10-10T17:47:28.218 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963759e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96381111d0 0x7f963819c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963659c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 0x7f96381a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963659c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 0x7f96381a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48454/0 (socket says 172.21.15.78:48454) 2023-10-10T17:47:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.215+0000 7f963659c700 1 -- 172.21.15.78:0/1097600380 learned_addr learned my addr 172.21.15.78:0/1097600380 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.216+0000 7f963659c700 1 -- 172.21.15.78:0/1097600380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96381111d0 msgr2=0x7f963819c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:28.219 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.216+0000 7f963659c700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96381111d0 0x7f963819c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.216+0000 7f963659c700 1 -- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9638107d60 msgr2=0x7f96381a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.216+0000 7f963659c700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9638107d60 0x7f96381a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.216+0000 7f963659c700 1 -- 172.21.15.78:0/1097600380 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96280071c0 con 0x7f9638108670 2023-10-10T17:47:28.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.216+0000 7f963659c700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 0x7f96381a1d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f962c00b530 tx=0x7f962c00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:28.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.219+0000 7f961ffff700 1 -- 172.21.15.78:0/1097600380 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c011830 con 0x7f9638108670 2023-10-10T17:47:28.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.219+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96381169b0 con 0x7f9638108670 2023-10-10T17:47:28.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.219+0000 7f961ffff700 1 -- 172.21.15.78:0/1097600380 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f962c011e50 con 0x7f9638108670 2023-10-10T17:47:28.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.219+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9638116820 con 0x7f9638108670 2023-10-10T17:47:28.222 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.219+0000 7f961ffff700 1 -- 172.21.15.78:0/1097600380 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c00f2e0 con 0x7f9638108670 2023-10-10T17:47:28.223 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.222+0000 7f961ffff700 1 -- 172.21.15.78:0/1097600380 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f962c010380 con 0x7f9638108670 2023-10-10T17:47:28.225 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.224+0000 7f9649312b80 1 Processor -- start 2023-10-10T17:47:28.226 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.224+0000 7f9649312b80 1 -- start start 2023-10-10T17:47:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9649312b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0b71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9649312b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9649312b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9649312b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42efb7000 con 0x55a42f0b7b90 2023-10-10T17:47:28.227 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9649312b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f009840 con 0x55a42ef0d6b0 2023-10-10T17:47:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9649312b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f002cd0 con 0x55a42f0b7720 2023-10-10T17:47:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f963659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f963659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46736/0 (socket says 172.21.15.78:46736) 2023-10-10T17:47:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f963659c700 1 -- 172.21.15.78:0/1142933596 learned_addr learned my addr 172.21.15.78:0/1142933596 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:28.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9636d9d700 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f961ffff700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96200a1490 0x7f96200a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:27 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:28.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:27 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f963759e700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96200a1490 0x7f96200a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9636d9d700 1 -- 172.21.15.78:0/1142933596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 msgr2=0x55a42f0b71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9636d9d700 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0b71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9636d9d700 1 -- 172.21.15.78:0/1142933596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7b90 msgr2=0x55a42f0b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9636d9d700 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f9636d9d700 1 -- 172.21.15.78:0/1142933596 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a42ee845d0 con 0x55a42f0b7720 2023-10-10T17:47:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.225+0000 7f961ffff700 1 -- 172.21.15.78:0/1097600380 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f962c0cca00 con 0x7f9638108670 2023-10-10T17:47:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.226+0000 7f963759e700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96200a1490 0x7f96200a3940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f963000b590 tx=0x7f9630009970 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.226+0000 7f9636d9d700 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9628006010 tx=0x7f9628005670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=724ca729c3b48da1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.227+0000 7f961d7fa700 1 -- 172.21.15.78:0/1142933596 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962800f070 con 0x55a42f0b7720 2023-10-10T17:47:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.227+0000 7f961d7fa700 1 -- 172.21.15.78:0/1142933596 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9628007e80 con 0x55a42f0b7720 2023-10-10T17:47:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.227+0000 7f961d7fa700 1 -- 172.21.15.78:0/1142933596 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962800c5c0 con 0x55a42f0b7720 2023-10-10T17:47:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.227+0000 7f9649312b80 1 -- 172.21.15.78:0/1142933596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7720 msgr2=0x55a42f0b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.227+0000 7f9649312b80 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9628006010 tx=0x7f9628005670 comp rx=0 tx=0).stop 2023-10-10T17:47:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 -- 172.21.15.78:0/1142933596 shutdown_connections 2023-10-10T17:47:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0b71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 --2- 172.21.15.78:0/1142933596 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 -- 172.21.15.78:0/1142933596 >> 172.21.15.78:0/1142933596 conn(0x55a42f0a6f50 msgr2=0x55a42f0ac270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 -- 172.21.15.78:0/1142933596 shutdown_connections 2023-10-10T17:47:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 -- 172.21.15.78:0/1142933596 wait complete. 2023-10-10T17:47:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 -- start start 2023-10-10T17:47:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 Processor -- start 2023-10-10T17:47:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0c0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0c0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.228+0000 7f9649312b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9649312b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42efa4880 con 0x55a42f0b7720 2023-10-10T17:47:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9649312b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42efa4520 con 0x55a42f0b7b90 2023-10-10T17:47:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9649312b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42ee868c0 con 0x55a42ef0d6b0 2023-10-10T17:47:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9649312b80 10 client.0 _ll_register_callbacks cb 0x55a42efc2720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0c0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9636d9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0c0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0c0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46744/0 (socket says 172.21.15.78:46744) 2023-10-10T17:47:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 -- 172.21.15.78:0/1062935131 learned_addr learned my addr 172.21.15.78:0/1062935131 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963759e700 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 -- 172.21.15.78:0/1062935131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 msgr2=0x55a42f0c0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0c0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 -- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 msgr2=0x55a42f0bace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 -- 172.21.15.78:0/1062935131 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9638114320 con 0x55a42f0b7720 2023-10-10T17:47:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f963659c700 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0c0f90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f962c0a1790 tx=0x7f962c0d2290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e6b491711906bc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:28.242 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f961aff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:47:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f961aff5700 1 -- 172.21.15.78:0/1062935131 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c0b7ea0 con 0x55a42f0b7720 2023-10-10T17:47:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f961aff5700 1 -- 172.21.15.78:0/1062935131 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f962c099870 con 0x55a42f0b7720 2023-10-10T17:47:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a42efc5040 con 0x55a42f0b7720 2023-10-10T17:47:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f961aff5700 1 -- 172.21.15.78:0/1062935131 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f962c099640 con 0x55a42f0b7720 2023-10-10T17:47:28.243 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9649312b80 10 client.16542 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.229+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a42efc9130 con 0x55a42f0b7720 2023-10-10T17:47:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f961aff5700 1 -- 172.21.15.78:0/1062935131 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a42efc5040 con 0x55a42f0b7720 2023-10-10T17:47:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f961aff5700 1 -- 172.21.15.78:0/1062935131 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f962c0a1830 con 0x55a42f0b7720 2023-10-10T17:47:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9649312b80 10 client.16542 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9649312b80 10 client.16542 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9649312b80 20 client.16542 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9649312b80 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f0cb070 0x55a42f0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9649312b80 4 client.16542 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:47:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a42efd1600 con 0x55a42f0cb070 2023-10-10T17:47:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9636d9d700 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f0cb070 0x55a42f0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f961aff5700 10 client.16542 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.230+0000 7f9636d9d700 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f0cb070 0x55a42f0cd480 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4506345420c08bd server_cookie=cb1b2aa61cb0a6cf in_seq=0 out_seq=0 2023-10-10T17:47:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.231+0000 7f961aff5700 1 -- 172.21.15.78:0/1062935131 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a42efd1600 con 0x55a42f0cb070 2023-10-10T17:47:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.231+0000 7f961aff5700 10 client.16542 handle_command_reply: tid=0 2023-10-10T17:47:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.231+0000 7f9649312b80 1 client.16542 shutdown 2023-10-10T17:47:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 msgr2=0x55a42f0c0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0c0f90 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f962c0a1790 tx=0x7f962c0d2290 comp rx=0 tx=0).stop 2023-10-10T17:47:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 shutdown_connections 2023-10-10T17:47:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0c0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f0cb070 0x55a42f0cd480 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 --2- 172.21.15.78:0/1062935131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0c0f90 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f961aff5700 0 client.16542 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 >> 172.21.15.78:0/1062935131 conn(0x55a42f0a6f50 msgr2=0x55a42f0ab7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 shutdown_connections 2023-10-10T17:47:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 1 -- 172.21.15.78:0/1062935131 wait complete. 2023-10-10T17:47:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.232+0000 7f9649312b80 20 client.16542 trim_cache size 0 max 16384 2023-10-10T17:47:28.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.248+0000 7f9649312b80 1 Processor -- start 2023-10-10T17:47:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.248+0000 7f9649312b80 1 -- start start 2023-10-10T17:47:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.248+0000 7f9649312b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0abf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.248+0000 7f9649312b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.248+0000 7f9649312b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.256 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f9649312b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f0b1fe0 con 0x55a42f0b7720 2023-10-10T17:47:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f9649312b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f0cdae0 con 0x55a42ef0d6b0 2023-10-10T17:47:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f9649312b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f0b9280 con 0x55a42f0b7b90 2023-10-10T17:47:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f9636d9d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0abf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.257 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46752/0 (socket says 172.21.15.78:46752) 2023-10-10T17:47:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f9636d9d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0abf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48472/0 (socket says 172.21.15.78:48472) 2023-10-10T17:47:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 -- 172.21.15.78:0/1857270321 learned_addr learned my addr 172.21.15.78:0/1857270321 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963759e700 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 -- 172.21.15.78:0/1857270321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7b90 msgr2=0x55a42f0b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.258 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 -- 172.21.15.78:0/1857270321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 msgr2=0x55a42f0abf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0abf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 -- 172.21.15.78:0/1857270321 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a42f0a86f0 con 0x55a42f0b7720 2023-10-10T17:47:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963759e700 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:47:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f963659c700 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f962c00bc40 tx=0x7f962c010870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13c305dc8297c49 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f961dffb700 1 -- 172.21.15.78:0/1857270321 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c0b7ea0 con 0x55a42f0b7720 2023-10-10T17:47:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f961dffb700 1 -- 172.21.15.78:0/1857270321 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f962c099640 con 0x55a42f0b7720 2023-10-10T17:47:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.249+0000 7f961dffb700 1 -- 172.21.15.78:0/1857270321 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c099900 con 0x55a42f0b7720 2023-10-10T17:47:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 -- 172.21.15.78:0/1857270321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 msgr2=0x55a42f0b1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f962c00bc40 tx=0x7f962c010870 comp rx=0 tx=0).stop 2023-10-10T17:47:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 -- 172.21.15.78:0/1857270321 shutdown_connections 2023-10-10T17:47:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42f0b7b90 0x55a42f0b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42ef0d6b0 0x55a42f0abf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 --2- 172.21.15.78:0/1857270321 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0b1560 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.261 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 -- 172.21.15.78:0/1857270321 >> 172.21.15.78:0/1857270321 conn(0x55a42f0a6f50 msgr2=0x55a42f0bb610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 -- 172.21.15.78:0/1857270321 shutdown_connections 2023-10-10T17:47:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 -- 172.21.15.78:0/1857270321 wait complete. 2023-10-10T17:47:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.250+0000 7f9649312b80 1 -- start start 2023-10-10T17:47:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 1 Processor -- start 2023-10-10T17:47:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0be600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0beb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f0c12d0 con 0x55a42f0b7720 2023-10-10T17:47:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f11def0 con 0x55a42f0b7b90 2023-10-10T17:47:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42f11e030 con 0x55a42ef0d6b0 2023-10-10T17:47:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9649312b80 10 client.0 _ll_register_callbacks cb 0x55a42efc2720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f963759e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f963759e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48480/0 (socket says 172.21.15.78:48480) 2023-10-10T17:47:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f963759e700 1 -- 172.21.15.78:0/3750418205 learned_addr learned my addr 172.21.15.78:0/3750418205 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f9636d9d700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0be600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.251+0000 7f963659c700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0beb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f963759e700 1 -- 172.21.15.78:0/3750418205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 msgr2=0x55a42f0be600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f963759e700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0be600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f963759e700 1 -- 172.21.15.78:0/3750418205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 msgr2=0x55a42f0beb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:28.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f963759e700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0beb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f963759e700 1 -- 172.21.15.78:0/3750418205 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a42efc9130 con 0x55a42f0b7b90 2023-10-10T17:47:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f963659c700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0beb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:47:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f963759e700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bf080 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9630015de0 tx=0x7f9630009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7303e3a7bca084c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f961aff5700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:47:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f961aff5700 1 -- 172.21.15.78:0/3750418205 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630015e10 con 0x55a42f0b7b90 2023-10-10T17:47:28.266 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f961aff5700 1 -- 172.21.15.78:0/3750418205 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f96300149f0 con 0x55a42f0b7b90 2023-10-10T17:47:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a42ee51f10 con 0x55a42f0b7b90 2023-10-10T17:47:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f961aff5700 1 -- 172.21.15.78:0/3750418205 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9630014bb0 con 0x55a42f0b7b90 2023-10-10T17:47:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f9649312b80 10 client.25913 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:28.267 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a42ee30d80 con 0x55a42f0b7b90 2023-10-10T17:47:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.252+0000 7f961aff5700 1 -- 172.21.15.78:0/3750418205 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a42ee51f10 con 0x55a42f0b7b90 2023-10-10T17:47:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f961aff5700 1 -- 172.21.15.78:0/3750418205 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f96300223d0 con 0x55a42f0b7b90 2023-10-10T17:47:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9649312b80 10 client.25913 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9649312b80 10 client.25913 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9649312b80 20 client.25913 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:28.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9649312b80 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f11f8d0 0x55a42f121ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9649312b80 4 client.25913 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:47:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a42ee86f00 con 0x55a42f11f8d0 2023-10-10T17:47:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9636d9d700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f11f8d0 0x55a42f121ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f961aff5700 10 client.25913 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:28.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:28.253+0000 7f9636d9d700 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f11f8d0 0x55a42f121ce0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a6326615d88549 server_cookie=6466e20d2b6badfd in_seq=0 out_seq=0 2023-10-10T17:47:29.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:28 smithi078 ceph-mon[99494]: pgmap v151: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.9 MiB/s wr, 733 op/s 2023-10-10T17:47:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:28 smithi190 ceph-mon[101220]: pgmap v151: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.9 MiB/s wr, 733 op/s 2023-10-10T17:47:29.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:28 smithi157 ceph-mon[100896]: pgmap v151: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.9 MiB/s wr, 733 op/s 2023-10-10T17:47:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.460+0000 7f961aff5700 1 -- 172.21.15.78:0/3750418205 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a42ee86f00 con 0x55a42f11f8d0 2023-10-10T17:47:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.460+0000 7f961aff5700 10 client.25913 handle_command_reply: tid=0 2023-10-10T17:47:30.462 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.460+0000 7f9649312b80 1 client.25913 shutdown 2023-10-10T17:47:30.463 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 msgr2=0x55a42f0bf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bf080 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9630015de0 tx=0x7f9630009630 comp rx=0 tx=0).stop 2023-10-10T17:47:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 shutdown_connections 2023-10-10T17:47:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a42ef0d6b0 0x55a42f0be600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a42f11f8d0 0x55a42f121ce0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.464 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a42f0b7b90 0x55a42f0bf080 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 --2- 172.21.15.78:0/3750418205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a42f0b7720 0x55a42f0beb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 >> 172.21.15.78:0/3750418205 conn(0x55a42f0a6f50 msgr2=0x55a42f0b5de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f961aff5700 0 client.25913 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 shutdown_connections 2023-10-10T17:47:30.465 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 1 -- 172.21.15.78:0/3750418205 wait complete. 2023-10-10T17:47:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.461+0000 7f9649312b80 20 client.25913 trim_cache size 0 max 16384 2023-10-10T17:47:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:47:30.466 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:47:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:47:30.467 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:47:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96200a1490 msgr2=0x7f96200a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96200a1490 0x7f96200a3940 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f963000b590 tx=0x7f9630009970 comp rx=0 tx=0).stop 2023-10-10T17:47:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 msgr2=0x7f96381a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:30.468 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 0x7f96381a1d30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f962c00b530 tx=0x7f962c00b840 comp rx=0 tx=0).stop 2023-10-10T17:47:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 shutdown_connections 2023-10-10T17:47:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96381111d0 0x7f963819c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96200a1490 0x7f96200a3940 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9638107d60 0x7f96381a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.469 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 --2- 172.21.15.78:0/1097600380 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9638108670 0x7f96381a1d30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 >> 172.21.15.78:0/1097600380 conn(0x7f963806ca10 msgr2=0x7f963806ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 shutdown_connections 2023-10-10T17:47:30.470 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:30.462+0000 7f963d186700 1 -- 172.21.15.78:0/1097600380 wait complete. 2023-10-10T17:47:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:30 smithi078 ceph-mon[99494]: pgmap v152: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 417 op/s 2023-10-10T17:47:31.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:30 smithi190 ceph-mon[101220]: pgmap v152: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 417 op/s 2023-10-10T17:47:31.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:30 smithi157 ceph-mon[100896]: pgmap v152: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.4 MiB/s wr, 417 op/s 2023-10-10T17:47:33.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:32 smithi078 ceph-mon[99494]: pgmap v153: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 452 op/s 2023-10-10T17:47:33.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:32 smithi190 ceph-mon[101220]: pgmap v153: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 452 op/s 2023-10-10T17:47:33.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:32 smithi157 ceph-mon[100896]: pgmap v153: 129 pgs: 129 active+clean; 697 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 4.8 MiB/s wr, 452 op/s 2023-10-10T17:47:35.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:34 smithi078 ceph-mon[99494]: pgmap v154: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 650 op/s 2023-10-10T17:47:35.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:34 smithi190 ceph-mon[101220]: pgmap v154: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 650 op/s 2023-10-10T17:47:35.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:34 smithi157 ceph-mon[100896]: pgmap v154: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 5.6 MiB/s wr, 650 op/s 2023-10-10T17:47:37.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:36 smithi078 ceph-mon[99494]: pgmap v155: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 439 op/s 2023-10-10T17:47:37.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:36 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:37.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:36 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:37.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:36 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:37.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:36 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:37.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:36 smithi190 ceph-mon[101220]: pgmap v155: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 439 op/s 2023-10-10T17:47:37.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:36 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:37.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:36 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:37.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:36 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:37.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:36 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:37.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:36 smithi157 ceph-mon[100896]: pgmap v155: 129 pgs: 129 active+clean; 698 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 MiB/s rd, 4.0 MiB/s wr, 439 op/s 2023-10-10T17:47:37.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:36 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:36 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:36 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:37.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:36 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:39.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:38 smithi078 ceph-mon[99494]: pgmap v156: 129 pgs: 129 active+clean; 720 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.9 MiB/s wr, 668 op/s 2023-10-10T17:47:39.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:38 smithi190 ceph-mon[101220]: pgmap v156: 129 pgs: 129 active+clean; 720 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.9 MiB/s wr, 668 op/s 2023-10-10T17:47:39.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:38 smithi157 ceph-mon[100896]: pgmap v156: 129 pgs: 129 active+clean; 720 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.9 MiB/s wr, 668 op/s 2023-10-10T17:47:41.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:40 smithi078 ceph-mon[99494]: pgmap v157: 129 pgs: 129 active+clean; 720 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 488 op/s 2023-10-10T17:47:41.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:40 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:41.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:40 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:41.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:40 smithi190 ceph-mon[101220]: pgmap v157: 129 pgs: 129 active+clean; 720 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 488 op/s 2023-10-10T17:47:41.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:40 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:41.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:40 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:41.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:40 smithi157 ceph-mon[100896]: pgmap v157: 129 pgs: 129 active+clean; 720 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.7 MiB/s wr, 488 op/s 2023-10-10T17:47:41.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:40 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:41.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:40 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:42.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:41 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:42.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:41 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:42.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:41 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:42.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:41 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:42.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:41 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:42.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:41 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:43.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:42 smithi078 ceph-mon[99494]: pgmap v158: 129 pgs: 129 active+clean; 728 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.3 MiB/s wr, 513 op/s 2023-10-10T17:47:43.193 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:42 smithi190 ceph-mon[101220]: pgmap v158: 129 pgs: 129 active+clean; 728 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.3 MiB/s wr, 513 op/s 2023-10-10T17:47:43.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:42 smithi157 ceph-mon[100896]: pgmap v158: 129 pgs: 129 active+clean; 728 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 4.3 MiB/s wr, 513 op/s 2023-10-10T17:47:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 -- 172.21.15.78:0/2170882288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c104610 msgr2=0x7f072c10c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:43.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 --2- 172.21.15.78:0/2170882288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c104610 0x7f072c10c1e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f072400b0d0 tx=0x7f072400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 -- 172.21.15.78:0/2170882288 shutdown_connections 2023-10-10T17:47:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 --2- 172.21.15.78:0/2170882288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c104610 0x7f072c10c1e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 --2- 172.21.15.78:0/2170882288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c0ffaf0 0x7f072c0ffec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 --2- 172.21.15.78:0/2170882288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c100490 0x7f072c103f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 -- 172.21.15.78:0/2170882288 >> 172.21.15.78:0/2170882288 conn(0x7f072c075810 msgr2=0x7f072c075c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 -- 172.21.15.78:0/2170882288 shutdown_connections 2023-10-10T17:47:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 -- 172.21.15.78:0/2170882288 wait complete. 2023-10-10T17:47:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 Processor -- start 2023-10-10T17:47:43.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.574+0000 7f0732dda700 1 -- start start 2023-10-10T17:47:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0732dda700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c0ffaf0 0x7f072c1966f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0732dda700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c100490 0x7f072c196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0732dda700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c104610 0x7f072c199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0732dda700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c19a5e0 con 0x7f072c0ffaf0 2023-10-10T17:47:43.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0732dda700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c19a750 con 0x7f072c100490 2023-10-10T17:47:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0732dda700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c19a910 con 0x7f072c104610 2023-10-10T17:47:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c0ffaf0 0x7f072c1966f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c0ffaf0 0x7f072c1966f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46770/0 (socket says 172.21.15.78:46770) 2023-10-10T17:47:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 -- 172.21.15.78:0/344714664 learned_addr learned my addr 172.21.15.78:0/344714664 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:43.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0731377700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c104610 0x7f072c199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f072bfff700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c100490 0x7f072c196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 -- 172.21.15.78:0/344714664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c104610 msgr2=0x7f072c199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c104610 0x7f072c199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 -- 172.21.15.78:0/344714664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c100490 msgr2=0x7f072c196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:43.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c100490 0x7f072c196c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 -- 172.21.15.78:0/344714664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0724009d00 con 0x7f072c0ffaf0 2023-10-10T17:47:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0730b76700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c0ffaf0 0x7f072c1966f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f072000e9a0 tx=0x7f072000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0729ffb700 1 -- 172.21.15.78:0/344714664 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720009790 con 0x7f072c0ffaf0 2023-10-10T17:47:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c10f3e0 con 0x7f072c0ffaf0 2023-10-10T17:47:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0729ffb700 1 -- 172.21.15.78:0/344714664 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0720010430 con 0x7f072c0ffaf0 2023-10-10T17:47:43.583 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.575+0000 7f0729ffb700 1 -- 172.21.15.78:0/344714664 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720018680 con 0x7f072c0ffaf0 2023-10-10T17:47:43.584 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.576+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c10f250 con 0x7f072c0ffaf0 2023-10-10T17:47:43.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.578+0000 7f0729ffb700 1 -- 172.21.15.78:0/344714664 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f07200105a0 con 0x7f072c0ffaf0 2023-10-10T17:47:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.578+0000 7f0729ffb700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f071c0a12c0 0x7f071c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.578+0000 7f072bfff700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f071c0a12c0 0x7f071c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.578+0000 7f0729ffb700 1 -- 172.21.15.78:0/344714664 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0720014070 con 0x7f072c0ffaf0 2023-10-10T17:47:43.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f072bfff700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f071c0a12c0 0x7f071c0a3770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f072c197d00 tx=0x7f0718009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 Processor -- start 2023-10-10T17:47:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 -- start start 2023-10-10T17:47:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe23ed66b0 0x55fe240845b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe2407e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe2407eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe23f80000 con 0x55fe24084af0 2023-10-10T17:47:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe23fd2840 con 0x55fe24084f60 2023-10-10T17:47:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.579+0000 7f073ef66b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe23fcbcd0 con 0x55fe23ed66b0 2023-10-10T17:47:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe2407e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f0730b76700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe23ed66b0 0x55fe240845b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe2407e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46780/0 (socket says 172.21.15.78:46780) 2023-10-10T17:47:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 -- 172.21.15.78:0/4004133360 learned_addr learned my addr 172.21.15.78:0/4004133360 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f0731377700 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe2407eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 -- 172.21.15.78:0/4004133360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe23ed66b0 msgr2=0x55fe240845b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe23ed66b0 0x55fe240845b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 -- 172.21.15.78:0/4004133360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 msgr2=0x55fe2407eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe2407eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 -- 172.21.15.78:0/4004133360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe23e4d5d0 con 0x55fe24084af0 2023-10-10T17:47:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f0730b76700 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe23ed66b0 0x55fe240845b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:47:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f072bfff700 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe2407e970 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0718011ca0 tx=0x7f0718014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9844d29d3bd97a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:43.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f0716ffd700 1 -- 172.21.15.78:0/4004133360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07180119b0 con 0x55fe24084af0 2023-10-10T17:47:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f0716ffd700 1 -- 172.21.15.78:0/4004133360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07180044e0 con 0x55fe24084af0 2023-10-10T17:47:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.580+0000 7f0716ffd700 1 -- 172.21.15.78:0/4004133360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07180039e0 con 0x55fe24084af0 2023-10-10T17:47:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 -- 172.21.15.78:0/4004133360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 msgr2=0x55fe2407e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe2407e970 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f0718011ca0 tx=0x7f0718014000 comp rx=0 tx=0).stop 2023-10-10T17:47:43.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 -- 172.21.15.78:0/4004133360 shutdown_connections 2023-10-10T17:47:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe23ed66b0 0x55fe240845b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe2407eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 --2- 172.21.15.78:0/4004133360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe2407e970 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 -- 172.21.15.78:0/4004133360 >> 172.21.15.78:0/4004133360 conn(0x55fe2406ff50 msgr2=0x55fe24075270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:43.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 -- 172.21.15.78:0/4004133360 shutdown_connections 2023-10-10T17:47:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.581+0000 7f073ef66b80 1 -- 172.21.15.78:0/4004133360 wait complete. 2023-10-10T17:47:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 -- start start 2023-10-10T17:47:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 Processor -- start 2023-10-10T17:47:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe23ed66b0 0x55fe2408ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084af0 0x55fe240880b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084f60 0x55fe24088670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe23f6d880 con 0x55fe23ed66b0 2023-10-10T17:47:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe23f6d520 con 0x55fe24084af0 2023-10-10T17:47:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe23e4f8c0 con 0x55fe24084f60 2023-10-10T17:47:43.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f073ef66b80 10 client.0 _ll_register_callbacks cb 0x55fe23f8b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f0730b76700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe23ed66b0 0x55fe2408ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f0730b76700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe23ed66b0 0x55fe2408ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46784/0 (socket says 172.21.15.78:46784) 2023-10-10T17:47:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f0731377700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084f60 0x55fe24088670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f0730b76700 1 -- 172.21.15.78:0/2755269691 learned_addr learned my addr 172.21.15.78:0/2755269691 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:43.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f072bfff700 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084af0 0x55fe240880b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f0730b76700 1 -- 172.21.15.78:0/2755269691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084f60 msgr2=0x55fe24088670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f0730b76700 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084f60 0x55fe24088670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.582+0000 7f0730b76700 1 -- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084af0 msgr2=0x55fe240880b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f0730b76700 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084af0 0x55fe240880b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f0730b76700 1 -- 172.21.15.78:0/2755269691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0718017070 con 0x55fe23ed66b0 2023-10-10T17:47:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f0731377700 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084f60 0x55fe24088670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:47:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f0730b76700 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe23ed66b0 0x55fe2408ad90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0720096a20 tx=0x7f07200cd190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9489de437d69ec server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f070bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:47:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f070bfff700 1 -- 172.21.15.78:0/2755269691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07200996a0 con 0x55fe23ed66b0 2023-10-10T17:47:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f070bfff700 1 -- 172.21.15.78:0/2755269691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f07200a2400 con 0x55fe23ed66b0 2023-10-10T17:47:43.601 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe2406d710 con 0x55fe23ed66b0 2023-10-10T17:47:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f070bfff700 1 -- 172.21.15.78:0/2755269691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f07200a1f70 con 0x55fe23ed66b0 2023-10-10T17:47:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f073ef66b80 10 client.16560 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55fe23f92130 con 0x55fe23ed66b0 2023-10-10T17:47:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.583+0000 7f070bfff700 1 -- 172.21.15.78:0/2755269691 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07200b8030 con 0x55fe23ed66b0 2023-10-10T17:47:43.602 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f070bfff700 1 -- 172.21.15.78:0/2755269691 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0720094bc0 con 0x55fe23ed66b0 2023-10-10T17:47:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f073ef66b80 10 client.16560 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f073ef66b80 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f073ef66b80 20 client.16560 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe24098420 0x55fe2409a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.603 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f073ef66b80 4 client.16560 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:47:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe23f8e040 con 0x55fe24098420 2023-10-10T17:47:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f072bfff700 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe24098420 0x55fe2409a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f072bfff700 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe24098420 0x55fe2409a830 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8c858162f0068dc server_cookie=7009c557e2d0ce9b in_seq=0 out_seq=0 2023-10-10T17:47:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.584+0000 7f070bfff700 10 client.16560 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:43.604 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f070bfff700 1 -- 172.21.15.78:0/2755269691 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55fe23f8e040 con 0x55fe24098420 2023-10-10T17:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f070bfff700 10 client.16560 handle_command_reply: tid=0 2023-10-10T17:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 client.16560 shutdown 2023-10-10T17:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe23ed66b0 msgr2=0x55fe2408ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe23ed66b0 0x55fe2408ad90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f0720096a20 tx=0x7f07200cd190 comp rx=0 tx=0).stop 2023-10-10T17:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 shutdown_connections 2023-10-10T17:47:43.605 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084f60 0x55fe24088670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe24098420 0x55fe2409a830 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe23ed66b0 0x55fe2408ad90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f070bfff700 0 client.16560 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2755269691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084af0 0x55fe240880b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.606 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 >> 172.21.15.78:0/2755269691 conn(0x55fe2406ff50 msgr2=0x55fe240747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 shutdown_connections 2023-10-10T17:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 1 -- 172.21.15.78:0/2755269691 wait complete. 2023-10-10T17:47:43.607 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.588+0000 7f073ef66b80 20 client.16560 trim_cache size 0 max 16384 2023-10-10T17:47:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 Processor -- start 2023-10-10T17:47:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 -- start start 2023-10-10T17:47:43.612 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe2406ff50 0x55fe2408f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084af0 0x55fe24074f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084f60 0x55fe2408a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe2407e970 con 0x55fe24084f60 2023-10-10T17:47:43.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24086630 con 0x55fe2406ff50 2023-10-10T17:47:43.613 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f073ef66b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24082d80 con 0x55fe24084af0 2023-10-10T17:47:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f0731377700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084f60 0x55fe2408a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f0731377700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084f60 0x55fe2408a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46796/0 (socket says 172.21.15.78:46796) 2023-10-10T17:47:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f0731377700 1 -- 172.21.15.78:0/2246721926 learned_addr learned my addr 172.21.15.78:0/2246721926 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f072bfff700 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084af0 0x55fe24074f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f0730b76700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe2406ff50 0x55fe2408f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.614 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f072bfff700 1 -- 172.21.15.78:0/2246721926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe2406ff50 msgr2=0x55fe2408f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f072bfff700 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe2406ff50 0x55fe2408f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f072bfff700 1 -- 172.21.15.78:0/2246721926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084f60 msgr2=0x55fe2408a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f072bfff700 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084f60 0x55fe2408a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f072bfff700 1 -- 172.21.15.78:0/2246721926 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe240716f0 con 0x55fe24084af0 2023-10-10T17:47:43.615 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f0731377700 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084f60 0x55fe2408a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:47:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f0730b76700 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe2406ff50 0x55fe2408f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:47:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.605+0000 7f072bfff700 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084af0 0x55fe24074f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f071800ef20 tx=0x7f07180119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2a4655063609144 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f07177fe700 1 -- 172.21.15.78:0/2246721926 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071802b040 con 0x55fe24084af0 2023-10-10T17:47:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f07177fe700 1 -- 172.21.15.78:0/2246721926 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f071800ed40 con 0x55fe24084af0 2023-10-10T17:47:43.616 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f07177fe700 1 -- 172.21.15.78:0/2246721926 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07180052a0 con 0x55fe24084af0 2023-10-10T17:47:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f073ef66b80 1 -- 172.21.15.78:0/2246721926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084af0 msgr2=0x55fe24074f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084af0 0x55fe24074f60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f071800ef20 tx=0x7f07180119b0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f073ef66b80 1 -- 172.21.15.78:0/2246721926 shutdown_connections 2023-10-10T17:47:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084f60 0x55fe2408a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.617 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe2406ff50 0x55fe2408f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f073ef66b80 1 --2- 172.21.15.78:0/2246721926 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe24084af0 0x55fe24074f60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.606+0000 7f073ef66b80 1 -- 172.21.15.78:0/2246721926 >> 172.21.15.78:0/2246721926 conn(0x55fe24076d60 msgr2=0x55fe2407ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.607+0000 7f073ef66b80 1 -- 172.21.15.78:0/2246721926 shutdown_connections 2023-10-10T17:47:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.607+0000 7f073ef66b80 1 -- 172.21.15.78:0/2246721926 wait complete. 2023-10-10T17:47:43.618 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.607+0000 7f073ef66b80 1 -- start start 2023-10-10T17:47:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.607+0000 7f073ef66b80 1 Processor -- start 2023-10-10T17:47:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.607+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2406ff50 0x55fe24081100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe24081640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f073ef66b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe24080010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f073ef66b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24081d50 con 0x55fe24084af0 2023-10-10T17:47:43.619 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f073ef66b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe24080550 con 0x55fe24084f60 2023-10-10T17:47:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f073ef66b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe240806c0 con 0x55fe2406ff50 2023-10-10T17:47:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f073ef66b80 10 client.0 _ll_register_callbacks cb 0x55fe23f8b720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe24081640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe24081640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46800/0 (socket says 172.21.15.78:46800) 2023-10-10T17:47:43.620 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 -- 172.21.15.78:0/1001193508 learned_addr learned my addr 172.21.15.78:0/1001193508 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f0730b76700 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2406ff50 0x55fe24081100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f0731377700 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe24080010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 -- 172.21.15.78:0/1001193508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2406ff50 msgr2=0x55fe24081100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2406ff50 0x55fe24081100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.621 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 -- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 msgr2=0x55fe24080010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:43.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe24080010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:43.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 -- 172.21.15.78:0/1001193508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c10ca20 con 0x55fe24084af0 2023-10-10T17:47:43.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f070bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:47:43.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f072bfff700 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe24081640 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0718005950 tx=0x7f071800ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddd844b44bafadb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:43.622 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f070bfff700 1 -- 172.21.15.78:0/1001193508 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718034e50 con 0x55fe24084af0 2023-10-10T17:47:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f070bfff700 1 -- 172.21.15.78:0/1001193508 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f07180340d0 con 0x55fe24084af0 2023-10-10T17:47:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.608+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f07180044e0 con 0x55fe24084af0 2023-10-10T17:47:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f070bfff700 1 -- 172.21.15.78:0/1001193508 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0718034290 con 0x55fe24084af0 2023-10-10T17:47:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 10 client.16566 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:43.623 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55fe23e4d5d0 con 0x55fe24084af0 2023-10-10T17:47:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f070bfff700 1 -- 172.21.15.78:0/1001193508 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07180044e0 con 0x55fe24084af0 2023-10-10T17:47:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f070bfff700 1 -- 172.21.15.78:0/1001193508 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f071800e3b0 con 0x55fe24084af0 2023-10-10T17:47:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 10 client.16566 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 10 client.16566 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:43.624 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 20 client.16566 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe240ecc20 0x55fe240ef030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 4 client.16566 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:47:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fe23f9a600 con 0x55fe240ecc20 2023-10-10T17:47:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.609+0000 7f0730b76700 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe240ecc20 0x55fe240ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.610+0000 7f070bfff700 10 client.16566 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:43.625 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:43.610+0000 7f0730b76700 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe240ecc20 0x55fe240ef030 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2502d3bb252a8e51 server_cookie=a13ab0f92389458c in_seq=0 out_seq=0 2023-10-10T17:47:45.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:47:45.102 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:44 smithi078 ceph-mon[99494]: pgmap v159: 129 pgs: 129 active+clean; 738 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.2 MiB/s wr, 710 op/s 2023-10-10T17:47:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:44 smithi190 ceph-mon[101220]: pgmap v159: 129 pgs: 129 active+clean; 738 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.2 MiB/s wr, 710 op/s 2023-10-10T17:47:45.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:44 smithi157 ceph-mon[100896]: pgmap v159: 129 pgs: 129 active+clean; 738 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 5.2 MiB/s wr, 710 op/s 2023-10-10T17:47:45.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 -- 172.21.15.78:0/3596905905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8418104420 msgr2=0x7f841810c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:45.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 --2- 172.21.15.78:0/3596905905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8418104420 0x7f841810c030 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f8410009fc0 tx=0x7f841000b330 comp rx=0 tx=0).stop 2023-10-10T17:47:45.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 -- 172.21.15.78:0/3596905905 shutdown_connections 2023-10-10T17:47:45.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 --2- 172.21.15.78:0/3596905905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f84180ffb70 0x7f84180fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 --2- 172.21.15.78:0/3596905905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8418104420 0x7f841810c030 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 --2- 172.21.15.78:0/3596905905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8418100510 0x7f8418103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 -- 172.21.15.78:0/3596905905 >> 172.21.15.78:0/3596905905 conn(0x7f8418075890 msgr2=0x7f8418075c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:45.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 -- 172.21.15.78:0/3596905905 shutdown_connections 2023-10-10T17:47:45.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.256+0000 7f841e7d4700 1 -- 172.21.15.78:0/3596905905 wait complete. 2023-10-10T17:47:45.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.257+0000 7f841e7d4700 1 Processor -- start 2023-10-10T17:47:45.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.257+0000 7f841e7d4700 1 -- start start 2023-10-10T17:47:45.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f841e7d4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84180ffb70 0x7f84181133c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f841e7d4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8418100510 0x7f841810f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f841e7d4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8418104420 0x7f841810fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f841e7d4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418110490 con 0x7f84180ffb70 2023-10-10T17:47:45.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f841e7d4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84181105d0 con 0x7f8418100510 2023-10-10T17:47:45.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f841e7d4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418110760 con 0x7f8418104420 2023-10-10T17:47:45.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f84177fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8418100510 0x7f841810f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f84177fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8418100510 0x7f841810f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48534/0 (socket says 172.21.15.78:48534) 2023-10-10T17:47:45.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.258+0000 7f84177fe700 1 -- 172.21.15.78:0/1972813233 learned_addr learned my addr 172.21.15.78:0/1972813233 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:45.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.259+0000 7f841cd71700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8418104420 0x7f841810fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.259+0000 7f8417fff700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84180ffb70 0x7f84181133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841cd71700 1 -- 172.21.15.78:0/1972813233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8418100510 msgr2=0x7f841810f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841cd71700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8418100510 0x7f841810f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841cd71700 1 -- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84180ffb70 msgr2=0x7f84181133c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:45.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841cd71700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84180ffb70 0x7f84181133c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841cd71700 1 -- 172.21.15.78:0/1972813233 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8410009c40 con 0x7f8418104420 2023-10-10T17:47:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841cd71700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8418104420 0x7f841810fe90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8410015000 tx=0x7f84100094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f84157fa700 1 -- 172.21.15.78:0/1972813233 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841000e040 con 0x7f8418104420 2023-10-10T17:47:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f84157fa700 1 -- 172.21.15.78:0/1972813233 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f841000bbf0 con 0x7f8418104420 2023-10-10T17:47:45.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f84157fa700 1 -- 172.21.15.78:0/1972813233 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410005040 con 0x7f8418104420 2023-10-10T17:47:45.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f841810e1a0 con 0x7f8418104420 2023-10-10T17:47:45.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.260+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f841810e010 con 0x7f8418104420 2023-10-10T17:47:45.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 Processor -- start 2023-10-10T17:47:45.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 -- start start 2023-10-10T17:47:45.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122edf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122f28b0 0x5569122f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569121f3060 con 0x5569122edb60 2023-10-10T17:47:45.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569122458a0 con 0x5569122f28b0 2023-10-10T17:47:45.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.265+0000 7f842a960b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691223ed30 con 0x5569122ee470 2023-10-10T17:47:45.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.266+0000 7f84177fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122f28b0 0x5569122f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.267+0000 7f84177fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122f28b0 0x5569122f2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48542/0 (socket says 172.21.15.78:48542) 2023-10-10T17:47:45.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.267+0000 7f84177fe700 1 -- 172.21.15.78:0/4212746725 learned_addr learned my addr 172.21.15.78:0/4212746725 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:45.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.267+0000 7f8417fff700 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.267+0000 7f84177fe700 1 -- 172.21.15.78:0/4212746725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 msgr2=0x5569122f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:45.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.267+0000 7f84177fe700 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.268+0000 7f84177fe700 1 -- 172.21.15.78:0/4212746725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 msgr2=0x5569122edf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:45.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.268+0000 7f84177fe700 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122edf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.268+0000 7f84177fe700 1 -- 172.21.15.78:0/4212746725 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5569120c0630 con 0x5569122f28b0 2023-10-10T17:47:45.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.268+0000 7f84177fe700 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122f28b0 0x5569122f2d00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8408009780 tx=0x7f8408011720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96a68fac47ad3370 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:45.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.268+0000 7f840e7fc700 1 -- 172.21.15.78:0/4212746725 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408016070 con 0x5569122f28b0 2023-10-10T17:47:45.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.268+0000 7f840e7fc700 1 -- 172.21.15.78:0/4212746725 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84080044e0 con 0x5569122f28b0 2023-10-10T17:47:45.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.268+0000 7f840e7fc700 1 -- 172.21.15.78:0/4212746725 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408003c60 con 0x5569122f28b0 2023-10-10T17:47:45.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.269+0000 7f84157fa700 1 -- 172.21.15.78:0/1972813233 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f841001d470 con 0x7f8418104420 2023-10-10T17:47:45.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.269+0000 7f84157fa700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f84040a1340 0x7f84040a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.269+0000 7f842a960b80 1 -- 172.21.15.78:0/4212746725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122f28b0 msgr2=0x5569122f2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:45.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.269+0000 7f842a960b80 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122f28b0 0x5569122f2d00 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8408009780 tx=0x7f8408011720 comp rx=0 tx=0).stop 2023-10-10T17:47:45.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.270+0000 7f84157fa700 1 -- 172.21.15.78:0/1972813233 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f84100cdc80 con 0x7f8418104420 2023-10-10T17:47:45.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.270+0000 7f8417fff700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f84040a1340 0x7f84040a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.270+0000 7f8417fff700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f84040a1340 0x7f84040a37f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8400009b80 tx=0x7f8400009190 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:45.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 -- 172.21.15.78:0/4212746725 shutdown_connections 2023-10-10T17:47:45.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122f28b0 0x5569122f2d00 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122edf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 --2- 172.21.15.78:0/4212746725 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 -- 172.21.15.78:0/4212746725 >> 172.21.15.78:0/4212746725 conn(0x5569122e3360 msgr2=0x5569122e8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:45.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 -- 172.21.15.78:0/4212746725 shutdown_connections 2023-10-10T17:47:45.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 -- 172.21.15.78:0/4212746725 wait complete. 2023-10-10T17:47:45.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 -- start start 2023-10-10T17:47:45.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.276+0000 7f842a960b80 1 Processor -- start 2023-10-10T17:47:45.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f842a960b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122edb60 0x5569122fa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f842a960b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122f72f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f842a960b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122f28b0 0x5569122f7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f842a960b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569121e08e0 con 0x5569122f28b0 2023-10-10T17:47:45.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f842a960b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569121e0580 con 0x5569122ee470 2023-10-10T17:47:45.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f842a960b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569120c2920 con 0x5569122edb60 2023-10-10T17:47:45.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f842a960b80 10 client.0 _ll_register_callbacks cb 0x5569121fe780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:45.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f841cd71700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122f28b0 0x5569122f7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f841cd71700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122f28b0 0x5569122f7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46826/0 (socket says 172.21.15.78:46826) 2023-10-10T17:47:45.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f841cd71700 1 -- 172.21.15.78:0/2583165288 learned_addr learned my addr 172.21.15.78:0/2583165288 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:45.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.277+0000 7f84177fe700 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122f72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f841cd71700 1 -- 172.21.15.78:0/2583165288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122edb60 msgr2=0x5569122fa0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:45.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f841cd71700 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122edb60 0x5569122fa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f841cd71700 1 -- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 msgr2=0x5569122f72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:45.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f841cd71700 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122f72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:45.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f841cd71700 1 -- 172.21.15.78:0/2583165288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8408009ae0 con 0x5569122f28b0 2023-10-10T17:47:45.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f841cd71700 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122f28b0 0x5569122f7a30 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8410097570 tx=0x7f841009a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72594614bcad005 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f83f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:47:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f83f77fe700 1 -- 172.21.15.78:0/2583165288 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841009ab00 con 0x5569122f28b0 2023-10-10T17:47:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f83f77fe700 1 -- 172.21.15.78:0/2583165288 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f841009eb30 con 0x5569122f28b0 2023-10-10T17:47:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5569122e0b20 con 0x5569122f28b0 2023-10-10T17:47:45.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f83f77fe700 1 -- 172.21.15.78:0/2583165288 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84100a2ec0 con 0x5569122f28b0 2023-10-10T17:47:45.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f842a960b80 10 client.16575 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:45.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.278+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x556912205190 con 0x5569122f28b0 2023-10-10T17:47:45.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f83f77fe700 1 -- 172.21.15.78:0/2583165288 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5569122e0b20 con 0x5569122f28b0 2023-10-10T17:47:45.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f83f77fe700 1 -- 172.21.15.78:0/2583165288 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8410099ff0 con 0x5569122f28b0 2023-10-10T17:47:45.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f842a960b80 10 client.16575 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:45.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f842a960b80 10 client.16575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:45.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f842a960b80 20 client.16575 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:45.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f842a960b80 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556912307520 0x556912309930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:45.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f842a960b80 4 client.16575 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:47:45.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5569122010a0 con 0x556912307520 2023-10-10T17:47:45.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.279+0000 7f8417fff700 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556912307520 0x556912309930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:45.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.280+0000 7f83f77fe700 10 client.16575 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:45.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:45.280+0000 7f8417fff700 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556912307520 0x556912309930 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3001e479d8772dcf server_cookie=f85862cd841f88c7 in_seq=0 out_seq=0 2023-10-10T17:47:47.007 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f070bfff700 1 -- 172.21.15.78:0/1001193508 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fe23f9a600 con 0x55fe240ecc20 2023-10-10T17:47:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f070bfff700 10 client.16566 handle_command_reply: tid=0 2023-10-10T17:47:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 client.16566 shutdown 2023-10-10T17:47:47.008 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 msgr2=0x55fe24081640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe24081640 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0718005950 tx=0x7f071800ebc0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 shutdown_connections 2023-10-10T17:47:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55fe2406ff50 0x55fe24081100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55fe240ecc20 0x55fe240ef030 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.011 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55fe24084f60 0x55fe24080010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.011 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 --2- 172.21.15.78:0/1001193508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55fe24084af0 0x55fe24081640 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.011 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f070bfff700 0 client.16566 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:47.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 >> 172.21.15.78:0/1001193508 conn(0x55fe24076d60 msgr2=0x55fe24082210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:47.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f83f77fe700 1 -- 172.21.15.78:0/2583165288 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5569122010a0 con 0x556912307520 2023-10-10T17:47:47.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f83f77fe700 10 client.16575 handle_command_reply: tid=0 2023-10-10T17:47:47.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f842a960b80 1 client.16575 shutdown 2023-10-10T17:47:47.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.005+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 shutdown_connections 2023-10-10T17:47:47.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122f28b0 msgr2=0x5569122f7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122f28b0 0x5569122f7a30 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8410097570 tx=0x7f841009a6b0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 shutdown_connections 2023-10-10T17:47:47.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122edb60 0x5569122fa0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556912307520 0x556912309930 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122f28b0 0x5569122f7a30 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 --2- 172.21.15.78:0/2583165288 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122f72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 >> 172.21.15.78:0/2583165288 conn(0x5569122e3360 msgr2=0x5569122f3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:47.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f83f77fe700 0 client.16575 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:47.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 shutdown_connections 2023-10-10T17:47:47.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 1 -- 172.21.15.78:0/2583165288 wait complete. 2023-10-10T17:47:47.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.006+0000 7f842a960b80 20 client.16575 trim_cache size 0 max 16384 2023-10-10T17:47:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f073ef66b80 1 -- 172.21.15.78:0/1001193508 wait complete. 2023-10-10T17:47:47.019 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f073ef66b80 20 client.16566 trim_cache size 0 max 16384 2023-10-10T17:47:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:47:47.020 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:47:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:47:47.021 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:47:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f071c0a12c0 msgr2=0x7f071c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f071c0a12c0 0x7f071c0a3770 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f072c197d00 tx=0x7f0718009260 comp rx=0 tx=0).stop 2023-10-10T17:47:47.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c0ffaf0 msgr2=0x7f072c1966f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c0ffaf0 0x7f072c1966f0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f072000e9a0 tx=0x7f072000ecb0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 shutdown_connections 2023-10-10T17:47:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f072c104610 0x7f072c199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f071c0a12c0 0x7f071c0a3770 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f072c0ffaf0 0x7f072c1966f0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 --2- 172.21.15.78:0/344714664 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f072c100490 0x7f072c196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.008+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 >> 172.21.15.78:0/344714664 conn(0x7f072c075810 msgr2=0x7f072c0fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.009+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 shutdown_connections 2023-10-10T17:47:47.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:47:47.009+0000 7f0732dda700 1 -- 172.21.15.78:0/344714664 wait complete. 2023-10-10T17:47:47.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 Processor -- start 2023-10-10T17:47:47.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 -- start start 2023-10-10T17:47:47.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122e3360 0x5569122fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:47.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122edb60 0x5569122f9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:47.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:47.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569122f1260 con 0x5569122e3360 2023-10-10T17:47:47.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569122f1480 con 0x5569122edb60 2023-10-10T17:47:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.026+0000 7f842a960b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569122f8780 con 0x5569122ee470 2023-10-10T17:47:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122e3360 0x5569122fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:47.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122e3360 0x5569122fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46834/0 (socket says 172.21.15.78:46834) 2023-10-10T17:47:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 -- 172.21.15.78:0/978071080 learned_addr learned my addr 172.21.15.78:0/978071080 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:47.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f84177fe700 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122edb60 0x5569122f9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f841cd71700 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 -- 172.21.15.78:0/978071080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 msgr2=0x5569122f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:47.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 -- 172.21.15.78:0/978071080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122edb60 msgr2=0x5569122f9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122edb60 0x5569122f9c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 -- 172.21.15.78:0/978071080 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556912204ee0 con 0x5569122e3360 2023-10-10T17:47:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f8417fff700 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122e3360 0x5569122fe9a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x55691230a1e0 tx=0x7f8400004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78731a8db80f15c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f840effd700 1 -- 172.21.15.78:0/978071080 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400026160 con 0x5569122e3360 2023-10-10T17:47:47.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f840effd700 1 -- 172.21.15.78:0/978071080 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f840002a070 con 0x5569122e3360 2023-10-10T17:47:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.027+0000 7f840effd700 1 -- 172.21.15.78:0/978071080 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400014440 con 0x5569122e3360 2023-10-10T17:47:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 -- 172.21.15.78:0/978071080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122e3360 msgr2=0x5569122fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122e3360 0x5569122fe9a0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x55691230a1e0 tx=0x7f8400004860 comp rx=0 tx=0).stop 2023-10-10T17:47:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 -- 172.21.15.78:0/978071080 shutdown_connections 2023-10-10T17:47:47.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122ee470 0x5569122f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122e3360 0x5569122fe9a0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 --2- 172.21.15.78:0/978071080 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122edb60 0x5569122f9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 -- 172.21.15.78:0/978071080 >> 172.21.15.78:0/978071080 conn(0x5569122f7ba0 msgr2=0x556912303c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:47.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 -- 172.21.15.78:0/978071080 shutdown_connections 2023-10-10T17:47:47.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.028+0000 7f842a960b80 1 -- 172.21.15.78:0/978071080 wait complete. 2023-10-10T17:47:47.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 -- start start 2023-10-10T17:47:47.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 Processor -- start 2023-10-10T17:47:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122e3360 0x5569122feba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122e7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122e7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:47.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569122e81a0 con 0x5569122edb60 2023-10-10T17:47:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5569122e8310 con 0x5569122ee470 2023-10-10T17:47:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55691235b3e0 con 0x5569122e3360 2023-10-10T17:47:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f842a960b80 10 client.0 _ll_register_callbacks cb 0x7f841009a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:47:47.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.029+0000 7f84177fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122e7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f8417fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122e3360 0x5569122feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122e7550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46844/0 (socket says 172.21.15.78:46844) 2023-10-10T17:47:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f841cd71700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122e7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:47.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 -- 172.21.15.78:0/1772952700 learned_addr learned my addr 172.21.15.78:0/1772952700 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:47:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 -- 172.21.15.78:0/1772952700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122e3360 msgr2=0x5569122feba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122e3360 0x5569122feba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 -- 172.21.15.78:0/1772952700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 msgr2=0x5569122e7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:47:47.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122e7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 -- 172.21.15.78:0/1772952700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8400011810 con 0x5569122edb60 2023-10-10T17:47:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f84177fe700 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122e7550 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8408009dd0 tx=0x7f8408005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967b0fdff8b49ca6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:47:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f83f77fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:47:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f83f77fe700 1 -- 172.21.15.78:0/1772952700 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8408016070 con 0x5569122edb60 2023-10-10T17:47:47.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f83f77fe700 1 -- 172.21.15.78:0/1772952700 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f8408000e40 con 0x5569122edb60 2023-10-10T17:47:47.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84080044e0 con 0x5569122edb60 2023-10-10T17:47:47.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f83f77fe700 1 -- 172.21.15.78:0/1772952700 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8408004ba0 con 0x5569122edb60 2023-10-10T17:47:47.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f842a960b80 10 client.16581 fetch_fsmap learned FSMap version 32 2023-10-10T17:47:47.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.030+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x556912055c60 con 0x5569122edb60 2023-10-10T17:47:47.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f83f77fe700 1 -- 172.21.15.78:0/1772952700 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f84080044e0 con 0x5569122edb60 2023-10-10T17:47:47.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f83f77fe700 1 -- 172.21.15.78:0/1772952700 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8408011440 con 0x5569122edb60 2023-10-10T17:47:47.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f842a960b80 10 client.16581 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:47:47.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f842a960b80 10 client.16581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:47:47.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f842a960b80 20 client.16581 populate_metadata read hostname 'smithi078' 2023-10-10T17:47:47.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f842a960b80 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569122f28b0 0x55691235ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:47:47.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f842a960b80 4 client.16581 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:47:47.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55691220d660 con 0x5569122f28b0 2023-10-10T17:47:47.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f8417fff700 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569122f28b0 0x55691235ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:47:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f83f77fe700 10 client.16581 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:47.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.031+0000 7f8417fff700 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569122f28b0 0x55691235ea50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f989c0711f4f8769 server_cookie=6396e873029500f in_seq=0 out_seq=0 2023-10-10T17:47:47.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f83f77fe700 1 -- 172.21.15.78:0/1772952700 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55691220d660 con 0x5569122f28b0 2023-10-10T17:47:47.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f83f77fe700 10 client.16581 handle_command_reply: tid=0 2023-10-10T17:47:47.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 client.16581 shutdown 2023-10-10T17:47:47.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 msgr2=0x5569122e7550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122e7550 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8408009dd0 tx=0x7f8408005750 comp rx=0 tx=0).stop 2023-10-10T17:47:47.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 shutdown_connections 2023-10-10T17:47:47.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5569122e3360 0x5569122feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5569122f28b0 0x55691235ea50 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5569122ee470 0x5569122e7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f83f77fe700 0 client.16581 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:47:47.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 --2- 172.21.15.78:0/1772952700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5569122edb60 0x5569122e7550 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 >> 172.21.15.78:0/1772952700 conn(0x5569122f7ba0 msgr2=0x5569122ec960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:47.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 shutdown_connections 2023-10-10T17:47:47.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.036+0000 7f842a960b80 1 -- 172.21.15.78:0/1772952700 wait complete. 2023-10-10T17:47:47.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f842a960b80 20 client.16581 trim_cache size 0 max 16384 2023-10-10T17:47:47.062 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:47:47.063 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:47:47.063 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:47:47.063 INFO:teuthology.orchestra.run.smithi078.stdout: "5ecf957f-7c91-40df-8738-d2beae2feba1": { 2023-10-10T17:47:47.064 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:47:47.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:47:47.064 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:47:47.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:47:47.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:47:47.065 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:47:47.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f84040a1340 msgr2=0x7f84040a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f84040a1340 0x7f84040a37f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8400009b80 tx=0x7f8400009190 comp rx=0 tx=0).stop 2023-10-10T17:47:47.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8418104420 msgr2=0x7f841810fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:47:47.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8418104420 0x7f841810fe90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8410015000 tx=0x7f84100094a0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 shutdown_connections 2023-10-10T17:47:47.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8418104420 0x7f841810fe90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f84040a1340 0x7f84040a37f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84180ffb70 0x7f84181133c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 --2- 172.21.15.78:0/1972813233 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8418100510 0x7f841810f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:47:47.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 >> 172.21.15.78:0/1972813233 conn(0x7f8418075890 msgr2=0x7f8418102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:47:47.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.037+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 shutdown_connections 2023-10-10T17:47:47.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:47:47.038+0000 7f841e7d4700 1 -- 172.21.15.78:0/1972813233 wait complete. 2023-10-10T17:47:47.072 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:46 smithi078 ceph-mon[99494]: pgmap v160: 129 pgs: 129 active+clean; 738 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 486 op/s 2023-10-10T17:47:47.073 INFO:tasks.cephfs.filesystem:scrub status for tag:5ecf957f-7c91-40df-8738-d2beae2feba1 - {'path': '/', 'tag': '5ecf957f-7c91-40df-8738-d2beae2feba1', 'options': 'recursive,force'} 2023-10-10T17:47:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:46 smithi190 ceph-mon[101220]: pgmap v160: 129 pgs: 129 active+clean; 738 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 486 op/s 2023-10-10T17:47:47.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:46 smithi157 ceph-mon[100896]: pgmap v160: 129 pgs: 129 active+clean; 738 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 3.7 MiB/s wr, 486 op/s 2023-10-10T17:47:48.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:47 smithi078 ceph-mon[99494]: pgmap v161: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 803 op/s 2023-10-10T17:47:48.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:47 smithi190 ceph-mon[101220]: pgmap v161: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 803 op/s 2023-10-10T17:47:48.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:47 smithi157 ceph-mon[100896]: pgmap v161: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 5.4 MiB/s wr, 803 op/s 2023-10-10T17:47:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:50 smithi157 ceph-mon[100896]: pgmap v162: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 573 op/s 2023-10-10T17:47:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:50 smithi078 ceph-mon[99494]: pgmap v162: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 573 op/s 2023-10-10T17:47:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:50 smithi190 ceph-mon[101220]: pgmap v162: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 573 op/s 2023-10-10T17:47:51.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:51 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:51 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:51 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:51.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:51 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:52.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:51 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:52.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:51 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:52.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:51 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:52.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:51 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:52.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:51 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:52.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:51 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:52.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:51 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:52.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:51 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:52 smithi157 ceph-mon[100896]: pgmap v163: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 596 op/s 2023-10-10T17:47:53.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:52 smithi078 ceph-mon[99494]: pgmap v163: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 596 op/s 2023-10-10T17:47:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:52 smithi190 ceph-mon[101220]: pgmap v163: 129 pgs: 129 active+clean; 753 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.5 MiB/s wr, 596 op/s 2023-10-10T17:47:54.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:54 smithi157 ceph-mon[100896]: pgmap v164: 129 pgs: 129 active+clean; 748 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 864 op/s 2023-10-10T17:47:55.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:54 smithi078 ceph-mon[99494]: pgmap v164: 129 pgs: 129 active+clean; 748 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 864 op/s 2023-10-10T17:47:55.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:54 smithi190 ceph-mon[101220]: pgmap v164: 129 pgs: 129 active+clean; 748 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 4.5 MiB/s wr, 864 op/s 2023-10-10T17:47:56.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:56 smithi157 ceph-mon[100896]: pgmap v165: 129 pgs: 129 active+clean; 748 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 632 op/s 2023-10-10T17:47:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:56 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:47:57.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:56 smithi078 ceph-mon[99494]: pgmap v165: 129 pgs: 129 active+clean; 748 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 632 op/s 2023-10-10T17:47:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:56 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:47:57.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:56 smithi190 ceph-mon[101220]: pgmap v165: 129 pgs: 129 active+clean; 748 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.3 MiB/s wr, 632 op/s 2023-10-10T17:47:57.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:56 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:47:57.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:57 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:47:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:57 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:47:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:57 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:47:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:57 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:47:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:57 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:47:58.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:57 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:47:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:57 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:47:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:57 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:47:58.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:57 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:47:58.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:58 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:47:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:58 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:47:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:58 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:47:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:58 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:47:58.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:58 smithi157 ceph-mon[100896]: pgmap v166: 129 pgs: 129 active+clean; 758 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.8 MiB/s wr, 844 op/s 2023-10-10T17:47:58.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:58 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:47:58.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:47:58 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:58 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:47:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:58 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:47:59.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:58 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:47:59.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:58 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:47:59.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:58 smithi078 ceph-mon[99494]: pgmap v166: 129 pgs: 129 active+clean; 758 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.8 MiB/s wr, 844 op/s 2023-10-10T17:47:59.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:58 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:47:59.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:47:58 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:47:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:58 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:47:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:58 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:47:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:58 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:47:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:58 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:47:59.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:58 smithi190 ceph-mon[101220]: pgmap v166: 129 pgs: 129 active+clean; 758 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 4.8 MiB/s wr, 844 op/s 2023-10-10T17:47:59.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:58 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:47:59.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:47:58 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:00.220 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.220+0000 7f5a92aa4700 1 -- 172.21.15.78:0/4046271757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a8c107d60 msgr2=0x7f5a8c108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:00.221 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.220+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/4046271757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a8c107d60 0x7f5a8c108130 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009990 tx=0x7f5a7c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.220+0000 7f5a92aa4700 1 -- 172.21.15.78:0/4046271757 shutdown_connections 2023-10-10T17:48:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.220+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/4046271757 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a8c107d60 0x7f5a8c108130 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c009990 tx=0x7f5a7c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.220+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/4046271757 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a8c1111d0 0x7f5a8c113660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.220+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/4046271757 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 0x7f5a8c110b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.220+0000 7f5a92aa4700 1 -- 172.21.15.78:0/4046271757 >> 172.21.15.78:0/4046271757 conn(0x7f5a8c06ca10 msgr2=0x7f5a8c0716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.227+0000 7f5a92aa4700 1 -- 172.21.15.78:0/4046271757 shutdown_connections 2023-10-10T17:48:00.234 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.234+0000 7f5a92aa4700 1 -- 172.21.15.78:0/4046271757 wait complete. 2023-10-10T17:48:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 Processor -- start 2023-10-10T17:48:00.235 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 -- start start 2023-10-10T17:48:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 0x7f5a8c11cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a8c1111d0 0x7f5a8c116b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a8c11d220 0x7f5a8c117170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c117940 con 0x7f5a8c108670 2023-10-10T17:48:00.236 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c1ba650 con 0x7f5a8c11d220 2023-10-10T17:48:00.237 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.235+0000 7f5a92aa4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a8c1ba790 con 0x7f5a8c1111d0 2023-10-10T17:48:00.240 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.240+0000 7f5a91041700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a8c11d220 0x7f5a8c117170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.245+0000 7f5a90840700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 0x7f5a8c11cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.245 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.245+0000 7f5a90840700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 0x7f5a8c11cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46856/0 (socket says 172.21.15.78:46856) 2023-10-10T17:48:00.246 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.245+0000 7f5a90840700 1 -- 172.21.15.78:0/1319042245 learned_addr learned my addr 172.21.15.78:0/1319042245 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.251+0000 7f5a90840700 1 -- 172.21.15.78:0/1319042245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a8c1111d0 msgr2=0x7f5a8c116b60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T17:48:00.251 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.251+0000 7f5a90840700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a8c1111d0 0x7f5a8c116b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.251+0000 7f5a90840700 1 -- 172.21.15.78:0/1319042245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a8c11d220 msgr2=0x7f5a8c117170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.251+0000 7f5a90840700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a8c11d220 0x7f5a8c117170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.251+0000 7f5a90840700 1 -- 172.21.15.78:0/1319042245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c0095f0 con 0x7f5a8c108670 2023-10-10T17:48:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.252+0000 7f5a90840700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 0x7f5a8c11cb80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0098e0 tx=0x7f5a7c003c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.253+0000 7f5a89ffb700 1 -- 172.21.15.78:0/1319042245 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c01d070 con 0x7f5a8c108670 2023-10-10T17:48:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.253+0000 7f5a89ffb700 1 -- 172.21.15.78:0/1319042245 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a7c003e20 con 0x7f5a8c108670 2023-10-10T17:48:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.253+0000 7f5a89ffb700 1 -- 172.21.15.78:0/1319042245 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00fe10 con 0x7f5a8c108670 2023-10-10T17:48:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.254+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c1156f0 con 0x7f5a8c108670 2023-10-10T17:48:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.254+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c115560 con 0x7f5a8c108670 2023-10-10T17:48:00.263 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.262+0000 7f5a89ffb700 1 -- 172.21.15.78:0/1319042245 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5a7c0052d0 con 0x7f5a8c108670 2023-10-10T17:48:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.263+0000 7f5a89ffb700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a740a14c0 0x7f5a740a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.265 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.264+0000 7f5a89ffb700 1 -- 172.21.15.78:0/1319042245 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5a7c0cd950 con 0x7f5a8c108670 2023-10-10T17:48:00.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.267+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a740a14c0 0x7f5a740a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.268 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.267+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a740a14c0 0x7f5a740a3970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c118350 tx=0x7f5a80006cb0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:00.269 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.268+0000 7f5a9ec30b80 1 Processor -- start 2023-10-10T17:48:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a9ec30b80 1 -- start start 2023-10-10T17:48:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 0x563fc2f0b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.273 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0b720 0x563fc2f05560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 0x563fc2f05b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2e0b000 con 0x563fc2f0b720 2023-10-10T17:48:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2e5d840 con 0x563fc2d616b0 2023-10-10T17:48:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2e56cd0 con 0x563fc2f0bb90 2023-10-10T17:48:00.274 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.272+0000 7f5a8bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0b720 0x563fc2f05560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a90840700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 0x563fc2f0b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0b720 0x563fc2f05560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46864/0 (socket says 172.21.15.78:46864) 2023-10-10T17:48:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 -- 172.21.15.78:0/2552096500 learned_addr learned my addr 172.21.15.78:0/2552096500 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:00.275 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 -- 172.21.15.78:0/2552096500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 msgr2=0x563fc2f05b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:48:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/2552096500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 0x563fc2f05b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 -- 172.21.15.78:0/2552096500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 msgr2=0x563fc2f0b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/2552096500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 0x563fc2f0b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.276 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 -- 172.21.15.78:0/2552096500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563fc2cd85d0 con 0x563fc2f0b720 2023-10-10T17:48:00.277 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.273+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/2552096500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0b720 0x563fc2f05560 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5a80015940 tx=0x7f5a8000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82225d2933c6000 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:00.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.289+0000 7f5a72ffd700 1 -- 172.21.15.78:0/2552096500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8001a070 con 0x563fc2f0b720 2023-10-10T17:48:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.289+0000 7f5a72ffd700 1 -- 172.21.15.78:0/2552096500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a800129b0 con 0x563fc2f0b720 2023-10-10T17:48:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.289+0000 7f5a72ffd700 1 -- 172.21.15.78:0/2552096500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a800138e0 con 0x563fc2f0b720 2023-10-10T17:48:00.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.290+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/2552096500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0b720 msgr2=0x563fc2f05560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.290+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/2552096500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0b720 0x563fc2f05560 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5a80015940 tx=0x7f5a8000e000 comp rx=0 tx=0).stop 2023-10-10T17:48:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.290+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/2552096500 shutdown_connections 2023-10-10T17:48:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.290+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/2552096500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 0x563fc2f05b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.291 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.290+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/2552096500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 0x563fc2f0b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.290+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/2552096500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0b720 0x563fc2f05560 secure :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5a80015940 tx=0x7f5a8000e000 comp rx=0 tx=0).stop 2023-10-10T17:48:00.292 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.290+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/2552096500 >> 172.21.15.78:0/2552096500 conn(0x563fc2efaf50 msgr2=0x563fc2f00270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:00.308 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.308+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/2552096500 shutdown_connections 2023-10-10T17:48:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.308+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/2552096500 wait complete. 2023-10-10T17:48:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.308+0000 7f5a9ec30b80 1 -- start start 2023-10-10T17:48:00.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.308+0000 7f5a9ec30b80 1 Processor -- start 2023-10-10T17:48:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2d616b0 0x563fc2f0eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f0f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f0fa70 0x563fc2f1b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2df8880 con 0x563fc2f0bb90 2023-10-10T17:48:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2df8520 con 0x563fc2f0fa70 2023-10-10T17:48:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2cda8c0 con 0x563fc2d616b0 2023-10-10T17:48:00.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a9ec30b80 10 client.0 _ll_register_callbacks cb 0x563fc2e16720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:00.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a90840700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2d616b0 0x563fc2f0eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a90840700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2d616b0 0x563fc2f0eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38396/0 (socket says 172.21.15.78:38396) 2023-10-10T17:48:00.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a90840700 1 -- 172.21.15.78:0/1763592569 learned_addr learned my addr 172.21.15.78:0/1763592569 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:00.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.309+0000 7f5a91041700 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f0fa70 0x563fc2f1b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.311+0000 7f5a91041700 1 -- 172.21.15.78:0/1763592569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2d616b0 msgr2=0x563fc2f0eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:00.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.311+0000 7f5a91041700 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2d616b0 0x563fc2f0eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.311+0000 7f5a91041700 1 -- 172.21.15.78:0/1763592569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 msgr2=0x563fc2f0f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:00.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.311+0000 7f5a91041700 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f0f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.311+0000 7f5a91041700 1 -- 172.21.15.78:0/1763592569 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a800155f0 con 0x563fc2f0fa70 2023-10-10T17:48:00.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.312+0000 7f5a91041700 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f0fa70 0x563fc2f1b160 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400a890 tx=0x7f5a8400ac50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16cc6eb1b3bd6e7e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:00.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.315+0000 7f5a6bfff700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:48:00.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.315+0000 7f5a6bfff700 1 -- 172.21.15.78:0/1763592569 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84010040 con 0x563fc2f0fa70 2023-10-10T17:48:00.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.315+0000 7f5a6bfff700 1 -- 172.21.15.78:0/1763592569 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5a8400e6b0 con 0x563fc2f0fa70 2023-10-10T17:48:00.317 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.316+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563fc2ef8710 con 0x563fc2f0fa70 2023-10-10T17:48:00.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.326+0000 7f5a6bfff700 1 -- 172.21.15.78:0/1763592569 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a8400e870 con 0x563fc2f0fa70 2023-10-10T17:48:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.329+0000 7f5a9ec30b80 10 client.25958 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.329+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x563fc2e1d130 con 0x563fc2f0fa70 2023-10-10T17:48:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a6bfff700 1 -- 172.21.15.78:0/1763592569 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563fc2ef8710 con 0x563fc2f0fa70 2023-10-10T17:48:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a6bfff700 1 -- 172.21.15.78:0/1763592569 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5a84008030 con 0x563fc2f0fa70 2023-10-10T17:48:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a9ec30b80 10 client.25958 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a9ec30b80 10 client.25958 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a9ec30b80 20 client.25958 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f1f7f0 0x563fc2f21c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a9ec30b80 4 client.25958 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.345+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563fc2e19040 con 0x563fc2f1f7f0 2023-10-10T17:48:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.348+0000 7f5a90840700 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f1f7f0 0x563fc2f21c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.349+0000 7f5a6bfff700 10 client.25958 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.349+0000 7f5a90840700 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f1f7f0 0x563fc2f21c00 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=652c60e1746915b5 server_cookie=606d6aecdd599904 in_seq=0 out_seq=0 2023-10-10T17:48:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.364+0000 7f5a6bfff700 1 -- 172.21.15.78:0/1763592569 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563fc2e19040 con 0x563fc2f1f7f0 2023-10-10T17:48:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.364+0000 7f5a6bfff700 10 client.25958 handle_command_reply: tid=0 2023-10-10T17:48:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.364+0000 7f5a9ec30b80 1 client.25958 shutdown 2023-10-10T17:48:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f0fa70 msgr2=0x563fc2f1b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f0fa70 0x563fc2f1b160 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5a8400a890 tx=0x7f5a8400ac50 comp rx=0 tx=0).stop 2023-10-10T17:48:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 shutdown_connections 2023-10-10T17:48:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2d616b0 0x563fc2f0eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f1f7f0 0x563fc2f21c00 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f0fa70 0x563fc2f1b160 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/1763592569 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f0f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 >> 172.21.15.78:0/1763592569 conn(0x563fc2efaf50 msgr2=0x563fc2eff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.365+0000 7f5a6bfff700 0 client.25958 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.371+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 shutdown_connections 2023-10-10T17:48:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.372+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/1763592569 wait complete. 2023-10-10T17:48:00.386 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.379+0000 7f5a9ec30b80 20 client.25958 trim_cache size 0 max 16384 2023-10-10T17:48:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 Processor -- start 2023-10-10T17:48:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 -- start start 2023-10-10T17:48:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 0x563fc2f1b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 0x563fc2f0a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0fa70 0x563fc2f0deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2f1cd40 con 0x563fc2f0fa70 2023-10-10T17:48:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2efcfd0 con 0x563fc2d616b0 2023-10-10T17:48:00.412 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.410+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2f11ea0 con 0x563fc2f0bb90 2023-10-10T17:48:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.413+0000 7f5a91041700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0fa70 0x563fc2f0deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.413+0000 7f5a91041700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0fa70 0x563fc2f0deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46874/0 (socket says 172.21.15.78:46874) 2023-10-10T17:48:00.414 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.413+0000 7f5a91041700 1 -- 172.21.15.78:0/612155142 learned_addr learned my addr 172.21.15.78:0/612155142 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a91041700 1 -- 172.21.15.78:0/612155142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 msgr2=0x563fc2f0a740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:48:00.415 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a91041700 1 --2- 172.21.15.78:0/612155142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 0x563fc2f0a740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a91041700 1 -- 172.21.15.78:0/612155142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 msgr2=0x563fc2f1b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:00.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a91041700 1 --2- 172.21.15.78:0/612155142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 0x563fc2f1b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a91041700 1 -- 172.21.15.78:0/612155142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563fc2e1ce80 con 0x563fc2f0fa70 2023-10-10T17:48:00.416 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a91041700 1 --2- 172.21.15.78:0/612155142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0fa70 0x563fc2f0deb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5a84001220 tx=0x7f5a84009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86615b4829dc9032 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:00.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a737fe700 1 -- 172.21.15.78:0/612155142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a84010050 con 0x563fc2f0fa70 2023-10-10T17:48:00.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a737fe700 1 -- 172.21.15.78:0/612155142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a84007000 con 0x563fc2f0fa70 2023-10-10T17:48:00.417 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.414+0000 7f5a737fe700 1 -- 172.21.15.78:0/612155142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a8400e4b0 con 0x563fc2f0fa70 2023-10-10T17:48:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.415+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/612155142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0fa70 msgr2=0x563fc2f0deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.415+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/612155142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0fa70 0x563fc2f0deb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f5a84001220 tx=0x7f5a84009b40 comp rx=0 tx=0).stop 2023-10-10T17:48:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.416+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/612155142 shutdown_connections 2023-10-10T17:48:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.416+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/612155142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0fa70 0x563fc2f0deb0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.416+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/612155142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2d616b0 0x563fc2f1b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.416+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/612155142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0bb90 0x563fc2f0a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.416+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/612155142 >> 172.21.15.78:0/612155142 conn(0x563fc2efaf50 msgr2=0x563fc2f0d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:00.424 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.422+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/612155142 shutdown_connections 2023-10-10T17:48:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.425+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/612155142 wait complete. 2023-10-10T17:48:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.425+0000 7f5a9ec30b80 1 -- start start 2023-10-10T17:48:00.430 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.425+0000 7f5a9ec30b80 1 Processor -- start 2023-10-10T17:48:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.430+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f14300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.430+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0fa70 0x563fc2f14840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.430+0000 7f5a9ec30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f706a0 0x563fc2f72a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.432 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.430+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2f72fc0 con 0x563fc2f0bb90 2023-10-10T17:48:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.430+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2f73100 con 0x563fc2f706a0 2023-10-10T17:48:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.430+0000 7f5a9ec30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fc2f73270 con 0x563fc2f0fa70 2023-10-10T17:48:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.431+0000 7f5a9ec30b80 10 client.0 _ll_register_callbacks cb 0x7f5a8400a400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:00.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.431+0000 7f5a90840700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f14300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.431+0000 7f5a90840700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f14300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46878/0 (socket says 172.21.15.78:46878) 2023-10-10T17:48:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.431+0000 7f5a90840700 1 -- 172.21.15.78:0/4230921789 learned_addr learned my addr 172.21.15.78:0/4230921789 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.431+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0fa70 0x563fc2f14840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.431+0000 7f5a91041700 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f706a0 0x563fc2f72a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.432+0000 7f5a90840700 1 -- 172.21.15.78:0/4230921789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0fa70 msgr2=0x563fc2f14840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.432+0000 7f5a90840700 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0fa70 0x563fc2f14840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.432+0000 7f5a90840700 1 -- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f706a0 msgr2=0x563fc2f72a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:00.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.432+0000 7f5a90840700 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f706a0 0x563fc2f72a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.432+0000 7f5a90840700 1 -- 172.21.15.78:0/4230921789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563fc2e1d130 con 0x563fc2f0bb90 2023-10-10T17:48:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.432+0000 7f5a90840700 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f14300 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0ba010 tx=0x7f5a7c009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0acfa4d761979da server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.432+0000 7f5a6bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:00.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.433+0000 7f5a6bfff700 1 -- 172.21.15.78:0/4230921789 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c09a7d0 con 0x563fc2f0bb90 2023-10-10T17:48:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.433+0000 7f5a6bfff700 1 -- 172.21.15.78:0/4230921789 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5a7c09e800 con 0x563fc2f0bb90 2023-10-10T17:48:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.433+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a800129b0 con 0x563fc2f0bb90 2023-10-10T17:48:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.433+0000 7f5a6bfff700 1 -- 172.21.15.78:0/4230921789 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a7c0a2b90 con 0x563fc2f0bb90 2023-10-10T17:48:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.433+0000 7f5a9ec30b80 10 client.16593 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:00.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.433+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x563fc2c6dc00 con 0x563fc2f0bb90 2023-10-10T17:48:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.433+0000 7f5a6bfff700 1 -- 172.21.15.78:0/4230921789 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a7c0c8020 con 0x563fc2f0bb90 2023-10-10T17:48:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a6bfff700 1 -- 172.21.15.78:0/4230921789 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5a7c0b88d0 con 0x563fc2f0bb90 2023-10-10T17:48:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a9ec30b80 10 client.16593 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a9ec30b80 10 client.16593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:00.440 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a9ec30b80 20 client.16593 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f74550 0x563fc2f76930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a9ec30b80 4 client.16593 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:48:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563fc2e25600 con 0x563fc2f74550 2023-10-10T17:48:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.434+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f74550 0x563fc2f76930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:00.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.435+0000 7f5a6bfff700 10 client.16593 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:00.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:00.435+0000 7f5a8bfff700 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f74550 0x563fc2f76930 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acbdb1d080d5d626 server_cookie=da4da7695720d169 in_seq=0 out_seq=0 2023-10-10T17:48:00.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:00 smithi157 ceph-mon[100896]: pgmap v167: 129 pgs: 129 active+clean; 758 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 527 op/s 2023-10-10T17:48:01.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:00 smithi078 ceph-mon[99494]: pgmap v167: 129 pgs: 129 active+clean; 758 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 527 op/s 2023-10-10T17:48:01.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:00 smithi190 ceph-mon[101220]: pgmap v167: 129 pgs: 129 active+clean; 758 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 3.2 MiB/s wr, 527 op/s 2023-10-10T17:48:01.642 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a6bfff700 1 -- 172.21.15.78:0/4230921789 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563fc2e25600 con 0x563fc2f74550 2023-10-10T17:48:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a6bfff700 10 client.16593 handle_command_reply: tid=0 2023-10-10T17:48:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 client.16593 shutdown 2023-10-10T17:48:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 msgr2=0x563fc2f14300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:01.643 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f14300 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0ba010 tx=0x7f5a7c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 shutdown_connections 2023-10-10T17:48:01.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x563fc2f74550 0x563fc2f76930 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563fc2f706a0 0x563fc2f72a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563fc2f0bb90 0x563fc2f14300 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.644 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 --2- 172.21.15.78:0/4230921789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563fc2f0fa70 0x563fc2f14840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 >> 172.21.15.78:0/4230921789 conn(0x563fc2efaf50 msgr2=0x563fc2f1afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:01.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.641+0000 7f5a6bfff700 0 client.16593 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:01.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.642+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 shutdown_connections 2023-10-10T17:48:01.645 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.642+0000 7f5a9ec30b80 1 -- 172.21.15.78:0/4230921789 wait complete. 2023-10-10T17:48:01.652 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.652+0000 7f5a9ec30b80 20 client.16593 trim_cache size 0 max 16384 2023-10-10T17:48:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:48:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:48:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:48:01.656 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:48:01.659 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.658+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a740a14c0 msgr2=0x7f5a740a3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.658+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a740a14c0 0x7f5a740a3970 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5a8c118350 tx=0x7f5a80006cb0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.658+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 msgr2=0x7f5a8c11cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:01.660 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.658+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 0x7f5a8c11cb80 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c0098e0 tx=0x7f5a7c003c00 comp rx=0 tx=0).stop 2023-10-10T17:48:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.663+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 shutdown_connections 2023-10-10T17:48:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.663+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a740a14c0 0x7f5a740a3970 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.663+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a8c11d220 0x7f5a8c117170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.663+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a8c108670 0x7f5a8c11cb80 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.663+0000 7f5a92aa4700 1 --2- 172.21.15.78:0/1319042245 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a8c1111d0 0x7f5a8c116b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.663+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 >> 172.21.15.78:0/1319042245 conn(0x7f5a8c06ca10 msgr2=0x7f5a8c071680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:01.665 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.665+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 shutdown_connections 2023-10-10T17:48:01.666 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:01.665+0000 7f5a92aa4700 1 -- 172.21.15.78:0/1319042245 wait complete. 2023-10-10T17:48:02.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:02 smithi157 ceph-mon[100896]: pgmap v168: 129 pgs: 129 active+clean; 761 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 561 op/s 2023-10-10T17:48:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:02 smithi078 ceph-mon[99494]: pgmap v168: 129 pgs: 129 active+clean; 761 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 561 op/s 2023-10-10T17:48:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:02 smithi190 ceph-mon[101220]: pgmap v168: 129 pgs: 129 active+clean; 761 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 3.4 MiB/s wr, 561 op/s 2023-10-10T17:48:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:03 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:03 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:03 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:03.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:03 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:03.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:03 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:03.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:03 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:04.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:03 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:04.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:03 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:04.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:03 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:04.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:03 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:04.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:03 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:04.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:03 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:04.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:03 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:04.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:03 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:04.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:03 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:04.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:03 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:04.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:03 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:04.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:03 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:04.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:04 smithi157 ceph-mon[100896]: pgmap v169: 129 pgs: 129 active+clean; 776 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 916 op/s 2023-10-10T17:48:05.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:04 smithi078 ceph-mon[99494]: pgmap v169: 129 pgs: 129 active+clean; 776 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 916 op/s 2023-10-10T17:48:05.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:04 smithi190 ceph-mon[101220]: pgmap v169: 129 pgs: 129 active+clean; 776 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 916 op/s 2023-10-10T17:48:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:06 smithi078 ceph-mon[99494]: pgmap v170: 129 pgs: 129 active+clean; 776 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 623 op/s 2023-10-10T17:48:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:06 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:06 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:07.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:06 smithi190 ceph-mon[101220]: pgmap v170: 129 pgs: 129 active+clean; 776 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 623 op/s 2023-10-10T17:48:07.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:06 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:07.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:06 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:06 smithi157 ceph-mon[100896]: pgmap v170: 129 pgs: 129 active+clean; 776 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 3.5 MiB/s wr, 623 op/s 2023-10-10T17:48:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:06 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:06 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:09.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:08 smithi078 ceph-mon[99494]: pgmap v171: 129 pgs: 129 active+clean; 768 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 1.03k op/s 2023-10-10T17:48:09.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:08 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:09.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:08 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:08 smithi190 ceph-mon[101220]: pgmap v171: 129 pgs: 129 active+clean; 768 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 1.03k op/s 2023-10-10T17:48:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:08 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:09.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:08 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:09.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:08 smithi157 ceph-mon[100896]: pgmap v171: 129 pgs: 129 active+clean; 768 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 5.2 MiB/s wr, 1.03k op/s 2023-10-10T17:48:09.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:08 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:09.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:08 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:11.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:10 smithi078 ceph-mon[99494]: pgmap v172: 129 pgs: 129 active+clean; 768 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 813 op/s 2023-10-10T17:48:11.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:10 smithi190 ceph-mon[101220]: pgmap v172: 129 pgs: 129 active+clean; 768 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 813 op/s 2023-10-10T17:48:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:10 smithi157 ceph-mon[100896]: pgmap v172: 129 pgs: 129 active+clean; 768 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.7 MiB/s wr, 813 op/s 2023-10-10T17:48:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:12 smithi078 ceph-mon[99494]: pgmap v173: 129 pgs: 129 active+clean; 771 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 848 op/s 2023-10-10T17:48:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:12 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:12 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:13.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:12 smithi190 ceph-mon[101220]: pgmap v173: 129 pgs: 129 active+clean; 771 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 848 op/s 2023-10-10T17:48:13.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:12 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:13.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:12 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:13.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:12 smithi157 ceph-mon[100896]: pgmap v173: 129 pgs: 129 active+clean; 771 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 4.2 MiB/s wr, 848 op/s 2023-10-10T17:48:13.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:12 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:13.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:12 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:14.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.858+0000 7f6a3feb9700 1 -- 172.21.15.78:0/1353179579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38102480 msgr2=0x7f6a381028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:14.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.858+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/1353179579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38102480 0x7f6a381028f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6a280098f0 tx=0x7f6a28009c00 comp rx=0 tx=0).stop 2023-10-10T17:48:14.859 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.858+0000 7f6a3feb9700 1 -- 172.21.15.78:0/1353179579 shutdown_connections 2023-10-10T17:48:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.858+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/1353179579 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a38102e30 0x7f6a3810f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.858+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/1353179579 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a38108490 0x7f6a38108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.858+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/1353179579 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38102480 0x7f6a381028f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.860 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.858+0000 7f6a3feb9700 1 -- 172.21.15.78:0/1353179579 >> 172.21.15.78:0/1353179579 conn(0x7f6a380fdf60 msgr2=0x7f6a38100370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:14.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.859+0000 7f6a3feb9700 1 -- 172.21.15.78:0/1353179579 shutdown_connections 2023-10-10T17:48:14.861 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.859+0000 7f6a3feb9700 1 -- 172.21.15.78:0/1353179579 wait complete. 2023-10-10T17:48:14.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 Processor -- start 2023-10-10T17:48:14.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 -- start start 2023-10-10T17:48:14.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a38102480 0x7f6a3810d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a38102e30 0x7f6a38119780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.862 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38108490 0x7f6a3810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3810f5a0 con 0x7f6a38102e30 2023-10-10T17:48:14.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3810de00 con 0x7f6a38108490 2023-10-10T17:48:14.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.860+0000 7f6a3feb9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3810dfc0 con 0x7f6a38102480 2023-10-10T17:48:14.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38108490 0x7f6a3810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.863 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38108490 0x7f6a3810d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48614/0 (socket says 172.21.15.78:48614) 2023-10-10T17:48:14.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 -- 172.21.15.78:0/572999698 learned_addr learned my addr 172.21.15.78:0/572999698 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:14.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 -- 172.21.15.78:0/572999698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a38102480 msgr2=0x7f6a3810d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:48:14.864 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a38102480 0x7f6a3810d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 -- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a38102e30 msgr2=0x7f6a38119780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:48:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a38102e30 0x7f6a38119780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.861+0000 7f6a3e456700 1 -- 172.21.15.78:0/572999698 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a28009520 con 0x7f6a38108490 2023-10-10T17:48:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.864+0000 7f6a3e456700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38108490 0x7f6a3810d8c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a30007ee0 tx=0x7f6a3000d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:14.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.864+0000 7f6a2effd700 1 -- 172.21.15.78:0/572999698 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30017070 con 0x7f6a38108490 2023-10-10T17:48:14.866 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.864+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a38075ce0 con 0x7f6a38108490 2023-10-10T17:48:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.864+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a38075b50 con 0x7f6a38108490 2023-10-10T17:48:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.865+0000 7f6a2effd700 1 -- 172.21.15.78:0/572999698 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a3000f040 con 0x7f6a38108490 2023-10-10T17:48:14.867 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.865+0000 7f6a2effd700 1 -- 172.21.15.78:0/572999698 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30005740 con 0x7f6a38108490 2023-10-10T17:48:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.869+0000 7f6a4c045b80 1 Processor -- start 2023-10-10T17:48:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.869+0000 7f6a4c045b80 1 -- start start 2023-10-10T17:48:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585aced56b0 0x5585ad07f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad079560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585ad07fb90 0x5585ad079b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585acf7f000 con 0x5585ad07f720 2023-10-10T17:48:14.871 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585acfd1840 con 0x5585ad07fb90 2023-10-10T17:48:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585acfcacd0 con 0x5585aced56b0 2023-10-10T17:48:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585aced56b0 0x5585ad07f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585aced56b0 0x5585ad07f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38424/0 (socket says 172.21.15.78:38424) 2023-10-10T17:48:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 -- 172.21.15.78:0/1623951455 learned_addr learned my addr 172.21.15.78:0/1623951455 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:14.872 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585ad07fb90 0x5585ad079b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad079560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.873 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 -- 172.21.15.78:0/1623951455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585aced56b0 msgr2=0x5585ad07f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585aced56b0 0x5585ad07f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 -- 172.21.15.78:0/1623951455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 msgr2=0x5585ad079560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad079560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.870+0000 7f6a3d454700 1 -- 172.21.15.78:0/1623951455 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585ace4c5d0 con 0x5585ad07fb90 2023-10-10T17:48:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.871+0000 7f6a3d454700 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585ad07fb90 0x5585ad079b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009810 tx=0x7f6a28000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22de0ebe35bc23fc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.871+0000 7f6a2effd700 1 -- 172.21.15.78:0/572999698 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6a300058a0 con 0x7f6a38108490 2023-10-10T17:48:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.871+0000 7f6a23fff700 1 -- 172.21.15.78:0/1623951455 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2800f630 con 0x5585ad07fb90 2023-10-10T17:48:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.871+0000 7f6a23fff700 1 -- 172.21.15.78:0/1623951455 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a280044e0 con 0x5585ad07fb90 2023-10-10T17:48:14.875 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.871+0000 7f6a23fff700 1 -- 172.21.15.78:0/1623951455 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a28005660 con 0x5585ad07fb90 2023-10-10T17:48:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.872+0000 7f6a2effd700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6a240a1340 0x7f6a240a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.872+0000 7f6a2effd700 1 -- 172.21.15.78:0/572999698 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f6a300cc7f0 con 0x7f6a38108490 2023-10-10T17:48:14.876 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.874+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6a240a1340 0x7f6a240a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.874+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1623951455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585ad07fb90 msgr2=0x5585ad079b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:14.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.874+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585ad07fb90 0x5585ad079b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6a28009810 tx=0x7f6a28000bc0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.876+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1623951455 shutdown_connections 2023-10-10T17:48:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.876+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585aced56b0 0x5585ad07f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.876+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585ad07fb90 0x5585ad079b00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.876+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1623951455 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad079560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.876+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1623951455 >> 172.21.15.78:0/1623951455 conn(0x5585ad06ef50 msgr2=0x5585ad074270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.883+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1623951455 shutdown_connections 2023-10-10T17:48:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.885+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1623951455 wait complete. 2023-10-10T17:48:14.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.886+0000 7f6a4c045b80 1 -- start start 2023-10-10T17:48:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.886+0000 7f6a4c045b80 1 Processor -- start 2023-10-10T17:48:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad085a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad082c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07fb90 0x5585ad083280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585acf6c880 con 0x5585ad07f720 2023-10-10T17:48:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585acf6c520 con 0x5585aced56b0 2023-10-10T17:48:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ace4e8c0 con 0x5585ad07fb90 2023-10-10T17:48:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a4c045b80 10 client.0 _ll_register_callbacks cb 0x5585acf8a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6a240a1340 0x7f6a240a37f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005fd0 tx=0x7f6a34009310 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:14.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a3dc55700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad085a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a3dc55700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad085a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48626/0 (socket says 172.21.15.78:48626) 2023-10-10T17:48:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.889+0000 7f6a3dc55700 1 -- 172.21.15.78:0/94811977 learned_addr learned my addr 172.21.15.78:0/94811977 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3e456700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07fb90 0x5585ad083280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3d454700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad082c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3d454700 1 -- 172.21.15.78:0/94811977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07fb90 msgr2=0x5585ad083280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3d454700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07fb90 0x5585ad083280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3d454700 1 -- 172.21.15.78:0/94811977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 msgr2=0x5585ad085a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3d454700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad085a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3d454700 1 -- 172.21.15.78:0/94811977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a28020040 con 0x5585ad07f720 2023-10-10T17:48:14.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.890+0000 7f6a3e456700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07fb90 0x5585ad083280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.891+0000 7f6a3d454700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad082c70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x5585ad07ff60 tx=0x7f6a28005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a269c7051382d17 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:14.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.891+0000 7f6a217fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.891+0000 7f6a217fa700 1 -- 172.21.15.78:0/94811977 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2802b070 con 0x5585ad07f720 2023-10-10T17:48:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.891+0000 7f6a217fa700 1 -- 172.21.15.78:0/94811977 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f6a28004230 con 0x5585ad07f720 2023-10-10T17:48:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.891+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585ad06c710 con 0x5585ad07f720 2023-10-10T17:48:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.892+0000 7f6a217fa700 1 -- 172.21.15.78:0/94811977 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a28005200 con 0x5585ad07f720 2023-10-10T17:48:14.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 10 client.16602 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5585acf91130 con 0x5585ad07f720 2023-10-10T17:48:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a217fa700 1 -- 172.21.15.78:0/94811977 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a28037030 con 0x5585ad07f720 2023-10-10T17:48:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a217fa700 1 -- 172.21.15.78:0/94811977 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f6a28024070 con 0x5585ad07f720 2023-10-10T17:48:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 10 client.16602 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:14.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 10 client.16602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 20 client.16602 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad092ad0 0x5585ad094ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 4 client.16602 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:14.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.895+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585acf8d040 con 0x5585ad092ad0 2023-10-10T17:48:14.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.899+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad092ad0 0x5585ad094ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.903+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad092ad0 0x5585ad094ee0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9b26a3f59f3f01d server_cookie=7d422a0a90a590e9 in_seq=0 out_seq=0 2023-10-10T17:48:14.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.903+0000 7f6a217fa700 10 client.16602 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.907+0000 7f6a217fa700 1 -- 172.21.15.78:0/94811977 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5585acf8d040 con 0x5585ad092ad0 2023-10-10T17:48:14.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.907+0000 7f6a217fa700 10 client.16602 handle_command_reply: tid=0 2023-10-10T17:48:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.908+0000 7f6a4c045b80 1 client.16602 shutdown 2023-10-10T17:48:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 msgr2=0x5585ad082c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad082c70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x5585ad07ff60 tx=0x7f6a28005dc0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 shutdown_connections 2023-10-10T17:48:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07fb90 0x5585ad083280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad092ad0 0x5585ad094ee0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad085a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/94811977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07f720 0x5585ad082c70 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.915+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 >> 172.21.15.78:0/94811977 conn(0x5585ad06ef50 msgr2=0x5585ad0737e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.917+0000 7f6a217fa700 0 client.16602 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.920+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 shutdown_connections 2023-10-10T17:48:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.920+0000 7f6a4c045b80 1 -- 172.21.15.78:0/94811977 wait complete. 2023-10-10T17:48:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.922+0000 7f6a4c045b80 20 client.16602 trim_cache size 0 max 16384 2023-10-10T17:48:14.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.949+0000 7f6a4c045b80 1 Processor -- start 2023-10-10T17:48:14.952 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.950+0000 7f6a4c045b80 1 -- start start 2023-10-10T17:48:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.950+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad08a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad073f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad094ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad081210 con 0x5585ad07fb90 2023-10-10T17:48:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad07d940 con 0x5585aced56b0 2023-10-10T17:48:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad07a2a0 con 0x5585ad07f720 2023-10-10T17:48:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad094ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad094ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46910/0 (socket says 172.21.15.78:46910) 2023-10-10T17:48:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 -- 172.21.15.78:0/1971008686 learned_addr learned my addr 172.21.15.78:0/1971008686 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad08a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 -- 172.21.15.78:0/1971008686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 msgr2=0x5585ad073f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 -- 172.21.15.78:0/1971008686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 msgr2=0x5585ad08a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad08a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.955 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.951+0000 7f6a3e456700 1 -- 172.21.15.78:0/1971008686 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585ad0706f0 con 0x5585ad07fb90 2023-10-10T17:48:14.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.952+0000 7f6a3e456700 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad094ae0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009510 tx=0x7f6a300b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea88ae5fbcc292c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:14.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.952+0000 7f6a2cff9700 1 -- 172.21.15.78:0/1971008686 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3009d5c0 con 0x5585ad07fb90 2023-10-10T17:48:14.956 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.952+0000 7f6a2cff9700 1 -- 172.21.15.78:0/1971008686 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a300a19e0 con 0x5585ad07fb90 2023-10-10T17:48:14.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.952+0000 7f6a2cff9700 1 -- 172.21.15.78:0/1971008686 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3000dbd0 con 0x5585ad07fb90 2023-10-10T17:48:14.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.953+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1971008686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 msgr2=0x5585ad094ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:14.957 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.953+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad094ae0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009510 tx=0x7f6a300b9040 comp rx=0 tx=0).stop 2023-10-10T17:48:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.959+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1971008686 shutdown_connections 2023-10-10T17:48:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.959+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad094ae0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.959+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad08a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.959+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/1971008686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.959+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1971008686 >> 172.21.15.78:0/1971008686 conn(0x5585ad06ef50 msgr2=0x5585ad075a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.960+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1971008686 shutdown_connections 2023-10-10T17:48:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.960+0000 7f6a4c045b80 1 -- 172.21.15.78:0/1971008686 wait complete. 2023-10-10T17:48:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.960+0000 7f6a4c045b80 1 -- start start 2023-10-10T17:48:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.960+0000 7f6a4c045b80 1 Processor -- start 2023-10-10T17:48:14.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad07c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad082040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.965 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a4c045b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad082580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad082ac0 con 0x5585ad07fb90 2023-10-10T17:48:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad082c30 con 0x5585aced56b0 2023-10-10T17:48:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a4c045b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ad082df0 con 0x5585ad07f720 2023-10-10T17:48:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a4c045b80 10 client.0 _ll_register_callbacks cb 0x5585acf8a720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:14.966 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad082040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad082040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38448/0 (socket says 172.21.15.78:38448) 2023-10-10T17:48:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 -- 172.21.15.78:0/4080560918 learned_addr learned my addr 172.21.15.78:0/4080560918 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 -- 172.21.15.78:0/4080560918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 msgr2=0x5585ad07c2a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T17:48:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad07c2a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.967 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 -- 172.21.15.78:0/4080560918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 msgr2=0x5585ad082580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:48:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad082580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:14.968 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.961+0000 7f6a3d454700 1 -- 172.21.15.78:0/4080560918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a38075b50 con 0x5585ad07f720 2023-10-10T17:48:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.967+0000 7f6a3d454700 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad082040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6a28001eb0 tx=0x7f6a2800f510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3d0008caa6b77de server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.967+0000 7f6a217fa700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:48:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.968+0000 7f6a217fa700 1 -- 172.21.15.78:0/4080560918 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2802b070 con 0x5585ad07f720 2023-10-10T17:48:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.968+0000 7f6a217fa700 1 -- 172.21.15.78:0/4080560918 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f6a28003680 con 0x5585ad07f720 2023-10-10T17:48:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.968+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6a280044e0 con 0x5585ad07f720 2023-10-10T17:48:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.969+0000 7f6a217fa700 1 -- 172.21.15.78:0/4080560918 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a28026650 con 0x5585ad07f720 2023-10-10T17:48:14.970 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.969+0000 7f6a4c045b80 10 client.26047 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.969+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5585acdf8d80 con 0x5585ad07f720 2023-10-10T17:48:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.969+0000 7f6a217fa700 1 -- 172.21.15.78:0/4080560918 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a280044e0 con 0x5585ad07f720 2023-10-10T17:48:14.971 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.969+0000 7f6a217fa700 1 -- 172.21.15.78:0/4080560918 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f6a28009920 con 0x5585ad07f720 2023-10-10T17:48:14.972 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.972+0000 7f6a4c045b80 10 client.26047 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.972+0000 7f6a4c045b80 10 client.26047 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.972+0000 7f6a4c045b80 20 client.26047 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.972+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad0e69f0 0x5585ad0e8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.972+0000 7f6a4c045b80 4 client.26047 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:48:14.973 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.972+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5585acf99600 con 0x5585ad0e69f0 2023-10-10T17:48:14.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.973+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad0e69f0 0x5585ad0e8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:14.974 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.973+0000 7f6a217fa700 10 client.26047 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:14.975 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:14.973+0000 7f6a3dc55700 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad0e69f0 0x5585ad0e8e00 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=469d9d62911a3501 server_cookie=ccfa4e93d693d587 in_seq=0 out_seq=0 2023-10-10T17:48:15.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:14 smithi078 ceph-mon[99494]: pgmap v174: 129 pgs: 129 active+clean; 780 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 1.07k op/s 2023-10-10T17:48:15.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:14 smithi190 ceph-mon[101220]: pgmap v174: 129 pgs: 129 active+clean; 780 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 1.07k op/s 2023-10-10T17:48:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:14 smithi157 ceph-mon[100896]: pgmap v174: 129 pgs: 129 active+clean; 780 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 5.2 MiB/s wr, 1.07k op/s 2023-10-10T17:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a217fa700 1 -- 172.21.15.78:0/4080560918 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5585acf99600 con 0x5585ad0e69f0 2023-10-10T17:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a217fa700 10 client.26047 handle_command_reply: tid=0 2023-10-10T17:48:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 client.26047 shutdown 2023-10-10T17:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 msgr2=0x5585ad082040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad082040 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6a28001eb0 tx=0x7f6a2800f510 comp rx=0 tx=0).stop 2023-10-10T17:48:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 shutdown_connections 2023-10-10T17:48:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5585ad0e69f0 0x5585ad0e8e00 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5585ad07fb90 0x5585ad082580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5585aced56b0 0x5585ad07c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a217fa700 0 client.26047 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 --2- 172.21.15.78:0/4080560918 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5585ad07f720 0x5585ad082040 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 >> 172.21.15.78:0/4080560918 conn(0x5585ad06ef50 msgr2=0x5585ad094ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 shutdown_connections 2023-10-10T17:48:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.738+0000 7f6a4c045b80 1 -- 172.21.15.78:0/4080560918 wait complete. 2023-10-10T17:48:15.744 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.739+0000 7f6a4c045b80 20 client.26047 trim_cache size 0 max 16384 2023-10-10T17:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:48:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6a240a1340 msgr2=0x7f6a240a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6a240a1340 0x7f6a240a37f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6a34005fd0 tx=0x7f6a34009310 comp rx=0 tx=0).stop 2023-10-10T17:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38108490 msgr2=0x7f6a3810d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38108490 0x7f6a3810d8c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6a30007ee0 tx=0x7f6a3000d3b0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 shutdown_connections 2023-10-10T17:48:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6a38102480 0x7f6a3810d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6a240a1340 0x7f6a240a37f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6a38108490 0x7f6a3810d8c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 --2- 172.21.15.78:0/572999698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6a38102e30 0x7f6a38119780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 >> 172.21.15.78:0/572999698 conn(0x7f6a380fdf60 msgr2=0x7f6a380ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:15.748 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 shutdown_connections 2023-10-10T17:48:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:15.740+0000 7f6a3feb9700 1 -- 172.21.15.78:0/572999698 wait complete. 2023-10-10T17:48:16.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:15 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:16.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:15 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:16.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:15 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:16.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:15 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:16.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:15 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:16.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:15 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:17.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:48:17.099 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:16 smithi078 ceph-mon[99494]: pgmap v175: 129 pgs: 129 active+clean; 780 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 694 op/s 2023-10-10T17:48:17.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:16 smithi190 ceph-mon[101220]: pgmap v175: 129 pgs: 129 active+clean; 780 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 694 op/s 2023-10-10T17:48:17.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 -- 172.21.15.78:0/1051535451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e000ffdf0 msgr2=0x7f0e00103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/1051535451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e000ffdf0 0x7f0e00103db0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f0df00099c0 tx=0x7f0df0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 -- 172.21.15.78:0/1051535451 shutdown_connections 2023-10-10T17:48:17.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/1051535451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e000ff450 0x7f0e000ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/1051535451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e00104490 0x7f0e0010c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/1051535451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e000ffdf0 0x7f0e00103db0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 -- 172.21.15.78:0/1051535451 >> 172.21.15.78:0/1051535451 conn(0x7f0e000770b0 msgr2=0x7f0e000774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 -- 172.21.15.78:0/1051535451 shutdown_connections 2023-10-10T17:48:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 -- 172.21.15.78:0/1051535451 wait complete. 2023-10-10T17:48:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.209+0000 7f0e05cfd700 1 Processor -- start 2023-10-10T17:48:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0e05cfd700 1 -- start start 2023-10-10T17:48:17.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0e05cfd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e000ff450 0x7f0e001078a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0e05cfd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e000ffdf0 0x7f0e00107de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0e05cfd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e00104490 0x7f0e001a86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0e05cfd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e001a8c00 con 0x7f0e00104490 2023-10-10T17:48:17.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0e05cfd700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e001a8d70 con 0x7f0e000ff450 2023-10-10T17:48:17.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0e05cfd700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e001a8f30 con 0x7f0e000ffdf0 2023-10-10T17:48:17.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e00104490 0x7f0e001a86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfeffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e000ffdf0 0x7f0e00107de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e00104490 0x7f0e001a86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46932/0 (socket says 172.21.15.78:46932) 2023-10-10T17:48:17.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 -- 172.21.15.78:0/3590437454 learned_addr learned my addr 172.21.15.78:0/3590437454 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:17.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e000ff450 0x7f0e001078a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 -- 172.21.15.78:0/3590437454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e000ffdf0 msgr2=0x7f0e00107de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e000ffdf0 0x7f0e00107de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 -- 172.21.15.78:0/3590437454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e000ff450 msgr2=0x7f0e001078a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e000ff450 0x7f0e001078a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.210+0000 7f0dfffff700 1 -- 172.21.15.78:0/3590437454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df00095f0 con 0x7f0e00104490 2023-10-10T17:48:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.211+0000 7f0dfffff700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e00104490 0x7f0e001a86c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0df8007d70 tx=0x7f0df800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.211+0000 7f0dfcff9700 1 -- 172.21.15.78:0/3590437454 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df800ac90 con 0x7f0e00104490 2023-10-10T17:48:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.211+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0010e120 con 0x7f0e00104490 2023-10-10T17:48:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.211+0000 7f0dfcff9700 1 -- 172.21.15.78:0/3590437454 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0df800adf0 con 0x7f0e00104490 2023-10-10T17:48:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.211+0000 7f0dfcff9700 1 -- 172.21.15.78:0/3590437454 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df80115e0 con 0x7f0e00104490 2023-10-10T17:48:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.211+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0010df90 con 0x7f0e00104490 2023-10-10T17:48:17.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.213+0000 7f0dfcff9700 1 -- 172.21.15.78:0/3590437454 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0df8012420 con 0x7f0e00104490 2023-10-10T17:48:17.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.214+0000 7f0dfcff9700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0dec0a12c0 0x7f0dec0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.214+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0dec0a12c0 0x7f0dec0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.214+0000 7f0dfcff9700 1 -- 172.21.15.78:0/3590437454 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0df80cd080 con 0x7f0e00104490 2023-10-10T17:48:17.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.215+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0dec0a12c0 0x7f0dec0a3770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009c60 tx=0x7f0de8009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:17.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:16 smithi157 ceph-mon[100896]: pgmap v175: 129 pgs: 129 active+clean; 780 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.4 MiB/s wr, 694 op/s 2023-10-10T17:48:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.215+0000 7f0e11e89b80 1 Processor -- start 2023-10-10T17:48:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.215+0000 7f0e11e89b80 1 -- start start 2023-10-10T17:48:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225ab60 0x55f27225af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f27225f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f272160060 con 0x55f27225b470 2023-10-10T17:48:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2721b28a0 con 0x55f27225f8b0 2023-10-10T17:48:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2721abd30 con 0x55f27225ab60 2023-10-10T17:48:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f27225f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f27225f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46938/0 (socket says 172.21.15.78:46938) 2023-10-10T17:48:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2565765397 learned_addr learned my addr 172.21.15.78:0/2565765397 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225ab60 0x55f27225af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfffff700 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2565765397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225ab60 msgr2=0x55f27225af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225ab60 0x55f27225af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2565765397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 msgr2=0x55f27225fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2565765397 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f27202d630 con 0x55f27225b470 2023-10-10T17:48:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.216+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f27225f370 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0df00036c0 tx=0x7f0df00047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89b0b40d396a067 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.217+0000 7f0df5ffb700 1 -- 172.21.15.78:0/2565765397 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0026070 con 0x55f27225b470 2023-10-10T17:48:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.217+0000 7f0df5ffb700 1 -- 172.21.15.78:0/2565765397 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0df001e040 con 0x55f27225b470 2023-10-10T17:48:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.217+0000 7f0df5ffb700 1 -- 172.21.15.78:0/2565765397 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df000fb00 con 0x55f27225b470 2023-10-10T17:48:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.217+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2565765397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 msgr2=0x55f27225f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.217+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f27225f370 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f0df00036c0 tx=0x7f0df00047e0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2565765397 shutdown_connections 2023-10-10T17:48:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225ab60 0x55f27225af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2565765397 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f27225f370 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2565765397 >> 172.21.15.78:0/2565765397 conn(0x55f272250360 msgr2=0x55f272255b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2565765397 shutdown_connections 2023-10-10T17:48:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2565765397 wait complete. 2023-10-10T17:48:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- start start 2023-10-10T17:48:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 Processor -- start 2023-10-10T17:48:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272264340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f272264880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225f8b0 0x55f272264dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f27214d8e0 con 0x55f27225b470 2023-10-10T17:48:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f27214d580 con 0x55f27225ab60 2023-10-10T17:48:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.218+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f27202f920 con 0x55f27225f8b0 2023-10-10T17:48:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0e11e89b80 10 client.0 _ll_register_callbacks cb 0x55f27216b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f272264880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f272264880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46944/0 (socket says 172.21.15.78:46944) 2023-10-10T17:48:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2598962834 learned_addr learned my addr 172.21.15.78:0/2598962834 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272264340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfffff700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225f8b0 0x55f272264dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2598962834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225f8b0 msgr2=0x55f272264dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225f8b0 0x55f272264dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 msgr2=0x55f272264340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272264340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 -- 172.21.15.78:0/2598962834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0010c8a0 con 0x55f27225b470 2023-10-10T17:48:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272264340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f272264880 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009d10 tx=0x7f0df000b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b395410e4971ff17 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0ddeffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0ddeffd700 1 -- 172.21.15.78:0/2598962834 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0026070 con 0x55f27225b470 2023-10-10T17:48:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0ddeffd700 1 -- 172.21.15.78:0/2598962834 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0df002b600 con 0x55f27225b470 2023-10-10T17:48:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f27224db20 con 0x55f27225b470 2023-10-10T17:48:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0ddeffd700 1 -- 172.21.15.78:0/2598962834 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0df000f460 con 0x55f27225b470 2023-10-10T17:48:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0e11e89b80 10 client.16617 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.219+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f272172190 con 0x55f27225b470 2023-10-10T17:48:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0ddeffd700 1 -- 172.21.15.78:0/2598962834 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f27224db20 con 0x55f27225b470 2023-10-10T17:48:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0ddeffd700 1 -- 172.21.15.78:0/2598962834 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0df0021070 con 0x55f27225b470 2023-10-10T17:48:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0e11e89b80 10 client.16617 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0e11e89b80 10 client.16617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0e11e89b80 20 client.16617 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f272274430 0x55f272276810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0e11e89b80 4 client.16617 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f27216e0a0 con 0x55f272274430 2023-10-10T17:48:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f272274430 0x55f272276810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f272274430 0x55f272276810 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9de989f9271efc25 server_cookie=173c494c90efaf19 in_seq=0 out_seq=0 2023-10-10T17:48:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.220+0000 7f0ddeffd700 10 client.16617 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.221+0000 7f0ddeffd700 1 -- 172.21.15.78:0/2598962834 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f27216e0a0 con 0x55f272274430 2023-10-10T17:48:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.221+0000 7f0ddeffd700 10 client.16617 handle_command_reply: tid=0 2023-10-10T17:48:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.221+0000 7f0e11e89b80 1 client.16617 shutdown 2023-10-10T17:48:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 msgr2=0x55f272264880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f272264880 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f0df0009d10 tx=0x7f0df000b9c0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 shutdown_connections 2023-10-10T17:48:17.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225f8b0 0x55f272264dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f272274430 0x55f272276810 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272264340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0ddeffd700 0 client.16617 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/2598962834 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225b470 0x55f272264880 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 >> 172.21.15.78:0/2598962834 conn(0x55f272250360 msgr2=0x55f272254de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 shutdown_connections 2023-10-10T17:48:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 1 -- 172.21.15.78:0/2598962834 wait complete. 2023-10-10T17:48:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.222+0000 7f0e11e89b80 20 client.16617 trim_cache size 0 max 16384 2023-10-10T17:48:17.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.241+0000 7f0e11e89b80 1 Processor -- start 2023-10-10T17:48:17.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.241+0000 7f0e11e89b80 1 -- start start 2023-10-10T17:48:17.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225ab60 0x55f272263650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f272271340 con 0x55f27225ab60 2023-10-10T17:48:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2722571c0 con 0x55f27225f8b0 2023-10-10T17:48:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f27225e4b0 con 0x55f27225b470 2023-10-10T17:48:17.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225ab60 0x55f272263650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225ab60 0x55f272263650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46950/0 (socket says 172.21.15.78:46950) 2023-10-10T17:48:17.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 -- 172.21.15.78:0/3754533328 learned_addr learned my addr 172.21.15.78:0/3754533328 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:17.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dfffff700 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 -- 172.21.15.78:0/3754533328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 msgr2=0x55f27226b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 -- 172.21.15.78:0/3754533328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 msgr2=0x55f27225efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 -- 172.21.15.78:0/3754533328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f272171ee0 con 0x55f27225ab60 2023-10-10T17:48:17.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225ab60 0x55f272263650 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x55f272277120 tx=0x7f0de8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4876cab03c0ff7cd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:17.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0df67fc700 1 -- 172.21.15.78:0/3754533328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8022070 con 0x55f27225ab60 2023-10-10T17:48:17.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0df67fc700 1 -- 172.21.15.78:0/3754533328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0de8023040 con 0x55f27225ab60 2023-10-10T17:48:17.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.242+0000 7f0df67fc700 1 -- 172.21.15.78:0/3754533328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de801d670 con 0x55f27225ab60 2023-10-10T17:48:17.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 -- 172.21.15.78:0/3754533328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225ab60 msgr2=0x55f272263650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225ab60 0x55f272263650 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x55f272277120 tx=0x7f0de8014000 comp rx=0 tx=0).stop 2023-10-10T17:48:17.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 -- 172.21.15.78:0/3754533328 shutdown_connections 2023-10-10T17:48:17.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225f8b0 0x55f27225efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225ab60 0x55f272263650 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/3754533328 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 -- 172.21.15.78:0/3754533328 >> 172.21.15.78:0/3754533328 conn(0x55f272250360 msgr2=0x55f27225e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:17.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 -- 172.21.15.78:0/3754533328 shutdown_connections 2023-10-10T17:48:17.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 -- 172.21.15.78:0/3754533328 wait complete. 2023-10-10T17:48:17.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.243+0000 7f0e11e89b80 1 -- start start 2023-10-10T17:48:17.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 1 Processor -- start 2023-10-10T17:48:17.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272269b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225f8b0 0x55f2722c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f27226a770 con 0x55f27225f8b0 2023-10-10T17:48:17.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2722c7800 con 0x55f27225ab60 2023-10-10T17:48:17.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2722c7940 con 0x55f27225b470 2023-10-10T17:48:17.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0e11e89b80 10 client.0 _ll_register_callbacks cb 0x7f0de80096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:17.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dff7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272269b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225f8b0 0x55f2722c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfeffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dff7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272269b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48678/0 (socket says 172.21.15.78:48678) 2023-10-10T17:48:17.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38488/0 (socket says 172.21.15.78:38488) 2023-10-10T17:48:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dff7fe700 1 -- 172.21.15.78:0/4188855389 learned_addr learned my addr 172.21.15.78:0/4188855389 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfeffd700 1 -- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 msgr2=0x55f272269b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272269b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfeffd700 1 -- 172.21.15.78:0/4188855389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225f8b0 msgr2=0x55f2722c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:17.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225f8b0 0x55f2722c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dfeffd700 1 -- 172.21.15.78:0/4188855389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de8017070 con 0x55f27225b470 2023-10-10T17:48:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.244+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272269b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0dfeffd700 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226a060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0df00058e0 tx=0x7f0df0009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c8f64b621fde5cc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0ddeffd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:48:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0ddeffd700 1 -- 172.21.15.78:0/4188855389 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df0026050 con 0x55f27225b470 2023-10-10T17:48:17.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0df001e040 con 0x55f27225b470 2023-10-10T17:48:17.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0ddeffd700 1 -- 172.21.15.78:0/4188855389 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0df0000e40 con 0x55f27225b470 2023-10-10T17:48:17.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0ddeffd700 1 -- 172.21.15.78:0/4188855389 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0df000b5b0 con 0x55f27225b470 2023-10-10T17:48:17.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0e11e89b80 10 client.26065 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:17.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f271fc2c60 con 0x55f27225b470 2023-10-10T17:48:17.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0ddeffd700 1 -- 172.21.15.78:0/4188855389 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0df0046080 con 0x55f27225b470 2023-10-10T17:48:17.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0ddeffd700 1 -- 172.21.15.78:0/4188855389 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0df00099f0 con 0x55f27225b470 2023-10-10T17:48:17.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0e11e89b80 10 client.26065 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:17.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0e11e89b80 10 client.26065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:17.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0e11e89b80 20 client.26065 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:17.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.245+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f2722c8b90 0x55f2722cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.246+0000 7f0e11e89b80 4 client.26065 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:48:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.246+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f27217a660 con 0x55f2722c8b90 2023-10-10T17:48:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.246+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f2722c8b90 0x55f2722cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.246+0000 7f0dff7fe700 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f2722c8b90 0x55f2722cafa0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2a90a98c54c71be server_cookie=8af36ccbbb0bef65 in_seq=0 out_seq=0 2023-10-10T17:48:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.246+0000 7f0ddeffd700 10 client.26065 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0ddeffd700 1 -- 172.21.15.78:0/4188855389 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f27217a660 con 0x55f2722c8b90 2023-10-10T17:48:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0ddeffd700 10 client.26065 handle_command_reply: tid=0 2023-10-10T17:48:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 client.26065 shutdown 2023-10-10T17:48:17.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 msgr2=0x55f27226a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226a060 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0df00058e0 tx=0x7f0df0009fd0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 shutdown_connections 2023-10-10T17:48:17.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f2722c8b90 0x55f2722cafa0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f27225f8b0 0x55f2722c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f27225ab60 0x55f272269b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 --2- 172.21.15.78:0/4188855389 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f27225b470 0x55f27226a060 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0ddeffd700 0 client.26065 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:17.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 >> 172.21.15.78:0/4188855389 conn(0x55f272250360 msgr2=0x55f272259960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:17.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 shutdown_connections 2023-10-10T17:48:17.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 1 -- 172.21.15.78:0/4188855389 wait complete. 2023-10-10T17:48:17.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.247+0000 7f0e11e89b80 20 client.26065 trim_cache size 0 max 16384 2023-10-10T17:48:17.276 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:48:17.277 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:48:17.277 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:48:17.277 INFO:teuthology.orchestra.run.smithi078.stdout: "5ecf957f-7c91-40df-8738-d2beae2feba1": { 2023-10-10T17:48:17.278 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:48:17.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "5ecf957f-7c91-40df-8738-d2beae2feba1", 2023-10-10T17:48:17.278 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:48:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:48:17.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:48:17.279 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:48:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0dec0a12c0 msgr2=0x7f0dec0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0dec0a12c0 0x7f0dec0a3770 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009c60 tx=0x7f0de8009260 comp rx=0 tx=0).stop 2023-10-10T17:48:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e00104490 msgr2=0x7f0e001a86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e00104490 0x7f0e001a86c0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f0df8007d70 tx=0x7f0df800a340 comp rx=0 tx=0).stop 2023-10-10T17:48:17.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 shutdown_connections 2023-10-10T17:48:17.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0dec0a12c0 0x7f0dec0a3770 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e00104490 0x7f0e001a86c0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0e000ff450 0x7f0e001078a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 --2- 172.21.15.78:0/3590437454 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0e000ffdf0 0x7f0e00107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:17.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 >> 172.21.15.78:0/3590437454 conn(0x7f0e000770b0 msgr2=0x7f0e000fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:17.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 shutdown_connections 2023-10-10T17:48:17.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:17.248+0000 7f0e05cfd700 1 -- 172.21.15.78:0/3590437454 wait complete. 2023-10-10T17:48:17.287 INFO:tasks.cephfs.filesystem:scrub status for tag:5ecf957f-7c91-40df-8738-d2beae2feba1 - {'path': '/', 'tag': '5ecf957f-7c91-40df-8738-d2beae2feba1', 'options': 'recursive,force'} 2023-10-10T17:48:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:18 smithi078 ceph-mon[99494]: pgmap v176: 129 pgs: 129 active+clean; 771 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 1.13k op/s 2023-10-10T17:48:19.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:18 smithi190 ceph-mon[101220]: pgmap v176: 129 pgs: 129 active+clean; 771 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 1.13k op/s 2023-10-10T17:48:19.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:18 smithi157 ceph-mon[100896]: pgmap v176: 129 pgs: 129 active+clean; 771 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 4.9 MiB/s wr, 1.13k op/s 2023-10-10T17:48:21.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:20 smithi078 ceph-mon[99494]: pgmap v177: 129 pgs: 129 active+clean; 771 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 726 op/s 2023-10-10T17:48:21.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:20 smithi190 ceph-mon[101220]: pgmap v177: 129 pgs: 129 active+clean; 771 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 726 op/s 2023-10-10T17:48:21.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:20 smithi157 ceph-mon[100896]: pgmap v177: 129 pgs: 129 active+clean; 771 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 3.1 MiB/s wr, 726 op/s 2023-10-10T17:48:23.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:22 smithi078 ceph-mon[99494]: pgmap v178: 129 pgs: 129 active+clean; 772 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 748 op/s 2023-10-10T17:48:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:22 smithi190 ceph-mon[101220]: pgmap v178: 129 pgs: 129 active+clean; 772 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 748 op/s 2023-10-10T17:48:23.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:22 smithi157 ceph-mon[100896]: pgmap v178: 129 pgs: 129 active+clean; 772 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 3.1 MiB/s wr, 748 op/s 2023-10-10T17:48:24.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:23 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:23 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:24.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:23 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:24.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:23 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:24.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:23 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:24.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:23 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:25.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:24 smithi078 ceph-mon[99494]: pgmap v179: 129 pgs: 129 active+clean; 785 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 930 op/s 2023-10-10T17:48:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:24 smithi190 ceph-mon[101220]: pgmap v179: 129 pgs: 129 active+clean; 785 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 930 op/s 2023-10-10T17:48:25.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:24 smithi157 ceph-mon[100896]: pgmap v179: 129 pgs: 129 active+clean; 785 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 930 op/s 2023-10-10T17:48:27.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:26 smithi078 ceph-mon[99494]: pgmap v180: 129 pgs: 129 active+clean; 785 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 672 op/s 2023-10-10T17:48:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:26 smithi190 ceph-mon[101220]: pgmap v180: 129 pgs: 129 active+clean; 785 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 672 op/s 2023-10-10T17:48:27.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:26 smithi157 ceph-mon[100896]: pgmap v180: 129 pgs: 129 active+clean; 785 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.9 MiB/s wr, 672 op/s 2023-10-10T17:48:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 -- 172.21.15.78:0/3333339437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 msgr2=0x7f0f50104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3333339437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 0x7f0f50104e80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009920 tx=0x7f0f40009c30 comp rx=0 tx=0).stop 2023-10-10T17:48:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 -- 172.21.15.78:0/3333339437 shutdown_connections 2023-10-10T17:48:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3333339437 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f50105560 0x7f0f5010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3333339437 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 0x7f0f50101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3333339437 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 0x7f0f50104e80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 -- 172.21.15.78:0/3333339437 >> 172.21.15.78:0/3333339437 conn(0x7f0f50077550 msgr2=0x7f0f50077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 -- 172.21.15.78:0/3333339437 shutdown_connections 2023-10-10T17:48:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.867+0000 7f0f565f3700 1 -- 172.21.15.78:0/3333339437 wait complete. 2023-10-10T17:48:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 Processor -- start 2023-10-10T17:48:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 -- start start 2023-10-10T17:48:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 0x7f0f50196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 0x7f0f50196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f50105560 0x7f0f5019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5019b3f0 con 0x7f0f50100c50 2023-10-10T17:48:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5019b560 con 0x7f0f50105560 2023-10-10T17:48:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f565f3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5019b720 con 0x7f0f501015f0 2023-10-10T17:48:28.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 0x7f0f50196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.868+0000 7f0f4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 0x7f0f50196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46970/0 (socket says 172.21.15.78:46970) 2023-10-10T17:48:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3653559362 learned_addr learned my addr 172.21.15.78:0/3653559362 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f54b90700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f50105560 0x7f0f5019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.888 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 0x7f0f50196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3653559362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 msgr2=0x7f0f50196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:28.889 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 0x7f0f50196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3653559362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f50105560 msgr2=0x7f0f5019adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:28.890 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f50105560 0x7f0f5019adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3653559362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f380095f0 con 0x7f0f50100c50 2023-10-10T17:48:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 0x7f0f50196410 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800ec10 tx=0x7f0f3800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4d7fa700 1 -- 172.21.15.78:0/3653559362 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3800cc90 con 0x7f0f50100c50 2023-10-10T17:48:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f40009520 con 0x7f0f50100c50 2023-10-10T17:48:28.891 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4d7fa700 1 -- 172.21.15.78:0/3653559362 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f380044e0 con 0x7f0f50100c50 2023-10-10T17:48:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f4d7fa700 1 -- 172.21.15.78:0/3653559362 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f38010430 con 0x7f0f50100c50 2023-10-10T17:48:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.869+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f50110320 con 0x7f0f50100c50 2023-10-10T17:48:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.871+0000 7f0f4d7fa700 1 -- 172.21.15.78:0/3653559362 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0f38003fb0 con 0x7f0f50100c50 2023-10-10T17:48:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.872+0000 7f0f4d7fa700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0f3c0a12c0 0x7f0f3c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.892 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.872+0000 7f0f4d7fa700 1 -- 172.21.15.78:0/3653559362 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0f38014070 con 0x7f0f50100c50 2023-10-10T17:48:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.872+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0f3c0a12c0 0x7f0f3c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.872+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0f3c0a12c0 0x7f0f3c0a3770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0f40005e90 tx=0x7f0f40019040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.873+0000 7f0f6277fb80 1 Processor -- start 2023-10-10T17:48:28.893 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.873+0000 7f0f6277fb80 1 -- start start 2023-10-10T17:48:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.873+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a0e46b0 0x55e27a2925f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.873+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292b30 0x55e27a28c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292fa0 0x55e27a28cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.894 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a18e000 con 0x55e27a292fa0 2023-10-10T17:48:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a1e0840 con 0x55e27a0e46b0 2023-10-10T17:48:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a1d9cd0 con 0x55e27a292b30 2023-10-10T17:48:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292fa0 0x55e27a28cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f4f7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292b30 0x55e27a28c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.895 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292fa0 0x55e27a28cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46982/0 (socket says 172.21.15.78:46982) 2023-10-10T17:48:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 -- 172.21.15.78:0/4008262214 learned_addr learned my addr 172.21.15.78:0/4008262214 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f4ffff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a0e46b0 0x55e27a2925f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 -- 172.21.15.78:0/4008262214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292b30 msgr2=0x55e27a28c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292b30 0x55e27a28c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.896 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 -- 172.21.15.78:0/4008262214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a0e46b0 msgr2=0x55e27a2925f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a0e46b0 0x55e27a2925f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 -- 172.21.15.78:0/4008262214 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e27a05b5d0 con 0x55e27a292fa0 2023-10-10T17:48:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a0e46b0 0x55e27a2925f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f54b90700 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292fa0 0x55e27a28cf10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0f48011680 tx=0x7f0f48009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eeab1d528d92096 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:28.897 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f467fc700 1 -- 172.21.15.78:0/4008262214 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800e040 con 0x55e27a292fa0 2023-10-10T17:48:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f467fc700 1 -- 172.21.15.78:0/4008262214 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f4800f040 con 0x55e27a292fa0 2023-10-10T17:48:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.874+0000 7f0f467fc700 1 -- 172.21.15.78:0/4008262214 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48004860 con 0x55e27a292fa0 2023-10-10T17:48:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 -- 172.21.15.78:0/4008262214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292fa0 msgr2=0x55e27a28cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292fa0 0x55e27a28cf10 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f0f48011680 tx=0x7f0f48009920 comp rx=0 tx=0).stop 2023-10-10T17:48:28.898 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 -- 172.21.15.78:0/4008262214 shutdown_connections 2023-10-10T17:48:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292fa0 0x55e27a28cf10 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a0e46b0 0x55e27a2925f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/4008262214 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292b30 0x55e27a28c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 -- 172.21.15.78:0/4008262214 >> 172.21.15.78:0/4008262214 conn(0x55e27a27df50 msgr2=0x55e27a283270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:28.899 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 -- 172.21.15.78:0/4008262214 shutdown_connections 2023-10-10T17:48:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 -- 172.21.15.78:0/4008262214 wait complete. 2023-10-10T17:48:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.875+0000 7f0f6277fb80 1 -- start start 2023-10-10T17:48:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 1 Processor -- start 2023-10-10T17:48:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a298f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.900 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a2961b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a2968f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a17b880 con 0x55e27a292b30 2023-10-10T17:48:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a17b520 con 0x55e27a292fa0 2023-10-10T17:48:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a05d8c0 con 0x55e27a0e46b0 2023-10-10T17:48:28.901 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f6277fb80 10 client.0 _ll_register_callbacks cb 0x55e27a199720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a2961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4ffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a298f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a2961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46986/0 (socket says 172.21.15.78:46986) 2023-10-10T17:48:28.902 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/1206804357 learned_addr learned my addr 172.21.15.78:0/1206804357 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a298f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38514/0 (socket says 172.21.15.78:38514) 2023-10-10T17:48:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f54b90700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a2968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.903 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/1206804357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 msgr2=0x55e27a298f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a298f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 msgr2=0x55e27a2968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a2968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/1206804357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f48011330 con 0x55e27a292b30 2023-10-10T17:48:28.904 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a298f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.876+0000 7f0f54b90700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a2968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a2961b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000c010 tx=0x7f0f40009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a84b2e951eaf9ea server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/1206804357 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4002a050 con 0x55e27a292b30 2023-10-10T17:48:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/1206804357 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0f40005450 con 0x55e27a292b30 2023-10-10T17:48:28.905 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e27a27b710 con 0x55e27a292b30 2023-10-10T17:48:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/1206804357 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f4000f950 con 0x55e27a292b30 2023-10-10T17:48:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 10 client.16635 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55e27a1a0130 con 0x55e27a292b30 2023-10-10T17:48:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/1206804357 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e27a27b710 con 0x55e27a292b30 2023-10-10T17:48:28.906 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/1206804357 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0f400234a0 con 0x55e27a292b30 2023-10-10T17:48:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 10 client.16635 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 10 client.16635 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 20 client.16635 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2a64b0 0x55e27a2a88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.907 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 4 client.16635 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.877+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e27a19c040 con 0x55e27a2a64b0 2023-10-10T17:48:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.878+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2a64b0 0x55e27a2a88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.878+0000 7f0f2f7fe700 10 client.16635 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.878+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2a64b0 0x55e27a2a88c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=743ab4ea238dc91b server_cookie=e66944b3f899b683 in_seq=0 out_seq=0 2023-10-10T17:48:28.908 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/1206804357 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55e27a19c040 con 0x55e27a2a64b0 2023-10-10T17:48:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f2f7fe700 10 client.16635 handle_command_reply: tid=0 2023-10-10T17:48:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 client.16635 shutdown 2023-10-10T17:48:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 msgr2=0x55e27a2961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a2961b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000c010 tx=0x7f0f40009c30 comp rx=0 tx=0).stop 2023-10-10T17:48:28.909 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 shutdown_connections 2023-10-10T17:48:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a298f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2a64b0 0x55e27a2a88c0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.910 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a2968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f2f7fe700 0 client.16635 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/1206804357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a2961b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.911 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 >> 172.21.15.78:0/1206804357 conn(0x55e27a27df50 msgr2=0x55e27a2827e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 shutdown_connections 2023-10-10T17:48:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 1 -- 172.21.15.78:0/1206804357 wait complete. 2023-10-10T17:48:28.912 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.879+0000 7f0f6277fb80 20 client.16635 trim_cache size 0 max 16384 2023-10-10T17:48:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 Processor -- start 2023-10-10T17:48:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 -- start start 2023-10-10T17:48:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a0e46b0 0x55e27a291980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.920 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292b30 0x55e27a282f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292fa0 0x55e27a28c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a283820 con 0x55e27a0e46b0 2023-10-10T17:48:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a280000 con 0x55e27a292b30 2023-10-10T17:48:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.894+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a283de0 con 0x55e27a292fa0 2023-10-10T17:48:28.921 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a0e46b0 0x55e27a291980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a0e46b0 0x55e27a291980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:46992/0 (socket says 172.21.15.78:46992) 2023-10-10T17:48:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3143321142 learned_addr learned my addr 172.21.15.78:0/3143321142 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f54b90700 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292fa0 0x55e27a28c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292b30 0x55e27a282f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.922 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3143321142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292fa0 msgr2=0x55e27a28c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292fa0 0x55e27a28c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3143321142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292b30 msgr2=0x55e27a282f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292b30 0x55e27a282f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 -- 172.21.15.78:0/3143321142 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e27a19fe80 con 0x55e27a0e46b0 2023-10-10T17:48:28.923 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292b30 0x55e27a282f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a0e46b0 0x55e27a291980 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800c5b0 tx=0x7f0f3809a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0eccebbf2410b14 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f46ffd700 1 -- 172.21.15.78:0/3143321142 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3809e330 con 0x55e27a0e46b0 2023-10-10T17:48:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f46ffd700 1 -- 172.21.15.78:0/3143321142 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f380a2750 con 0x55e27a0e46b0 2023-10-10T17:48:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.895+0000 7f0f46ffd700 1 -- 172.21.15.78:0/3143321142 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f380987a0 con 0x55e27a0e46b0 2023-10-10T17:48:28.924 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3143321142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a0e46b0 msgr2=0x55e27a291980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a0e46b0 0x55e27a291980 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800c5b0 tx=0x7f0f3809a180 comp rx=0 tx=0).stop 2023-10-10T17:48:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3143321142 shutdown_connections 2023-10-10T17:48:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a292fa0 0x55e27a28c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a0e46b0 0x55e27a291980 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.925 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3143321142 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292b30 0x55e27a282f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3143321142 >> 172.21.15.78:0/3143321142 conn(0x55e27a27df50 msgr2=0x55e27a291270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3143321142 shutdown_connections 2023-10-10T17:48:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3143321142 wait complete. 2023-10-10T17:48:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.896+0000 7f0f6277fb80 1 -- start start 2023-10-10T17:48:28.926 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 1 Processor -- start 2023-10-10T17:48:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a299a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a299fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a29a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a2f9410 con 0x55e27a292b30 2023-10-10T17:48:28.927 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a2f9580 con 0x55e27a292fa0 2023-10-10T17:48:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e27a2f9740 con 0x55e27a0e46b0 2023-10-10T17:48:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f6277fb80 10 client.0 _ll_register_callbacks cb 0x7f0f4000b870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a299fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4ffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a299a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.928 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a299fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47000/0 (socket says 172.21.15.78:47000) 2023-10-10T17:48:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/3762733128 learned_addr learned my addr 172.21.15.78:0/3762733128 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f54b90700 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a29a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/3762733128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 msgr2=0x55e27a299a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a299a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.929 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 msgr2=0x55e27a29a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a29a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:28.930 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 -- 172.21.15.78:0/3762733128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e27a1a0130 con 0x55e27a292b30 2023-10-10T17:48:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f4f7fe700 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a299fa0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000f530 tx=0x7f0f40009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed2151ee98b666a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.897+0000 7f0f2f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:28.931 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/3762733128 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4002a040 con 0x55e27a292b30 2023-10-10T17:48:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f4800f040 con 0x55e27a292b30 2023-10-10T17:48:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/3762733128 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f0f40023e20 con 0x55e27a292b30 2023-10-10T17:48:28.932 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/3762733128 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f4000f950 con 0x55e27a292b30 2023-10-10T17:48:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 10 client.16641 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:28.933 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7f0f40009520 con 0x55e27a292b30 2023-10-10T17:48:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/3762733128 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f4800f040 con 0x55e27a292b30 2023-10-10T17:48:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/3762733128 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0f40041420 con 0x55e27a292b30 2023-10-10T17:48:28.934 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 10 client.16641 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:28.935 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 20 client.16641 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2faa40 0x55e27a2fce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 4 client.16641 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:48:28.936 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e27a1a8600 con 0x55e27a2faa40 2023-10-10T17:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.898+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2faa40 0x55e27a2fce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.899+0000 7f0f2f7fe700 10 client.16641 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:28.937 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:28.899+0000 7f0f4ffff700 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2faa40 0x55e27a2fce50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd9a8ffc265f484a server_cookie=e81bf866e83642ff in_seq=0 out_seq=0 2023-10-10T17:48:29.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:28 smithi078 ceph-mon[99494]: pgmap v181: 129 pgs: 129 active+clean; 794 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 880 op/s 2023-10-10T17:48:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:28 smithi190 ceph-mon[101220]: pgmap v181: 129 pgs: 129 active+clean; 794 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 880 op/s 2023-10-10T17:48:29.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:28 smithi157 ceph-mon[100896]: pgmap v181: 129 pgs: 129 active+clean; 794 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 4.1 MiB/s wr, 880 op/s 2023-10-10T17:48:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.875+0000 7f0f2f7fe700 1 -- 172.21.15.78:0/3762733128 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e27a1a8600 con 0x55e27a2faa40 2023-10-10T17:48:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.875+0000 7f0f2f7fe700 10 client.16641 handle_command_reply: tid=0 2023-10-10T17:48:29.877 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.875+0000 7f0f6277fb80 1 client.16641 shutdown 2023-10-10T17:48:29.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 msgr2=0x55e27a299fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:29.880 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a299fa0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000f530 tx=0x7f0f40009300 comp rx=0 tx=0).stop 2023-10-10T17:48:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 shutdown_connections 2023-10-10T17:48:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55e27a0e46b0 0x55e27a299a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55e27a2faa40 0x55e27a2fce50 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55e27a292fa0 0x55e27a29a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.881 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 --2- 172.21.15.78:0/3762733128 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e27a292b30 0x55e27a299fa0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 >> 172.21.15.78:0/3762733128 conn(0x55e27a27df50 msgr2=0x55e27a2933a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f2f7fe700 0 client.16641 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 shutdown_connections 2023-10-10T17:48:29.882 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.878+0000 7f0f6277fb80 1 -- 172.21.15.78:0/3762733128 wait complete. 2023-10-10T17:48:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.879+0000 7f0f6277fb80 20 client.16641 trim_cache size 0 max 16384 2023-10-10T17:48:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:48:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:48:29.883 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:48:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:48:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0f3c0a12c0 msgr2=0x7f0f3c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:29.884 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0f3c0a12c0 0x7f0f3c0a3770 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0f40005e90 tx=0x7f0f40019040 comp rx=0 tx=0).stop 2023-10-10T17:48:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 msgr2=0x7f0f50196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 0x7f0f50196410 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f0f3800ec10 tx=0x7f0f3800ef20 comp rx=0 tx=0).stop 2023-10-10T17:48:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 shutdown_connections 2023-10-10T17:48:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0f3c0a12c0 0x7f0f3c0a3770 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.885 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0f50105560 0x7f0f5019adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f50100c50 0x7f0f50196410 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 --2- 172.21.15.78:0/3653559362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0f501015f0 0x7f0f50196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.882+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 >> 172.21.15.78:0/3653559362 conn(0x7f0f50077550 msgr2=0x7f0f500feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:29.886 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.884+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 shutdown_connections 2023-10-10T17:48:29.887 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:29.884+0000 7f0f565f3700 1 -- 172.21.15.78:0/3653559362 wait complete. 2023-10-10T17:48:30.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:29 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:30.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:29 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:30.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:29 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:30.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:29 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:30.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:29 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:30.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:29 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:31.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:30 smithi078 ceph-mon[99494]: pgmap v182: 129 pgs: 129 active+clean; 794 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 446 op/s 2023-10-10T17:48:31.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:30 smithi190 ceph-mon[101220]: pgmap v182: 129 pgs: 129 active+clean; 794 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 446 op/s 2023-10-10T17:48:31.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:30 smithi157 ceph-mon[100896]: pgmap v182: 129 pgs: 129 active+clean; 794 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.6 MiB/s wr, 446 op/s 2023-10-10T17:48:32.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:31 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:32.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:31 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:32.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:31 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:32.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:31 smithi078 ceph-mon[99494]: pgmap v183: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 497 op/s 2023-10-10T17:48:32.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:31 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:32.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:31 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:32.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:31 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:32.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:31 smithi190 ceph-mon[101220]: pgmap v183: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 497 op/s 2023-10-10T17:48:32.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:31 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:32.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:31 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:32.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:31 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:32.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:31 smithi157 ceph-mon[100896]: pgmap v183: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 497 op/s 2023-10-10T17:48:34.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:34 smithi157 ceph-mon[100896]: pgmap v184: 129 pgs: 129 active+clean; 783 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 783 op/s 2023-10-10T17:48:35.015 INFO:tasks.workunit.client.0.smithi078.stdout:cleaned up hierarchy 2023-10-10T17:48:35.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:34 smithi078 ceph-mon[99494]: pgmap v184: 129 pgs: 129 active+clean; 783 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 783 op/s 2023-10-10T17:48:35.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:34 smithi190 ceph-mon[101220]: pgmap v184: 129 pgs: 129 active+clean; 783 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 3.8 MiB/s wr, 783 op/s 2023-10-10T17:48:36.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:36 smithi157 ceph-mon[100896]: pgmap v185: 129 pgs: 129 active+clean; 783 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 566 op/s 2023-10-10T17:48:37.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:36 smithi078 ceph-mon[99494]: pgmap v185: 129 pgs: 129 active+clean; 783 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 566 op/s 2023-10-10T17:48:37.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:36 smithi190 ceph-mon[101220]: pgmap v185: 129 pgs: 129 active+clean; 783 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.3 MiB/s wr, 566 op/s 2023-10-10T17:48:38.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:38 smithi157 ceph-mon[100896]: pgmap v186: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 729 op/s 2023-10-10T17:48:39.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:38 smithi078 ceph-mon[99494]: pgmap v186: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 729 op/s 2023-10-10T17:48:39.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:38 smithi190 ceph-mon[101220]: pgmap v186: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 3.9 MiB/s wr, 729 op/s 2023-10-10T17:48:40.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:40 smithi157 ceph-mon[100896]: pgmap v187: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 520 op/s 2023-10-10T17:48:41.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:40 smithi078 ceph-mon[99494]: pgmap v187: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 520 op/s 2023-10-10T17:48:41.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:40 smithi190 ceph-mon[101220]: pgmap v187: 129 pgs: 129 active+clean; 793 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 2.7 MiB/s wr, 520 op/s 2023-10-10T17:48:41.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:41 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:48:42.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:41 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:48:42.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:41 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:48:42.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:42 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:48:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:42 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:48:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:42 smithi157 ceph-mon[100896]: pgmap v188: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 549 op/s 2023-10-10T17:48:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 -- 172.21.15.78:0/3401891686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c1006d0 msgr2=0x7f536c103d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 --2- 172.21.15.78:0/3401891686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c1006d0 0x7f536c103d50 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f53600099c0 tx=0x7f5360009cd0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 -- 172.21.15.78:0/3401891686 shutdown_connections 2023-10-10T17:48:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 --2- 172.21.15.78:0/3401891686 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f536c0ffd30 0x7f536c100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.009 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 --2- 172.21.15.78:0/3401891686 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f536c104430 0x7f536c10c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 --2- 172.21.15.78:0/3401891686 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c1006d0 0x7f536c103d50 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 -- 172.21.15.78:0/3401891686 >> 172.21.15.78:0/3401891686 conn(0x7f536c075770 msgr2=0x7f536c075b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 -- 172.21.15.78:0/3401891686 shutdown_connections 2023-10-10T17:48:43.010 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.007+0000 7f537287a700 1 -- 172.21.15.78:0/3401891686 wait complete. 2023-10-10T17:48:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 Processor -- start 2023-10-10T17:48:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 -- start start 2023-10-10T17:48:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f536c0ffd30 0x7f536c195340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.012 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f536c1006d0 0x7f536c195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c104430 0x7f536c199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c19a320 con 0x7f536c104430 2023-10-10T17:48:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c19a490 con 0x7f536c0ffd30 2023-10-10T17:48:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f537287a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c19a650 con 0x7f536c1006d0 2023-10-10T17:48:43.013 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f5370e17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c104430 0x7f536c199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f536bfff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f536c0ffd30 0x7f536c195340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f5370e17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c104430 0x7f536c199ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47016/0 (socket says 172.21.15.78:47016) 2023-10-10T17:48:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f5370e17700 1 -- 172.21.15.78:0/822378123 learned_addr learned my addr 172.21.15.78:0/822378123 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.008+0000 7f536b7fe700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f536c1006d0 0x7f536c195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.014 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f5370e17700 1 -- 172.21.15.78:0/822378123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f536c1006d0 msgr2=0x7f536c195880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f5370e17700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f536c1006d0 0x7f536c195880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f5370e17700 1 -- 172.21.15.78:0/822378123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f536c0ffd30 msgr2=0x7f536c195340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f5370e17700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f536c0ffd30 0x7f536c195340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f5370e17700 1 -- 172.21.15.78:0/822378123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600095f0 con 0x7f536c104430 2023-10-10T17:48:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f5370e17700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c104430 0x7f536c199ce0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f536400cfb0 tx=0x7f536400e330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:43.015 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f53697fa700 1 -- 172.21.15.78:0/822378123 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364011070 con 0x7f536c104430 2023-10-10T17:48:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f53697fa700 1 -- 172.21.15.78:0/822378123 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5364007b30 con 0x7f536c104430 2023-10-10T17:48:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c10f1a0 con 0x7f536c104430 2023-10-10T17:48:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f53697fa700 1 -- 172.21.15.78:0/822378123 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536400aca0 con 0x7f536c104430 2023-10-10T17:48:43.016 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.009+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c10f010 con 0x7f536c104430 2023-10-10T17:48:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.011+0000 7f53697fa700 1 -- 172.21.15.78:0/822378123 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f536400c2e0 con 0x7f536c104430 2023-10-10T17:48:43.022 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.012+0000 7f53697fa700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f53540a12c0 0x7f53540a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.012+0000 7f536bfff700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f53540a12c0 0x7f53540a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.012+0000 7f53697fa700 1 -- 172.21.15.78:0/822378123 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f53640cea30 con 0x7f536c104430 2023-10-10T17:48:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.012+0000 7f536bfff700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f53540a12c0 0x7f53540a3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f535c009c60 tx=0x7f535c009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 Processor -- start 2023-10-10T17:48:43.023 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 -- start start 2023-10-10T17:48:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc98a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc984b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc88a000 con 0x55eedc98a720 2023-10-10T17:48:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc8dc840 con 0x55eedc7e06b0 2023-10-10T17:48:43.024 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f537ea06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc8d5cd0 con 0x55eedc98ab90 2023-10-10T17:48:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47022/0 (socket says 172.21.15.78:47022) 2023-10-10T17:48:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536bfff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc98a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 -- 172.21.15.78:0/833962979 learned_addr learned my addr 172.21.15.78:0/833962979 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:43.025 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f5370e17700 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc984b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 -- 172.21.15.78:0/833962979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 msgr2=0x55eedc984b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc984b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 -- 172.21.15.78:0/833962979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 msgr2=0x55eedc98a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc98a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 -- 172.21.15.78:0/833962979 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eedc7575d0 con 0x55eedc98a720 2023-10-10T17:48:43.026 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536bfff700 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc98a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.013+0000 7f536b7fe700 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc984560 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f53600036c0 tx=0x7f5360004890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535b487d4e7c88e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f53527fc700 1 -- 172.21.15.78:0/833962979 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360026070 con 0x55eedc98a720 2023-10-10T17:48:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f53527fc700 1 -- 172.21.15.78:0/833962979 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f536001e040 con 0x55eedc98a720 2023-10-10T17:48:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f53527fc700 1 -- 172.21.15.78:0/833962979 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536000faa0 con 0x55eedc98a720 2023-10-10T17:48:43.027 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 -- 172.21.15.78:0/833962979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 msgr2=0x55eedc984560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc984560 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f53600036c0 tx=0x7f5360004890 comp rx=0 tx=0).stop 2023-10-10T17:48:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 -- 172.21.15.78:0/833962979 shutdown_connections 2023-10-10T17:48:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc984b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc98a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 --2- 172.21.15.78:0/833962979 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.028 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 -- 172.21.15.78:0/833962979 >> 172.21.15.78:0/833962979 conn(0x55eedc979f50 msgr2=0x55eedc97f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 -- 172.21.15.78:0/833962979 shutdown_connections 2023-10-10T17:48:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.014+0000 7f537ea06b80 1 -- 172.21.15.78:0/833962979 wait complete. 2023-10-10T17:48:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 -- start start 2023-10-10T17:48:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 Processor -- start 2023-10-10T17:48:43.029 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc993b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc994050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc98dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc877880 con 0x55eedc98a720 2023-10-10T17:48:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc877520 con 0x55eedc7e06b0 2023-10-10T17:48:43.030 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc7598c0 con 0x55eedc98ab90 2023-10-10T17:48:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f537ea06b80 10 client.0 _ll_register_callbacks cb 0x55eedc895720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f536b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc994050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f536bfff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc993b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f536b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc994050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47028/0 (socket says 172.21.15.78:47028) 2023-10-10T17:48:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f536b7fe700 1 -- 172.21.15.78:0/4154981618 learned_addr learned my addr 172.21.15.78:0/4154981618 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:43.031 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.015+0000 7f5370e17700 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc98dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f536bfff700 1 -- 172.21.15.78:0/4154981618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 msgr2=0x55eedc98dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f536bfff700 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc98dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f536bfff700 1 -- 172.21.15.78:0/4154981618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 msgr2=0x55eedc994050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f536bfff700 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc994050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.032 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f536bfff700 1 -- 172.21.15.78:0/4154981618 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c10c810 con 0x55eedc7e06b0 2023-10-10T17:48:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f5370e17700 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc98dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f536bfff700 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc993b10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f535c011ab0 tx=0x7f535c0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c251238330a3a4ad server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f534b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:48:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f534b7fe700 1 -- 172.21.15.78:0/4154981618 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c022070 con 0x55eedc7e06b0 2023-10-10T17:48:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eedc977710 con 0x55eedc7e06b0 2023-10-10T17:48:43.033 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f534b7fe700 1 -- 172.21.15.78:0/4154981618 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f535c01d620 con 0x55eedc7e06b0 2023-10-10T17:48:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f534b7fe700 1 -- 172.21.15.78:0/4154981618 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f535c01d7e0 con 0x55eedc7e06b0 2023-10-10T17:48:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f537ea06b80 10 client.26018 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55eedc89c130 con 0x55eedc7e06b0 2023-10-10T17:48:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.016+0000 7f534b7fe700 1 -- 172.21.15.78:0/4154981618 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eedc977710 con 0x55eedc7e06b0 2023-10-10T17:48:43.034 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f534b7fe700 1 -- 172.21.15.78:0/4154981618 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f535c027a20 con 0x55eedc7e06b0 2023-10-10T17:48:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f537ea06b80 10 client.26018 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f537ea06b80 10 client.26018 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f537ea06b80 20 client.26018 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc99e0c0 0x55eedc9a04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f537ea06b80 4 client.26018 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:43.035 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eedc898040 con 0x55eedc99e0c0 2023-10-10T17:48:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f536b7fe700 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc99e0c0 0x55eedc9a04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f536b7fe700 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc99e0c0 0x55eedc9a04d0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91c266841ca98f36 server_cookie=ff9005af928e8d28 in_seq=0 out_seq=0 2023-10-10T17:48:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.017+0000 7f534b7fe700 10 client.26018 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f534b7fe700 1 -- 172.21.15.78:0/4154981618 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55eedc898040 con 0x55eedc99e0c0 2023-10-10T17:48:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f534b7fe700 10 client.26018 handle_command_reply: tid=0 2023-10-10T17:48:43.036 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 client.26018 shutdown 2023-10-10T17:48:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 msgr2=0x55eedc993b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc993b10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f535c011ab0 tx=0x7f535c0047c0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 shutdown_connections 2023-10-10T17:48:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc98dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.037 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc99e0c0 0x55eedc9a04d0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc7e06b0 0x55eedc993b10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f534b7fe700 0 client.26018 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:43.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4154981618 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc98a720 0x55eedc994050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 >> 172.21.15.78:0/4154981618 conn(0x55eedc979f50 msgr2=0x55eedc97e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:43.038 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 shutdown_connections 2023-10-10T17:48:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 1 -- 172.21.15.78:0/4154981618 wait complete. 2023-10-10T17:48:43.039 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.018+0000 7f537ea06b80 20 client.26018 trim_cache size 0 max 16384 2023-10-10T17:48:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 Processor -- start 2023-10-10T17:48:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 -- start start 2023-10-10T17:48:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc98dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98ab90 0x55eedc97ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.044 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc988a50 con 0x55eedc7e06b0 2023-10-10T17:48:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc98e290 con 0x55eedc98ab90 2023-10-10T17:48:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f537ea06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc977550 con 0x55eedc98a720 2023-10-10T17:48:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f536b7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f536b7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38564/0 (socket says 172.21.15.78:38564) 2023-10-10T17:48:43.045 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f5370e17700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98ab90 0x55eedc97ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f536bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc98dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f536b7fe700 1 -- 172.21.15.78:0/1871901213 learned_addr learned my addr 172.21.15.78:0/1871901213 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f5370e17700 1 -- 172.21.15.78:0/1871901213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98a720 msgr2=0x55eedc984560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f5370e17700 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.046 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f5370e17700 1 -- 172.21.15.78:0/1871901213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 msgr2=0x55eedc98dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f5370e17700 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc98dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f5370e17700 1 -- 172.21.15.78:0/1871901213 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eedc89be80 con 0x55eedc98ab90 2023-10-10T17:48:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f536b7fe700 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.033+0000 7f536bfff700 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc98dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f5370e17700 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98ab90 0x55eedc97ef60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f53640cbe10 tx=0x7f53640bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ca63bbc789f6b16 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:43.047 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f5352ffd700 1 -- 172.21.15.78:0/1871901213 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53640a3be0 con 0x55eedc98ab90 2023-10-10T17:48:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f5352ffd700 1 -- 172.21.15.78:0/1871901213 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5364099b30 con 0x55eedc98ab90 2023-10-10T17:48:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f5352ffd700 1 -- 172.21.15.78:0/1871901213 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536409aee0 con 0x55eedc98ab90 2023-10-10T17:48:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 -- 172.21.15.78:0/1871901213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98ab90 msgr2=0x55eedc97ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98ab90 0x55eedc97ef60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f53640cbe10 tx=0x7f53640bb040 comp rx=0 tx=0).stop 2023-10-10T17:48:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 -- 172.21.15.78:0/1871901213 shutdown_connections 2023-10-10T17:48:43.048 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98ab90 0x55eedc97ef60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc98dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 --2- 172.21.15.78:0/1871901213 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98a720 0x55eedc984560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 -- 172.21.15.78:0/1871901213 >> 172.21.15.78:0/1871901213 conn(0x55eedc979f50 msgr2=0x55eedc98d640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 -- 172.21.15.78:0/1871901213 shutdown_connections 2023-10-10T17:48:43.049 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.034+0000 7f537ea06b80 1 -- 172.21.15.78:0/1871901213 wait complete. 2023-10-10T17:48:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 -- start start 2023-10-10T17:48:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 Processor -- start 2023-10-10T17:48:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc9916c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98a720 0x55eedc991c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.050 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc992140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc9f0f80 con 0x55eedc7e06b0 2023-10-10T17:48:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc9f10c0 con 0x55eedc98a720 2023-10-10T17:48:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedc9f1200 con 0x55eedc98ab90 2023-10-10T17:48:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f537ea06b80 10 client.0 _ll_register_callbacks cb 0x55eedc895720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f536bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc9916c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.051 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f536b7fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98a720 0x55eedc991c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f536bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc9916c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47040/0 (socket says 172.21.15.78:47040) 2023-10-10T17:48:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f5370e17700 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc992140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.035+0000 7f536bfff700 1 -- 172.21.15.78:0/4030614785 learned_addr learned my addr 172.21.15.78:0/4030614785 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f536b7fe700 1 -- 172.21.15.78:0/4030614785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 msgr2=0x55eedc992140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f536b7fe700 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc992140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.052 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f536b7fe700 1 -- 172.21.15.78:0/4030614785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 msgr2=0x55eedc9916c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f536b7fe700 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc9916c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f536b7fe700 1 -- 172.21.15.78:0/4030614785 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eedc89c130 con 0x55eedc98a720 2023-10-10T17:48:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f536b7fe700 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98a720 0x55eedc991c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f53600058e0 tx=0x7f536000bb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5155bfb7e3e5ed37 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f534b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:48:43.053 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f534b7fe700 1 -- 172.21.15.78:0/4030614785 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360026040 con 0x55eedc98a720 2023-10-10T17:48:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f536001e040 con 0x55eedc98a720 2023-10-10T17:48:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f534b7fe700 1 -- 172.21.15.78:0/4030614785 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f5360030cc0 con 0x55eedc98a720 2023-10-10T17:48:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f534b7fe700 1 -- 172.21.15.78:0/4030614785 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5360030e80 con 0x55eedc98a720 2023-10-10T17:48:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f537ea06b80 10 client.26024 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:43.054 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55eedc7575d0 con 0x55eedc98a720 2023-10-10T17:48:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f534b7fe700 1 -- 172.21.15.78:0/4030614785 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f536001e040 con 0x55eedc98a720 2023-10-10T17:48:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.036+0000 7f534b7fe700 1 -- 172.21.15.78:0/4030614785 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5360021070 con 0x55eedc98a720 2023-10-10T17:48:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f537ea06b80 10 client.26024 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f537ea06b80 10 client.26024 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f537ea06b80 20 client.26024 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:43.055 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc9f24e0 0x55eedc9f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f537ea06b80 4 client.26024 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:48:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55eedc8a4600 con 0x55eedc9f24e0 2023-10-10T17:48:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f536bfff700 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc9f24e0 0x55eedc9f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f534b7fe700 10 client.26024 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:43.056 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.037+0000 7f536bfff700 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc9f24e0 0x55eedc9f48c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aed9f63c1404a1e3 server_cookie=111ceb393ebabe54 in_seq=0 out_seq=0 2023-10-10T17:48:43.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:42 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:48:43.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:42 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:48:43.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:42 smithi078 ceph-mon[99494]: pgmap v188: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 549 op/s 2023-10-10T17:48:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f534b7fe700 1 -- 172.21.15.78:0/4030614785 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55eedc8a4600 con 0x55eedc9f24e0 2023-10-10T17:48:43.148 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f534b7fe700 10 client.26024 handle_command_reply: tid=0 2023-10-10T17:48:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 client.26024 shutdown 2023-10-10T17:48:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98a720 msgr2=0x55eedc991c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98a720 0x55eedc991c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f53600058e0 tx=0x7f536000bb10 comp rx=0 tx=0).stop 2023-10-10T17:48:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 shutdown_connections 2023-10-10T17:48:43.149 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55eedc98ab90 0x55eedc992140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55eedc9f24e0 0x55eedc9f48c0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55eedc7e06b0 0x55eedc9916c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 --2- 172.21.15.78:0/4030614785 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55eedc98a720 0x55eedc991c00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.150 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f534b7fe700 0 client.26024 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.146+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 >> 172.21.15.78:0/4030614785 conn(0x55eedc979f50 msgr2=0x55eedc988e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 shutdown_connections 2023-10-10T17:48:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537ea06b80 1 -- 172.21.15.78:0/4030614785 wait complete. 2023-10-10T17:48:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537ea06b80 20 client.26024 trim_cache size 0 max 16384 2023-10-10T17:48:43.151 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:48:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:48:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:48:43.152 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:48:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f53540a12c0 msgr2=0x7f53540a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f53540a12c0 0x7f53540a3770 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f535c009c60 tx=0x7f535c009260 comp rx=0 tx=0).stop 2023-10-10T17:48:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c104430 msgr2=0x7f536c199ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c104430 0x7f536c199ce0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f536400cfb0 tx=0x7f536400e330 comp rx=0 tx=0).stop 2023-10-10T17:48:43.153 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 shutdown_connections 2023-10-10T17:48:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f53540a12c0 0x7f53540a3770 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f536c104430 0x7f536c199ce0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f536c0ffd30 0x7f536c195340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.154 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 --2- 172.21.15.78:0/822378123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f536c1006d0 0x7f536c195880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 >> 172.21.15.78:0/822378123 conn(0x7f536c075770 msgr2=0x7f536c0fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 shutdown_connections 2023-10-10T17:48:43.155 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:43.147+0000 7f537287a700 1 -- 172.21.15.78:0/822378123 wait complete. 2023-10-10T17:48:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:42 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:48:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:42 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:48:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:42 smithi190 ceph-mon[101220]: pgmap v188: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 2.8 MiB/s wr, 549 op/s 2023-10-10T17:48:44.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:44 smithi157 ceph-mon[100896]: pgmap v189: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 582 op/s 2023-10-10T17:48:45.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:44 smithi078 ceph-mon[99494]: pgmap v189: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 582 op/s 2023-10-10T17:48:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:44 smithi190 ceph-mon[101220]: pgmap v189: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 582 op/s 2023-10-10T17:48:46.980 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:46 smithi157 ceph-mon[100896]: pgmap v190: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 275 op/s 2023-10-10T17:48:47.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:46 smithi078 ceph-mon[99494]: pgmap v190: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 275 op/s 2023-10-10T17:48:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:46 smithi190 ceph-mon[101220]: pgmap v190: 129 pgs: 129 active+clean; 790 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 275 op/s 2023-10-10T17:48:47.288 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:48:47.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 -- 172.21.15.78:0/1548010778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96140ff990 msgr2=0x7f96140ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 --2- 172.21.15.78:0/1548010778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96140ff990 0x7f96140ffe00 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f96040099c0 tx=0x7f9604009cd0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 -- 172.21.15.78:0/1548010778 shutdown_connections 2023-10-10T17:48:47.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 --2- 172.21.15.78:0/1548010778 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9614104180 0x7f9614104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 --2- 172.21.15.78:0/1548010778 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9614100340 0x7f961410c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 --2- 172.21.15.78:0/1548010778 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96140ff990 0x7f96140ffe00 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 -- 172.21.15.78:0/1548010778 >> 172.21.15.78:0/1548010778 conn(0x7f9614075920 msgr2=0x7f9614075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 -- 172.21.15.78:0/1548010778 shutdown_connections 2023-10-10T17:48:47.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.447+0000 7f9619032700 1 -- 172.21.15.78:0/1548010778 wait complete. 2023-10-10T17:48:47.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 Processor -- start 2023-10-10T17:48:47.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 -- start start 2023-10-10T17:48:47.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96140ff990 0x7f96141963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9614100340 0x7f9614196910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9614104180 0x7f961419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961419b3b0 con 0x7f9614104180 2023-10-10T17:48:47.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961419b520 con 0x7f96140ff990 2023-10-10T17:48:47.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9619032700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961419b6e0 con 0x7f9614100340 2023-10-10T17:48:47.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f961359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9614104180 0x7f961419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f9612d9d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96140ff990 0x7f96141963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f961359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9614104180 0x7f961419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47058/0 (socket says 172.21.15.78:47058) 2023-10-10T17:48:47.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f961359e700 1 -- 172.21.15.78:0/518035002 learned_addr learned my addr 172.21.15.78:0/518035002 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:47.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.448+0000 7f961259c700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9614100340 0x7f9614196910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f961359e700 1 -- 172.21.15.78:0/518035002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9614100340 msgr2=0x7f9614196910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f961359e700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9614100340 0x7f9614196910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f961359e700 1 -- 172.21.15.78:0/518035002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96140ff990 msgr2=0x7f96141963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f961359e700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96140ff990 0x7f96141963d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f961359e700 1 -- 172.21.15.78:0/518035002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96040095f0 con 0x7f9614104180 2023-10-10T17:48:47.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f961359e700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9614104180 0x7f961419ad70 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f960c007ee0 tx=0x7f960c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f95fbfff700 1 -- 172.21.15.78:0/518035002 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c004db0 con 0x7f9614104180 2023-10-10T17:48:47.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961410f210 con 0x7f9614104180 2023-10-10T17:48:47.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f95fbfff700 1 -- 172.21.15.78:0/518035002 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f960c00f040 con 0x7f9614104180 2023-10-10T17:48:47.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f95fbfff700 1 -- 172.21.15.78:0/518035002 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0058a0 con 0x7f9614104180 2023-10-10T17:48:47.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.449+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f961410f080 con 0x7f9614104180 2023-10-10T17:48:47.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.451+0000 7f95fbfff700 1 -- 172.21.15.78:0/518035002 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f960c00db20 con 0x7f9614104180 2023-10-10T17:48:47.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.451+0000 7f95fbfff700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f95fc0a12c0 0x7f95fc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.452+0000 7f9612d9d700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f95fc0a12c0 0x7f95fc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.452+0000 7f95fbfff700 1 -- 172.21.15.78:0/518035002 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f960c0cc960 con 0x7f9614104180 2023-10-10T17:48:47.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.452+0000 7f9612d9d700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f95fc0a12c0 0x7f95fc0a3770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9608009be0 tx=0x7f9608009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.452+0000 7f96251beb80 1 Processor -- start 2023-10-10T17:48:47.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f96251beb80 1 -- start start 2023-10-10T17:48:47.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f96251beb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071b5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f96251beb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b5880 0x557c071b9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f96251beb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b9cc0 0x557c071ba110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f96251beb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c070b6060 con 0x557c071b4f70 2023-10-10T17:48:47.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f96251beb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c071088a0 con 0x557c071b5880 2023-10-10T17:48:47.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f96251beb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c07101d30 con 0x557c071b9cc0 2023-10-10T17:48:47.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071b5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47062/0 (socket says 172.21.15.78:47062) 2023-10-10T17:48:47.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 -- 172.21.15.78:0/3686004862 learned_addr learned my addr 172.21.15.78:0/3686004862 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:47.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f961259c700 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b5880 0x557c071b9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f961359e700 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b9cc0 0x557c071ba110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 -- 172.21.15.78:0/3686004862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b9cc0 msgr2=0x557c071ba110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b9cc0 0x557c071ba110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 -- 172.21.15.78:0/3686004862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b5880 msgr2=0x557c071b9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b5880 0x557c071b9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 -- 172.21.15.78:0/3686004862 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c06f83630 con 0x557c071b4f70 2023-10-10T17:48:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f9612d9d700 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071b5340 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9608011d70 tx=0x7f9608014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd54aea08fdab388 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f95f97fa700 1 -- 172.21.15.78:0/3686004862 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608011da0 con 0x557c071b4f70 2023-10-10T17:48:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f95f97fa700 1 -- 172.21.15.78:0/3686004862 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9608004020 con 0x557c071b4f70 2023-10-10T17:48:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.453+0000 7f95f97fa700 1 -- 172.21.15.78:0/3686004862 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608003bc0 con 0x557c071b4f70 2023-10-10T17:48:47.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 -- 172.21.15.78:0/3686004862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 msgr2=0x557c071b5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071b5340 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9608011d70 tx=0x7f9608014000 comp rx=0 tx=0).stop 2023-10-10T17:48:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 -- 172.21.15.78:0/3686004862 shutdown_connections 2023-10-10T17:48:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b9cc0 0x557c071ba110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071b5340 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 --2- 172.21.15.78:0/3686004862 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b5880 0x557c071b9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 -- 172.21.15.78:0/3686004862 >> 172.21.15.78:0/3686004862 conn(0x557c071a6360 msgr2=0x557c071abb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 -- 172.21.15.78:0/3686004862 shutdown_connections 2023-10-10T17:48:47.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 -- 172.21.15.78:0/3686004862 wait complete. 2023-10-10T17:48:47.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.454+0000 7f96251beb80 1 -- start start 2023-10-10T17:48:47.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 1 Processor -- start 2023-10-10T17:48:47.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071be740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c070a38e0 con 0x557c071b4f70 2023-10-10T17:48:47.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c070a3580 con 0x557c071b9cc0 2023-10-10T17:48:47.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c06f85920 con 0x557c071b5880 2023-10-10T17:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f96251beb80 10 client.0 _ll_register_callbacks cb 0x557c070c1780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47068/0 (socket says 172.21.15.78:47068) 2023-10-10T17:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 -- 172.21.15.78:0/733468107 learned_addr learned my addr 172.21.15.78:0/733468107 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f961259c700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071be740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f961359e700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 -- 172.21.15.78:0/733468107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 msgr2=0x557c071be740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071be740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 -- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 msgr2=0x557c071bedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 -- 172.21.15.78:0/733468107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9608017070 con 0x557c071b4f70 2023-10-10T17:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f961359e700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f961259c700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071be740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f9612d9d700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c1450 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9608009210 tx=0x7f9608011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a13d6505068982d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f95f6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f95f6ff5700 1 -- 172.21.15.78:0/733468107 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9608003a60 con 0x557c071b4f70 2023-10-10T17:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.455+0000 7f95f6ff5700 1 -- 172.21.15.78:0/733468107 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f9608003bc0 con 0x557c071b4f70 2023-10-10T17:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c071a3b20 con 0x557c071b4f70 2023-10-10T17:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f95f6ff5700 1 -- 172.21.15.78:0/733468107 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9608003d80 con 0x557c071b4f70 2023-10-10T17:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 10 client.16671 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x557c070c8190 con 0x557c071b4f70 2023-10-10T17:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f95f6ff5700 1 -- 172.21.15.78:0/733468107 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557c071a3b20 con 0x557c071b4f70 2023-10-10T17:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f95f6ff5700 1 -- 172.21.15.78:0/733468107 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f9608024030 con 0x557c071b4f70 2023-10-10T17:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 10 client.16671 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 10 client.16671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 20 client.16671 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c071ceb60 0x557c071d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 4 client.16671 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c070c40a0 con 0x557c071ceb60 2023-10-10T17:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.456+0000 7f961259c700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c071ceb60 0x557c071d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f95f6ff5700 10 client.16671 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f961259c700 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c071ceb60 0x557c071d0f70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c546632e0b40fa4 server_cookie=e2f29b02bf76d126 in_seq=0 out_seq=0 2023-10-10T17:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f95f6ff5700 1 -- 172.21.15.78:0/733468107 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x557c070c40a0 con 0x557c071ceb60 2023-10-10T17:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f95f6ff5700 10 client.16671 handle_command_reply: tid=0 2023-10-10T17:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 client.16671 shutdown 2023-10-10T17:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 msgr2=0x557c071c1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c1450 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9608009210 tx=0x7f9608011da0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 shutdown_connections 2023-10-10T17:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c071ceb60 0x557c071d0f70 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c1450 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 --2- 172.21.15.78:0/733468107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071be740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f95f6ff5700 0 client.16671 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 >> 172.21.15.78:0/733468107 conn(0x557c071a6360 msgr2=0x557c071aade0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 shutdown_connections 2023-10-10T17:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.457+0000 7f96251beb80 1 -- 172.21.15.78:0/733468107 wait complete. 2023-10-10T17:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.458+0000 7f96251beb80 20 client.16671 trim_cache size 0 max 16384 2023-10-10T17:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 Processor -- start 2023-10-10T17:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 -- start start 2023-10-10T17:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071caff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071ab560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c071bfcb0 con 0x557c071b4f70 2023-10-10T17:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c071ac950 con 0x557c071b9cc0 2023-10-10T17:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.474+0000 7f96251beb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c071bccc0 con 0x557c071b5880 2023-10-10T17:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47076/0 (socket says 172.21.15.78:47076) 2023-10-10T17:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 -- 172.21.15.78:0/3240699298 learned_addr learned my addr 172.21.15.78:0/3240699298 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f961359e700 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f961259c700 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 -- 172.21.15.78:0/3240699298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 msgr2=0x557c071caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 -- 172.21.15.78:0/3240699298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 msgr2=0x557c071ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071ab560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 -- 172.21.15.78:0/3240699298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c070c7ee0 con 0x557c071b4f70 2023-10-10T17:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f9612d9d700 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c5db0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f960801e420 tx=0x7f9608005b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d11f0abf56a19e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f95f9ffb700 1 -- 172.21.15.78:0/3240699298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960801e690 con 0x557c071b4f70 2023-10-10T17:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f95f9ffb700 1 -- 172.21.15.78:0/3240699298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9608005040 con 0x557c071b4f70 2023-10-10T17:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.475+0000 7f95f9ffb700 1 -- 172.21.15.78:0/3240699298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960801a600 con 0x557c071b4f70 2023-10-10T17:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- 172.21.15.78:0/3240699298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 msgr2=0x557c071c5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c5db0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f960801e420 tx=0x7f9608005b40 comp rx=0 tx=0).stop 2023-10-10T17:48:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- 172.21.15.78:0/3240699298 shutdown_connections 2023-10-10T17:48:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b4f70 0x557c071c5db0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 --2- 172.21.15.78:0/3240699298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b5880 0x557c071caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- 172.21.15.78:0/3240699298 >> 172.21.15.78:0/3240699298 conn(0x557c071a6360 msgr2=0x557c071b8300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- 172.21.15.78:0/3240699298 shutdown_connections 2023-10-10T17:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- 172.21.15.78:0/3240699298 wait complete. 2023-10-10T17:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- start start 2023-10-10T17:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 Processor -- start 2023-10-10T17:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b4f70 0x557c071baad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b5880 0x557c071bb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c071bbce0 con 0x557c071b5880 2023-10-10T17:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c071bbe50 con 0x557c071b9cc0 2023-10-10T17:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.476+0000 7f96251beb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c071bc010 con 0x557c071b4f70 2023-10-10T17:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f96251beb80 10 client.0 _ll_register_callbacks cb 0x7f9608005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b5880 0x557c071bb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b5880 0x557c071bb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47084/0 (socket says 172.21.15.78:47084) 2023-10-10T17:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 -- 172.21.15.78:0/2009383352 learned_addr learned my addr 172.21.15.78:0/2009383352 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f9612d9d700 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b4f70 0x557c071baad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961359e700 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 -- 172.21.15.78:0/2009383352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b4f70 msgr2=0x557c071baad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b4f70 0x557c071baad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 -- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 msgr2=0x557c071bb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:47.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 -- 172.21.15.78:0/2009383352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c070c8190 con 0x557c071b5880 2023-10-10T17:48:47.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f961259c700 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b5880 0x557c071bb010 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9604009990 tx=0x7f9604027510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad515ec83d087cc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f95f6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:47.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f95f6ff5700 1 -- 172.21.15.78:0/2009383352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9604027cc0 con 0x557c071b5880 2023-10-10T17:48:47.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f95f6ff5700 1 -- 172.21.15.78:0/2009383352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f9604027e20 con 0x557c071b5880 2023-10-10T17:48:47.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9608004020 con 0x557c071b5880 2023-10-10T17:48:47.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f95f6ff5700 1 -- 172.21.15.78:0/2009383352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96040056e0 con 0x557c071b5880 2023-10-10T17:48:47.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f96251beb80 10 client.16677 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:47.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.477+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x557c06f18c60 con 0x557c071b5880 2023-10-10T17:48:47.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f95f6ff5700 1 -- 172.21.15.78:0/2009383352 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9608004020 con 0x557c071b5880 2023-10-10T17:48:47.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f95f6ff5700 1 -- 172.21.15.78:0/2009383352 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f960400bcc0 con 0x557c071b5880 2023-10-10T17:48:47.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f96251beb80 10 client.16677 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:47.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f96251beb80 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:47.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f96251beb80 20 client.16677 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:47.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f96251beb80 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c072238a0 0x557c07225cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f96251beb80 4 client.16677 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:48:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557c070d0660 con 0x557c072238a0 2023-10-10T17:48:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f9612d9d700 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c072238a0 0x557c07225cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f95f6ff5700 10 client.16677 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f9612d9d700 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c072238a0 0x557c07225cb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e4d1e22b8b80672 server_cookie=5e02675ce797dee9 in_seq=0 out_seq=0 2023-10-10T17:48:47.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f95f6ff5700 1 -- 172.21.15.78:0/2009383352 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557c070d0660 con 0x557c072238a0 2023-10-10T17:48:47.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f95f6ff5700 10 client.16677 handle_command_reply: tid=0 2023-10-10T17:48:47.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.478+0000 7f96251beb80 1 client.16677 shutdown 2023-10-10T17:48:47.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b5880 msgr2=0x557c071bb010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b5880 0x557c071bb010 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9604009990 tx=0x7f9604027510 comp rx=0 tx=0).stop 2023-10-10T17:48:47.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 shutdown_connections 2023-10-10T17:48:47.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x557c071b4f70 0x557c071baad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x557c072238a0 0x557c07225cb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x557c071b9cc0 0x557c071bb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f95f6ff5700 0 client.16677 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:47.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 --2- 172.21.15.78:0/2009383352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x557c071b5880 0x557c071bb010 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 >> 172.21.15.78:0/2009383352 conn(0x557c071a6360 msgr2=0x557c071b8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 shutdown_connections 2023-10-10T17:48:47.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 1 -- 172.21.15.78:0/2009383352 wait complete. 2023-10-10T17:48:47.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f96251beb80 20 client.16677 trim_cache size 0 max 16384 2023-10-10T17:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:48:47.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f95fc0a12c0 msgr2=0x7f95fc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f95fc0a12c0 0x7f95fc0a3770 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9608009be0 tx=0x7f9608009260 comp rx=0 tx=0).stop 2023-10-10T17:48:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9614104180 msgr2=0x7f961419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9614104180 0x7f961419ad70 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f960c007ee0 tx=0x7f960c00d3b0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 shutdown_connections 2023-10-10T17:48:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f95fc0a12c0 0x7f95fc0a3770 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9614104180 0x7f961419ad70 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96140ff990 0x7f96141963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 --2- 172.21.15.78:0/518035002 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9614100340 0x7f9614196910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 >> 172.21.15.78:0/518035002 conn(0x7f9614075920 msgr2=0x7f96140fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 shutdown_connections 2023-10-10T17:48:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.479+0000 7f9619032700 1 -- 172.21.15.78:0/518035002 wait complete. 2023-10-10T17:48:47.507 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:48:47.507 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 -- 172.21.15.78:0/2845582252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b241014c0 msgr2=0x7f4b24101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 --2- 172.21.15.78:0/2845582252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b241014c0 0x7f4b24101890 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4b14009990 tx=0x7f4b14009ca0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 -- 172.21.15.78:0/2845582252 shutdown_connections 2023-10-10T17:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 --2- 172.21.15.78:0/2845582252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b2410a800 0x7f4b2410cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 --2- 172.21.15.78:0/2845582252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b241014c0 0x7f4b24101890 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 --2- 172.21.15.78:0/2845582252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b24101dd0 0x7f4b2410a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 -- 172.21.15.78:0/2845582252 >> 172.21.15.78:0/2845582252 conn(0x7f4b24076310 msgr2=0x7f4b24076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 -- 172.21.15.78:0/2845582252 shutdown_connections 2023-10-10T17:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 -- 172.21.15.78:0/2845582252 wait complete. 2023-10-10T17:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.638+0000 7f4b298fc700 1 Processor -- start 2023-10-10T17:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b298fc700 1 -- start start 2023-10-10T17:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b298fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b241014c0 0x7f4b24196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b298fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b24101dd0 0x7f4b241969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b298fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b2410a800 0x7f4b2419ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b298fc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2419b450 con 0x7f4b24101dd0 2023-10-10T17:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b298fc700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2419b5c0 con 0x7f4b2410a800 2023-10-10T17:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b298fc700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2419b780 con 0x7f4b241014c0 2023-10-10T17:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b24101dd0 0x7f4b241969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b24101dd0 0x7f4b241969b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47098/0 (socket says 172.21.15.78:47098) 2023-10-10T17:48:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b22ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b241014c0 0x7f4b24196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 -- 172.21.15.78:0/189219662 learned_addr learned my addr 172.21.15.78:0/189219662 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b237fe700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b2410a800 0x7f4b2419ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 -- 172.21.15.78:0/189219662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b241014c0 msgr2=0x7f4b24196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b241014c0 0x7f4b24196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 -- 172.21.15.78:0/189219662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b2410a800 msgr2=0x7f4b2419ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b2410a800 0x7f4b2419ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 -- 172.21.15.78:0/189219662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b140095f0 con 0x7f4b24101dd0 2023-10-10T17:48:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.639+0000 7f4b227fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b24101dd0 0x7f4b241969b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e9a0 tx=0x7f4b1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.640+0000 7f4b288fa700 1 -- 172.21.15.78:0/189219662 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18009790 con 0x7f4b24101dd0 2023-10-10T17:48:47.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.640+0000 7f4b288fa700 1 -- 172.21.15.78:0/189219662 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b18010430 con 0x7f4b24101dd0 2023-10-10T17:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.640+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b24110340 con 0x7f4b24101dd0 2023-10-10T17:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.640+0000 7f4b288fa700 1 -- 172.21.15.78:0/189219662 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18018680 con 0x7f4b24101dd0 2023-10-10T17:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.640+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b241101b0 con 0x7f4b24101dd0 2023-10-10T17:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.641+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b241104d0 con 0x7f4b24101dd0 2023-10-10T17:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.642+0000 7f4b288fa700 1 -- 172.21.15.78:0/189219662 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4b180105a0 con 0x7f4b24101dd0 2023-10-10T17:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.642+0000 7f4b288fa700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4b0c0a12c0 0x7f4b0c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.642+0000 7f4b288fa700 1 -- 172.21.15.78:0/189219662 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f4b18014070 con 0x7f4b24101dd0 2023-10-10T17:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.642+0000 7f4b22ffd700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4b0c0a12c0 0x7f4b0c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.643+0000 7f4b22ffd700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4b0c0a12c0 0x7f4b0c0a3770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400b2e0 tx=0x7f4b14005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.648+0000 7f4b288fa700 1 -- 172.21.15.78:0/189219662 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4b18051730 con 0x7f4b24101dd0 2023-10-10T17:48:47.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.915+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b24067b00 con 0x7f4b24101dd0 2023-10-10T17:48:47.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.916+0000 7f4b288fa700 1 -- 172.21.15.78:0/189219662 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10817 (secure 0 0 0) 0x7f4b180962e0 con 0x7f4b24101dd0 2023-10-10T17:48:47.918 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:48:47.918 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:46:23.648656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:48:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4b0c0a12c0 msgr2=0x7f4b0c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4b0c0a12c0 0x7f4b0c0a3770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400b2e0 tx=0x7f4b14005fd0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b24101dd0 msgr2=0x7f4b241969b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:47.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b24101dd0 0x7f4b241969b0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800e9a0 tx=0x7f4b1800ecb0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 shutdown_connections 2023-10-10T17:48:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4b241014c0 0x7f4b24196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f4b0c0a12c0 0x7f4b0c0a3770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4b2410a800 0x7f4b2419ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 --2- 172.21.15.78:0/189219662 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4b24101dd0 0x7f4b241969b0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 >> 172.21.15.78:0/189219662 conn(0x7f4b24076310 msgr2=0x7f4b24102260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 shutdown_connections 2023-10-10T17:48:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:47.918+0000 7f4b298fc700 1 -- 172.21.15.78:0/189219662 wait complete. 2023-10-10T17:48:47.922 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2023-10-10T17:48:47.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:48:48.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 -- 172.21.15.78:0/2456515279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 msgr2=0x7f911c10c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 --2- 172.21.15.78:0/2456515279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c10c060 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f911400b0d0 tx=0x7f911400b3e0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 -- 172.21.15.78:0/2456515279 shutdown_connections 2023-10-10T17:48:48.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 --2- 172.21.15.78:0/2456515279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f911c0ff450 0x7f911c0ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 --2- 172.21.15.78:0/2456515279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c10c060 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 --2- 172.21.15.78:0/2456515279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911c0ffdf0 0x7f911c103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 -- 172.21.15.78:0/2456515279 >> 172.21.15.78:0/2456515279 conn(0x7f911c0770b0 msgr2=0x7f911c0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 -- 172.21.15.78:0/2456515279 shutdown_connections 2023-10-10T17:48:48.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.056+0000 7f9123ab6700 1 -- 172.21.15.78:0/2456515279 wait complete. 2023-10-10T17:48:48.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 Processor -- start 2023-10-10T17:48:48.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 -- start start 2023-10-10T17:48:48.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911c0ff450 0x7f911c195450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f911c0ffdf0 0x7f911c195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c19a430 con 0x7f911c104490 2023-10-10T17:48:48.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c19a5a0 con 0x7f911c0ff450 2023-10-10T17:48:48.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9123ab6700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c19a760 con 0x7f911c0ffdf0 2023-10-10T17:48:48.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9121852700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911c0ff450 0x7f911c195450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c199df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47112/0 (socket says 172.21.15.78:47112) 2023-10-10T17:48:48.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 -- 172.21.15.78:0/3337529748 learned_addr learned my addr 172.21.15.78:0/3337529748 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9121051700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f911c0ffdf0 0x7f911c195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 -- 172.21.15.78:0/3337529748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f911c0ffdf0 msgr2=0x7f911c195990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f911c0ffdf0 0x7f911c195990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 -- 172.21.15.78:0/3337529748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911c0ff450 msgr2=0x7f911c195450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911c0ff450 0x7f911c195450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.057+0000 7f9122053700 1 -- 172.21.15.78:0/3337529748 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9114009d00 con 0x7f911c104490 2023-10-10T17:48:48.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.058+0000 7f9122053700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c199df0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9114007820 tx=0x7f9114007850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.058+0000 7f9112ffd700 1 -- 172.21.15.78:0/3337529748 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911400e040 con 0x7f911c104490 2023-10-10T17:48:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.058+0000 7f9112ffd700 1 -- 172.21.15.78:0/3337529748 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f911400bcb0 con 0x7f911c104490 2023-10-10T17:48:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.058+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c10f2b0 con 0x7f911c104490 2023-10-10T17:48:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.058+0000 7f9112ffd700 1 -- 172.21.15.78:0/3337529748 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91140058a0 con 0x7f911c104490 2023-10-10T17:48:48.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.058+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c10f120 con 0x7f911c104490 2023-10-10T17:48:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 Processor -- start 2023-10-10T17:48:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 -- start start 2023-10-10T17:48:48.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e3470 0x559f083e7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e78b0 0x559f083e7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f082e8060 con 0x559f083e78b0 2023-10-10T17:48:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0833a8a0 con 0x559f083e2b60 2023-10-10T17:48:48.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.061+0000 7f912fc42b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f08333d30 con 0x559f083e3470 2023-10-10T17:48:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48836/0 (socket says 172.21.15.78:48836) 2023-10-10T17:48:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121852700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e3470 0x559f083e7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 -- 172.21.15.78:0/3753732179 learned_addr learned my addr 172.21.15.78:0/3753732179 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e78b0 0x559f083e7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 -- 172.21.15.78:0/3753732179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e3470 msgr2=0x559f083e7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e3470 0x559f083e7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 -- 172.21.15.78:0/3753732179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e78b0 msgr2=0x559f083e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e78b0 0x559f083e7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 -- 172.21.15.78:0/3753732179 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f081b5630 con 0x559f083e2b60 2023-10-10T17:48:48.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121852700 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e3470 0x559f083e7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9121051700 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e2f30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9118013820 tx=0x7f9118004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5b6aa237fe126f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9107fff700 1 -- 172.21.15.78:0/3753732179 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180130d0 con 0x559f083e2b60 2023-10-10T17:48:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9107fff700 1 -- 172.21.15.78:0/3753732179 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9118013230 con 0x559f083e2b60 2023-10-10T17:48:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.062+0000 7f9107fff700 1 -- 172.21.15.78:0/3753732179 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118010430 con 0x559f083e2b60 2023-10-10T17:48:48.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f9112ffd700 1 -- 172.21.15.78:0/3337529748 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f9114004a50 con 0x7f911c104490 2023-10-10T17:48:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f9112ffd700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f91080a1340 0x7f91080a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 -- 172.21.15.78:0/3753732179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 msgr2=0x559f083e2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e2f30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9118013820 tx=0x7f9118004750 comp rx=0 tx=0).stop 2023-10-10T17:48:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f9121852700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f91080a1340 0x7f91080a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f9112ffd700 1 -- 172.21.15.78:0/3337529748 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f91140cdd90 con 0x7f911c104490 2023-10-10T17:48:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 -- 172.21.15.78:0/3753732179 shutdown_connections 2023-10-10T17:48:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e78b0 0x559f083e7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e2f30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3753732179 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e3470 0x559f083e7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 -- 172.21.15.78:0/3753732179 >> 172.21.15.78:0/3753732179 conn(0x559f083d8360 msgr2=0x559f083ddb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.063+0000 7f912fc42b80 1 -- 172.21.15.78:0/3753732179 shutdown_connections 2023-10-10T17:48:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 -- 172.21.15.78:0/3753732179 wait complete. 2023-10-10T17:48:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f9121852700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f91080a1340 0x7f91080a37f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f910c009f20 tx=0x7f910c015040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 -- start start 2023-10-10T17:48:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 Processor -- start 2023-10-10T17:48:48.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083ef100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083ec310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f083eca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f082d58e0 con 0x559f083e3470 2023-10-10T17:48:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f082d5580 con 0x559f083e78b0 2023-10-10T17:48:48.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f081b7920 con 0x559f083e2b60 2023-10-10T17:48:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f912fc42b80 10 client.0 _ll_register_callbacks cb 0x559f082f3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.064+0000 7f9121051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083ec310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9121852700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083ef100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9121051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083ec310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47124/0 (socket says 172.21.15.78:47124) 2023-10-10T17:48:48.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9121051700 1 -- 172.21.15.78:0/3178050900 learned_addr learned my addr 172.21.15.78:0/3178050900 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9122053700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f083eca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9121852700 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083ef100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38652/0 (socket says 172.21.15.78:38652) 2023-10-10T17:48:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9122053700 1 -- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 msgr2=0x559f083ef100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9122053700 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083ef100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9122053700 1 -- 172.21.15.78:0/3178050900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 msgr2=0x559f083ec310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9122053700 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083ec310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9122053700 1 -- 172.21.15.78:0/3178050900 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911801b040 con 0x559f083e78b0 2023-10-10T17:48:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9121051700 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f9122053700 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f083eca50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f91140cb1b0 tx=0x7f9114096f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0f0cdb5dee25b4c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f91057fa700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:48:48.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f91057fa700 1 -- 172.21.15.78:0/3178050900 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911409b020 con 0x559f083e78b0 2023-10-10T17:48:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559f083d5b20 con 0x559f083e78b0 2023-10-10T17:48:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f91057fa700 1 -- 172.21.15.78:0/3178050900 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f911400be20 con 0x559f083e78b0 2023-10-10T17:48:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f91057fa700 1 -- 172.21.15.78:0/3178050900 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f911409abc0 con 0x559f083e78b0 2023-10-10T17:48:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f912fc42b80 10 client.26057 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:48.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x559f082fa190 con 0x559f083e78b0 2023-10-10T17:48:48.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.065+0000 7f91057fa700 1 -- 172.21.15.78:0/3178050900 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559f083d5b20 con 0x559f083e78b0 2023-10-10T17:48:48.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f91057fa700 1 -- 172.21.15.78:0/3178050900 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f911409a0b0 con 0x559f083e78b0 2023-10-10T17:48:48.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f912fc42b80 10 client.26057 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:48.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f912fc42b80 10 client.26057 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:48:48.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f912fc42b80 20 client.26057 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:48.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f083fc5a0 0x559f083fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f912fc42b80 4 client.26057 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:48.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559f082f60a0 con 0x559f083fc5a0 2023-10-10T17:48:48.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f9121852700 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f083fc5a0 0x559f083fe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f91057fa700 10 client.26057 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:48:48.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.066+0000 7f9121852700 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f083fc5a0 0x559f083fe9b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=427bd9e2184cec1f server_cookie=116ceb0ac5c2e5b5 in_seq=0 out_seq=0 2023-10-10T17:48:48.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.068+0000 7f91057fa700 1 -- 172.21.15.78:0/3178050900 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559f082f60a0 con 0x559f083fc5a0 2023-10-10T17:48:48.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.068+0000 7f91057fa700 10 client.26057 handle_command_reply: tid=0 2023-10-10T17:48:48.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.068+0000 7f912fc42b80 1 client.26057 shutdown 2023-10-10T17:48:48.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 msgr2=0x559f083eca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f083eca50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f91140cb1b0 tx=0x7f9114096f80 comp rx=0 tx=0).stop 2023-10-10T17:48:48.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 shutdown_connections 2023-10-10T17:48:48.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f083fc5a0 0x559f083fe9b0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083ef100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f91057fa700 0 client.26057 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:48:48.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f083eca50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3178050900 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083ec310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 >> 172.21.15.78:0/3178050900 conn(0x559f083d8360 msgr2=0x559f083e8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 shutdown_connections 2023-10-10T17:48:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 1 -- 172.21.15.78:0/3178050900 wait complete. 2023-10-10T17:48:48.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.069+0000 7f912fc42b80 20 client.26057 trim_cache size 0 max 16384 2023-10-10T17:48:48.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 Processor -- start 2023-10-10T17:48:48.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 -- start start 2023-10-10T17:48:48.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083eec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e78b0 0x559f083dd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f083e3840 con 0x559f083e3470 2023-10-10T17:48:48.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f083e64a0 con 0x559f083e2b60 2023-10-10T17:48:48.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f912fc42b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f083ec9b0 con 0x559f083e78b0 2023-10-10T17:48:48.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083eec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083eec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47132/0 (socket says 172.21.15.78:47132) 2023-10-10T17:48:48.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 -- 172.21.15.78:0/3758002648 learned_addr learned my addr 172.21.15.78:0/3758002648 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9122053700 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e78b0 0x559f083dd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121852700 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 -- 172.21.15.78:0/3758002648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e78b0 msgr2=0x559f083dd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e78b0 0x559f083dd560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 -- 172.21.15.78:0/3758002648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 msgr2=0x559f083e8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 -- 172.21.15.78:0/3758002648 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559f082f9ee0 con 0x559f083e3470 2023-10-10T17:48:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9121051700 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083eec30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9118009ba0 tx=0x7f9118011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e09d8c4b8e8770 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.082+0000 7f9110dac700 1 -- 172.21.15.78:0/3758002648 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118018040 con 0x559f083e3470 2023-10-10T17:48:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f9110dac700 1 -- 172.21.15.78:0/3758002648 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f91180133a0 con 0x559f083e3470 2023-10-10T17:48:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f9110dac700 1 -- 172.21.15.78:0/3758002648 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118004a20 con 0x559f083e3470 2023-10-10T17:48:48.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 -- 172.21.15.78:0/3758002648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 msgr2=0x559f083eec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083eec30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f9118009ba0 tx=0x7f9118011cf0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 -- 172.21.15.78:0/3758002648 shutdown_connections 2023-10-10T17:48:48.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e78b0 0x559f083dd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e2b60 0x559f083e8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 --2- 172.21.15.78:0/3758002648 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083eec30 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 -- 172.21.15.78:0/3758002648 >> 172.21.15.78:0/3758002648 conn(0x559f083d8360 msgr2=0x559f083e5930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 -- 172.21.15.78:0/3758002648 shutdown_connections 2023-10-10T17:48:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 -- 172.21.15.78:0/3758002648 wait complete. 2023-10-10T17:48:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.083+0000 7f912fc42b80 1 -- start start 2023-10-10T17:48:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 1 Processor -- start 2023-10-10T17:48:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083f1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f0844f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f083f27c0 con 0x559f083e3470 2023-10-10T17:48:48.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0844f9a0 con 0x559f083e78b0 2023-10-10T17:48:48.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559f0844fae0 con 0x559f083e2b60 2023-10-10T17:48:48.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f912fc42b80 10 client.0 _ll_register_callbacks cb 0x559f082f3780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:48.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083f1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47138/0 (socket says 172.21.15.78:47138) 2023-10-10T17:48:48.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 -- 172.21.15.78:0/826807133 learned_addr learned my addr 172.21.15.78:0/826807133 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121852700 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083f1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9122053700 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f0844f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 -- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 msgr2=0x559f083f1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083f1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 -- 172.21.15.78:0/826807133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 msgr2=0x559f0844f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f0844f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.084+0000 7f9121051700 1 -- 172.21.15.78:0/826807133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559f081b5630 con 0x559f083e3470 2023-10-10T17:48:48.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f9121051700 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083f1ed0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9118000c00 tx=0x7f9118022b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf34cfbf4b95dd2d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f91057fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:48.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f91057fa700 1 -- 172.21.15.78:0/826807133 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9118009dd0 con 0x559f083e3470 2023-10-10T17:48:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f91057fa700 1 -- 172.21.15.78:0/826807133 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f9118022d00 con 0x559f083e3470 2023-10-10T17:48:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9118013230 con 0x559f083e3470 2023-10-10T17:48:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f91057fa700 1 -- 172.21.15.78:0/826807133 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9118022ec0 con 0x559f083e3470 2023-10-10T17:48:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 10 client.16701 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:48.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x559f0814ac60 con 0x559f083e3470 2023-10-10T17:48:48.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f91057fa700 1 -- 172.21.15.78:0/826807133 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9118013230 con 0x559f083e3470 2023-10-10T17:48:48.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f91057fa700 1 -- 172.21.15.78:0/826807133 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f911802ac90 con 0x559f083e3470 2023-10-10T17:48:48.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 10 client.16701 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:48.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 10 client.16701 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:48:48.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 20 client.16701 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f08450de0 0x559f084531f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 4 client.16701 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:48:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559f08302660 con 0x559f08450de0 2023-10-10T17:48:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.085+0000 7f9121852700 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f08450de0 0x559f084531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.086+0000 7f91057fa700 10 client.16701 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:48:48.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.086+0000 7f9121852700 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f08450de0 0x559f084531f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=582018bce5f13ea0 server_cookie=15a228473b7c9303 in_seq=0 out_seq=0 2023-10-10T17:48:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f91057fa700 1 -- 172.21.15.78:0/826807133 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559f08302660 con 0x559f08450de0 2023-10-10T17:48:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f91057fa700 10 client.16701 handle_command_reply: tid=0 2023-10-10T17:48:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 client.16701 shutdown 2023-10-10T17:48:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 msgr2=0x559f083f1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083f1ed0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f9118000c00 tx=0x7f9118022b10 comp rx=0 tx=0).stop 2023-10-10T17:48:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 shutdown_connections 2023-10-10T17:48:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x559f08450de0 0x559f084531f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559f083e2b60 0x559f083f1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f91057fa700 0 client.16701 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:48:48.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559f083e78b0 0x559f0844f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 --2- 172.21.15.78:0/826807133 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559f083e3470 0x559f083f1ed0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 >> 172.21.15.78:0/826807133 conn(0x559f083d8360 msgr2=0x559f083e68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 shutdown_connections 2023-10-10T17:48:48.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 1 -- 172.21.15.78:0/826807133 wait complete. 2023-10-10T17:48:48.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.090+0000 7f912fc42b80 20 client.16701 trim_cache size 0 max 16384 2023-10-10T17:48:48.111 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:48:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f91080a1340 msgr2=0x7f91080a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f91080a1340 0x7f91080a37f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f910c009f20 tx=0x7f910c015040 comp rx=0 tx=0).stop 2023-10-10T17:48:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 msgr2=0x7f911c199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c199df0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f9114007820 tx=0x7f9114007850 comp rx=0 tx=0).stop 2023-10-10T17:48:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 shutdown_connections 2023-10-10T17:48:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f91080a1340 0x7f91080a37f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f911c104490 0x7f911c199df0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f911c0ff450 0x7f911c195450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 --2- 172.21.15.78:0/3337529748 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f911c0ffdf0 0x7f911c195990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 >> 172.21.15.78:0/3337529748 conn(0x7f911c0770b0 msgr2=0x7f911c0fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 shutdown_connections 2023-10-10T17:48:48.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.091+0000 7f9123ab6700 1 -- 172.21.15.78:0/3337529748 wait complete. 2023-10-10T17:48:48.115 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:48:48.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 -- 172.21.15.78:0/1093435712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0104560 msgr2=0x7f6ca010c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1093435712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0104560 0x7f6ca010c130 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b0d0 tx=0x7f6c9800b3e0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 -- 172.21.15.78:0/1093435712 shutdown_connections 2023-10-10T17:48:48.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1093435712 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca00ff520 0x7f6ca00ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1093435712 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca0104560 0x7f6ca010c130 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1093435712 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ca00ffec0 0x7f6ca0103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 -- 172.21.15.78:0/1093435712 >> 172.21.15.78:0/1093435712 conn(0x7f6ca00760d0 msgr2=0x7f6ca00764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 -- 172.21.15.78:0/1093435712 shutdown_connections 2023-10-10T17:48:48.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 -- 172.21.15.78:0/1093435712 wait complete. 2023-10-10T17:48:48.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 Processor -- start 2023-10-10T17:48:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.245+0000 7f6ca779f700 1 -- start start 2023-10-10T17:48:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00ff520 0x7f6ca0197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ca00ffec0 0x7f6ca0197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca0104560 0x7f6ca01a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca01981a0 con 0x7f6ca00ff520 2023-10-10T17:48:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca01a2910 con 0x7f6ca00ffec0 2023-10-10T17:48:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca01a2a80 con 0x7f6ca0104560 2023-10-10T17:48:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00ff520 0x7f6ca0197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00ff520 0x7f6ca0197350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47150/0 (socket says 172.21.15.78:47150) 2023-10-10T17:48:48.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca4d3a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ca00ffec0 0x7f6ca0197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 -- 172.21.15.78:0/1083435171 learned_addr learned my addr 172.21.15.78:0/1083435171 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca0104560 0x7f6ca01a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 -- 172.21.15.78:0/1083435171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca0104560 msgr2=0x7f6ca01a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca0104560 0x7f6ca01a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 -- 172.21.15.78:0/1083435171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ca00ffec0 msgr2=0x7f6ca0197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ca00ffec0 0x7f6ca0197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 -- 172.21.15.78:0/1083435171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c98009d00 con 0x7f6ca00ff520 2023-10-10T17:48:48.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00ff520 0x7f6ca0197350 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9a0 tx=0x7f6c9000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6c967fc700 1 -- 172.21.15.78:0/1083435171 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90009790 con 0x7f6ca00ff520 2023-10-10T17:48:48.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6c967fc700 1 -- 172.21.15.78:0/1083435171 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c90010430 con 0x7f6ca00ff520 2023-10-10T17:48:48.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca010f340 con 0x7f6ca00ff520 2023-10-10T17:48:48.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6c967fc700 1 -- 172.21.15.78:0/1083435171 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90018680 con 0x7f6ca00ff520 2023-10-10T17:48:48.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.246+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca010f1b0 con 0x7f6ca00ff520 2023-10-10T17:48:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.249+0000 7f6c967fc700 1 -- 172.21.15.78:0/1083435171 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f6c900105a0 con 0x7f6ca00ff520 2023-10-10T17:48:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.250+0000 7f6c967fc700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6c8c0a12c0 0x7f6c8c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.250+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6c8c0a12c0 0x7f6c8c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.250+0000 7f6c967fc700 1 -- 172.21.15.78:0/1083435171 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f6c90014070 con 0x7f6ca00ff520 2023-10-10T17:48:48.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.250+0000 7f6cb392bb80 1 Processor -- start 2023-10-10T17:48:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.250+0000 7f6cb392bb80 1 -- start start 2023-10-10T17:48:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.250+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6c8c0a12c0 0x7f6c8c0a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00696c0 tx=0x7f6c9c009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e54360f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54365370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e54365d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e54266060 con 0x559e543658b0 2023-10-10T17:48:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e542b88a0 con 0x559e54360b60 2023-10-10T17:48:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e542b1d30 con 0x559e54361470 2023-10-10T17:48:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca5d3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e54365d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e54360f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e54360f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48880/0 (socket says 172.21.15.78:48880) 2023-10-10T17:48:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca553b700 1 -- 172.21.15.78:0/2054882852 learned_addr learned my addr 172.21.15.78:0/2054882852 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54365370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca5d3c700 1 -- 172.21.15.78:0/2054882852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 msgr2=0x559e54365370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54365370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca5d3c700 1 -- 172.21.15.78:0/2054882852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 msgr2=0x559e54360f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e54360f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca5d3c700 1 -- 172.21.15.78:0/2054882852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e54133630 con 0x559e543658b0 2023-10-10T17:48:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54365370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:48:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca553b700 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e54360f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:48.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.251+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e54365d00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b770 tx=0x7f6c98009c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfdb1da695bec8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6c8b7fe700 1 -- 172.21.15.78:0/2054882852 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9800e040 con 0x559e543658b0 2023-10-10T17:48:48.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6c8b7fe700 1 -- 172.21.15.78:0/2054882852 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c980074e0 con 0x559e543658b0 2023-10-10T17:48:48.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6c8b7fe700 1 -- 172.21.15.78:0/2054882852 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003e80 con 0x559e543658b0 2023-10-10T17:48:48.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2054882852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 msgr2=0x559e54365d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e54365d00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f6c9800b770 tx=0x7f6c98009c60 comp rx=0 tx=0).stop 2023-10-10T17:48:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2054882852 shutdown_connections 2023-10-10T17:48:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e54365d00 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e54360f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2054882852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54365370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.252+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2054882852 >> 172.21.15.78:0/2054882852 conn(0x559e54356360 msgr2=0x559e5435bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.253+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2054882852 shutdown_connections 2023-10-10T17:48:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.253+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2054882852 wait complete. 2023-10-10T17:48:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.253+0000 7f6cb392bb80 1 -- start start 2023-10-10T17:48:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 1 Processor -- start 2023-10-10T17:48:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e543746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54374c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e5436e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e542538e0 con 0x559e543658b0 2023-10-10T17:48:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e54253580 con 0x559e54360b60 2023-10-10T17:48:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e54135920 con 0x559e54361470 2023-10-10T17:48:48.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6cb392bb80 10 client.0 _ll_register_callbacks cb 0x559e54271780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e543746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e543746f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48886/0 (socket says 172.21.15.78:48886) 2023-10-10T17:48:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.254+0000 7f6ca553b700 1 -- 172.21.15.78:0/1986157915 learned_addr learned my addr 172.21.15.78:0/1986157915 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54374c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e5436e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca553b700 1 -- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 msgr2=0x559e54374c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54374c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca553b700 1 -- 172.21.15.78:0/1986157915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 msgr2=0x559e5436e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e5436e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca553b700 1 -- 172.21.15.78:0/1986157915 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca010c970 con 0x559e54360b60 2023-10-10T17:48:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e5436e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:48:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e543746f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000c510 tx=0x7f6c9009a020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ddb9d6803091eef server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.255+0000 7f6c88ff9700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:48:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1986157915 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90099690 con 0x559e54360b60 2023-10-10T17:48:48.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1986157915 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f6c900a23f0 con 0x559e54360b60 2023-10-10T17:48:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e54353b20 con 0x559e54360b60 2023-10-10T17:48:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1986157915 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c9009e050 con 0x559e54360b60 2023-10-10T17:48:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 10 client.26075 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x559e54278190 con 0x559e54360b60 2023-10-10T17:48:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1986157915 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e54353b20 con 0x559e54360b60 2023-10-10T17:48:48.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1986157915 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f6c900b80d0 con 0x559e54360b60 2023-10-10T17:48:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 10 client.26075 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 10 client.26075 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:48:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 20 client.26075 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e5437ec40 0x559e54381050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 4 client.26075 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.256+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e542740a0 con 0x559e5437ec40 2023-10-10T17:48:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.257+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e5437ec40 0x559e54381050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.257+0000 7f6c88ff9700 10 client.26075 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:48:48.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.257+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e5437ec40 0x559e54381050 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb75587e3c371d8b server_cookie=c8152f0f2d37a50f in_seq=0 out_seq=0 2023-10-10T17:48:48.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1986157915 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x559e542740a0 con 0x559e5437ec40 2023-10-10T17:48:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6c88ff9700 10 client.26075 handle_command_reply: tid=0 2023-10-10T17:48:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 client.26075 shutdown 2023-10-10T17:48:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 msgr2=0x559e543746f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e543746f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000c510 tx=0x7f6c9009a020 comp rx=0 tx=0).stop 2023-10-10T17:48:48.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 shutdown_connections 2023-10-10T17:48:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e543658b0 0x559e5436e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e543746f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e5437ec40 0x559e54381050 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1986157915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e54361470 0x559e54374c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6c88ff9700 0 client.26075 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:48:48.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 >> 172.21.15.78:0/1986157915 conn(0x559e54356360 msgr2=0x559e54366d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 shutdown_connections 2023-10-10T17:48:48.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1986157915 wait complete. 2023-10-10T17:48:48.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.261+0000 7f6cb392bb80 20 client.26075 trim_cache size 0 max 16384 2023-10-10T17:48:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 Processor -- start 2023-10-10T17:48:48.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 -- start start 2023-10-10T17:48:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e5436f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54361470 0x559e5436d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e5436c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e54362750 con 0x559e54361470 2023-10-10T17:48:48.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e54361900 con 0x559e54360b60 2023-10-10T17:48:48.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e54353960 con 0x559e543658b0 2023-10-10T17:48:48.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54361470 0x559e5436d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54361470 0x559e5436d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47174/0 (socket says 172.21.15.78:47174) 2023-10-10T17:48:48.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 -- 172.21.15.78:0/2060962801 learned_addr learned my addr 172.21.15.78:0/2060962801 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca553b700 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e5436f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e5436c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 -- 172.21.15.78:0/2060962801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 msgr2=0x559e5436c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e5436c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 -- 172.21.15.78:0/2060962801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 msgr2=0x559e5436f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e5436f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.275+0000 7f6ca4d3a700 1 -- 172.21.15.78:0/2060962801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e54277ee0 con 0x559e54361470 2023-10-10T17:48:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54361470 0x559e5436d980 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6ca006a3d0 tx=0x7f6c9c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6326fc369c567934 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6c8bfff700 1 -- 172.21.15.78:0/2060962801 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c024040 con 0x559e54361470 2023-10-10T17:48:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6c8bfff700 1 -- 172.21.15.78:0/2060962801 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c9c004510 con 0x559e54361470 2023-10-10T17:48:48.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6c8bfff700 1 -- 172.21.15.78:0/2060962801 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c003d80 con 0x559e54361470 2023-10-10T17:48:48.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2060962801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54361470 msgr2=0x559e5436d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54361470 0x559e5436d980 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f6ca006a3d0 tx=0x7f6c9c014000 comp rx=0 tx=0).stop 2023-10-10T17:48:48.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2060962801 shutdown_connections 2023-10-10T17:48:48.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e5436c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54360b60 0x559e5436f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/2060962801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54361470 0x559e5436d980 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.276+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2060962801 >> 172.21.15.78:0/2060962801 conn(0x559e54356360 msgr2=0x559e543859b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2060962801 shutdown_connections 2023-10-10T17:48:48.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 -- 172.21.15.78:0/2060962801 wait complete. 2023-10-10T17:48:48.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 -- start start 2023-10-10T17:48:48.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 Processor -- start 2023-10-10T17:48:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54360b60 0x559e54373f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54361470 0x559e54374490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e543d1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e54374c10 con 0x559e54360b60 2023-10-10T17:48:48.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e543d2070 con 0x559e54361470 2023-10-10T17:48:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e543d2230 con 0x559e543658b0 2023-10-10T17:48:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.277+0000 7f6cb392bb80 10 client.0 _ll_register_callbacks cb 0x7f6c900999b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54360b60 0x559e54373f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54360b60 0x559e54373f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47178/0 (socket says 172.21.15.78:47178) 2023-10-10T17:48:48.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 -- 172.21.15.78:0/1509914307 learned_addr learned my addr 172.21.15.78:0/1509914307 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca5d3c700 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e543d1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54361470 0x559e54374490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 -- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 msgr2=0x559e543d1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e543d1b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 -- 172.21.15.78:0/1509914307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54361470 msgr2=0x559e54374490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54361470 0x559e54374490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 -- 172.21.15.78:0/1509914307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c017070 con 0x559e54360b60 2023-10-10T17:48:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6ca553b700 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54360b60 0x559e54373f50 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6c900964a0 tx=0x7f6c900b2880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c08815174ac332 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6c88ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1509914307 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900b2430 con 0x559e54360b60 2023-10-10T17:48:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1509914307 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f6c900a1f60 con 0x559e54360b60 2023-10-10T17:48:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.278+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6c980074e0 con 0x559e54360b60 2023-10-10T17:48:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1509914307 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c900a2120 con 0x559e54360b60 2023-10-10T17:48:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 10 client.16716 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:48.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x559e540c8c60 con 0x559e54360b60 2023-10-10T17:48:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1509914307 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c90063030 con 0x559e54360b60 2023-10-10T17:48:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1509914307 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f6c90094bb0 con 0x559e54360b60 2023-10-10T17:48:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 10 client.16716 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 10 client.16716 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:48:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 20 client.16716 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:48.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e543d35b0 0x559e543d59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 4 client.16716 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:48:48.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559e54280660 con 0x559e543d35b0 2023-10-10T17:48:48.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.279+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e543d35b0 0x559e543d59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.280+0000 7f6c88ff9700 10 client.16716 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:48:48.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.280+0000 7f6ca4d3a700 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e543d35b0 0x559e543d59c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b52c9d17b34b77f8 server_cookie=941157b8ace92841 in_seq=0 out_seq=0 2023-10-10T17:48:48.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.281+0000 7f6c88ff9700 1 -- 172.21.15.78:0/1509914307 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559e54280660 con 0x559e543d35b0 2023-10-10T17:48:48.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.281+0000 7f6c88ff9700 10 client.16716 handle_command_reply: tid=0 2023-10-10T17:48:48.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.281+0000 7f6cb392bb80 1 client.16716 shutdown 2023-10-10T17:48:48.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.281+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54360b60 msgr2=0x559e54373f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.281+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54360b60 0x559e54373f50 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f6c900964a0 tx=0x7f6c900b2880 comp rx=0 tx=0).stop 2023-10-10T17:48:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.281+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 shutdown_connections 2023-10-10T17:48:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.281+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x559e543d35b0 0x559e543d59c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x559e543658b0 0x559e543d1b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559e54360b60 0x559e54373f50 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6cb392bb80 1 --2- 172.21.15.78:0/1509914307 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x559e54361470 0x559e54374490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 >> 172.21.15.78:0/1509914307 conn(0x559e54356360 msgr2=0x559e5435f960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6c88ff9700 0 client.16716 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:48:48.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 shutdown_connections 2023-10-10T17:48:48.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6cb392bb80 1 -- 172.21.15.78:0/1509914307 wait complete. 2023-10-10T17:48:48.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6cb392bb80 20 client.16716 trim_cache size 0 max 16384 2023-10-10T17:48:48.300 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:48:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6c8c0a12c0 msgr2=0x7f6c8c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6c8c0a12c0 0x7f6c8c0a3770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00696c0 tx=0x7f6c9c009260 comp rx=0 tx=0).stop 2023-10-10T17:48:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00ff520 msgr2=0x7f6ca0197350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.282+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00ff520 0x7f6ca0197350 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000e9a0 tx=0x7f6c9000ecb0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 shutdown_connections 2023-10-10T17:48:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f6ca0104560 0x7f6ca01a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f6c8c0a12c0 0x7f6c8c0a3770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6ca00ff520 0x7f6ca0197350 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 --2- 172.21.15.78:0/1083435171 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f6ca00ffec0 0x7f6ca0197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 >> 172.21.15.78:0/1083435171 conn(0x7f6ca00760d0 msgr2=0x7f6ca00fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 shutdown_connections 2023-10-10T17:48:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.283+0000 7f6ca779f700 1 -- 172.21.15.78:0/1083435171 wait complete. 2023-10-10T17:48:48.303 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:48:48.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.446+0000 7fa34a627700 1 -- 172.21.15.78:0/3503329957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3440ff590 msgr2=0x7fa3440ff960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.446+0000 7fa34a627700 1 --2- 172.21.15.78:0/3503329957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3440ff590 0x7fa3440ff960 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fa3340099c0 tx=0x7fa334009cd0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 -- 172.21.15.78:0/3503329957 shutdown_connections 2023-10-10T17:48:48.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 --2- 172.21.15.78:0/3503329957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3441045d0 0x7fa34410c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 --2- 172.21.15.78:0/3503329957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3440ff590 0x7fa3440ff960 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 --2- 172.21.15.78:0/3503329957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3440fff30 0x7fa344103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 -- 172.21.15.78:0/3503329957 >> 172.21.15.78:0/3503329957 conn(0x7fa3440760d0 msgr2=0x7fa3440764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 -- 172.21.15.78:0/3503329957 shutdown_connections 2023-10-10T17:48:48.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 -- 172.21.15.78:0/3503329957 wait complete. 2023-10-10T17:48:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 Processor -- start 2023-10-10T17:48:48.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.447+0000 7fa34a627700 1 -- start start 2023-10-10T17:48:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440ff590 0x7fa3441964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3440fff30 0x7fa344196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3441045d0 0x7fa34419ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34419b4b0 con 0x7fa3441045d0 2023-10-10T17:48:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34419b620 con 0x7fa3440fff30 2023-10-10T17:48:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34419b7e0 con 0x7fa3440ff590 2023-10-10T17:48:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3441045d0 0x7fa34419ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3441045d0 0x7fa34419ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47196/0 (socket says 172.21.15.78:47196) 2023-10-10T17:48:48.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 -- 172.21.15.78:0/3918753388 learned_addr learned my addr 172.21.15.78:0/3918753388 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa343fff700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440ff590 0x7fa3441964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa3437fe700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3440fff30 0x7fa344196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 -- 172.21.15.78:0/3918753388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440ff590 msgr2=0x7fa3441964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440ff590 0x7fa3441964d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 -- 172.21.15.78:0/3918753388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3440fff30 msgr2=0x7fa344196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3440fff30 0x7fa344196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 -- 172.21.15.78:0/3918753388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3340095f0 con 0x7fa3441045d0 2023-10-10T17:48:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa348bc4700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3441045d0 0x7fa34419ae70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fa33c007ee0 tx=0x7fa33c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa3417fa700 1 -- 172.21.15.78:0/3918753388 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c00dd20 con 0x7fa3441045d0 2023-10-10T17:48:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa3417fa700 1 -- 172.21.15.78:0/3918753388 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa33c00f040 con 0x7fa3441045d0 2023-10-10T17:48:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa3417fa700 1 -- 172.21.15.78:0/3918753388 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0048a0 con 0x7fa3441045d0 2023-10-10T17:48:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa344110500 con 0x7fa3441045d0 2023-10-10T17:48:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.448+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa344110370 con 0x7fa3441045d0 2023-10-10T17:48:48.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.451+0000 7fa3417fa700 1 -- 172.21.15.78:0/3918753388 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa33c003a00 con 0x7fa3441045d0 2023-10-10T17:48:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.451+0000 7fa3417fa700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa32c0a1340 0x7fa32c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.452+0000 7fa343fff700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa32c0a1340 0x7fa32c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.452+0000 7fa3417fa700 1 -- 172.21.15.78:0/3918753388 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa33c0cc7e0 con 0x7fa3441045d0 2023-10-10T17:48:48.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.452+0000 7fa343fff700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa32c0a1340 0x7fa32c0a37f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa334000c00 tx=0x7fa334003680 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 Processor -- start 2023-10-10T17:48:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 -- start start 2023-10-10T17:48:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a7b60 0x562fc04a7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04ac370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04ac8b0 0x562fc04acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc03ad060 con 0x562fc04a8470 2023-10-10T17:48:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc03ff8a0 con 0x562fc04ac8b0 2023-10-10T17:48:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc03f8d30 con 0x562fc04a7b60 2023-10-10T17:48:48.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04ac370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04ac370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47204/0 (socket says 172.21.15.78:47204) 2023-10-10T17:48:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa343fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a7b60 0x562fc04a7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa3437fe700 1 -- 172.21.15.78:0/3528096130 learned_addr learned my addr 172.21.15.78:0/3528096130 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.453+0000 7fa348bc4700 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04ac8b0 0x562fc04acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa3437fe700 1 -- 172.21.15.78:0/3528096130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a7b60 msgr2=0x562fc04a7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa3437fe700 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a7b60 0x562fc04a7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa3437fe700 1 -- 172.21.15.78:0/3528096130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04ac8b0 msgr2=0x562fc04acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa3437fe700 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04ac8b0 0x562fc04acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa3437fe700 1 -- 172.21.15.78:0/3528096130 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fc027a630 con 0x562fc04a8470 2023-10-10T17:48:48.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa3437fe700 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04ac370 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa33800bcb0 tx=0x7fa3380103d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d210ceb7d2a75127 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa32a7fc700 1 -- 172.21.15.78:0/3528096130 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338010d10 con 0x562fc04a8470 2023-10-10T17:48:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa32a7fc700 1 -- 172.21.15.78:0/3528096130 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa338015070 con 0x562fc04a8470 2023-10-10T17:48:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.454+0000 7fa32a7fc700 1 -- 172.21.15.78:0/3528096130 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338005020 con 0x562fc04a8470 2023-10-10T17:48:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 -- 172.21.15.78:0/3528096130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 msgr2=0x562fc04ac370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04ac370 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa33800bcb0 tx=0x7fa3380103d0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 -- 172.21.15.78:0/3528096130 shutdown_connections 2023-10-10T17:48:48.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a7b60 0x562fc04a7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04ac8b0 0x562fc04acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/3528096130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04ac370 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 -- 172.21.15.78:0/3528096130 >> 172.21.15.78:0/3528096130 conn(0x562fc049d360 msgr2=0x562fc04a2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 -- 172.21.15.78:0/3528096130 shutdown_connections 2023-10-10T17:48:48.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.455+0000 7fa3567b3b80 1 -- 172.21.15.78:0/3528096130 wait complete. 2023-10-10T17:48:48.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 -- start start 2023-10-10T17:48:48.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 Processor -- start 2023-10-10T17:48:48.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a8470 0x562fc04b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04ac8b0 0x562fc04b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc039a8e0 con 0x562fc04ac8b0 2023-10-10T17:48:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc039a580 con 0x562fc04a7b60 2023-10-10T17:48:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc027c920 con 0x562fc04a8470 2023-10-10T17:48:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3567b3b80 10 client.0 _ll_register_callbacks cb 0x562fc03b8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:48.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa348bc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04ac8b0 0x562fc04b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa343fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.456+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a8470 0x562fc04b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04ac8b0 0x562fc04b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47210/0 (socket says 172.21.15.78:47210) 2023-10-10T17:48:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa343fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48928/0 (socket says 172.21.15.78:48928) 2023-10-10T17:48:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a8470 0x562fc04b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38738/0 (socket says 172.21.15.78:38738) 2023-10-10T17:48:48.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 -- 172.21.15.78:0/1143511828 learned_addr learned my addr 172.21.15.78:0/1143511828 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 -- 172.21.15.78:0/1143511828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a8470 msgr2=0x562fc04b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a8470 0x562fc04b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 -- 172.21.15.78:0/1143511828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 msgr2=0x562fc04b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 -- 172.21.15.78:0/1143511828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33800b8e0 con 0x562fc04ac8b0 2023-10-10T17:48:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa348bc4700 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04ac8b0 0x562fc04b1870 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0c9c60 tx=0x7fa33c0990c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9be0cc091799092 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3237fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3237fe700 1 -- 172.21.15.78:0/1143511828 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0995e0 con 0x562fc04ac8b0 2023-10-10T17:48:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3237fe700 1 -- 172.21.15.78:0/1143511828 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa33c09d5b0 con 0x562fc04ac8b0 2023-10-10T17:48:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562fc049ab20 con 0x562fc04ac8b0 2023-10-10T17:48:48.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3237fe700 1 -- 172.21.15.78:0/1143511828 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa33c0a19d0 con 0x562fc04ac8b0 2023-10-10T17:48:48.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3567b3b80 10 client.16728 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:48.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x562fc03bf190 con 0x562fc04ac8b0 2023-10-10T17:48:48.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.457+0000 7fa3237fe700 1 -- 172.21.15.78:0/1143511828 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa33c0c7070 con 0x562fc04ac8b0 2023-10-10T17:48:48.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3237fe700 1 -- 172.21.15.78:0/1143511828 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa33c0997e0 con 0x562fc04ac8b0 2023-10-10T17:48:48.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3567b3b80 10 client.16728 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:48.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3567b3b80 10 client.16728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:48.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3567b3b80 20 client.16728 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:48.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc04c1920 0x562fc04c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3567b3b80 4 client.16728 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:48.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562fc03bb0a0 con 0x562fc04c1920 2023-10-10T17:48:48.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa343fff700 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc04c1920 0x562fc04c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa343fff700 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc04c1920 0x562fc04c3d30 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fe1dad7aa4a36d server_cookie=d96f276a3b5053cf in_seq=0 out_seq=0 2023-10-10T17:48:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.458+0000 7fa3237fe700 10 client.16728 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3237fe700 1 -- 172.21.15.78:0/1143511828 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x562fc03bb0a0 con 0x562fc04c1920 2023-10-10T17:48:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3237fe700 10 client.16728 handle_command_reply: tid=0 2023-10-10T17:48:48.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 client.16728 shutdown 2023-10-10T17:48:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04ac8b0 msgr2=0x562fc04b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04ac8b0 0x562fc04b1870 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fa33c0c9c60 tx=0x7fa33c0990c0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 shutdown_connections 2023-10-10T17:48:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc04c1920 0x562fc04c3d30 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04ac8b0 0x562fc04b1870 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3237fe700 0 client.16728 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1143511828 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04a8470 0x562fc04b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 >> 172.21.15.78:0/1143511828 conn(0x562fc049d360 msgr2=0x562fc04add60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 shutdown_connections 2023-10-10T17:48:48.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.459+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1143511828 wait complete. 2023-10-10T17:48:48.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.460+0000 7fa3567b3b80 20 client.16728 trim_cache size 0 max 16384 2023-10-10T17:48:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.475+0000 7fa3567b3b80 1 Processor -- start 2023-10-10T17:48:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.475+0000 7fa3567b3b80 1 -- start start 2023-10-10T17:48:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a7b60 0x562fc04b89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a8470 0x562fc04a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc04b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc04a8bc0 con 0x562fc04a7b60 2023-10-10T17:48:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc04a38b0 con 0x562fc04a8470 2023-10-10T17:48:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc04af800 con 0x562fc04ac8b0 2023-10-10T17:48:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a7b60 0x562fc04b89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a7b60 0x562fc04b89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47214/0 (socket says 172.21.15.78:47214) 2023-10-10T17:48:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 -- 172.21.15.78:0/1018263944 learned_addr learned my addr 172.21.15.78:0/1018263944 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa3437fe700 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a8470 0x562fc04a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa348bc4700 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc04b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 -- 172.21.15.78:0/1018263944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 msgr2=0x562fc04b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc04b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 -- 172.21.15.78:0/1018263944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a8470 msgr2=0x562fc04a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a8470 0x562fc04a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 -- 172.21.15.78:0/1018263944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fc04a1820 con 0x562fc04a7b60 2023-10-10T17:48:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa348bc4700 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc04b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:48:48.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.476+0000 7fa343fff700 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a7b60 0x562fc04b89a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa3340099c0 tx=0x7fa334020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67363b96962cef1e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.477+0000 7fa32affd700 1 -- 172.21.15.78:0/1018263944 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33400bc70 con 0x562fc04a7b60 2023-10-10T17:48:48.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.477+0000 7fa32affd700 1 -- 172.21.15.78:0/1018263944 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa33401e040 con 0x562fc04a7b60 2023-10-10T17:48:48.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.477+0000 7fa32affd700 1 -- 172.21.15.78:0/1018263944 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334003b20 con 0x562fc04a7b60 2023-10-10T17:48:48.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1018263944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a7b60 msgr2=0x562fc04b89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a7b60 0x562fc04b89a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fa3340099c0 tx=0x7fa334020000 comp rx=0 tx=0).stop 2023-10-10T17:48:48.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1018263944 shutdown_connections 2023-10-10T17:48:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc04b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a7b60 0x562fc04b89a0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/1018263944 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a8470 0x562fc04a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1018263944 >> 172.21.15.78:0/1018263944 conn(0x562fc049d360 msgr2=0x562fc04aec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1018263944 shutdown_connections 2023-10-10T17:48:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 -- 172.21.15.78:0/1018263944 wait complete. 2023-10-10T17:48:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 -- start start 2023-10-10T17:48:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.478+0000 7fa3567b3b80 1 Processor -- start 2023-10-10T17:48:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04b8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3567b3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc05147e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc04b89f0 con 0x562fc04a8470 2023-10-10T17:48:48.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc0514d20 con 0x562fc04a7b60 2023-10-10T17:48:48.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3567b3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fc0514e60 con 0x562fc04ac8b0 2023-10-10T17:48:48.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3567b3b80 10 client.0 _ll_register_callbacks cb 0x7fa33c0958c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:48.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04b8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04b8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47222/0 (socket says 172.21.15.78:47222) 2023-10-10T17:48:48.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 -- 172.21.15.78:0/421911852 learned_addr learned my addr 172.21.15.78:0/421911852 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:48.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa343fff700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa348bc4700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc05147e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 -- 172.21.15.78:0/421911852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 msgr2=0x562fc05147e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc05147e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 -- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 msgr2=0x562fc04b7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:48.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa3437fe700 1 -- 172.21.15.78:0/421911852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa344068e30 con 0x562fc04a8470 2023-10-10T17:48:48.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.479+0000 7fa343fff700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:48.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3437fe700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04b8300 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa33800bcb0 tx=0x7fa338010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b15f55b310a4eb server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:48.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3237fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:48.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3237fe700 1 -- 172.21.15.78:0/421911852 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338020040 con 0x562fc04a8470 2023-10-10T17:48:48.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa338015070 con 0x562fc04a8470 2023-10-10T17:48:48.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3237fe700 1 -- 172.21.15.78:0/421911852 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa338017070 con 0x562fc04a8470 2023-10-10T17:48:48.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3237fe700 1 -- 172.21.15.78:0/421911852 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa338003a90 con 0x562fc04a8470 2023-10-10T17:48:48.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3567b3b80 10 client.16734 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:48.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x562fc0226de0 con 0x562fc04a8470 2023-10-10T17:48:48.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3237fe700 1 -- 172.21.15.78:0/421911852 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa338015070 con 0x562fc04a8470 2023-10-10T17:48:48.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3237fe700 1 -- 172.21.15.78:0/421911852 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa338027450 con 0x562fc04a8470 2023-10-10T17:48:48.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.480+0000 7fa3567b3b80 10 client.16734 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:48.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:48.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 20 client.16734 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:48.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc05163f0 0x562fc0518800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:48.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 4 client.16734 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:48:48.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562fc03c7660 con 0x562fc05163f0 2023-10-10T17:48:48.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa343fff700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc05163f0 0x562fc0518800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:48.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3237fe700 10 client.16734 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:48.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa343fff700 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc05163f0 0x562fc0518800 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c94e3d7fba196455 server_cookie=e2b73799ca6d598f in_seq=0 out_seq=0 2023-10-10T17:48:48.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3237fe700 1 -- 172.21.15.78:0/421911852 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562fc03c7660 con 0x562fc05163f0 2023-10-10T17:48:48.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3237fe700 10 client.16734 handle_command_reply: tid=0 2023-10-10T17:48:48.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 1 client.16734 shutdown 2023-10-10T17:48:48.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 msgr2=0x562fc04b8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.481+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04b8300 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fa33800bcb0 tx=0x7fa338010db0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 shutdown_connections 2023-10-10T17:48:48.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x562fc04ac8b0 0x562fc05147e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x562fc05163f0 0x562fc0518800 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x562fc04a7b60 0x562fc04b7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 --2- 172.21.15.78:0/421911852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x562fc04a8470 0x562fc04b8300 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3237fe700 0 client.16734 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:48.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 >> 172.21.15.78:0/421911852 conn(0x562fc049d360 msgr2=0x562fc04aa8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 shutdown_connections 2023-10-10T17:48:48.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 1 -- 172.21.15.78:0/421911852 wait complete. 2023-10-10T17:48:48.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa3567b3b80 20 client.16734 trim_cache size 0 max 16384 2023-10-10T17:48:48.501 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:48:48.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa32c0a1340 msgr2=0x7fa32c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa32c0a1340 0x7fa32c0a37f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa334000c00 tx=0x7fa334003680 comp rx=0 tx=0).stop 2023-10-10T17:48:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3441045d0 msgr2=0x7fa34419ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3441045d0 0x7fa34419ae70 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fa33c007ee0 tx=0x7fa33c00d3b0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 shutdown_connections 2023-10-10T17:48:48.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa3440ff590 0x7fa3441964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa32c0a1340 0x7fa32c0a37f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa3441045d0 0x7fa34419ae70 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 --2- 172.21.15.78:0/3918753388 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa3440fff30 0x7fa344196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:48.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 >> 172.21.15.78:0/3918753388 conn(0x7fa3440760d0 msgr2=0x7fa3440fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:48.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 shutdown_connections 2023-10-10T17:48:48.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:48.482+0000 7fa34a627700 1 -- 172.21.15.78:0/3918753388 wait complete. 2023-10-10T17:48:48.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:48 smithi157 ceph-mon[100896]: pgmap v191: 129 pgs: 129 active+clean; 786 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 449 op/s 2023-10-10T17:48:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:48 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/189219662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:48:49.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:48 smithi078 ceph-mon[99494]: pgmap v191: 129 pgs: 129 active+clean; 786 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 449 op/s 2023-10-10T17:48:49.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:48 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/189219662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:48:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:48 smithi190 ceph-mon[101220]: pgmap v191: 129 pgs: 129 active+clean; 786 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.7 MiB/s wr, 449 op/s 2023-10-10T17:48:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:48 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/189219662' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:48:49.506 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:48:49.506 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:48:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 -- 172.21.15.78:0/3115703905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684104490 msgr2=0x7f468410c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 --2- 172.21.15.78:0/3115703905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684104490 0x7f468410c060 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f467c009fc0 tx=0x7f467c00b330 comp rx=0 tx=0).stop 2023-10-10T17:48:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 -- 172.21.15.78:0/3115703905 shutdown_connections 2023-10-10T17:48:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 --2- 172.21.15.78:0/3115703905 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4684104490 0x7f468410c060 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 --2- 172.21.15.78:0/3115703905 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 0x7f46840ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 --2- 172.21.15.78:0/3115703905 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46840ffdf0 0x7f4684103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 -- 172.21.15.78:0/3115703905 >> 172.21.15.78:0/3115703905 conn(0x7f46840770b0 msgr2=0x7f46840774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 -- 172.21.15.78:0/3115703905 shutdown_connections 2023-10-10T17:48:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.632+0000 7f468c3ca700 1 -- 172.21.15.78:0/3115703905 wait complete. 2023-10-10T17:48:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 Processor -- start 2023-10-10T17:48:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 -- start start 2023-10-10T17:48:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 0x7f4684194310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46840ffdf0 0x7f4684194850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4684104490 0x7f4684198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46841992f0 con 0x7f46840ff450 2023-10-10T17:48:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684199460 con 0x7f4684104490 2023-10-10T17:48:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468c3ca700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684199620 con 0x7f46840ffdf0 2023-10-10T17:48:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468a166700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 0x7f4684194310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468a166700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 0x7f4684194310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47234/0 (socket says 172.21.15.78:47234) 2023-10-10T17:48:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468a166700 1 -- 172.21.15.78:0/1482171005 learned_addr learned my addr 172.21.15.78:0/1482171005 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f4689965700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46840ffdf0 0x7f4684194850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.633+0000 7f468a967700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4684104490 0x7f4684198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468a166700 1 -- 172.21.15.78:0/1482171005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46840ffdf0 msgr2=0x7f4684194850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468a166700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46840ffdf0 0x7f4684194850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468a166700 1 -- 172.21.15.78:0/1482171005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4684104490 msgr2=0x7f4684198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468a166700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4684104490 0x7f4684198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468a166700 1 -- 172.21.15.78:0/1482171005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c009c40 con 0x7f46840ff450 2023-10-10T17:48:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468a166700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 0x7f4684194310 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f468000cad0 tx=0x7f468000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f467b7fe700 1 -- 172.21.15.78:0/1482171005 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680012040 con 0x7f46840ff450 2023-10-10T17:48:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468410e1c0 con 0x7f46840ff450 2023-10-10T17:48:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f467b7fe700 1 -- 172.21.15.78:0/1482171005 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f468000ce50 con 0x7f46840ff450 2023-10-10T17:48:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f467b7fe700 1 -- 172.21.15.78:0/1482171005 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46800107c0 con 0x7f46840ff450 2023-10-10T17:48:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.634+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468410e030 con 0x7f46840ff450 2023-10-10T17:48:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.637+0000 7f467b7fe700 1 -- 172.21.15.78:0/1482171005 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f4680005000 con 0x7f46840ff450 2023-10-10T17:48:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.637+0000 7f467b7fe700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46700a12c0 0x7f46700a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.637+0000 7f467b7fe700 1 -- 172.21.15.78:0/1482171005 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f46800cea10 con 0x7f46840ff450 2023-10-10T17:48:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.637+0000 7f4689965700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46700a12c0 0x7f46700a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.638+0000 7f4689965700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46700a12c0 0x7f46700a3770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4684195920 tx=0x7f4674009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.638+0000 7f4698556b80 1 Processor -- start 2023-10-10T17:48:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.638+0000 7f4698556b80 1 -- start start 2023-10-10T17:48:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4698556b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf2fdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4698556b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3023f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4698556b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf302930 0x55f1cf302d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4698556b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf2030e0 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4698556b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf255920 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4698556b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf24edb0 con 0x55f1cf302930 2023-10-10T17:48:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf2fdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf2fdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47242/0 (socket says 172.21.15.78:47242) 2023-10-10T17:48:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a967700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf302930 0x55f1cf302d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 -- 172.21.15.78:0/111391527 learned_addr learned my addr 172.21.15.78:0/111391527 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4689965700 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3023f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 -- 172.21.15.78:0/111391527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf302930 msgr2=0x55f1cf302d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf302930 0x55f1cf302d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 -- 172.21.15.78:0/111391527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fe4f0 msgr2=0x55f1cf3023f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3023f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 -- 172.21.15.78:0/111391527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1cf0d06b0 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a967700 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf302930 0x55f1cf302d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f468a166700 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf2fdfb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4680098220 tx=0x7f468009b190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2814bce4351405 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4678dac700 1 -- 172.21.15.78:0/111391527 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468009f7e0 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4678dac700 1 -- 172.21.15.78:0/111391527 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46800a3c00 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.639+0000 7f4678dac700 1 -- 172.21.15.78:0/111391527 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468009aca0 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 -- 172.21.15.78:0/111391527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 msgr2=0x55f1cf2fdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf2fdfb0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f4680098220 tx=0x7f468009b190 comp rx=0 tx=0).stop 2023-10-10T17:48:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 -- 172.21.15.78:0/111391527 shutdown_connections 2023-10-10T17:48:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf302930 0x55f1cf302d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf2fdfb0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 --2- 172.21.15.78:0/111391527 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 -- 172.21.15.78:0/111391527 >> 172.21.15.78:0/111391527 conn(0x55f1cf2f33e0 msgr2=0x55f1cf2f8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 -- 172.21.15.78:0/111391527 shutdown_connections 2023-10-10T17:48:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.640+0000 7f4698556b80 1 -- 172.21.15.78:0/111391527 wait complete. 2023-10-10T17:48:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 -- start start 2023-10-10T17:48:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 Processor -- start 2023-10-10T17:48:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf307410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf307950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf302930 0x55f1cf307e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf1f0960 con 0x55f1cf302930 2023-10-10T17:48:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf1f0600 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf0d29a0 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4698556b80 10 client.0 _ll_register_callbacks cb 0x55f1cf20e800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f468a967700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf302930 0x55f1cf307e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4689965700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf307950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f468a166700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf307410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f468a967700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf302930 0x55f1cf307e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47252/0 (socket says 172.21.15.78:47252) 2023-10-10T17:48:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f468a967700 1 -- 172.21.15.78:0/150661511 learned_addr learned my addr 172.21.15.78:0/150661511 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f4689965700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf307950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38780/0 (socket says 172.21.15.78:38780) 2023-10-10T17:48:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.641+0000 7f468a166700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf307410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:48970/0 (socket says 172.21.15.78:48970) 2023-10-10T17:48:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f468a967700 1 -- 172.21.15.78:0/150661511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 msgr2=0x55f1cf307950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f468a967700 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf307950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f468a967700 1 -- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fdbe0 msgr2=0x55f1cf307410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f468a967700 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf307410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f468a967700 1 -- 172.21.15.78:0/150661511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468410e030 con 0x55f1cf302930 2023-10-10T17:48:49.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f468a166700 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf307410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:48:49.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f468a967700 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf302930 0x55f1cf307e90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f467c0073e0 tx=0x7f467c007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f05d928f5d32f83 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f466dffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:49.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f466dffb700 1 -- 172.21.15.78:0/150661511 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c00e040 con 0x55f1cf302930 2023-10-10T17:48:49.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f466dffb700 1 -- 172.21.15.78:0/150661511 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f467c003a00 con 0x55f1cf302930 2023-10-10T17:48:49.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1cf2f0ba0 con 0x55f1cf302930 2023-10-10T17:48:49.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f466dffb700 1 -- 172.21.15.78:0/150661511 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f467c003bc0 con 0x55f1cf302930 2023-10-10T17:48:49.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f4698556b80 10 client.16746 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:49.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f1cf215210 con 0x55f1cf302930 2023-10-10T17:48:49.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f466dffb700 1 -- 172.21.15.78:0/150661511 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f1cf2f0ba0 con 0x55f1cf302930 2023-10-10T17:48:49.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.642+0000 7f466dffb700 1 -- 172.21.15.78:0/150661511 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f467c019070 con 0x55f1cf302930 2023-10-10T17:48:49.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f4698556b80 10 client.16746 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:49.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f4698556b80 10 client.16746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:49.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f4698556b80 20 client.16746 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:49.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f4698556b80 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf317a60 0x55f1cf319e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f4698556b80 4 client.16746 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1cf211120 con 0x55f1cf317a60 2023-10-10T17:48:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f468a166700 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf317a60 0x55f1cf319e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f466dffb700 10 client.16746 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.643+0000 7f468a166700 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf317a60 0x55f1cf319e70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78b1e9586dd78121 server_cookie=34054c753fa8f320 in_seq=0 out_seq=0 2023-10-10T17:48:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f466dffb700 1 -- 172.21.15.78:0/150661511 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f1cf211120 con 0x55f1cf317a60 2023-10-10T17:48:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f466dffb700 10 client.16746 handle_command_reply: tid=0 2023-10-10T17:48:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 client.16746 shutdown 2023-10-10T17:48:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf302930 msgr2=0x55f1cf307e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf302930 0x55f1cf307e90 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f467c0073e0 tx=0x7f467c007410 comp rx=0 tx=0).stop 2023-10-10T17:48:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 shutdown_connections 2023-10-10T17:48:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf317a60 0x55f1cf319e70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf302930 0x55f1cf307e90 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f466dffb700 0 client.16746 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf307410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 --2- 172.21.15.78:0/150661511 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf307950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 >> 172.21.15.78:0/150661511 conn(0x55f1cf2f33e0 msgr2=0x55f1cf303de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 shutdown_connections 2023-10-10T17:48:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 1 -- 172.21.15.78:0/150661511 wait complete. 2023-10-10T17:48:49.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.644+0000 7f4698556b80 20 client.16746 trim_cache size 0 max 16384 2023-10-10T17:48:49.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 Processor -- start 2023-10-10T17:48:49.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 -- start start 2023-10-10T17:48:49.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf300220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf30ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf3019b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf305480 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf314380 con 0x55f1cf302930 2023-10-10T17:48:49.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4698556b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf2fa190 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf300220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf300220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47256/0 (socket says 172.21.15.78:47256) 2023-10-10T17:48:49.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 -- 172.21.15.78:0/2255577029 learned_addr learned my addr 172.21.15.78:0/2255577029 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f4689965700 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf30ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a967700 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf3019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 -- 172.21.15.78:0/2255577029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 msgr2=0x55f1cf30ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf30ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 -- 172.21.15.78:0/2255577029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 msgr2=0x55f1cf3019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf3019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 -- 172.21.15.78:0/2255577029 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1cf214f60 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a967700 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf3019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:48:49.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f468a166700 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf300220 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4680000c00 tx=0x7f4680005350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52452ec42eef2ba4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.662+0000 7f46795ad700 1 -- 172.21.15.78:0/2255577029 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680026f00 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f46795ad700 1 -- 172.21.15.78:0/2255577029 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46800bb0d0 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f46795ad700 1 -- 172.21.15.78:0/2255577029 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468009bc70 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 -- 172.21.15.78:0/2255577029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 msgr2=0x55f1cf300220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf300220 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4680000c00 tx=0x7f4680005350 comp rx=0 tx=0).stop 2023-10-10T17:48:49.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 -- 172.21.15.78:0/2255577029 shutdown_connections 2023-10-10T17:48:49.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf3019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf300220 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 --2- 172.21.15.78:0/2255577029 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf30ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 -- 172.21.15.78:0/2255577029 >> 172.21.15.78:0/2255577029 conn(0x55f1cf2f33e0 msgr2=0x55f1cf3079d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 -- 172.21.15.78:0/2255577029 shutdown_connections 2023-10-10T17:48:49.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 -- 172.21.15.78:0/2255577029 wait complete. 2023-10-10T17:48:49.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.663+0000 7f4698556b80 1 -- start start 2023-10-10T17:48:49.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 1 Processor -- start 2023-10-10T17:48:49.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf365280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3657c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf35f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf35fe80 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf35fff0 con 0x55f1cf302930 2023-10-10T17:48:49.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1cf3601b0 con 0x55f1cf2fdbe0 2023-10-10T17:48:49.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4698556b80 10 client.0 _ll_register_callbacks cb 0x7f467c007de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:49.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3657c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f468a166700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf365280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3657c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47264/0 (socket says 172.21.15.78:47264) 2023-10-10T17:48:49.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 -- 172.21.15.78:0/905383974 learned_addr learned my addr 172.21.15.78:0/905383974 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f468a967700 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf35f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 -- 172.21.15.78:0/905383974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fdbe0 msgr2=0x55f1cf365280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf365280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 -- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 msgr2=0x55f1cf35f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf35f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 -- 172.21.15.78:0/905383974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1cf0d06b0 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f468a166700 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf365280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f4689965700 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3657c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f468406a4a0 tx=0x7f46740047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457967a03520221c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.664+0000 7f466dffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f466dffb700 1 -- 172.21.15.78:0/905383974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674004020 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f466dffb700 1 -- 172.21.15.78:0/905383974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f4674004180 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46800a3c00 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f466dffb700 1 -- 172.21.15.78:0/905383974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4674004340 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 10 client.16752 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f1cf065ce0 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f466dffb700 1 -- 172.21.15.78:0/905383974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46800a3c00 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f466dffb700 1 -- 172.21.15.78:0/905383974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4674003840 con 0x55f1cf2fe4f0 2023-10-10T17:48:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 10 client.16752 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 10 client.16752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 20 client.16752 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf36d300 0x55f1cf36f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 4 client.16752 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55f1cf21d6e0 con 0x55f1cf36d300 2023-10-10T17:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.665+0000 7f468a166700 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf36d300 0x55f1cf36f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.666+0000 7f466dffb700 10 client.16752 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.666+0000 7f468a166700 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf36d300 0x55f1cf36f710 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a241100e320881ab server_cookie=27e18d9cf2aaf297 in_seq=0 out_seq=0 2023-10-10T17:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f466dffb700 1 -- 172.21.15.78:0/905383974 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55f1cf21d6e0 con 0x55f1cf36d300 2023-10-10T17:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f466dffb700 10 client.16752 handle_command_reply: tid=0 2023-10-10T17:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 client.16752 shutdown 2023-10-10T17:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fe4f0 msgr2=0x55f1cf3657c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3657c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f468406a4a0 tx=0x7f46740047c0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 shutdown_connections 2023-10-10T17:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f1cf2fdbe0 0x55f1cf365280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f1cf36d300 0x55f1cf36f710 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f1cf302930 0x55f1cf35f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f466dffb700 0 client.16752 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 --2- 172.21.15.78:0/905383974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f1cf2fe4f0 0x55f1cf3657c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 >> 172.21.15.78:0/905383974 conn(0x55f1cf2f33e0 msgr2=0x55f1cf3019b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 shutdown_connections 2023-10-10T17:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 1 -- 172.21.15.78:0/905383974 wait complete. 2023-10-10T17:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f4698556b80 20 client.16752 trim_cache size 0 max 16384 2023-10-10T17:48:49.750 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:48:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:48:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "732ae985-17c1-48a1-ade3-ad5abcdc2599", 2023-10-10T17:48:49.750 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:48:49.751 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46700a12c0 msgr2=0x7f46700a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46700a12c0 0x7f46700a3770 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4684195920 tx=0x7f4674009260 comp rx=0 tx=0).stop 2023-10-10T17:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 msgr2=0x7f4684194310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 0x7f4684194310 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f468000cad0 tx=0x7f468000cde0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 shutdown_connections 2023-10-10T17:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46700a12c0 0x7f46700a3770 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4684104490 0x7f4684198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46840ff450 0x7f4684194310 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 --2- 172.21.15.78:0/1482171005 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46840ffdf0 0x7f4684194850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 >> 172.21.15.78:0/1482171005 conn(0x7f46840770b0 msgr2=0x7f46840fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 shutdown_connections 2023-10-10T17:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.667+0000 7f468c3ca700 1 -- 172.21.15.78:0/1482171005 wait complete. 2023-10-10T17:48:49.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:48:49.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 -- 172.21.15.78:0/2394128893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 msgr2=0x7f7128104ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 --2- 172.21.15.78:0/2394128893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128104ef0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f71180099c0 tx=0x7f7118009cd0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 -- 172.21.15.78:0/2394128893 shutdown_connections 2023-10-10T17:48:49.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 --2- 172.21.15.78:0/2394128893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71281055d0 0x7f712810d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 --2- 172.21.15.78:0/2394128893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71280feaf0 0x7f71280feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 --2- 172.21.15.78:0/2394128893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128104ef0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 -- 172.21.15.78:0/2394128893 >> 172.21.15.78:0/2394128893 conn(0x7f7128075920 msgr2=0x7f7128075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 -- 172.21.15.78:0/2394128893 shutdown_connections 2023-10-10T17:48:49.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 -- 172.21.15.78:0/2394128893 wait complete. 2023-10-10T17:48:49.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 Processor -- start 2023-10-10T17:48:49.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.883+0000 7f712dee4700 1 -- start start 2023-10-10T17:48:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71280feaf0 0x7f7128196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71281055d0 0x7f712819aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712819b500 con 0x7f71280ff490 2023-10-10T17:48:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712819b670 con 0x7f71280feaf0 2023-10-10T17:48:49.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712819b830 con 0x7f71281055d0 2023-10-10T17:48:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47278/0 (socket says 172.21.15.78:47278) 2023-10-10T17:48:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 -- 172.21.15.78:0/3278908293 learned_addr learned my addr 172.21.15.78:0/3278908293 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f71277fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71280feaf0 0x7f7128196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7127fff700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71281055d0 0x7f712819aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 -- 172.21.15.78:0/3278908293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71281055d0 msgr2=0x7f712819aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71281055d0 0x7f712819aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 -- 172.21.15.78:0/3278908293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71280feaf0 msgr2=0x7f7128196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71280feaf0 0x7f7128196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 -- 172.21.15.78:0/3278908293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71180095f0 con 0x7f71280ff490 2023-10-10T17:48:49.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7126ffd700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128196a60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f71180098e0 tx=0x7f711800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7124ff9700 1 -- 172.21.15.78:0/3278908293 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711801d070 con 0x7f71280ff490 2023-10-10T17:48:49.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7124ff9700 1 -- 172.21.15.78:0/3278908293 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7118022470 con 0x7f71280ff490 2023-10-10T17:48:49.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71281103e0 con 0x7f71280ff490 2023-10-10T17:48:49.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f7124ff9700 1 -- 172.21.15.78:0/3278908293 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118017650 con 0x7f71280ff490 2023-10-10T17:48:49.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.884+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7128110250 con 0x7f71280ff490 2023-10-10T17:48:49.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.886+0000 7f7124ff9700 1 -- 172.21.15.78:0/3278908293 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f7118022850 con 0x7f71280ff490 2023-10-10T17:48:49.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.887+0000 7f7124ff9700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71100a12c0 0x7f71100a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.887+0000 7f7124ff9700 1 -- 172.21.15.78:0/3278908293 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f71180ce020 con 0x7f71280ff490 2023-10-10T17:48:49.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.887+0000 7f71277fe700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71100a12c0 0x7f71100a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f71277fe700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71100a12c0 0x7f71100a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f711c009c60 tx=0x7f711c009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 Processor -- start 2023-10-10T17:48:49.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 -- start start 2023-10-10T17:48:49.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f2473fd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f2473fd880 0x55f247401780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f247401cc0 0x55f247402110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2472fe060 con 0x55f247401cc0 2023-10-10T17:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2473508a0 con 0x55f2473fd880 2023-10-10T17:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f713a070b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f247349d30 con 0x55f2473fcf70 2023-10-10T17:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.888+0000 7f7127fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f247401cc0 0x55f247402110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f247401cc0 0x55f247402110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47288/0 (socket says 172.21.15.78:47288) 2023-10-10T17:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 -- 172.21.15.78:0/2999892526 learned_addr learned my addr 172.21.15.78:0/2999892526 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f71277fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f2473fd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7126ffd700 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f2473fd880 0x55f247401780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 -- 172.21.15.78:0/2999892526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 msgr2=0x55f2473fd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f2473fd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 -- 172.21.15.78:0/2999892526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f2473fd880 msgr2=0x55f247401780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f2473fd880 0x55f247401780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 -- 172.21.15.78:0/2999892526 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2471cb630 con 0x55f247401cc0 2023-10-10T17:48:49.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7126ffd700 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f2473fd880 0x55f247401780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:49.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f7127fff700 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f247401cc0 0x55f247402110 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7120011670 tx=0x7f7120009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=148925a21de71d8b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f710dffb700 1 -- 172.21.15.78:0/2999892526 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712000e040 con 0x55f247401cc0 2023-10-10T17:48:49.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f710dffb700 1 -- 172.21.15.78:0/2999892526 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f712000f040 con 0x55f247401cc0 2023-10-10T17:48:49.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.889+0000 7f710dffb700 1 -- 172.21.15.78:0/2999892526 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7120004740 con 0x55f247401cc0 2023-10-10T17:48:49.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 -- 172.21.15.78:0/2999892526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f247401cc0 msgr2=0x55f247402110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f247401cc0 0x55f247402110 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7120011670 tx=0x7f7120009920 comp rx=0 tx=0).stop 2023-10-10T17:48:49.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 -- 172.21.15.78:0/2999892526 shutdown_connections 2023-10-10T17:48:49.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f2473fd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f247401cc0 0x55f247402110 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 --2- 172.21.15.78:0/2999892526 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f2473fd880 0x55f247401780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 -- 172.21.15.78:0/2999892526 >> 172.21.15.78:0/2999892526 conn(0x55f2473ee360 msgr2=0x55f2473f3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 -- 172.21.15.78:0/2999892526 shutdown_connections 2023-10-10T17:48:49.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 -- 172.21.15.78:0/2999892526 wait complete. 2023-10-10T17:48:49.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.890+0000 7f713a070b80 1 -- start start 2023-10-10T17:48:49.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 1 Processor -- start 2023-10-10T17:48:49.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fcf70 0x55f247409450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fd880 0x55f247406660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247406c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2472eb8e0 con 0x55f2473fcf70 2023-10-10T17:48:49.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2472eb580 con 0x55f247401cc0 2023-10-10T17:48:49.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2471cd920 con 0x55f2473fd880 2023-10-10T17:48:49.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f713a070b80 10 client.0 _ll_register_callbacks cb 0x55f247309780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:49.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fcf70 0x55f247409450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fcf70 0x55f247409450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47290/0 (socket says 172.21.15.78:47290) 2023-10-10T17:48:49.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 -- 172.21.15.78:0/2464219123 learned_addr learned my addr 172.21.15.78:0/2464219123 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f7127fff700 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247406c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f7126ffd700 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fd880 0x55f247406660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 -- 172.21.15.78:0/2464219123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fd880 msgr2=0x55f247406660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fd880 0x55f247406660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 -- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 msgr2=0x55f247406c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247406c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 -- 172.21.15.78:0/2464219123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7120011350 con 0x55f2473fcf70 2023-10-10T17:48:49.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f71277fe700 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fcf70 0x55f247409450 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f711c003900 tx=0x7f711c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24244adb04cc5cad server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f7106ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:49.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.891+0000 7f7106ffd700 1 -- 172.21.15.78:0/2464219123 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711c022070 con 0x55f2473fcf70 2023-10-10T17:48:49.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f7106ffd700 1 -- 172.21.15.78:0/2464219123 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f711c027420 con 0x55f2473fcf70 2023-10-10T17:48:49.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2473ebb20 con 0x55f2473fcf70 2023-10-10T17:48:49.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f7106ffd700 1 -- 172.21.15.78:0/2464219123 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f711c0275e0 con 0x55f2473fcf70 2023-10-10T17:48:49.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 10 client.16764 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:49.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f247310190 con 0x55f2473fcf70 2023-10-10T17:48:49.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f7106ffd700 1 -- 172.21.15.78:0/2464219123 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f2473ebb20 con 0x55f2473fcf70 2023-10-10T17:48:49.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f7106ffd700 1 -- 172.21.15.78:0/2464219123 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f711c01d930 con 0x55f2473fcf70 2023-10-10T17:48:49.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 10 client.16764 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:49.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 10 client.16764 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:49.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 20 client.16764 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:49.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f247416ce0 0x55f2474190f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 4 client.16764 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:49.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f24730c0a0 con 0x55f247416ce0 2023-10-10T17:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.892+0000 7f7126ffd700 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f247416ce0 0x55f2474190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.893+0000 7f7106ffd700 10 client.16764 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.893+0000 7f7126ffd700 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f247416ce0 0x55f2474190f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e337026ad42d151 server_cookie=f36b97706b9c23d8 in_seq=0 out_seq=0 2023-10-10T17:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f7106ffd700 1 -- 172.21.15.78:0/2464219123 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f24730c0a0 con 0x55f247416ce0 2023-10-10T17:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f7106ffd700 10 client.16764 handle_command_reply: tid=0 2023-10-10T17:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 client.16764 shutdown 2023-10-10T17:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fcf70 msgr2=0x55f247409450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fcf70 0x55f247409450 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f711c003900 tx=0x7f711c0047c0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 shutdown_connections 2023-10-10T17:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f247416ce0 0x55f2474190f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247406c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fcf70 0x55f247409450 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 --2- 172.21.15.78:0/2464219123 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fd880 0x55f247406660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 >> 172.21.15.78:0/2464219123 conn(0x55f2473ee360 msgr2=0x55f247403170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f7106ffd700 0 client.16764 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 shutdown_connections 2023-10-10T17:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 1 -- 172.21.15.78:0/2464219123 wait complete. 2023-10-10T17:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.896+0000 7f713a070b80 20 client.16764 trim_cache size 0 max 16384 2023-10-10T17:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 Processor -- start 2023-10-10T17:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 -- start start 2023-10-10T17:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f2474012f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247408f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f247404c10 con 0x55f2473fd880 2023-10-10T17:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2474007d0 con 0x55f247401cc0 2023-10-10T17:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.914+0000 7f713a070b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f247400570 con 0x55f2473fcf70 2023-10-10T17:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f2474012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f2474012f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47300/0 (socket says 172.21.15.78:47300) 2023-10-10T17:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 -- 172.21.15.78:0/272217708 learned_addr learned my addr 172.21.15.78:0/272217708 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f71277fe700 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7127fff700 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247408f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 -- 172.21.15.78:0/272217708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 msgr2=0x55f24740ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 -- 172.21.15.78:0/272217708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 msgr2=0x55f247408f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247408f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 -- 172.21.15.78:0/272217708 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2473efb00 con 0x55f2473fd880 2023-10-10T17:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7127fff700 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247408f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f7126ffd700 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f2474012f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f71180979e0 tx=0x7f7118009370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bbfbc4bc2cfb3a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f710e7fc700 1 -- 172.21.15.78:0/272217708 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71180b7070 con 0x55f2473fd880 2023-10-10T17:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f710e7fc700 1 -- 172.21.15.78:0/272217708 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f711809efa0 con 0x55f2473fd880 2023-10-10T17:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.915+0000 7f710e7fc700 1 -- 172.21.15.78:0/272217708 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71180aebe0 con 0x55f2473fd880 2023-10-10T17:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 -- 172.21.15.78:0/272217708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 msgr2=0x55f2474012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f2474012f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f71180979e0 tx=0x7f7118009370 comp rx=0 tx=0).stop 2023-10-10T17:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 -- 172.21.15.78:0/272217708 shutdown_connections 2023-10-10T17:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247408f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 --2- 172.21.15.78:0/272217708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f2474012f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 -- 172.21.15.78:0/272217708 >> 172.21.15.78:0/272217708 conn(0x55f2473ee360 msgr2=0x55f247400be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 -- 172.21.15.78:0/272217708 shutdown_connections 2023-10-10T17:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 -- 172.21.15.78:0/272217708 wait complete. 2023-10-10T17:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 -- start start 2023-10-10T17:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.916+0000 7f713a070b80 1 Processor -- start 2023-10-10T17:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f24740c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247469ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24746a0e0 con 0x55f2473fd880 2023-10-10T17:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24746a220 con 0x55f247401cc0 2023-10-10T17:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f24746a360 con 0x55f2473fcf70 2023-10-10T17:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 10 client.0 _ll_register_callbacks cb 0x55f247309780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f24740c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f24740c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47306/0 (socket says 172.21.15.78:47306) 2023-10-10T17:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 -- 172.21.15.78:0/3112397083 learned_addr learned my addr 172.21.15.78:0/3112397083 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7127fff700 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247469ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f71277fe700 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 -- 172.21.15.78:0/3112397083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 msgr2=0x55f24740bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 -- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 msgr2=0x55f247469ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247469ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 -- 172.21.15.78:0/3112397083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71281103e0 con 0x55f2473fd880 2023-10-10T17:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7126ffd700 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f24740c4d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7118024000 tx=0x7f71180a3470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9db39ea4831593 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7106ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7106ffd700 1 -- 172.21.15.78:0/3112397083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71180b7070 con 0x55f2473fd880 2023-10-10T17:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f7106ffd700 1 -- 172.21.15.78:0/3112397083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7f71180b91c0 con 0x55f2473fd880 2023-10-10T17:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.917+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f712000f040 con 0x55f2473fd880 2023-10-10T17:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f7106ffd700 1 -- 172.21.15.78:0/3112397083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71180cb2b0 con 0x55f2473fd880 2023-10-10T17:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 10 client.16770 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f247177de0 con 0x55f2473fd880 2023-10-10T17:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f7106ffd700 1 -- 172.21.15.78:0/3112397083 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f712000f040 con 0x55f2473fd880 2023-10-10T17:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f7106ffd700 1 -- 172.21.15.78:0/3112397083 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f711809b320 con 0x55f2473fd880 2023-10-10T17:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 10 client.16770 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 10 client.16770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 20 client.16770 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f24746b700 0x55f24746db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 4 client.16770 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f247318660 con 0x55f24746b700 2023-10-10T17:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f71277fe700 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f24746b700 0x55f24746db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f7106ffd700 10 client.16770 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.918+0000 7f71277fe700 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f24746b700 0x55f24746db10 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae9f0818db1fe289 server_cookie=9d3ea92304ac39f4 in_seq=0 out_seq=0 2023-10-10T17:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f7106ffd700 1 -- 172.21.15.78:0/3112397083 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+267 (crc 0 0 0) 0x55f247318660 con 0x55f24746b700 2023-10-10T17:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f7106ffd700 10 client.16770 handle_command_reply: tid=0 2023-10-10T17:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 client.16770 shutdown 2023-10-10T17:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 msgr2=0x55f24740c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f24740c4d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f7118024000 tx=0x7f71180a3470 comp rx=0 tx=0).stop 2023-10-10T17:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 shutdown_connections 2023-10-10T17:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f2473fcf70 0x55f24740bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f24746b700 0x55f24746db10 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f247401cc0 0x55f247469ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 --2- 172.21.15.78:0/3112397083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f2473fd880 0x55f24740c4d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f7106ffd700 0 client.16770 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 >> 172.21.15.78:0/3112397083 conn(0x55f2473ee360 msgr2=0x55f247400be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 shutdown_connections 2023-10-10T17:48:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.921+0000 7f713a070b80 1 -- 172.21.15.78:0/3112397083 wait complete. 2023-10-10T17:48:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f713a070b80 20 client.16770 trim_cache size 0 max 16384 2023-10-10T17:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "scrub active (103 inodes in the stack)", 2023-10-10T17:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "732ae985-17c1-48a1-ade3-ad5abcdc2599": { 2023-10-10T17:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "732ae985-17c1-48a1-ade3-ad5abcdc2599", 2023-10-10T17:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:48:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71100a12c0 msgr2=0x7f71100a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71100a12c0 0x7f71100a3770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f711c009c60 tx=0x7f711c009260 comp rx=0 tx=0).stop 2023-10-10T17:48:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 msgr2=0x7f7128196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128196a60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f71180098e0 tx=0x7f711800f690 comp rx=0 tx=0).stop 2023-10-10T17:48:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 shutdown_connections 2023-10-10T17:48:49.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f71281055d0 0x7f712819aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71100a12c0 0x7f71100a3770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f71280feaf0 0x7f7128196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 --2- 172.21.15.78:0/3278908293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f71280ff490 0x7f7128196a60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 >> 172.21.15.78:0/3278908293 conn(0x7f7128075920 msgr2=0x7f71280fc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:49.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 shutdown_connections 2023-10-10T17:48:49.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:48:49.922+0000 7f712dee4700 1 -- 172.21.15.78:0/3278908293 wait complete. 2023-10-10T17:48:49.946 INFO:tasks.cephfs.filesystem:scrub status for tag:732ae985-17c1-48a1-ade3-ad5abcdc2599 - {'path': '/', 'tag': '732ae985-17c1-48a1-ade3-ad5abcdc2599', 'options': 'recursive,force'} 2023-10-10T17:48:50.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:50 smithi157 ceph-mon[100896]: pgmap v192: 129 pgs: 129 active+clean; 786 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 169 KiB/s wr, 286 op/s 2023-10-10T17:48:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:50 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:48:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:50 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:50 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:50.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:50 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:50.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:50 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:51.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:50 smithi078 ceph-mon[99494]: pgmap v192: 129 pgs: 129 active+clean; 786 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 169 KiB/s wr, 286 op/s 2023-10-10T17:48:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:50 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:48:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:50 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:50 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:50 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:50 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:50 smithi190 ceph-mon[101220]: pgmap v192: 129 pgs: 129 active+clean; 786 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 169 KiB/s wr, 286 op/s 2023-10-10T17:48:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:50 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:48:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:50 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:50 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:51.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:50 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:51.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:50 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:52 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:48:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:52 smithi157 ceph-mon[100896]: pgmap v193: 129 pgs: 129 active+clean; 788 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 279 KiB/s wr, 341 op/s 2023-10-10T17:48:52.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:52 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:52 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:48:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:52 smithi078 ceph-mon[99494]: pgmap v193: 129 pgs: 129 active+clean; 788 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 279 KiB/s wr, 341 op/s 2023-10-10T17:48:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:52 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:52 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:48:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:52 smithi190 ceph-mon[101220]: pgmap v193: 129 pgs: 129 active+clean; 788 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 279 KiB/s wr, 341 op/s 2023-10-10T17:48:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:52 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:53.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:53 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:54.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:53 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:54.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:53 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:48:54.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:54 smithi157 ceph-mon[100896]: pgmap v194: 129 pgs: 129 active+clean; 810 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.1 MiB/s wr, 597 op/s 2023-10-10T17:48:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:54 smithi078 ceph-mon[99494]: pgmap v194: 129 pgs: 129 active+clean; 810 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.1 MiB/s wr, 597 op/s 2023-10-10T17:48:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:54 smithi190 ceph-mon[101220]: pgmap v194: 129 pgs: 129 active+clean; 810 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.1 MiB/s wr, 597 op/s 2023-10-10T17:48:56.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.279+0000 7fa584543700 1 -- 172.21.15.78:0/2034072562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c100c50 msgr2=0x7fa57c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.279+0000 7fa584543700 1 --2- 172.21.15.78:0/2034072562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c100c50 0x7fa57c101020 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0099c0 tx=0x7fa56c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.279+0000 7fa584543700 1 -- 172.21.15.78:0/2034072562 shutdown_connections 2023-10-10T17:48:56.281 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.279+0000 7fa584543700 1 --2- 172.21.15.78:0/2034072562 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c100c50 0x7fa57c101020 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.279+0000 7fa584543700 1 --2- 172.21.15.78:0/2034072562 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa57c105560 0x7fa57c10d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.279+0000 7fa584543700 1 --2- 172.21.15.78:0/2034072562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c1015f0 0x7fa57c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.279+0000 7fa584543700 1 -- 172.21.15.78:0/2034072562 >> 172.21.15.78:0/2034072562 conn(0x7fa57c077550 msgr2=0x7fa57c077950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:56.282 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.280+0000 7fa584543700 1 -- 172.21.15.78:0/2034072562 shutdown_connections 2023-10-10T17:48:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.280+0000 7fa584543700 1 -- 172.21.15.78:0/2034072562 wait complete. 2023-10-10T17:48:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.280+0000 7fa584543700 1 Processor -- start 2023-10-10T17:48:56.283 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.280+0000 7fa584543700 1 -- start start 2023-10-10T17:48:56.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa584543700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c100c50 0x7fa57c196390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa584543700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa57c1015f0 0x7fa57c1968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa584543700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c105560 0x7fa57c19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.285 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa584543700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c19b370 con 0x7fa57c100c50 2023-10-10T17:48:56.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa584543700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c19b4e0 con 0x7fa57c1015f0 2023-10-10T17:48:56.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa584543700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57c19b6a0 con 0x7fa57c105560 2023-10-10T17:48:56.286 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c100c50 0x7fa57c196390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c100c50 0x7fa57c196390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47318/0 (socket says 172.21.15.78:47318) 2023-10-10T17:48:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 -- 172.21.15.78:0/2671484418 learned_addr learned my addr 172.21.15.78:0/2671484418 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa581ade700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa57c1015f0 0x7fa57c1968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.287 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa582ae0700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c105560 0x7fa57c19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 -- 172.21.15.78:0/2671484418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c105560 msgr2=0x7fa57c19ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c105560 0x7fa57c19ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 -- 172.21.15.78:0/2671484418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa57c1015f0 msgr2=0x7fa57c1968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa57c1015f0 0x7fa57c1968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.288 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 -- 172.21.15.78:0/2671484418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c0095f0 con 0x7fa57c100c50 2023-10-10T17:48:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5822df700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c100c50 0x7fa57c196390 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00b580 tx=0x7fa56c00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.281+0000 7fa5737fe700 1 -- 172.21.15.78:0/2671484418 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c01d070 con 0x7fa57c100c50 2023-10-10T17:48:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.282+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57c110360 con 0x7fa57c100c50 2023-10-10T17:48:56.289 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.282+0000 7fa5737fe700 1 -- 172.21.15.78:0/2671484418 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa56c022470 con 0x7fa57c100c50 2023-10-10T17:48:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.282+0000 7fa5737fe700 1 -- 172.21.15.78:0/2671484418 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56c017650 con 0x7fa57c100c50 2023-10-10T17:48:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.282+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57c1101d0 con 0x7fa57c100c50 2023-10-10T17:48:56.290 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.284+0000 7fa5737fe700 1 -- 172.21.15.78:0/2671484418 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa56c0225e0 con 0x7fa57c100c50 2023-10-10T17:48:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.285+0000 7fa5737fe700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5680a12c0 0x7fa5680a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.285+0000 7fa5737fe700 1 -- 172.21.15.78:0/2671484418 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa56c0ce080 con 0x7fa57c100c50 2023-10-10T17:48:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.285+0000 7fa581ade700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5680a12c0 0x7fa5680a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.296 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.286+0000 7fa581ade700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5680a12c0 0x7fa5680a3770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa57c1979a0 tx=0x7fa578009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.286+0000 7fa5906cfb80 1 Processor -- start 2023-10-10T17:48:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.286+0000 7fa5906cfb80 1 -- start start 2023-10-10T17:48:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825dc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.297 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825d6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcf60 0x561e825d6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e824d8000 con 0x561e825dcaf0 2023-10-10T17:48:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e8252a840 con 0x561e825dcf60 2023-10-10T17:48:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e82523cd0 con 0x561e8242e6b0 2023-10-10T17:48:56.298 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa581ade700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825d6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa581ade700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825d6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47328/0 (socket says 172.21.15.78:47328) 2023-10-10T17:48:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa582ae0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcf60 0x561e825d6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa581ade700 1 -- 172.21.15.78:0/1736854736 learned_addr learned my addr 172.21.15.78:0/1736854736 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:56.299 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa5822df700 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825dc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa582ae0700 1 -- 172.21.15.78:0/1736854736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 msgr2=0x561e825dc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa582ae0700 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825dc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa582ae0700 1 -- 172.21.15.78:0/1736854736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 msgr2=0x561e825d6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.300 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa582ae0700 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825d6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa582ae0700 1 -- 172.21.15.78:0/1736854736 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e823a55d0 con 0x561e825dcf60 2023-10-10T17:48:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.287+0000 7fa581ade700 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825d6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.288+0000 7fa582ae0700 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcf60 0x561e825d6ef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa574011580 tx=0x7fa574009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8130e0c35b5e33e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.288+0000 7fa570ff9700 1 -- 172.21.15.78:0/1736854736 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57400e040 con 0x561e825dcf60 2023-10-10T17:48:56.301 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.288+0000 7fa570ff9700 1 -- 172.21.15.78:0/1736854736 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa57400f040 con 0x561e825dcf60 2023-10-10T17:48:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.288+0000 7fa570ff9700 1 -- 172.21.15.78:0/1736854736 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574004490 con 0x561e825dcf60 2023-10-10T17:48:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 -- 172.21.15.78:0/1736854736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcf60 msgr2=0x561e825d6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcf60 0x561e825d6ef0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa574011580 tx=0x7fa574009920 comp rx=0 tx=0).stop 2023-10-10T17:48:56.302 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 -- 172.21.15.78:0/1736854736 shutdown_connections 2023-10-10T17:48:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825dc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcf60 0x561e825d6ef0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/1736854736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825d6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 -- 172.21.15.78:0/1736854736 >> 172.21.15.78:0/1736854736 conn(0x561e825c7f50 msgr2=0x561e825cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:56.303 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 -- 172.21.15.78:0/1736854736 shutdown_connections 2023-10-10T17:48:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 -- 172.21.15.78:0/1736854736 wait complete. 2023-10-10T17:48:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 -- start start 2023-10-10T17:48:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.289+0000 7fa5906cfb80 1 Processor -- start 2023-10-10T17:48:56.304 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825e2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcaf0 0x561e825e00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcf60 0x561e825e0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e824c5880 con 0x561e825dcf60 2023-10-10T17:48:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e824c5520 con 0x561e825dcaf0 2023-10-10T17:48:56.305 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e823a78c0 con 0x561e8242e6b0 2023-10-10T17:48:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5906cfb80 10 client.0 _ll_register_callbacks cb 0x561e824e3720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5822df700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825e2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa581ade700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcaf0 0x561e825e00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.306 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa582ae0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcf60 0x561e825e0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5822df700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825e2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:38862/0 (socket says 172.21.15.78:38862) 2023-10-10T17:48:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa581ade700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcaf0 0x561e825e00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:49056/0 (socket says 172.21.15.78:49056) 2023-10-10T17:48:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa5822df700 1 -- 172.21.15.78:0/3629370551 learned_addr learned my addr 172.21.15.78:0/3629370551 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:56.307 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa582ae0700 1 -- 172.21.15.78:0/3629370551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 msgr2=0x561e825e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa582ae0700 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825e2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa582ae0700 1 -- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcaf0 msgr2=0x561e825e00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa582ae0700 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcaf0 0x561e825e00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.308 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa582ae0700 1 -- 172.21.15.78:0/3629370551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa574011230 con 0x561e825dcf60 2023-10-10T17:48:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.290+0000 7fa581ade700 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcaf0 0x561e825e00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa582ae0700 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcf60 0x561e825e0720 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa57400b010 tx=0x7fa574011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b696a1f15c6c2cca server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa565ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:56.309 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa565ffb700 1 -- 172.21.15.78:0/3629370551 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57400e040 con 0x561e825dcf60 2023-10-10T17:48:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e825c5710 con 0x561e825dcf60 2023-10-10T17:48:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa565ffb700 1 -- 172.21.15.78:0/3629370551 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa574003d80 con 0x561e825dcf60 2023-10-10T17:48:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa565ffb700 1 -- 172.21.15.78:0/3629370551 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa574003f40 con 0x561e825dcf60 2023-10-10T17:48:56.310 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa5906cfb80 10 client.16782 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561e824ea130 con 0x561e825dcf60 2023-10-10T17:48:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa565ffb700 1 -- 172.21.15.78:0/3629370551 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561e825c5710 con 0x561e825dcf60 2023-10-10T17:48:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa565ffb700 1 -- 172.21.15.78:0/3629370551 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa57401b020 con 0x561e825dcf60 2023-10-10T17:48:56.311 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa5906cfb80 10 client.16782 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.291+0000 7fa5906cfb80 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.292+0000 7fa5906cfb80 20 client.16782 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.292+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e825f0450 0x561e825f2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.312 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.292+0000 7fa5906cfb80 4 client.16782 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:48:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.292+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e824e6040 con 0x561e825f0450 2023-10-10T17:48:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.292+0000 7fa5822df700 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e825f0450 0x561e825f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.292+0000 7fa565ffb700 10 client.16782 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:56.313 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.292+0000 7fa5822df700 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e825f0450 0x561e825f2860 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1eadb89f43b15dba server_cookie=b519626756b42f5f in_seq=0 out_seq=0 2023-10-10T17:48:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa565ffb700 1 -- 172.21.15.78:0/3629370551 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561e824e6040 con 0x561e825f0450 2023-10-10T17:48:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa565ffb700 10 client.16782 handle_command_reply: tid=0 2023-10-10T17:48:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 client.16782 shutdown 2023-10-10T17:48:56.314 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcf60 msgr2=0x561e825e0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcf60 0x561e825e0720 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa57400b010 tx=0x7fa574011f70 comp rx=0 tx=0).stop 2023-10-10T17:48:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 shutdown_connections 2023-10-10T17:48:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e8242e6b0 0x561e825e2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.315 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e825f0450 0x561e825f2860 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcf60 0x561e825e0720 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3629370551 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e825dcaf0 0x561e825e00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 >> 172.21.15.78:0/3629370551 conn(0x561e825c7f50 msgr2=0x561e825d1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:56.316 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa565ffb700 0 client.16782 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 shutdown_connections 2023-10-10T17:48:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.295+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3629370551 wait complete. 2023-10-10T17:48:56.322 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.296+0000 7fa5906cfb80 20 client.16782 trim_cache size 0 max 16384 2023-10-10T17:48:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.312+0000 7fa5906cfb80 1 Processor -- start 2023-10-10T17:48:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.312+0000 7fa5906cfb80 1 -- start start 2023-10-10T17:48:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.323 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825df8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e825e1710 con 0x561e825dcaf0 2023-10-10T17:48:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e825de650 con 0x561e8242e6b0 2023-10-10T17:48:56.324 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e825dadc0 con 0x561e825dcf60 2023-10-10T17:48:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825df8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825df8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47340/0 (socket says 172.21.15.78:47340) 2023-10-10T17:48:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5822df700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa582ae0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.325 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 -- 172.21.15.78:0/3191711331 learned_addr learned my addr 172.21.15.78:0/3191711331 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 -- 172.21.15.78:0/3191711331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 msgr2=0x561e825e00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 -- 172.21.15.78:0/3191711331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 msgr2=0x561e825e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.326 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 -- 172.21.15.78:0/3191711331 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e824e9e80 con 0x561e825dcaf0 2023-10-10T17:48:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa5822df700 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.313+0000 7fa581ade700 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825df8b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa578004360 tx=0x7fa578014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8a550f5583a8697 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5717fa700 1 -- 172.21.15.78:0/3191711331 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578022030 con 0x561e825dcaf0 2023-10-10T17:48:56.327 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5717fa700 1 -- 172.21.15.78:0/3191711331 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa578023040 con 0x561e825dcaf0 2023-10-10T17:48:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5717fa700 1 -- 172.21.15.78:0/3191711331 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578027400 con 0x561e825dcaf0 2023-10-10T17:48:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3191711331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 msgr2=0x561e825df8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825df8b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fa578004360 tx=0x7fa578014000 comp rx=0 tx=0).stop 2023-10-10T17:48:56.328 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3191711331 shutdown_connections 2023-10-10T17:48:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/3191711331 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825df8b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3191711331 >> 172.21.15.78:0/3191711331 conn(0x561e825c7f50 msgr2=0x561e825cd050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:56.329 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3191711331 shutdown_connections 2023-10-10T17:48:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.314+0000 7fa5906cfb80 1 -- 172.21.15.78:0/3191711331 wait complete. 2023-10-10T17:48:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 -- start start 2023-10-10T17:48:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 Processor -- start 2023-10-10T17:48:56.330 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825e3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e825e78b0 con 0x561e825dcaf0 2023-10-10T17:48:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e82643300 con 0x561e8242e6b0 2023-10-10T17:48:56.331 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.315+0000 7fa5906cfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e82643440 con 0x561e825dcf60 2023-10-10T17:48:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa5906cfb80 10 client.0 _ll_register_callbacks cb 0x7fa578011a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:48:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825e3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa5822df700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.332 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825e3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47348/0 (socket says 172.21.15.78:47348) 2023-10-10T17:48:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa582ae0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 -- 172.21.15.78:0/2367778318 learned_addr learned my addr 172.21.15.78:0/2367778318 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:48:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 -- 172.21.15.78:0/2367778318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 msgr2=0x561e825e3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.333 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 -- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 msgr2=0x561e825e34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:48:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 -- 172.21.15.78:0/2367778318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57c1101d0 con 0x561e825dcaf0 2023-10-10T17:48:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa5822df700 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:48:56.334 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa581ade700 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825e3a30 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa5780046b0 tx=0x7fa5780057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46ec33cddeca713f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:48:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa565ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:48:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa565ffb700 1 -- 172.21.15.78:0/2367778318 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578035070 con 0x561e825dcaf0 2023-10-10T17:48:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa565ffb700 1 -- 172.21.15.78:0/2367778318 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9344+0+0 (secure 0 0 0) 0x7fa578003d20 con 0x561e825dcaf0 2023-10-10T17:48:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.316+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa57400f040 con 0x561e825dcaf0 2023-10-10T17:48:56.335 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa565ffb700 1 -- 172.21.15.78:0/2367778318 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa578022070 con 0x561e825dcaf0 2023-10-10T17:48:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 10 client.16788 fetch_fsmap learned FSMap version 32 2023-10-10T17:48:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7fa578004040 con 0x561e825dcaf0 2023-10-10T17:48:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa565ffb700 1 -- 172.21.15.78:0/2367778318 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa57400f040 con 0x561e825dcaf0 2023-10-10T17:48:56.336 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa565ffb700 1 -- 172.21.15.78:0/2367778318 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa578038030 con 0x561e825dcaf0 2023-10-10T17:48:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 10 client.16788 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:48:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 10 client.16788 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:48:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 20 client.16788 populate_metadata read hostname 'smithi078' 2023-10-10T17:48:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e82644a90 0x561e82646ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:48:56.337 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 4 client.16788 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:48:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561e824f2600 con 0x561e82644a90 2023-10-10T17:48:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.317+0000 7fa5822df700 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e82644a90 0x561e82646ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:48:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.318+0000 7fa565ffb700 10 client.16788 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:56.338 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.318+0000 7fa5822df700 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e82644a90 0x561e82646ea0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=113b30be450e694 server_cookie=44540fc6c4606f90 in_seq=0 out_seq=0 2023-10-10T17:48:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa565ffb700 1 -- 172.21.15.78:0/2367778318 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561e824f2600 con 0x561e82644a90 2023-10-10T17:48:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa565ffb700 10 client.16788 handle_command_reply: tid=0 2023-10-10T17:48:56.433 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 client.16788 shutdown 2023-10-10T17:48:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 msgr2=0x561e825e3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825e3a30 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa5780046b0 tx=0x7fa5780057a0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 shutdown_connections 2023-10-10T17:48:56.434 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561e825dcf60 0x561e825e3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561e82644a90 0x561e82646ea0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561e8242e6b0 0x561e825e34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa565ffb700 0 client.16788 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:48:56.435 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 --2- 172.21.15.78:0/2367778318 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561e825dcaf0 0x561e825e3a30 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 >> 172.21.15.78:0/2367778318 conn(0x561e825c7f50 msgr2=0x561e825d1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 shutdown_connections 2023-10-10T17:48:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 1 -- 172.21.15.78:0/2367778318 wait complete. 2023-10-10T17:48:56.436 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.432+0000 7fa5906cfb80 20 client.16788 trim_cache size 0 max 16384 2023-10-10T17:48:56.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:48:56.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:48:56.437 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:48:56.438 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:48:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5680a12c0 msgr2=0x7fa5680a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5680a12c0 0x7fa5680a3770 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa57c1979a0 tx=0x7fa578009260 comp rx=0 tx=0).stop 2023-10-10T17:48:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c100c50 msgr2=0x7fa57c196390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:48:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c100c50 0x7fa57c196390 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fa56c00b580 tx=0x7fa56c00bfd0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 shutdown_connections 2023-10-10T17:48:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa57c105560 0x7fa57c19ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5680a12c0 0x7fa5680a3770 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa57c100c50 0x7fa57c196390 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 --2- 172.21.15.78:0/2671484418 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa57c1015f0 0x7fa57c1968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:48:56.442 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 >> 172.21.15.78:0/2671484418 conn(0x7fa57c077550 msgr2=0x7fa57c0fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:48:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 shutdown_connections 2023-10-10T17:48:56.443 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:48:56.433+0000 7fa584543700 1 -- 172.21.15.78:0/2671484418 wait complete. 2023-10-10T17:48:56.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:56 smithi157 ceph-mon[100896]: pgmap v195: 129 pgs: 129 active+clean; 810 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 513 op/s 2023-10-10T17:48:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:56 smithi078 ceph-mon[99494]: pgmap v195: 129 pgs: 129 active+clean; 810 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 513 op/s 2023-10-10T17:48:57.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:56 smithi190 ceph-mon[101220]: pgmap v195: 129 pgs: 129 active+clean; 810 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 2.1 MiB/s wr, 513 op/s 2023-10-10T17:48:57.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:57 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:48:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:57 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:48:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:57 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:48:58.536 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T17:48:58.536 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T17:48:58.571 INFO:tasks.workunit:Running workunit fs/misc/filelock_deadlock.py... 2023-10-10T17:48:58.572 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/filelock_deadlock.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_deadlock.py 2023-10-10T17:48:58.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:58 smithi078 ceph-mon[99494]: pgmap v196: 129 pgs: 129 active+clean; 806 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.1 MiB/s wr, 668 op/s 2023-10-10T17:48:58.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:58 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:48:58.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:58 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:48:58.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:58 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:48:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:58 smithi190 ceph-mon[101220]: pgmap v196: 129 pgs: 129 active+clean; 806 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.1 MiB/s wr, 668 op/s 2023-10-10T17:48:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:58 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:48:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:58 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:48:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:58 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:48:59.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:58 smithi157 ceph-mon[100896]: pgmap v196: 129 pgs: 129 active+clean; 806 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 2.1 MiB/s wr, 668 op/s 2023-10-10T17:48:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:58 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:48:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:58 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:48:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:58 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:49:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:59 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:49:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:59 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:49:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:59 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:49:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:48:59 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:49:00.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:59 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:49:00.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:59 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:49:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:59 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:49:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:48:59 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:49:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:59 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:49:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:59 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:49:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:59 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:49:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:48:59 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:49:01.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:00 smithi078 ceph-mon[99494]: pgmap v197: 129 pgs: 129 active+clean; 806 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 494 op/s 2023-10-10T17:49:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:00 smithi190 ceph-mon[101220]: pgmap v197: 129 pgs: 129 active+clean; 806 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 494 op/s 2023-10-10T17:49:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:00 smithi157 ceph-mon[100896]: pgmap v197: 129 pgs: 129 active+clean; 806 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 2.0 MiB/s wr, 494 op/s 2023-10-10T17:49:02.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:01 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:49:02.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:01 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:49:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:01 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:49:03.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:02 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:49:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:02 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:49:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:02 smithi078 ceph-mon[99494]: pgmap v198: 129 pgs: 129 active+clean; 802 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.0 MiB/s wr, 518 op/s 2023-10-10T17:49:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:02 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:49:03.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:02 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:49:03.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:02 smithi190 ceph-mon[101220]: pgmap v198: 129 pgs: 129 active+clean; 802 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.0 MiB/s wr, 518 op/s 2023-10-10T17:49:03.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:02 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:49:03.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:02 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:49:03.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:02 smithi157 ceph-mon[100896]: pgmap v198: 129 pgs: 129 active+clean; 802 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 2.0 MiB/s wr, 518 op/s 2023-10-10T17:49:05.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:04 smithi078 ceph-mon[99494]: pgmap v199: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 481 op/s 2023-10-10T17:49:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:04 smithi190 ceph-mon[101220]: pgmap v199: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 481 op/s 2023-10-10T17:49:05.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:04 smithi157 ceph-mon[100896]: pgmap v199: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 2.0 MiB/s wr, 481 op/s 2023-10-10T17:49:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:06 smithi078 ceph-mon[99494]: pgmap v200: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 120 KiB/s wr, 196 op/s 2023-10-10T17:49:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:06 smithi190 ceph-mon[101220]: pgmap v200: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 120 KiB/s wr, 196 op/s 2023-10-10T17:49:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:06 smithi157 ceph-mon[100896]: pgmap v200: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 120 KiB/s wr, 196 op/s 2023-10-10T17:49:08.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:07 smithi078 ceph-mon[99494]: osdmap e95: 12 total, 12 up, 12 in 2023-10-10T17:49:08.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:07 smithi190 ceph-mon[101220]: osdmap e95: 12 total, 12 up, 12 in 2023-10-10T17:49:08.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:07 smithi157 ceph-mon[100896]: osdmap e95: 12 total, 12 up, 12 in 2023-10-10T17:49:08.685 INFO:tasks.workunit.client.0.smithi078.stdout:ok 2023-10-10T17:49:08.689 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T17:49:08.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T17:49:08.745 INFO:tasks.workunit:Running workunit fs/misc/filelock_interrupt.py... 2023-10-10T17:49:08.745 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/filelock_interrupt.py> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/filelock_interrupt.py 2023-10-10T17:49:09.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:08 smithi078 ceph-mon[99494]: pgmap v202: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 166 KiB/s wr, 54 op/s 2023-10-10T17:49:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:08 smithi190 ceph-mon[101220]: pgmap v202: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 166 KiB/s wr, 54 op/s 2023-10-10T17:49:09.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:08 smithi157 ceph-mon[100896]: pgmap v202: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 166 KiB/s wr, 54 op/s 2023-10-10T17:49:09.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 -- 172.21.15.78:0/3497481285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c0ffd60 msgr2=0x7f5a5c103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 --2- 172.21.15.78:0/3497481285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c0ffd60 0x7f5a5c103d20 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f5a480099c0 tx=0x7f5a48009cd0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 -- 172.21.15.78:0/3497481285 shutdown_connections 2023-10-10T17:49:09.551 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 --2- 172.21.15.78:0/3497481285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a5c0ff3c0 0x7f5a5c0ff790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 --2- 172.21.15.78:0/3497481285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c104400 0x7f5a5c10bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 --2- 172.21.15.78:0/3497481285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c0ffd60 0x7f5a5c103d20 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 -- 172.21.15.78:0/3497481285 >> 172.21.15.78:0/3497481285 conn(0x7f5a5c0763a0 msgr2=0x7f5a5c0767a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:09.552 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 -- 172.21.15.78:0/3497481285 shutdown_connections 2023-10-10T17:49:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.548+0000 7f5a62e87700 1 -- 172.21.15.78:0/3497481285 wait complete. 2023-10-10T17:49:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 Processor -- start 2023-10-10T17:49:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 -- start start 2023-10-10T17:49:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a5c0ff3c0 0x7f5a5c196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c0ffd60 0x7f5a5c196940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.553 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c104400 0x7f5a5c19ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c19b3e0 con 0x7f5a5c104400 2023-10-10T17:49:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c19b550 con 0x7f5a5c0ffd60 2023-10-10T17:49:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a62e87700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5c19b710 con 0x7f5a5c0ff3c0 2023-10-10T17:49:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c104400 0x7f5a5c19ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.554 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c104400 0x7f5a5c19ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47364/0 (socket says 172.21.15.78:47364) 2023-10-10T17:49:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 -- 172.21.15.78:0/383715081 learned_addr learned my addr 172.21.15.78:0/383715081 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a60c23700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a5c0ff3c0 0x7f5a5c196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a5bfff700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c0ffd60 0x7f5a5c196940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 -- 172.21.15.78:0/383715081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a5c0ff3c0 msgr2=0x7f5a5c196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.555 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a5c0ff3c0 0x7f5a5c196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 -- 172.21.15.78:0/383715081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c0ffd60 msgr2=0x7f5a5c196940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c0ffd60 0x7f5a5c196940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 -- 172.21.15.78:0/383715081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a480095f0 con 0x7f5a5c104400 2023-10-10T17:49:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a5bfff700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c0ffd60 0x7f5a5c196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:09.556 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.549+0000 7f5a61424700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c104400 0x7f5a5c19ada0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400ada0 tx=0x7f5a5400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.550+0000 7f5a59ffb700 1 -- 172.21.15.78:0/383715081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5400f070 con 0x7f5a5c104400 2023-10-10T17:49:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.550+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5c110390 con 0x7f5a5c104400 2023-10-10T17:49:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.550+0000 7f5a59ffb700 1 -- 172.21.15.78:0/383715081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a54007b30 con 0x7f5a5c104400 2023-10-10T17:49:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.550+0000 7f5a59ffb700 1 -- 172.21.15.78:0/383715081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540053e0 con 0x7f5a5c104400 2023-10-10T17:49:09.557 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.550+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c110200 con 0x7f5a5c104400 2023-10-10T17:49:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.552+0000 7f5a59ffb700 1 -- 172.21.15.78:0/383715081 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5a5400c980 con 0x7f5a5c104400 2023-10-10T17:49:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.552+0000 7f5a59ffb700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a4c0a12c0 0x7f5a4c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.553+0000 7f5a59ffb700 1 -- 172.21.15.78:0/383715081 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f5a540cc6d0 con 0x7f5a5c104400 2023-10-10T17:49:09.564 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.553+0000 7f5a60c23700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a4c0a12c0 0x7f5a4c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.553+0000 7f5a60c23700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a4c0a12c0 0x7f5a4c0a3770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a50005950 tx=0x7f5a5000b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.553+0000 7f5a6f013b80 1 Processor -- start 2023-10-10T17:49:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.553+0000 7f5a6f013b80 1 -- start start 2023-10-10T17:49:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.553+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c2fe5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.565 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.553+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c2f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c2f8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c1fa000 con 0x56493c2fef60 2023-10-10T17:49:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c24c840 con 0x56493c1506b0 2023-10-10T17:49:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c245cd0 con 0x56493c2feaf0 2023-10-10T17:49:09.566 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c2f8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c2f8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47372/0 (socket says 172.21.15.78:47372) 2023-10-10T17:49:09.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 -- 172.21.15.78:0/1107049813 learned_addr learned my addr 172.21.15.78:0/1107049813 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:09.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a60c23700 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c2fe5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a5bfff700 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c2f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.567 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 -- 172.21.15.78:0/1107049813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 msgr2=0x56493c2f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c2f8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.568 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 -- 172.21.15.78:0/1107049813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 msgr2=0x56493c2fe5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c2fe5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 -- 172.21.15.78:0/1107049813 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56493c0c75d0 con 0x56493c2fef60 2023-10-10T17:49:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a60c23700 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c2fe5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a61424700 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c2f8ef0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5a54095e80 tx=0x7f5a54099310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25f2dcfb3f48fbc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:09.569 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a46ffd700 1 -- 172.21.15.78:0/1107049813 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540192b0 con 0x56493c2fef60 2023-10-10T17:49:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a46ffd700 1 -- 172.21.15.78:0/1107049813 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a54099410 con 0x56493c2fef60 2023-10-10T17:49:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.554+0000 7f5a46ffd700 1 -- 172.21.15.78:0/1107049813 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5409d1f0 con 0x56493c2fef60 2023-10-10T17:49:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 -- 172.21.15.78:0/1107049813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 msgr2=0x56493c2f8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c2f8ef0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f5a54095e80 tx=0x7f5a54099310 comp rx=0 tx=0).stop 2023-10-10T17:49:09.570 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 -- 172.21.15.78:0/1107049813 shutdown_connections 2023-10-10T17:49:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c2f8ef0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c2fe5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/1107049813 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c2f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 -- 172.21.15.78:0/1107049813 >> 172.21.15.78:0/1107049813 conn(0x56493c2e9f50 msgr2=0x56493c2ef270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:09.571 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 -- 172.21.15.78:0/1107049813 shutdown_connections 2023-10-10T17:49:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 -- 172.21.15.78:0/1107049813 wait complete. 2023-10-10T17:49:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 -- start start 2023-10-10T17:49:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.555+0000 7f5a6f013b80 1 Processor -- start 2023-10-10T17:49:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c1506b0 0x56493c304ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c302150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.572 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c302890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c1e7880 con 0x56493c2fef60 2023-10-10T17:49:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c1e7520 con 0x56493c2feaf0 2023-10-10T17:49:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c0c98c0 con 0x56493c1506b0 2023-10-10T17:49:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a6f013b80 10 client.0 _ll_register_callbacks cb 0x56493c205720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:09.573 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c302890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c302890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47378/0 (socket says 172.21.15.78:47378) 2023-10-10T17:49:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 -- 172.21.15.78:0/223734202 learned_addr learned my addr 172.21.15.78:0/223734202 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a60c23700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c1506b0 0x56493c304ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.574 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a5bfff700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c302150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 -- 172.21.15.78:0/223734202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c1506b0 msgr2=0x56493c304ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c1506b0 0x56493c304ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 -- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 msgr2=0x56493c302150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c302150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a61424700 1 -- 172.21.15.78:0/223734202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a5c110200 con 0x56493c2fef60 2023-10-10T17:49:09.575 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a5bfff700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c302150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.556+0000 7f5a60c23700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c1506b0 0x56493c304ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a61424700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c302890 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5a540c9f80 tx=0x7f5a54004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68d1af4d99bbaa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a3bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a3bfff700 1 -- 172.21.15.78:0/223734202 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a540b1050 con 0x56493c2fef60 2023-10-10T17:49:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a3bfff700 1 -- 172.21.15.78:0/223734202 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f5a5409d440 con 0x56493c2fef60 2023-10-10T17:49:09.576 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56493c2e7710 con 0x56493c2fef60 2023-10-10T17:49:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a3bfff700 1 -- 172.21.15.78:0/223734202 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a5409f2e0 con 0x56493c2fef60 2023-10-10T17:49:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 10 client.16800 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56493c20c130 con 0x56493c2fef60 2023-10-10T17:49:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a3bfff700 1 -- 172.21.15.78:0/223734202 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56493c2e7710 con 0x56493c2fef60 2023-10-10T17:49:09.577 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a3bfff700 1 -- 172.21.15.78:0/223734202 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5a54098900 con 0x56493c2fef60 2023-10-10T17:49:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 10 client.16800 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 10 client.16800 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 20 client.16800 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c312080 0x56493c314490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 4 client.16800 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:09.578 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56493c208040 con 0x56493c312080 2023-10-10T17:49:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.557+0000 7f5a60c23700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c312080 0x56493c314490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.558+0000 7f5a3bfff700 10 client.16800 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.558+0000 7f5a60c23700 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c312080 0x56493c314490 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48e13d3f53744290 server_cookie=91d0af39efffb5e1 in_seq=0 out_seq=0 2023-10-10T17:49:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a3bfff700 1 -- 172.21.15.78:0/223734202 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56493c208040 con 0x56493c312080 2023-10-10T17:49:09.579 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a3bfff700 10 client.16800 handle_command_reply: tid=0 2023-10-10T17:49:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 client.16800 shutdown 2023-10-10T17:49:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 msgr2=0x56493c302890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c302890 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f5a540c9f80 tx=0x7f5a54004dc0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 shutdown_connections 2023-10-10T17:49:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c1506b0 0x56493c304ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.580 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c312080 0x56493c314490 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c302890 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/223734202 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c302150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a3bfff700 0 client.16800 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 >> 172.21.15.78:0/223734202 conn(0x56493c2e9f50 msgr2=0x56493c2f3550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:09.581 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 shutdown_connections 2023-10-10T17:49:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 1 -- 172.21.15.78:0/223734202 wait complete. 2023-10-10T17:49:09.582 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.559+0000 7f5a6f013b80 20 client.16800 trim_cache size 0 max 16384 2023-10-10T17:49:09.586 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 Processor -- start 2023-10-10T17:49:09.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 -- start start 2023-10-10T17:49:09.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c1506b0 0x56493c309520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c2eef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.587 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2fef60 0x56493c2f8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c2f9510 con 0x56493c1506b0 2023-10-10T17:49:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c2ebfd0 con 0x56493c2feaf0 2023-10-10T17:49:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c2f9720 con 0x56493c2fef60 2023-10-10T17:49:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c1506b0 0x56493c309520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c1506b0 0x56493c309520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47382/0 (socket says 172.21.15.78:47382) 2023-10-10T17:49:09.588 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 -- 172.21.15.78:0/2073158086 learned_addr learned my addr 172.21.15.78:0/2073158086 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a61424700 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2fef60 0x56493c2f8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a5bfff700 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c2eef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 -- 172.21.15.78:0/2073158086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2fef60 msgr2=0x56493c2f8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2fef60 0x56493c2f8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.589 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 -- 172.21.15.78:0/2073158086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 msgr2=0x56493c2eef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c2eef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 -- 172.21.15.78:0/2073158086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56493c20be80 con 0x56493c1506b0 2023-10-10T17:49:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.574+0000 7f5a60c23700 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c1506b0 0x56493c309520 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x56493c314d60 tx=0x7f5a50013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d99c6147ecff6684 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a477fe700 1 -- 172.21.15.78:0/2073158086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5002a070 con 0x56493c1506b0 2023-10-10T17:49:09.590 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a477fe700 1 -- 172.21.15.78:0/2073158086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a50015070 con 0x56493c1506b0 2023-10-10T17:49:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a477fe700 1 -- 172.21.15.78:0/2073158086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a50024070 con 0x56493c1506b0 2023-10-10T17:49:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2073158086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c1506b0 msgr2=0x56493c309520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c1506b0 0x56493c309520 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x56493c314d60 tx=0x7f5a50013000 comp rx=0 tx=0).stop 2023-10-10T17:49:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2073158086 shutdown_connections 2023-10-10T17:49:09.591 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2fef60 0x56493c2f8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c1506b0 0x56493c309520 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2073158086 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c2feaf0 0x56493c2eef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2073158086 >> 172.21.15.78:0/2073158086 conn(0x56493c2e9f50 msgr2=0x56493c30e730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2073158086 shutdown_connections 2023-10-10T17:49:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.575+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2073158086 wait complete. 2023-10-10T17:49:09.592 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 -- start start 2023-10-10T17:49:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 Processor -- start 2023-10-10T17:49:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c3076f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c307c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c35ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.593 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c35f480 con 0x56493c2fef60 2023-10-10T17:49:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c35f5c0 con 0x56493c1506b0 2023-10-10T17:49:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56493c35f780 con 0x56493c2feaf0 2023-10-10T17:49:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a6f013b80 10 client.0 _ll_register_callbacks cb 0x7f5a54098fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c35ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.594 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a5bfff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c307c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c35ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47392/0 (socket says 172.21.15.78:47392) 2023-10-10T17:49:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 -- 172.21.15.78:0/2098553625 learned_addr learned my addr 172.21.15.78:0/2098553625 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a60c23700 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c3076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 -- 172.21.15.78:0/2098553625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 msgr2=0x56493c307c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.595 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c307c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 -- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 msgr2=0x56493c3076f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c3076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.576+0000 7f5a61424700 1 -- 172.21.15.78:0/2098553625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a50010c50 con 0x56493c2fef60 2023-10-10T17:49:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a61424700 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c35ef40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5a54098c60 tx=0x7f5a540992e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bb64e1c4ce996c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:09.596 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a3bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a3bfff700 1 -- 172.21.15.78:0/2098553625 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5409f2e0 con 0x56493c2fef60 2023-10-10T17:49:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a3bfff700 1 -- 172.21.15.78:0/2098553625 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f5a5400a1d0 con 0x56493c2fef60 2023-10-10T17:49:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a54099410 con 0x56493c2fef60 2023-10-10T17:49:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a3bfff700 1 -- 172.21.15.78:0/2098553625 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a54004ad0 con 0x56493c2fef60 2023-10-10T17:49:09.597 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 10 client.16806 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56493c05cc00 con 0x56493c2fef60 2023-10-10T17:49:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a3bfff700 1 -- 172.21.15.78:0/2098553625 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a54099410 con 0x56493c2fef60 2023-10-10T17:49:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a3bfff700 1 -- 172.21.15.78:0/2098553625 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5a540cc3c0 con 0x56493c2fef60 2023-10-10T17:49:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 10 client.16806 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:09.598 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 10 client.16806 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 20 client.16806 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c366b10 0x56493c368f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 4 client.16806 mds_command: new command op to 24500 tid=0[{"prefix": "flush journal"}] 2023-10-10T17:49:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56493c214600 con 0x56493c366b10 2023-10-10T17:49:09.599 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.577+0000 7f5a60c23700 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c366b10 0x56493c368f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.578+0000 7f5a3bfff700 10 client.16806 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:09.600 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.578+0000 7f5a60c23700 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c366b10 0x56493c368f20 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=507b72b56f63621e server_cookie=67b2f3c08c77a9c2 in_seq=0 out_seq=0 2023-10-10T17:49:09.697 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a3bfff700 1 -- 172.21.15.78:0/2098553625 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56493c214600 con 0x56493c366b10 2023-10-10T17:49:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a3bfff700 10 client.16806 handle_command_reply: tid=0 2023-10-10T17:49:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 client.16806 shutdown 2023-10-10T17:49:09.698 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 msgr2=0x56493c35ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c35ef40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f5a54098c60 tx=0x7f5a540992e0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 shutdown_connections 2023-10-10T17:49:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56493c366b10 0x56493c368f20 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56493c2fef60 0x56493c35ef40 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.699 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a3bfff700 0 client.16806 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56493c1506b0 0x56493c3076f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 --2- 172.21.15.78:0/2098553625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56493c2feaf0 0x56493c307c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.696+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 >> 172.21.15.78:0/2098553625 conn(0x56493c2e9f50 msgr2=0x56493c2f8970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 shutdown_connections 2023-10-10T17:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a6f013b80 1 -- 172.21.15.78:0/2098553625 wait complete. 2023-10-10T17:49:09.700 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a6f013b80 20 client.16806 trim_cache size 0 max 16384 2023-10-10T17:49:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:{ 2023-10-10T17:49:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "message": "", 2023-10-10T17:49:09.701 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout: "return_code": 0 2023-10-10T17:49:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stdout:} 2023-10-10T17:49:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a4c0a12c0 msgr2=0x7f5a4c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a4c0a12c0 0x7f5a4c0a3770 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5a50005950 tx=0x7f5a5000b410 comp rx=0 tx=0).stop 2023-10-10T17:49:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c104400 msgr2=0x7f5a5c19ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c104400 0x7f5a5c19ada0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f5a5400ada0 tx=0x7f5a5400c330 comp rx=0 tx=0).stop 2023-10-10T17:49:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 shutdown_connections 2023-10-10T17:49:09.703 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5a5c0ff3c0 0x7f5a5c196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5a4c0a12c0 0x7f5a4c0a3770 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5a5c104400 0x7f5a5c19ada0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 --2- 172.21.15.78:0/383715081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5a5c0ffd60 0x7f5a5c196940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 >> 172.21.15.78:0/383715081 conn(0x7f5a5c0763a0 msgr2=0x7f5a5c0fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 shutdown_connections 2023-10-10T17:49:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi078.front.sepia.ceph.com.smithi078.stderr:2023-10-10T17:49:09.697+0000 7f5a62e87700 1 -- 172.21.15.78:0/383715081 wait complete. 2023-10-10T17:49:11.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:10 smithi078 ceph-mon[99494]: pgmap v203: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 166 KiB/s wr, 54 op/s 2023-10-10T17:49:11.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:10 smithi190 ceph-mon[101220]: pgmap v203: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 166 KiB/s wr, 54 op/s 2023-10-10T17:49:11.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:10 smithi157 ceph-mon[100896]: pgmap v203: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 166 KiB/s wr, 54 op/s 2023-10-10T17:49:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:12 smithi078 ceph-mon[99494]: pgmap v204: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 142 KiB/s wr, 53 op/s 2023-10-10T17:49:13.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:12 smithi190 ceph-mon[101220]: pgmap v204: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 142 KiB/s wr, 53 op/s 2023-10-10T17:49:13.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:12 smithi157 ceph-mon[100896]: pgmap v204: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 142 KiB/s wr, 53 op/s 2023-10-10T17:49:15.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:14 smithi078 ceph-mon[99494]: pgmap v205: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 114 op/s 2023-10-10T17:49:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:14 smithi190 ceph-mon[101220]: pgmap v205: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 114 op/s 2023-10-10T17:49:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:14 smithi157 ceph-mon[100896]: pgmap v205: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 114 op/s 2023-10-10T17:49:17.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:16 smithi078 ceph-mon[99494]: pgmap v206: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 114 op/s 2023-10-10T17:49:17.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:16 smithi190 ceph-mon[101220]: pgmap v206: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 114 op/s 2023-10-10T17:49:17.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:16 smithi157 ceph-mon[100896]: pgmap v206: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 114 op/s 2023-10-10T17:49:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:18 smithi078 ceph-mon[99494]: pgmap v207: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 107 op/s 2023-10-10T17:49:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:18 smithi078 ceph-mon[99494]: osdmap e96: 12 total, 12 up, 12 in 2023-10-10T17:49:19.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:18 smithi190 ceph-mon[101220]: pgmap v207: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 107 op/s 2023-10-10T17:49:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:18 smithi190 ceph-mon[101220]: osdmap e96: 12 total, 12 up, 12 in 2023-10-10T17:49:19.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:18 smithi157 ceph-mon[100896]: pgmap v207: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 107 op/s 2023-10-10T17:49:19.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:18 smithi157 ceph-mon[100896]: osdmap e96: 12 total, 12 up, 12 in 2023-10-10T17:49:19.729 INFO:tasks.workunit.client.0.smithi078.stdout:ok 2023-10-10T17:49:19.733 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-10T17:49:19.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-10T17:49:19.780 INFO:tasks.workunit:Running workunit fs/misc/general_vxattrs.sh... 2023-10-10T17:49:19.780 DEBUG:teuthology.orchestra.run.smithi078:workunit test fs/misc/general_vxattrs.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh 2023-10-10T17:49:19.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:49:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3907952076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb684104490 msgr2=0x7fb68410c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3907952076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb684104490 0x7fb68410c060 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fb67c00b0d0 tx=0x7fb67c00b3e0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3907952076 shutdown_connections 2023-10-10T17:49:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3907952076 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6840ff450 0x7fb6840ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3907952076 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb684104490 0x7fb68410c060 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3907952076 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb6840ffdf0 0x7fb684103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3907952076 >> 172.21.15.78:0/3907952076 conn(0x7fb6840770b0 msgr2=0x7fb6840774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3907952076 shutdown_connections 2023-10-10T17:49:20.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.091+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3907952076 wait complete. 2023-10-10T17:49:20.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 Processor -- start 2023-10-10T17:49:20.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 -- start start 2023-10-10T17:49:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6840ff450 0x7fb6841973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6840ffdf0 0x7fb6841978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb684104490 0x7fb6841a2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb684198200 con 0x7fb6840ffdf0 2023-10-10T17:49:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6841a2970 con 0x7fb684104490 2023-10-10T17:49:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68bf6d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6841a2ae0 con 0x7fb6840ff450 2023-10-10T17:49:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6840ffdf0 0x7fb6841978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6840ffdf0 0x7fb6841978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47402/0 (socket says 172.21.15.78:47402) 2023-10-10T17:49:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 -- 172.21.15.78:0/3174330904 learned_addr learned my addr 172.21.15.78:0/3174330904 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb68a50a700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb684104490 0x7fb6841a2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689d09700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6840ff450 0x7fb6841973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 -- 172.21.15.78:0/3174330904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6840ff450 msgr2=0x7fb6841973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6840ff450 0x7fb6841973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 -- 172.21.15.78:0/3174330904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb684104490 msgr2=0x7fb6841a2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb684104490 0x7fb6841a2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 -- 172.21.15.78:0/3174330904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c009d00 con 0x7fb6840ffdf0 2023-10-10T17:49:20.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689d09700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6840ff450 0x7fb6841973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:20.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.092+0000 7fb689508700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6840ffdf0 0x7fb6841978f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb67400d950 tx=0x7fb67400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.093+0000 7fb67affd700 1 -- 172.21.15.78:0/3174330904 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674004220 con 0x7fb6840ffdf0 2023-10-10T17:49:20.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.093+0000 7fb67affd700 1 -- 172.21.15.78:0/3174330904 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb674004cf0 con 0x7fb6840ffdf0 2023-10-10T17:49:20.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.093+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68410f2b0 con 0x7fb6840ffdf0 2023-10-10T17:49:20.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.093+0000 7fb67affd700 1 -- 172.21.15.78:0/3174330904 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740056c0 con 0x7fb6840ffdf0 2023-10-10T17:49:20.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.093+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68410f120 con 0x7fb6840ffdf0 2023-10-10T17:49:20.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.095+0000 7fb67affd700 1 -- 172.21.15.78:0/3174330904 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fb674004380 con 0x7fb6840ffdf0 2023-10-10T17:49:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.095+0000 7fb67affd700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb6700a12c0 0x7fb6700a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.096+0000 7fb67affd700 1 -- 172.21.15.78:0/3174330904 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb6740cce50 con 0x7fb6840ffdf0 2023-10-10T17:49:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.096+0000 7fb689d09700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb6700a12c0 0x7fb6700a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.096+0000 7fb689d09700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb6700a12c0 0x7fb6700a3770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb680009cb0 tx=0x7fb680009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 Processor -- start 2023-10-10T17:49:20.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 -- start start 2023-10-10T17:49:20.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fbf70 0x56194f6fc340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fc880 0x56194f700780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f701110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f5fd060 con 0x56194f6fbf70 2023-10-10T17:49:20.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f64f8a0 con 0x56194f700cc0 2023-10-10T17:49:20.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f648d30 con 0x56194f6fc880 2023-10-10T17:49:20.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689d09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fbf70 0x56194f6fc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689d09700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fbf70 0x56194f6fc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47408/0 (socket says 172.21.15.78:47408) 2023-10-10T17:49:20.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689d09700 1 -- 172.21.15.78:0/2392985081 learned_addr learned my addr 172.21.15.78:0/2392985081 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689508700 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fc880 0x56194f700780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb68a50a700 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f701110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689d09700 1 -- 172.21.15.78:0/2392985081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fc880 msgr2=0x56194f700780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689d09700 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fc880 0x56194f700780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689d09700 1 -- 172.21.15.78:0/2392985081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 msgr2=0x56194f701110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.097+0000 7fb689d09700 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f701110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.098+0000 7fb689d09700 1 -- 172.21.15.78:0/2392985081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56194f4ca630 con 0x56194f6fbf70 2023-10-10T17:49:20.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.098+0000 7fb689d09700 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fbf70 0x56194f6fc340 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb680011fb0 tx=0x7fb680014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb188375205ef42a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.098+0000 7fb66ffff700 1 -- 172.21.15.78:0/2392985081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680003680 con 0x56194f6fbf70 2023-10-10T17:49:20.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.098+0000 7fb66ffff700 1 -- 172.21.15.78:0/2392985081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb680003ca0 con 0x56194f6fbf70 2023-10-10T17:49:20.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.098+0000 7fb66ffff700 1 -- 172.21.15.78:0/2392985081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680012ce0 con 0x56194f6fbf70 2023-10-10T17:49:20.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2392985081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fbf70 msgr2=0x56194f6fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fbf70 0x56194f6fc340 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fb680011fb0 tx=0x7fb680014000 comp rx=0 tx=0).stop 2023-10-10T17:49:20.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2392985081 shutdown_connections 2023-10-10T17:49:20.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f701110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fbf70 0x56194f6fc340 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2392985081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fc880 0x56194f700780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2392985081 >> 172.21.15.78:0/2392985081 conn(0x56194f6ed360 msgr2=0x56194f6f2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2392985081 shutdown_connections 2023-10-10T17:49:20.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2392985081 wait complete. 2023-10-10T17:49:20.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 -- start start 2023-10-10T17:49:20.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.099+0000 7fb6980f9b80 1 Processor -- start 2023-10-10T17:49:20.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f708480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f705770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f705e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f5ea8e0 con 0x56194f6fc880 2023-10-10T17:49:20.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f5ea580 con 0x56194f700cc0 2023-10-10T17:49:20.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f4cc920 con 0x56194f6fbf70 2023-10-10T17:49:20.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 10 client.0 _ll_register_callbacks cb 0x56194f608780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:20.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f705770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f705770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47416/0 (socket says 172.21.15.78:47416) 2023-10-10T17:49:20.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 -- 172.21.15.78:0/1199236860 learned_addr learned my addr 172.21.15.78:0/1199236860 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689d09700 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f708480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb68a50a700 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f705e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 -- 172.21.15.78:0/1199236860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 msgr2=0x56194f708480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f708480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 -- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 msgr2=0x56194f705e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f705e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 -- 172.21.15.78:0/1199236860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb680011c60 con 0x56194f6fc880 2023-10-10T17:49:20.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb689508700 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f705770 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb67400b540 tx=0x7fb6740995e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a84588563a5dba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb66d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:20.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb66d7fa700 1 -- 172.21.15.78:0/1199236860 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740b1070 con 0x56194f6fc880 2023-10-10T17:49:20.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb66d7fa700 1 -- 172.21.15.78:0/1199236860 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb6740a1fe0 con 0x56194f6fc880 2023-10-10T17:49:20.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.100+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56194f6eab20 con 0x56194f6fc880 2023-10-10T17:49:20.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb66d7fa700 1 -- 172.21.15.78:0/1199236860 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6740b2260 con 0x56194f6fc880 2023-10-10T17:49:20.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 10 client.16818 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:20.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56194f60f190 con 0x56194f6fc880 2023-10-10T17:49:20.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb66d7fa700 1 -- 172.21.15.78:0/1199236860 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6740c9080 con 0x56194f6fc880 2023-10-10T17:49:20.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb66d7fa700 1 -- 172.21.15.78:0/1199236860 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb6740a1b50 con 0x56194f6fc880 2023-10-10T17:49:20.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 10 client.16818 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:20.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 10 client.16818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:20.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 20 client.16818 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:20.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f715cb0 0x56194f7180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 4 client.16818 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:20.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56194f60b0a0 con 0x56194f715cb0 2023-10-10T17:49:20.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.101+0000 7fb689d09700 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f715cb0 0x56194f7180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.102+0000 7fb66d7fa700 10 client.16818 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:20.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.102+0000 7fb689d09700 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f715cb0 0x56194f7180c0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8afa452b3dbf5987 server_cookie=d137a8f7f59126c8 in_seq=0 out_seq=0 2023-10-10T17:49:20.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.102+0000 7fb66d7fa700 1 -- 172.21.15.78:0/1199236860 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56194f60b0a0 con 0x56194f715cb0 2023-10-10T17:49:20.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.102+0000 7fb66d7fa700 10 client.16818 handle_command_reply: tid=0 2023-10-10T17:49:20.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.102+0000 7fb6980f9b80 1 client.16818 shutdown 2023-10-10T17:49:20.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 msgr2=0x56194f705770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f705770 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb67400b540 tx=0x7fb6740995e0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 shutdown_connections 2023-10-10T17:49:20.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f708480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f715cb0 0x56194f7180c0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f705e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/1199236860 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f705770 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb66d7fa700 0 client.16818 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:20.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 >> 172.21.15.78:0/1199236860 conn(0x56194f6ed360 msgr2=0x56194f702170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 shutdown_connections 2023-10-10T17:49:20.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 1 -- 172.21.15.78:0/1199236860 wait complete. 2023-10-10T17:49:20.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.103+0000 7fb6980f9b80 20 client.16818 trim_cache size 0 max 16384 2023-10-10T17:49:20.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 Processor -- start 2023-10-10T17:49:20.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 -- start start 2023-10-10T17:49:20.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f6fe5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f70cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f704f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f708ba0 con 0x56194f6fc880 2023-10-10T17:49:20.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f6f38b0 con 0x56194f700cc0 2023-10-10T17:49:20.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f6ea740 con 0x56194f6fbf70 2023-10-10T17:49:20.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f70cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f70cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47424/0 (socket says 172.21.15.78:47424) 2023-10-10T17:49:20.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 -- 172.21.15.78:0/355603477 learned_addr learned my addr 172.21.15.78:0/355603477 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb68a50a700 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f704f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689d09700 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f6fe5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 -- 172.21.15.78:0/355603477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 msgr2=0x56194f6fe5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f6fe5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 -- 172.21.15.78:0/355603477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 msgr2=0x56194f704f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f704f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 -- 172.21.15.78:0/355603477 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56194f60eee0 con 0x56194f6fc880 2023-10-10T17:49:20.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb68a50a700 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f704f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:20.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.120+0000 7fb689508700 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f70cde0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb6740ca810 tx=0x7fb674098140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34f4f316a770485 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb678ff9700 1 -- 172.21.15.78:0/355603477 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740b1030 con 0x56194f6fc880 2023-10-10T17:49:20.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb678ff9700 1 -- 172.21.15.78:0/355603477 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56194f6eab20 con 0x56194f6fc880 2023-10-10T17:49:20.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb678ff9700 1 -- 172.21.15.78:0/355603477 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740992e0 con 0x56194f6fc880 2023-10-10T17:49:20.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 -- 172.21.15.78:0/355603477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 msgr2=0x56194f70cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f70cde0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fb6740ca810 tx=0x7fb674098140 comp rx=0 tx=0).stop 2023-10-10T17:49:20.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 -- 172.21.15.78:0/355603477 shutdown_connections 2023-10-10T17:49:20.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f6fbf70 0x56194f6fe5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f700cc0 0x56194f704f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/355603477 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f70cde0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 -- 172.21.15.78:0/355603477 >> 172.21.15.78:0/355603477 conn(0x56194f6ed360 msgr2=0x56194f6fee80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 -- 172.21.15.78:0/355603477 shutdown_connections 2023-10-10T17:49:20.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.121+0000 7fb6980f9b80 1 -- 172.21.15.78:0/355603477 wait complete. 2023-10-10T17:49:20.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 -- start start 2023-10-10T17:49:20.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 Processor -- start 2023-10-10T17:49:20.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f6fbf70 0x56194f6f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f6f1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f700cc0 0x56194f6f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f6f2fc0 con 0x56194f6fc880 2023-10-10T17:49:20.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f6f1fd0 con 0x56194f6fbf70 2023-10-10T17:49:20.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56194f6f2190 con 0x56194f700cc0 2023-10-10T17:49:20.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb6980f9b80 10 client.0 _ll_register_callbacks cb 0x7fb674099a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:20.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f6f1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb689508700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f6f1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47430/0 (socket says 172.21.15.78:47430) 2023-10-10T17:49:20.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb689508700 1 -- 172.21.15.78:0/2649532218 learned_addr learned my addr 172.21.15.78:0/2649532218 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb689d09700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f6fbf70 0x56194f6f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.122+0000 7fb68a50a700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f700cc0 0x56194f6f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb689508700 1 -- 172.21.15.78:0/2649532218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f700cc0 msgr2=0x56194f6f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb689508700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f700cc0 0x56194f6f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb689508700 1 -- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f6fbf70 msgr2=0x56194f6f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb689508700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f6fbf70 0x56194f6f29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb689508700 1 -- 172.21.15.78:0/2649532218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56194f60f190 con 0x56194f6fc880 2023-10-10T17:49:20.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb68a50a700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f700cc0 0x56194f6f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:20.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb689508700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f6f1550 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb674099890 tx=0x7fb67400df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fb19e204d4d9f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb66d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb66d7fa700 1 -- 172.21.15.78:0/2649532218 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6740b1050 con 0x56194f6fc880 2023-10-10T17:49:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb66d7fa700 1 -- 172.21.15.78:0/2649532218 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb6740b2070 con 0x56194f6fc880 2023-10-10T17:49:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb680003ca0 con 0x56194f6fc880 2023-10-10T17:49:20.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb66d7fa700 1 -- 172.21.15.78:0/2649532218 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6740ca0f0 con 0x56194f6fc880 2023-10-10T17:49:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 10 client.16824 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56194f45fc60 con 0x56194f6fc880 2023-10-10T17:49:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb66d7fa700 1 -- 172.21.15.78:0/2649532218 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb680003ca0 con 0x56194f6fc880 2023-10-10T17:49:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb66d7fa700 1 -- 172.21.15.78:0/2649532218 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb674095980 con 0x56194f6fc880 2023-10-10T17:49:20.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 10 client.16824 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:20.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 10 client.16824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:20.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 20 client.16824 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:20.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f76a8d0 0x56194f76ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 4 client.16824 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:49:20.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.123+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56194f617660 con 0x56194f76a8d0 2023-10-10T17:49:20.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb689d09700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f76a8d0 0x56194f76ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb66d7fa700 10 client.16824 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:20.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb689d09700 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f76a8d0 0x56194f76ccb0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f4d5b8822585c8 server_cookie=8a9314c9522b026 in_seq=0 out_seq=0 2023-10-10T17:49:20.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb66d7fa700 1 -- 172.21.15.78:0/2649532218 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56194f617660 con 0x56194f76a8d0 2023-10-10T17:49:20.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb66d7fa700 10 client.16824 handle_command_reply: tid=0 2023-10-10T17:49:20.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb6980f9b80 1 client.16824 shutdown 2023-10-10T17:49:20.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 msgr2=0x56194f6f1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.124+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f6f1550 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb674099890 tx=0x7fb67400df90 comp rx=0 tx=0).stop 2023-10-10T17:49:20.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 shutdown_connections 2023-10-10T17:49:20.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56194f700cc0 0x56194f6f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56194f76a8d0 0x56194f76ccb0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56194f6fbf70 0x56194f6f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 --2- 172.21.15.78:0/2649532218 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56194f6fc880 0x56194f6f1550 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb66d7fa700 0 client.16824 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:20.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 >> 172.21.15.78:0/2649532218 conn(0x56194f6ed360 msgr2=0x56194f6ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 shutdown_connections 2023-10-10T17:49:20.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 1 -- 172.21.15.78:0/2649532218 wait complete. 2023-10-10T17:49:20.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb6980f9b80 20 client.16824 trim_cache size 0 max 16384 2023-10-10T17:49:20.149 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:49:20.149 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:49:20.149 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:49:20.149 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:49:20.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb6700a12c0 msgr2=0x7fb6700a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb6700a12c0 0x7fb6700a3770 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb680009cb0 tx=0x7fb680009260 comp rx=0 tx=0).stop 2023-10-10T17:49:20.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6840ffdf0 msgr2=0x7fb6841978f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6840ffdf0 0x7fb6841978f0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb67400d950 tx=0x7fb67400dc60 comp rx=0 tx=0).stop 2023-10-10T17:49:20.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 shutdown_connections 2023-10-10T17:49:20.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb6840ff450 0x7fb6841973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb6700a12c0 0x7fb6700a3770 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb684104490 0x7fb6841a2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 --2- 172.21.15.78:0/3174330904 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb6840ffdf0 0x7fb6841978f0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 >> 172.21.15.78:0/3174330904 conn(0x7fb6840770b0 msgr2=0x7fb6840fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 shutdown_connections 2023-10-10T17:49:20.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.125+0000 7fb68bf6d700 1 -- 172.21.15.78:0/3174330904 wait complete. 2023-10-10T17:49:20.154 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:49:20.154 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:49:20.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.274+0000 7f5427504700 1 -- 172.21.15.78:0/387580541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 msgr2=0x7f5420104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.274+0000 7f5427504700 1 --2- 172.21.15.78:0/387580541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f5420104ec0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f54100099c0 tx=0x7f5410009cd0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.274+0000 7f5427504700 1 -- 172.21.15.78:0/387580541 shutdown_connections 2023-10-10T17:49:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.274+0000 7f5427504700 1 --2- 172.21.15.78:0/387580541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54201055f0 0x7f542010d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.274+0000 7f5427504700 1 --2- 172.21.15.78:0/387580541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54200fea50 0x7f54200fee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.274+0000 7f5427504700 1 --2- 172.21.15.78:0/387580541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f5420104ec0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.274+0000 7f5427504700 1 -- 172.21.15.78:0/387580541 >> 172.21.15.78:0/387580541 conn(0x7f5420075820 msgr2=0x7f5420075c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 -- 172.21.15.78:0/387580541 shutdown_connections 2023-10-10T17:49:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 -- 172.21.15.78:0/387580541 wait complete. 2023-10-10T17:49:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 Processor -- start 2023-10-10T17:49:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 -- start start 2023-10-10T17:49:20.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54200fea50 0x7f54200ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f54200fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54201055f0 0x7f54201a4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5420100730 con 0x7f54200ff3f0 2023-10-10T17:49:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54201a4a80 con 0x7f54201055f0 2023-10-10T17:49:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5427504700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54201a4bc0 con 0x7f54200fea50 2023-10-10T17:49:20.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5424a9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f54200fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5424a9f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f54200fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47444/0 (socket says 172.21.15.78:47444) 2023-10-10T17:49:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.275+0000 7f5424a9f700 1 -- 172.21.15.78:0/3776097421 learned_addr learned my addr 172.21.15.78:0/3776097421 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f54252a0700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54200fea50 0x7f54200ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5425aa1700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54201055f0 0x7f54201a4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5424a9f700 1 -- 172.21.15.78:0/3776097421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54200fea50 msgr2=0x7f54200ff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5424a9f700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54200fea50 0x7f54200ff9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5424a9f700 1 -- 172.21.15.78:0/3776097421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54201055f0 msgr2=0x7f54201a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5424a9f700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54201055f0 0x7f54201a4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5424a9f700 1 -- 172.21.15.78:0/3776097421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54100095f0 con 0x7f54200ff3f0 2023-10-10T17:49:20.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5424a9f700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f54200fff00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5410009990 tx=0x7f5410004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f54167fc700 1 -- 172.21.15.78:0/3776097421 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541001d070 con 0x7f54200ff3f0 2023-10-10T17:49:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f54167fc700 1 -- 172.21.15.78:0/3776097421 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5410003f90 con 0x7f54200ff3f0 2023-10-10T17:49:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54201102f0 con 0x7f54200ff3f0 2023-10-10T17:49:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f54167fc700 1 -- 172.21.15.78:0/3776097421 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5410022c00 con 0x7f54200ff3f0 2023-10-10T17:49:20.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.276+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5420110160 con 0x7f54200ff3f0 2023-10-10T17:49:20.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.277+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5420110480 con 0x7f54200ff3f0 2023-10-10T17:49:20.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.278+0000 7f54167fc700 1 -- 172.21.15.78:0/3776097421 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5410003ad0 con 0x7f54200ff3f0 2023-10-10T17:49:20.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.279+0000 7f54167fc700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f540c0a12c0 0x7f540c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.279+0000 7f54252a0700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f540c0a12c0 0x7f540c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.279+0000 7f54167fc700 1 -- 172.21.15.78:0/3776097421 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f54100ce6f0 con 0x7f54200ff3f0 2023-10-10T17:49:20.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.279+0000 7f54252a0700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f540c0a12c0 0x7f540c0a3770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f541c00ac50 tx=0x7f541c00a380 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.282+0000 7f54167fc700 1 -- 172.21.15.78:0/3776097421 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f54100977a0 con 0x7f54200ff3f0 2023-10-10T17:49:20.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.510+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54200671d0 con 0x7f54200ff3f0 2023-10-10T17:49:20.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.511+0000 7f54167fc700 1 -- 172.21.15.78:0/3776097421 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10817 (secure 0 0 0) 0x7f541009adc0 con 0x7f54200ff3f0 2023-10-10T17:49:20.512 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:49:20.512 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:46:23.648656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:49:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f540c0a12c0 msgr2=0x7f540c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f540c0a12c0 0x7f540c0a3770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f541c00ac50 tx=0x7f541c00a380 comp rx=0 tx=0).stop 2023-10-10T17:49:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 msgr2=0x7f54200fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f54200fff00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f5410009990 tx=0x7f5410004730 comp rx=0 tx=0).stop 2023-10-10T17:49:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 shutdown_connections 2023-10-10T17:49:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54200fea50 0x7f54200ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f540c0a12c0 0x7f540c0a3770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54201055f0 0x7f54201a4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 --2- 172.21.15.78:0/3776097421 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54200ff3f0 0x7f54200fff00 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.513+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 >> 172.21.15.78:0/3776097421 conn(0x7f5420075820 msgr2=0x7f54200fc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.514+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 shutdown_connections 2023-10-10T17:49:20.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.514+0000 7f5427504700 1 -- 172.21.15.78:0/3776097421 wait complete. 2023-10-10T17:49:20.517 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2023-10-10T17:49:20.526 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:49:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 -- 172.21.15.78:0/897291622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 msgr2=0x7fa5a4101760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 --2- 172.21.15.78:0/897291622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a4101760 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa598009990 tx=0x7fa598009ca0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 -- 172.21.15.78:0/897291622 shutdown_connections 2023-10-10T17:49:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 --2- 172.21.15.78:0/897291622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5a410a8b0 0x7fa5a410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 --2- 172.21.15.78:0/897291622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a4101760 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 --2- 172.21.15.78:0/897291622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a4101ca0 0x7fa5a410a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 -- 172.21.15.78:0/897291622 >> 172.21.15.78:0/897291622 conn(0x7fa5a4077080 msgr2=0x7fa5a4077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 -- 172.21.15.78:0/897291622 shutdown_connections 2023-10-10T17:49:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 -- 172.21.15.78:0/897291622 wait complete. 2023-10-10T17:49:20.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.648+0000 7fa5a9283700 1 Processor -- start 2023-10-10T17:49:20.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a9283700 1 -- start start 2023-10-10T17:49:20.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a9283700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a41964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a9283700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a4101ca0 0x7fa5a4196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a9283700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5a410a8b0 0x7fa5a419ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a9283700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a419b4c0 con 0x7fa5a4101390 2023-10-10T17:49:20.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a9283700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a419b630 con 0x7fa5a410a8b0 2023-10-10T17:49:20.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a9283700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a419b7f0 con 0x7fa5a4101ca0 2023-10-10T17:49:20.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a41964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a41964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47454/0 (socket says 172.21.15.78:47454) 2023-10-10T17:49:20.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1214916236 learned_addr learned my addr 172.21.15.78:0/1214916236 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a4101ca0 0x7fa5a4196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5a410a8b0 0x7fa5a419ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1214916236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a4101ca0 msgr2=0x7fa5a4196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a4101ca0 0x7fa5a4196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1214916236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5a410a8b0 msgr2=0x7fa5a419ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5a410a8b0 0x7fa5a419ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1214916236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5980095f0 con 0x7fa5a4101390 2023-10-10T17:49:20.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a4101ca0 0x7fa5a4196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:20.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.649+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a41964e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa59800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.650+0000 7fa58bfff700 1 -- 172.21.15.78:0/1214916236 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59801d070 con 0x7fa5a4101390 2023-10-10T17:49:20.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.650+0000 7fa58bfff700 1 -- 172.21.15.78:0/1214916236 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa59800fc20 con 0x7fa5a4101390 2023-10-10T17:49:20.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.650+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a4110330 con 0x7fa5a4101390 2023-10-10T17:49:20.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.650+0000 7fa58bfff700 1 -- 172.21.15.78:0/1214916236 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598021ea0 con 0x7fa5a4101390 2023-10-10T17:49:20.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.650+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a41101a0 con 0x7fa5a4101390 2023-10-10T17:49:20.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.652+0000 7fa58bfff700 1 -- 172.21.15.78:0/1214916236 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa598021410 con 0x7fa5a4101390 2023-10-10T17:49:20.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.652+0000 7fa58bfff700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa58c0a12c0 0x7fa58c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.652+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa58c0a12c0 0x7fa58c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.652+0000 7fa58bfff700 1 -- 172.21.15.78:0/1214916236 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa5980ce480 con 0x7fa5a4101390 2023-10-10T17:49:20.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa58c0a12c0 0x7fa58c0a3770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4197af0 tx=0x7fa594009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 Processor -- start 2023-10-10T17:49:20.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 -- start start 2023-10-10T17:49:20.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de1b60 0x5630a5de1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de2470 0x5630a5de6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5de6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5ce7060 con 0x5630a5de1b60 2023-10-10T17:49:20.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5d398a0 con 0x5630a5de68b0 2023-10-10T17:49:20.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5d32d30 con 0x5630a5de2470 2023-10-10T17:49:20.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de1b60 0x5630a5de1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5a2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de1b60 0x5630a5de1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47462/0 (socket says 172.21.15.78:47462) 2023-10-10T17:49:20.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.653+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1964095493 learned_addr learned my addr 172.21.15.78:0/1964095493 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de2470 0x5630a5de6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1964095493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de2470 msgr2=0x5630a5de6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5de6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de2470 0x5630a5de6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1964095493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 msgr2=0x5630a5de6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5de6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a2ffd700 1 -- 172.21.15.78:0/1964095493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630a5bb4630 con 0x5630a5de1b60 2023-10-10T17:49:20.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de1b60 0x5630a5de1f30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa598006010 tx=0x7fa598009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=725367f23a4ee11d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5897fa700 1 -- 172.21.15.78:0/1964095493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5980a3730 con 0x5630a5de1b60 2023-10-10T17:49:20.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5897fa700 1 -- 172.21.15.78:0/1964095493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa59800fd90 con 0x5630a5de1b60 2023-10-10T17:49:20.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.654+0000 7fa5897fa700 1 -- 172.21.15.78:0/1964095493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59809b5d0 con 0x5630a5de1b60 2023-10-10T17:49:20.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- 172.21.15.78:0/1964095493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de1b60 msgr2=0x5630a5de1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de1b60 0x5630a5de1f30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa598006010 tx=0x7fa598009ca0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- 172.21.15.78:0/1964095493 shutdown_connections 2023-10-10T17:49:20.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5de6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de1b60 0x5630a5de1f30 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/1964095493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de2470 0x5630a5de6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- 172.21.15.78:0/1964095493 >> 172.21.15.78:0/1964095493 conn(0x5630a5dd7360 msgr2=0x5630a5ddcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- 172.21.15.78:0/1964095493 shutdown_connections 2023-10-10T17:49:20.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- 172.21.15.78:0/1964095493 wait complete. 2023-10-10T17:49:20.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- start start 2023-10-10T17:49:20.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 Processor -- start 2023-10-10T17:49:20.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dedfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de2470 0x5630a5deb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de68b0 0x5630a5deb730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5cd48e0 con 0x5630a5de68b0 2023-10-10T17:49:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5cd4580 con 0x5630a5de2470 2023-10-10T17:49:20.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5bb6920 con 0x5630a5de1b60 2023-10-10T17:49:20.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.655+0000 7fa5b540fb80 10 client.0 _ll_register_callbacks cb 0x5630a5cf2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:20.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de68b0 0x5630a5deb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de68b0 0x5630a5deb730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47472/0 (socket says 172.21.15.78:47472) 2023-10-10T17:49:20.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 -- 172.21.15.78:0/2741146264 learned_addr learned my addr 172.21.15.78:0/2741146264 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dedfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de2470 0x5630a5deb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 -- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 msgr2=0x5630a5dedfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dedfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 -- 172.21.15.78:0/2741146264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de2470 msgr2=0x5630a5deb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de2470 0x5630a5deb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 -- 172.21.15.78:0/2741146264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a41101a0 con 0x5630a5de68b0 2023-10-10T17:49:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de68b0 0x5630a5deb730 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa59c008b00 tx=0x7fa59c008e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3536d26fef88de42 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa586ff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa586ff5700 1 -- 172.21.15.78:0/2741146264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c019070 con 0x5630a5de68b0 2023-10-10T17:49:20.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa586ff5700 1 -- 172.21.15.78:0/2741146264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa59c003a00 con 0x5630a5de68b0 2023-10-10T17:49:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630a5dd4b20 con 0x5630a5de68b0 2023-10-10T17:49:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa586ff5700 1 -- 172.21.15.78:0/2741146264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa59c007a30 con 0x5630a5de68b0 2023-10-10T17:49:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5b540fb80 10 client.16842 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5630a5cf9190 con 0x5630a5de68b0 2023-10-10T17:49:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa586ff5700 1 -- 172.21.15.78:0/2741146264 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5630a5dd4b20 con 0x5630a5de68b0 2023-10-10T17:49:20.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.656+0000 7fa586ff5700 1 -- 172.21.15.78:0/2741146264 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa59c01c030 con 0x5630a5de68b0 2023-10-10T17:49:20.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5b540fb80 10 client.16842 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:20.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5b540fb80 10 client.16842 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:49:20.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5b540fb80 20 client.16842 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:20.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5dfb640 0x5630a5dfda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5b540fb80 4 client.16842 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:20.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630a5cf50a0 con 0x5630a5dfb640 2023-10-10T17:49:20.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5dfb640 0x5630a5dfda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa586ff5700 10 client.16842 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:20.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.657+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5dfb640 0x5630a5dfda50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ac763771d88e1b0c server_cookie=25552e49614f65b1 in_seq=0 out_seq=0 2023-10-10T17:49:20.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa586ff5700 1 -- 172.21.15.78:0/2741146264 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5630a5cf50a0 con 0x5630a5dfb640 2023-10-10T17:49:20.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa586ff5700 10 client.16842 handle_command_reply: tid=0 2023-10-10T17:49:20.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 client.16842 shutdown 2023-10-10T17:49:20.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de68b0 msgr2=0x5630a5deb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de68b0 0x5630a5deb730 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa59c008b00 tx=0x7fa59c008e10 comp rx=0 tx=0).stop 2023-10-10T17:49:20.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 shutdown_connections 2023-10-10T17:49:20.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5dfb640 0x5630a5dfda50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dedfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de68b0 0x5630a5deb730 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/2741146264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de2470 0x5630a5deb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa586ff5700 0 client.16842 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:20.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 >> 172.21.15.78:0/2741146264 conn(0x5630a5dd7360 msgr2=0x5630a5de7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 shutdown_connections 2023-10-10T17:49:20.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 1 -- 172.21.15.78:0/2741146264 wait complete. 2023-10-10T17:49:20.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.658+0000 7fa5b540fb80 20 client.16842 trim_cache size 0 max 16384 2023-10-10T17:49:20.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 Processor -- start 2023-10-10T17:49:20.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 -- start start 2023-10-10T17:49:20.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5df2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5dea970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5df6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5debd10 con 0x5630a5de2470 2023-10-10T17:49:20.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5de8340 con 0x5630a5de68b0 2023-10-10T17:49:20.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5df8bf0 con 0x5630a5de1b60 2023-10-10T17:49:20.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.671+0000 7fa5a27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5dea970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5dea970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47478/0 (socket says 172.21.15.78:47478) 2023-10-10T17:49:20.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 -- 172.21.15.78:0/3659729515 learned_addr learned my addr 172.21.15.78:0/3659729515 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5df2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5df6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 -- 172.21.15.78:0/3659729515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 msgr2=0x5630a5df2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5df2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 -- 172.21.15.78:0/3659729515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 msgr2=0x5630a5df6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5df6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 -- 172.21.15.78:0/3659729515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630a5cf8ee0 con 0x5630a5de2470 2023-10-10T17:49:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5df6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:20.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5dea970 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa594012010 tx=0x7fa594014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46a718026d97874b server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa589ffb700 1 -- 172.21.15.78:0/3659729515 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594003880 con 0x5630a5de2470 2023-10-10T17:49:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa589ffb700 1 -- 172.21.15.78:0/3659729515 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa594012450 con 0x5630a5de2470 2023-10-10T17:49:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.672+0000 7fa589ffb700 1 -- 172.21.15.78:0/3659729515 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594003ea0 con 0x5630a5de2470 2023-10-10T17:49:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- 172.21.15.78:0/3659729515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 msgr2=0x5630a5dea970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5dea970 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa594012010 tx=0x7fa594014000 comp rx=0 tx=0).stop 2023-10-10T17:49:20.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- 172.21.15.78:0/3659729515 shutdown_connections 2023-10-10T17:49:20.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5df2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5df6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/3659729515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5dea970 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- 172.21.15.78:0/3659729515 >> 172.21.15.78:0/3659729515 conn(0x5630a5dd7360 msgr2=0x5630a5ddb550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- 172.21.15.78:0/3659729515 shutdown_connections 2023-10-10T17:49:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- 172.21.15.78:0/3659729515 wait complete. 2023-10-10T17:49:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- start start 2023-10-10T17:49:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 Processor -- start 2023-10-10T17:49:20.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5deee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5e4e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5e4ea30 con 0x5630a5de2470 2023-10-10T17:49:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5e4eb70 con 0x5630a5de68b0 2023-10-10T17:49:20.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630a5e4ecb0 con 0x5630a5de1b60 2023-10-10T17:49:20.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5b540fb80 10 client.0 _ll_register_callbacks cb 0x7fa59c007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:20.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5a27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5deee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5a27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5deee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47484/0 (socket says 172.21.15.78:47484) 2023-10-10T17:49:20.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5a27fc700 1 -- 172.21.15.78:0/83774271 learned_addr learned my addr 172.21.15.78:0/83774271 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.673+0000 7fa5a37fe700 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5e4e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5a27fc700 1 -- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 msgr2=0x5630a5dee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dee8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5a27fc700 1 -- 172.21.15.78:0/83774271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 msgr2=0x5630a5e4e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5e4e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5a27fc700 1 -- 172.21.15.78:0/83774271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa594011cc0 con 0x5630a5de2470 2023-10-10T17:49:20.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5a27fc700 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5deee00 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa594028000 tx=0x7fa594012b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a91ee5c5b289925 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa586ff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:20.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa586ff5700 1 -- 172.21.15.78:0/83774271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59402f070 con 0x5630a5de2470 2023-10-10T17:49:20.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa586ff5700 1 -- 172.21.15.78:0/83774271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa594018040 con 0x5630a5de2470 2023-10-10T17:49:20.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa59800fd90 con 0x5630a5de2470 2023-10-10T17:49:20.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa586ff5700 1 -- 172.21.15.78:0/83774271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa594009a40 con 0x5630a5de2470 2023-10-10T17:49:20.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 10 client.16848 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:20.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5630a5b49c60 con 0x5630a5de2470 2023-10-10T17:49:20.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa586ff5700 1 -- 172.21.15.78:0/83774271 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa59800fd90 con 0x5630a5de2470 2023-10-10T17:49:20.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa586ff5700 1 -- 172.21.15.78:0/83774271 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa594027020 con 0x5630a5de2470 2023-10-10T17:49:20.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 10 client.16848 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:20.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 10 client.16848 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:49:20.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 20 client.16848 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:20.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5e50380 0x5630a5e52790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 4 client.16848 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:49:20.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.674+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5630a5d01660 con 0x5630a5e50380 2023-10-10T17:49:20.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5e50380 0x5630a5e52790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa586ff5700 10 client.16848 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:20.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5a2ffd700 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5e50380 0x5630a5e52790 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7b376b4df9efb52e server_cookie=136969a30d8747ae in_seq=0 out_seq=0 2023-10-10T17:49:20.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa586ff5700 1 -- 172.21.15.78:0/83774271 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5630a5d01660 con 0x5630a5e50380 2023-10-10T17:49:20.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa586ff5700 10 client.16848 handle_command_reply: tid=0 2023-10-10T17:49:20.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 client.16848 shutdown 2023-10-10T17:49:20.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 msgr2=0x5630a5deee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5deee00 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fa594028000 tx=0x7fa594012b00 comp rx=0 tx=0).stop 2023-10-10T17:49:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 shutdown_connections 2023-10-10T17:49:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5630a5e50380 0x5630a5e52790 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5630a5de1b60 0x5630a5dee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa586ff5700 0 client.16848 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:20.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5630a5de68b0 0x5630a5e4e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 --2- 172.21.15.78:0/83774271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5630a5de2470 0x5630a5deee00 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 >> 172.21.15.78:0/83774271 conn(0x5630a5dd7360 msgr2=0x5630a5df78b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 shutdown_connections 2023-10-10T17:49:20.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.675+0000 7fa5b540fb80 1 -- 172.21.15.78:0/83774271 wait complete. 2023-10-10T17:49:20.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5b540fb80 20 client.16848 trim_cache size 0 max 16384 2023-10-10T17:49:20.706 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:49:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa58c0a12c0 msgr2=0x7fa58c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa58c0a12c0 0x7fa58c0a3770 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4197af0 tx=0x7fa594009260 comp rx=0 tx=0).stop 2023-10-10T17:49:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 msgr2=0x7fa5a41964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a41964e0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa598000c00 tx=0x7fa59800f510 comp rx=0 tx=0).stop 2023-10-10T17:49:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 shutdown_connections 2023-10-10T17:49:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa58c0a12c0 0x7fa58c0a3770 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5a410a8b0 0x7fa5a419ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5a4101390 0x7fa5a41964e0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 --2- 172.21.15.78:0/1214916236 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5a4101ca0 0x7fa5a4196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 >> 172.21.15.78:0/1214916236 conn(0x7fa5a4077080 msgr2=0x7fa5a40fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 shutdown_connections 2023-10-10T17:49:20.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.676+0000 7fa5a9283700 1 -- 172.21.15.78:0/1214916236 wait complete. 2023-10-10T17:49:20.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:49:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 -- 172.21.15.78:0/1609564154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c105210 msgr2=0x7fba5c1055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 --2- 172.21.15.78:0/1609564154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c105210 0x7fba5c1055e0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00b0a0 tx=0x7fba4c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 -- 172.21.15.78:0/1609564154 shutdown_connections 2023-10-10T17:49:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 --2- 172.21.15.78:0/1609564154 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba5c100530 0x7fba5c10c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 --2- 172.21.15.78:0/1609564154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c105210 0x7fba5c1055e0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 --2- 172.21.15.78:0/1609564154 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba5c0ffba0 0x7fba5c0ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 -- 172.21.15.78:0/1609564154 >> 172.21.15.78:0/1609564154 conn(0x7fba5c0748f0 msgr2=0x7fba5c074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 -- 172.21.15.78:0/1609564154 shutdown_connections 2023-10-10T17:49:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.835+0000 7fba63aae700 1 -- 172.21.15.78:0/1609564154 wait complete. 2023-10-10T17:49:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 Processor -- start 2023-10-10T17:49:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 -- start start 2023-10-10T17:49:20.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c0ffba0 0x7fba5c1974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba5c100530 0x7fba5c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba5c105210 0x7fba5c1a2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c198330 con 0x7fba5c0ffba0 2023-10-10T17:49:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1a2a70 con 0x7fba5c100530 2023-10-10T17:49:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba63aae700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1a2c30 con 0x7fba5c105210 2023-10-10T17:49:20.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c0ffba0 0x7fba5c1974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c0ffba0 0x7fba5c1974b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47496/0 (socket says 172.21.15.78:47496) 2023-10-10T17:49:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 -- 172.21.15.78:0/3480411424 learned_addr learned my addr 172.21.15.78:0/3480411424 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6204b700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba5c105210 0x7fba5c1a2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 -- 172.21.15.78:0/3480411424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba5c105210 msgr2=0x7fba5c1a2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba5c105210 0x7fba5c1a2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 -- 172.21.15.78:0/3480411424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba5c100530 msgr2=0x7fba5c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba5c100530 0x7fba5c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 -- 172.21.15.78:0/3480411424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba4c009d00 con 0x7fba5c0ffba0 2023-10-10T17:49:20.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.836+0000 7fba6184a700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c0ffba0 0x7fba5c1974b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00b070 tx=0x7fba4c004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.837+0000 7fba5affd700 1 -- 172.21.15.78:0/3480411424 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c00e040 con 0x7fba5c0ffba0 2023-10-10T17:49:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.837+0000 7fba5affd700 1 -- 172.21.15.78:0/3480411424 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba4c004020 con 0x7fba5c0ffba0 2023-10-10T17:49:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.837+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c10f280 con 0x7fba5c0ffba0 2023-10-10T17:49:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.837+0000 7fba5affd700 1 -- 172.21.15.78:0/3480411424 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c01b6b0 con 0x7fba5c0ffba0 2023-10-10T17:49:20.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.837+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c10f0f0 con 0x7fba5c0ffba0 2023-10-10T17:49:20.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.839+0000 7fba5affd700 1 -- 172.21.15.78:0/3480411424 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fba4c004190 con 0x7fba5c0ffba0 2023-10-10T17:49:20.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.840+0000 7fba5affd700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fba440a1340 0x7fba440a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.840+0000 7fba5affd700 1 -- 172.21.15.78:0/3480411424 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fba4c0ce9d0 con 0x7fba5c0ffba0 2023-10-10T17:49:20.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.840+0000 7fba61049700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fba440a1340 0x7fba440a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.840+0000 7fba61049700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fba440a1340 0x7fba440a37f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0ff360 tx=0x7fba50006d20 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.840+0000 7fba6fc3ab80 1 Processor -- start 2023-10-10T17:49:20.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6fc3ab80 1 -- start start 2023-10-10T17:49:20.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcbdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcc28b0 0x55a02fcc2d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fbc3060 con 0x55a02fcbdb60 2023-10-10T17:49:20.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fc158a0 con 0x55a02fcc28b0 2023-10-10T17:49:20.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fc0ed30 con 0x55a02fcbe470 2023-10-10T17:49:20.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcbdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcbdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47504/0 (socket says 172.21.15.78:47504) 2023-10-10T17:49:20.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 -- 172.21.15.78:0/4147741845 learned_addr learned my addr 172.21.15.78:0/4147741845 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba61049700 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6204b700 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcc28b0 0x55a02fcc2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 -- 172.21.15.78:0/4147741845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 msgr2=0x55a02fcc2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 -- 172.21.15.78:0/4147741845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcc28b0 msgr2=0x55a02fcc2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcc28b0 0x55a02fcc2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 -- 172.21.15.78:0/4147741845 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a02fa90630 con 0x55a02fcbdb60 2023-10-10T17:49:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba6184a700 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcbdf30 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fba4c09b730 tx=0x7fba4c003fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2177fc6092d4fb06 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.841+0000 7fba43fff700 1 -- 172.21.15.78:0/4147741845 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c09f7d0 con 0x55a02fcbdb60 2023-10-10T17:49:20.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba43fff700 1 -- 172.21.15.78:0/4147741845 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba4c0a3b60 con 0x55a02fcbdb60 2023-10-10T17:49:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba43fff700 1 -- 172.21.15.78:0/4147741845 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c09bc00 con 0x55a02fcbdb60 2023-10-10T17:49:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/4147741845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 msgr2=0x55a02fcbdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcbdf30 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fba4c09b730 tx=0x7fba4c003fb0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/4147741845 shutdown_connections 2023-10-10T17:49:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcc28b0 0x55a02fcc2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcbdf30 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/4147741845 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/4147741845 >> 172.21.15.78:0/4147741845 conn(0x55a02fcb3360 msgr2=0x55a02fcb8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/4147741845 shutdown_connections 2023-10-10T17:49:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.842+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/4147741845 wait complete. 2023-10-10T17:49:20.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 -- start start 2023-10-10T17:49:20.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 Processor -- start 2023-10-10T17:49:20.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbdb60 0x55a02fcca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fbb08e0 con 0x55a02fcc28b0 2023-10-10T17:49:20.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fbb0580 con 0x55a02fcbdb60 2023-10-10T17:49:20.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fa92920 con 0x55a02fcbe470 2023-10-10T17:49:20.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6fc3ab80 10 client.0 _ll_register_callbacks cb 0x55a02fbce780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:20.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6204b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6204b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc7b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47514/0 (socket says 172.21.15.78:47514) 2023-10-10T17:49:20.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6204b700 1 -- 172.21.15.78:0/2374650274 learned_addr learned my addr 172.21.15.78:0/2374650274 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba6184a700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbdb60 0x55a02fcca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.843+0000 7fba61049700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6204b700 1 -- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 msgr2=0x55a02fcc73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6204b700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6204b700 1 -- 172.21.15.78:0/2374650274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbdb60 msgr2=0x55a02fcca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6204b700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbdb60 0x55a02fcca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6204b700 1 -- 172.21.15.78:0/2374650274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c10f0f0 con 0x55a02fcc28b0 2023-10-10T17:49:20.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba61049700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6204b700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc7b20 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fba5400f760 tx=0x7fba5400fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a9e161ff720823 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba417fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba417fa700 1 -- 172.21.15.78:0/2374650274 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54010900 con 0x55a02fcc28b0 2023-10-10T17:49:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba417fa700 1 -- 172.21.15.78:0/2374650274 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fba54010a60 con 0x55a02fcc28b0 2023-10-10T17:49:20.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a02fcb0b20 con 0x55a02fcc28b0 2023-10-10T17:49:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba417fa700 1 -- 172.21.15.78:0/2374650274 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba54010c20 con 0x55a02fcc28b0 2023-10-10T17:49:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6fc3ab80 10 client.16860 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a02fbd5190 con 0x55a02fcc28b0 2023-10-10T17:49:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba417fa700 1 -- 172.21.15.78:0/2374650274 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a02fcb0b20 con 0x55a02fcc28b0 2023-10-10T17:49:20.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba417fa700 1 -- 172.21.15.78:0/2374650274 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fba54018030 con 0x55a02fcc28b0 2023-10-10T17:49:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6fc3ab80 10 client.16860 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6fc3ab80 10 client.16860 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:49:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.844+0000 7fba6fc3ab80 20 client.16860 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.845+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fcd78f0 0x55a02fcd9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.845+0000 7fba6fc3ab80 4 client.16860 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.845+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a02fbd10a0 con 0x55a02fcd78f0 2023-10-10T17:49:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.845+0000 7fba6184a700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fcd78f0 0x55a02fcd9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.845+0000 7fba417fa700 10 client.16860 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.845+0000 7fba6184a700 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fcd78f0 0x55a02fcd9d00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2e453d344e564170 server_cookie=f2ef8c9fce385194 in_seq=0 out_seq=0 2023-10-10T17:49:20.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba417fa700 1 -- 172.21.15.78:0/2374650274 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a02fbd10a0 con 0x55a02fcd78f0 2023-10-10T17:49:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba417fa700 10 client.16860 handle_command_reply: tid=0 2023-10-10T17:49:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 client.16860 shutdown 2023-10-10T17:49:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 msgr2=0x55a02fcc7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc7b20 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fba5400f760 tx=0x7fba5400fa70 comp rx=0 tx=0).stop 2023-10-10T17:49:20.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 shutdown_connections 2023-10-10T17:49:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc7b20 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbdb60 0x55a02fcca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fcd78f0 0x55a02fcd9d00 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/2374650274 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbe470 0x55a02fcc73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba417fa700 0 client.16860 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:20.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 >> 172.21.15.78:0/2374650274 conn(0x55a02fcb3360 msgr2=0x55a02fcc3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 shutdown_connections 2023-10-10T17:49:20.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/2374650274 wait complete. 2023-10-10T17:49:20.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.846+0000 7fba6fc3ab80 20 client.16860 trim_cache size 0 max 16384 2023-10-10T17:49:20.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 Processor -- start 2023-10-10T17:49:20.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 -- start start 2023-10-10T17:49:20.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbdb60 0x55a02fcc01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcce840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fcb2ee0 con 0x55a02fcc28b0 2023-10-10T17:49:20.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fcd4e70 con 0x55a02fcbe470 2023-10-10T17:49:20.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.859+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fcc5940 con 0x55a02fcbdb60 2023-10-10T17:49:20.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47522/0 (socket says 172.21.15.78:47522) 2023-10-10T17:49:20.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 -- 172.21.15.78:0/3978561363 learned_addr learned my addr 172.21.15.78:0/3978561363 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6184a700 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbdb60 0x55a02fcc01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba61049700 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 -- 172.21.15.78:0/3978561363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbdb60 msgr2=0x55a02fcc01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbdb60 0x55a02fcc01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 -- 172.21.15.78:0/3978561363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 msgr2=0x55a02fcce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 -- 172.21.15.78:0/3978561363 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a02fbd4ee0 con 0x55a02fcc28b0 2023-10-10T17:49:20.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba6204b700 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc0a10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fba54001eb0 tx=0x7fba54005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdc665c9e18a739 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba58dac700 1 -- 172.21.15.78:0/3978561363 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5400b4c0 con 0x55a02fcc28b0 2023-10-10T17:49:20.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba58dac700 1 -- 172.21.15.78:0/3978561363 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba54014070 con 0x55a02fcc28b0 2023-10-10T17:49:20.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.860+0000 7fba58dac700 1 -- 172.21.15.78:0/3978561363 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54010d70 con 0x55a02fcc28b0 2023-10-10T17:49:20.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3978561363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 msgr2=0x55a02fcc0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc0a10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fba54001eb0 tx=0x7fba54005d70 comp rx=0 tx=0).stop 2023-10-10T17:49:20.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3978561363 shutdown_connections 2023-10-10T17:49:20.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcbdb60 0x55a02fcc01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcc28b0 0x55a02fcc0a10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3978561363 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3978561363 >> 172.21.15.78:0/3978561363 conn(0x55a02fcb3360 msgr2=0x55a02fcd3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3978561363 shutdown_connections 2023-10-10T17:49:20.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3978561363 wait complete. 2023-10-10T17:49:20.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 -- start start 2023-10-10T17:49:20.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 Processor -- start 2023-10-10T17:49:20.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcb8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcb8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.861+0000 7fba6fc3ab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcc28b0 0x55a02fcb7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fcb7ac0 con 0x55a02fcbdb60 2023-10-10T17:49:20.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fcb7c30 con 0x55a02fcbe470 2023-10-10T17:49:20.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6fc3ab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02fcb7df0 con 0x55a02fcc28b0 2023-10-10T17:49:20.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6fc3ab80 10 client.0 _ll_register_callbacks cb 0x55a02fbce780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:20.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcb8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcb8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47524/0 (socket says 172.21.15.78:47524) 2023-10-10T17:49:20.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 -- 172.21.15.78:0/3210620520 learned_addr learned my addr 172.21.15.78:0/3210620520 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:20.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6204b700 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcc28b0 0x55a02fcb7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba61049700 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcb8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 -- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcc28b0 msgr2=0x55a02fcb7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcc28b0 0x55a02fcb7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 -- 172.21.15.78:0/3210620520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 msgr2=0x55a02fcb8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcb8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 -- 172.21.15.78:0/3210620520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a02fbd5190 con 0x55a02fcbdb60 2023-10-10T17:49:20.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6184a700 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcb8720 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fba4c099af0 tx=0x7fba4c025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f9a008390c824a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:20.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba417fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:20.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba417fa700 1 -- 172.21.15.78:0/3210620520 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4c0bb0b0 con 0x55a02fcbdb60 2023-10-10T17:49:20.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba417fa700 1 -- 172.21.15.78:0/3210620520 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fba4c0b6020 con 0x55a02fcbdb60 2023-10-10T17:49:20.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba4c0a3b60 con 0x55a02fcbdb60 2023-10-10T17:49:20.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba417fa700 1 -- 172.21.15.78:0/3210620520 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba4c0bb210 con 0x55a02fcbdb60 2023-10-10T17:49:20.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6fc3ab80 10 client.16866 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:20.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a02fa25c60 con 0x55a02fcbdb60 2023-10-10T17:49:20.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.862+0000 7fba417fa700 1 -- 172.21.15.78:0/3210620520 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba4c0a3b60 con 0x55a02fcbdb60 2023-10-10T17:49:20.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba417fa700 1 -- 172.21.15.78:0/3210620520 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fba4c0b98a0 con 0x55a02fcbdb60 2023-10-10T17:49:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba6fc3ab80 10 client.16866 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba6fc3ab80 10 client.16866 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:49:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba6fc3ab80 20 client.16866 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fd2c310 0x55a02fd2e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:20.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba6fc3ab80 4 client.16866 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:49:20.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a02fbdd660 con 0x55a02fd2c310 2023-10-10T17:49:20.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba61049700 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fd2c310 0x55a02fd2e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:20.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba417fa700 10 client.16866 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:20.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.863+0000 7fba61049700 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fd2c310 0x55a02fd2e720 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4696b9af08978897 server_cookie=316c2e4ea4f8442f in_seq=0 out_seq=0 2023-10-10T17:49:20.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba417fa700 1 -- 172.21.15.78:0/3210620520 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a02fbdd660 con 0x55a02fd2c310 2023-10-10T17:49:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba417fa700 10 client.16866 handle_command_reply: tid=0 2023-10-10T17:49:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 client.16866 shutdown 2023-10-10T17:49:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 msgr2=0x55a02fcb8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcb8720 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fba4c099af0 tx=0x7fba4c025000 comp rx=0 tx=0).stop 2023-10-10T17:49:20.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 shutdown_connections 2023-10-10T17:49:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a02fd2c310 0x55a02fd2e720 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a02fcc28b0 0x55a02fcb7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a02fcbdb60 0x55a02fcb8720 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 --2- 172.21.15.78:0/3210620520 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a02fcbe470 0x55a02fcb8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba417fa700 0 client.16866 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 >> 172.21.15.78:0/3210620520 conn(0x55a02fcb3360 msgr2=0x55a02fcc0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 shutdown_connections 2023-10-10T17:49:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 1 -- 172.21.15.78:0/3210620520 wait complete. 2023-10-10T17:49:20.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba6fc3ab80 20 client.16866 trim_cache size 0 max 16384 2023-10-10T17:49:20.895 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:49:20.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fba440a1340 msgr2=0x7fba440a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba63aae700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fba440a1340 0x7fba440a37f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fba5c0ff360 tx=0x7fba50006d20 comp rx=0 tx=0).stop 2023-10-10T17:49:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c0ffba0 msgr2=0x7fba5c1974b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.864+0000 7fba63aae700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c0ffba0 0x7fba5c1974b0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fba4c00b070 tx=0x7fba4c004a50 comp rx=0 tx=0).stop 2023-10-10T17:49:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 shutdown_connections 2023-10-10T17:49:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fba5c105210 0x7fba5c1a2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fba440a1340 0x7fba440a37f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fba5c0ffba0 0x7fba5c1974b0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 --2- 172.21.15.78:0/3480411424 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fba5c100530 0x7fba5c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 >> 172.21.15.78:0/3480411424 conn(0x7fba5c0748f0 msgr2=0x7fba5c0fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 shutdown_connections 2023-10-10T17:49:20.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:20.865+0000 7fba63aae700 1 -- 172.21.15.78:0/3480411424 wait complete. 2023-10-10T17:49:20.899 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:49:21.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.022+0000 7f09af242700 1 -- 172.21.15.78:0/3473520938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 msgr2=0x7f09a8100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.022+0000 7f09af242700 1 --2- 172.21.15.78:0/3473520938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a8100980 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f099c0099c0 tx=0x7f099c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.022+0000 7f09af242700 1 -- 172.21.15.78:0/3473520938 shutdown_connections 2023-10-10T17:49:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.022+0000 7f09af242700 1 --2- 172.21.15.78:0/3473520938 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a81055f0 0x7f09a810d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.022+0000 7f09af242700 1 --2- 172.21.15.78:0/3473520938 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a8100980 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.022+0000 7f09af242700 1 --2- 172.21.15.78:0/3473520938 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a8100f50 0x7f09a8104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.022+0000 7f09af242700 1 -- 172.21.15.78:0/3473520938 >> 172.21.15.78:0/3473520938 conn(0x7f09a8077170 msgr2=0x7f09a8077570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 -- 172.21.15.78:0/3473520938 shutdown_connections 2023-10-10T17:49:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 -- 172.21.15.78:0/3473520938 wait complete. 2023-10-10T17:49:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 Processor -- start 2023-10-10T17:49:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 -- start start 2023-10-10T17:49:21.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a819d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a8100f50 0x7f09a819d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a81055f0 0x7f09a8198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a81995f0 con 0x7f09a81005b0 2023-10-10T17:49:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8199760 con 0x7f09a81055f0 2023-10-10T17:49:21.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09af242700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8199920 con 0x7f09a8100f50 2023-10-10T17:49:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09acfde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a819d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09acfde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a819d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47538/0 (socket says 172.21.15.78:47538) 2023-10-10T17:49:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.023+0000 7f09acfde700 1 -- 172.21.15.78:0/3507452863 learned_addr learned my addr 172.21.15.78:0/3507452863 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09ad7df700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a81055f0 0x7f09a8198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09acfde700 1 -- 172.21.15.78:0/3507452863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a8100f50 msgr2=0x7f09a819d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09acfde700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a8100f50 0x7f09a819d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09acfde700 1 -- 172.21.15.78:0/3507452863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a81055f0 msgr2=0x7f09a8198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09acfde700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a81055f0 0x7f09a8198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09acfde700 1 -- 172.21.15.78:0/3507452863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c0095f0 con 0x7f09a81005b0 2023-10-10T17:49:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09ad7df700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a81055f0 0x7f09a8198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09acfde700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a819d3d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f099c004730 tx=0x7f099c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09a5ffb700 1 -- 172.21.15.78:0/3507452863 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c01d070 con 0x7f09a81005b0 2023-10-10T17:49:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09a5ffb700 1 -- 172.21.15.78:0/3507452863 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f099c004cd0 con 0x7f09a81005b0 2023-10-10T17:49:21.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a8073340 con 0x7f09a81005b0 2023-10-10T17:49:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09a5ffb700 1 -- 172.21.15.78:0/3507452863 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c005290 con 0x7f09a81005b0 2023-10-10T17:49:21.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.024+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a80731b0 con 0x7f09a81005b0 2023-10-10T17:49:21.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.026+0000 7f09a5ffb700 1 -- 172.21.15.78:0/3507452863 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f099c003c50 con 0x7f09a81005b0 2023-10-10T17:49:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.027+0000 7f09a5ffb700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f09980a12c0 0x7f09980a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.027+0000 7f09a5ffb700 1 -- 172.21.15.78:0/3507452863 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f099c0cddb0 con 0x7f09a81005b0 2023-10-10T17:49:21.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.027+0000 7f09a7fff700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f09980a12c0 0x7f09980a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.027+0000 7f09a7fff700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f09980a12c0 0x7f09980a3770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0994005950 tx=0x7f09940058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 Processor -- start 2023-10-10T17:49:21.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 -- start start 2023-10-10T17:49:21.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2553b60 0x5628b2553f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2554470 0x5628b2558370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b2558d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2459060 con 0x5628b25588b0 2023-10-10T17:49:21.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b24ab8a0 con 0x5628b2553b60 2023-10-10T17:49:21.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b24a4d30 con 0x5628b2554470 2023-10-10T17:49:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b2558d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b2558d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47550/0 (socket says 172.21.15.78:47550) 2023-10-10T17:49:21.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 -- 172.21.15.78:0/1806008632 learned_addr learned my addr 172.21.15.78:0/1806008632 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2554470 0x5628b2558370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09acfde700 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2553b60 0x5628b2553f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 -- 172.21.15.78:0/1806008632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2554470 msgr2=0x5628b2558370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2554470 0x5628b2558370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 -- 172.21.15.78:0/1806008632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2553b60 msgr2=0x5628b2553f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2553b60 0x5628b2553f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09ad7df700 1 -- 172.21.15.78:0/1806008632 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628b2326630 con 0x5628b25588b0 2023-10-10T17:49:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.028+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2554470 0x5628b2558370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09ad7df700 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b2558d00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009fa0 tx=0x7f09a000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe9b31770369385 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f0992ffd700 1 -- 172.21.15.78:0/1806008632 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0011070 con 0x5628b25588b0 2023-10-10T17:49:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f0992ffd700 1 -- 172.21.15.78:0/1806008632 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09a0009ad0 con 0x5628b25588b0 2023-10-10T17:49:21.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f0992ffd700 1 -- 172.21.15.78:0/1806008632 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a000a400 con 0x5628b25588b0 2023-10-10T17:49:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1806008632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 msgr2=0x5628b2558d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b2558d00 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009fa0 tx=0x7f09a000c420 comp rx=0 tx=0).stop 2023-10-10T17:49:21.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1806008632 shutdown_connections 2023-10-10T17:49:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b2558d00 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2553b60 0x5628b2553f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1806008632 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2554470 0x5628b2558370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1806008632 >> 172.21.15.78:0/1806008632 conn(0x5628b2549360 msgr2=0x5628b254eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:21.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1806008632 shutdown_connections 2023-10-10T17:49:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.029+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1806008632 wait complete. 2023-10-10T17:49:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 -- start start 2023-10-10T17:49:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 Processor -- start 2023-10-10T17:49:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2553b60 0x5628b2560040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b255d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b25588b0 0x5628b255d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b24468e0 con 0x5628b2553b60 2023-10-10T17:49:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2446580 con 0x5628b2554470 2023-10-10T17:49:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2328920 con 0x5628b25588b0 2023-10-10T17:49:21.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09bb3ceb80 10 client.0 _ll_register_callbacks cb 0x5628b2464780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09acfde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2553b60 0x5628b2560040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09acfde700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2553b60 0x5628b2560040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47552/0 (socket says 172.21.15.78:47552) 2023-10-10T17:49:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.030+0000 7f09acfde700 1 -- 172.21.15.78:0/1616478251 learned_addr learned my addr 172.21.15.78:0/1616478251 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:21.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b255d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09ad7df700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b25588b0 0x5628b255d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09acfde700 1 -- 172.21.15.78:0/1616478251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b25588b0 msgr2=0x5628b255d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09acfde700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b25588b0 0x5628b255d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09acfde700 1 -- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 msgr2=0x5628b255d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09acfde700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b255d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09acfde700 1 -- 172.21.15.78:0/1616478251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a0009c50 con 0x5628b2553b60 2023-10-10T17:49:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b255d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09acfde700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2553b60 0x5628b2560040 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f099c0cb1d0 tx=0x7f099c09a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84704c4cbf838701 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f0987fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f0987fff700 1 -- 172.21.15.78:0/1616478251 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c0b2070 con 0x5628b2553b60 2023-10-10T17:49:21.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f0987fff700 1 -- 172.21.15.78:0/1616478251 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f099c09ebb0 con 0x5628b2553b60 2023-10-10T17:49:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5628b24670a0 con 0x5628b2553b60 2023-10-10T17:49:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f0987fff700 1 -- 172.21.15.78:0/1616478251 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f099c0a2f40 con 0x5628b2553b60 2023-10-10T17:49:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 10 client.16878 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5628b246b190 con 0x5628b2553b60 2023-10-10T17:49:21.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f0987fff700 1 -- 172.21.15.78:0/1616478251 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5628b24670a0 con 0x5628b2553b60 2023-10-10T17:49:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f0987fff700 1 -- 172.21.15.78:0/1616478251 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f099c09a100 con 0x5628b2553b60 2023-10-10T17:49:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 10 client.16878 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 10 client.16878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 20 client.16878 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:21.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b256d4b0 0x5628b256f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 4 client.16878 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.031+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628b2473660 con 0x5628b256d4b0 2023-10-10T17:49:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.032+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b256d4b0 0x5628b256f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.032+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b256d4b0 0x5628b256f8c0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c83230d38d81b57a server_cookie=2920d8d43273d6c4 in_seq=0 out_seq=0 2023-10-10T17:49:21.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.032+0000 7f0987fff700 10 client.16878 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f0987fff700 1 -- 172.21.15.78:0/1616478251 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5628b2473660 con 0x5628b256d4b0 2023-10-10T17:49:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f0987fff700 10 client.16878 handle_command_reply: tid=0 2023-10-10T17:49:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 client.16878 shutdown 2023-10-10T17:49:21.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2553b60 msgr2=0x5628b2560040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2553b60 0x5628b2560040 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f099c0cb1d0 tx=0x7f099c09a590 comp rx=0 tx=0).stop 2023-10-10T17:49:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 shutdown_connections 2023-10-10T17:49:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b25588b0 0x5628b255d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b256d4b0 0x5628b256f8c0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2553b60 0x5628b2560040 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f0987fff700 0 client.16878 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1616478251 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b255d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 >> 172.21.15.78:0/1616478251 conn(0x5628b2549360 msgr2=0x5628b2559d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 shutdown_connections 2023-10-10T17:49:21.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1616478251 wait complete. 2023-10-10T17:49:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.033+0000 7f09bb3ceb80 20 client.16878 trim_cache size 0 max 16384 2023-10-10T17:49:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.046+0000 7f09bb3ceb80 1 Processor -- start 2023-10-10T17:49:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.046+0000 7f09bb3ceb80 1 -- start start 2023-10-10T17:49:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b25649a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b2559c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b255fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2554bc0 con 0x5628b25588b0 2023-10-10T17:49:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b255e250 con 0x5628b2554470 2023-10-10T17:49:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b25573e0 con 0x5628b2553b60 2023-10-10T17:49:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b255fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b255fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47564/0 (socket says 172.21.15.78:47564) 2023-10-10T17:49:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 -- 172.21.15.78:0/121527382 learned_addr learned my addr 172.21.15.78:0/121527382 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09acfde700 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b25649a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09a7fff700 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b2559c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 -- 172.21.15.78:0/121527382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 msgr2=0x5628b25649a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b25649a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 -- 172.21.15.78:0/121527382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 msgr2=0x5628b2559c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b2559c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 -- 172.21.15.78:0/121527382 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628b246aee0 con 0x5628b25588b0 2023-10-10T17:49:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09a7fff700 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b2559c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:21.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09ad7df700 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b255fbe0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009a10 tx=0x7f09a0004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ea03b95d8eb502 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09937fe700 1 -- 172.21.15.78:0/121527382 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0011050 con 0x5628b25588b0 2023-10-10T17:49:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09937fe700 1 -- 172.21.15.78:0/121527382 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f09a0012040 con 0x5628b25588b0 2023-10-10T17:49:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.047+0000 7f09937fe700 1 -- 172.21.15.78:0/121527382 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a001e420 con 0x5628b25588b0 2023-10-10T17:49:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/121527382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 msgr2=0x5628b255fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b255fbe0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009a10 tx=0x7f09a0004a50 comp rx=0 tx=0).stop 2023-10-10T17:49:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/121527382 shutdown_connections 2023-10-10T17:49:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b25649a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b25588b0 0x5628b255fbe0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/121527382 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b2554470 0x5628b2559c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/121527382 >> 172.21.15.78:0/121527382 conn(0x5628b2549360 msgr2=0x5628b2569be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/121527382 shutdown_connections 2023-10-10T17:49:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/121527382 wait complete. 2023-10-10T17:49:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 -- start start 2023-10-10T17:49:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.048+0000 7f09bb3ceb80 1 Processor -- start 2023-10-10T17:49:21.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b255ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2554470 0x5628b255bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09bb3ceb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b25588b0 0x5628b255c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b255cd00 con 0x5628b2554470 2023-10-10T17:49:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b255ce70 con 0x5628b25588b0 2023-10-10T17:49:21.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09bb3ceb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b255d030 con 0x5628b2553b60 2023-10-10T17:49:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09bb3ceb80 10 client.0 _ll_register_callbacks cb 0x7f099c09a950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2554470 0x5628b255bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2554470 0x5628b255bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47568/0 (socket says 172.21.15.78:47568) 2023-10-10T17:49:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 -- 172.21.15.78:0/1784947580 learned_addr learned my addr 172.21.15.78:0/1784947580 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:21.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09acfde700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b255ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09ad7df700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b25588b0 0x5628b255c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 -- 172.21.15.78:0/1784947580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 msgr2=0x5628b255ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b255ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 -- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b25588b0 msgr2=0x5628b255c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:21.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b25588b0 0x5628b255c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 -- 172.21.15.78:0/1784947580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628b2326630 con 0x5628b2554470 2023-10-10T17:49:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09acfde700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b255ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f09a7fff700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2554470 0x5628b255bfb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f09a810d5d0 tx=0x7f0994005d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b739591550cd41 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f0987fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:21.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.049+0000 7f0987fff700 1 -- 172.21.15.78:0/1784947580 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099400dee0 con 0x5628b2554470 2023-10-10T17:49:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f0987fff700 1 -- 172.21.15.78:0/1784947580 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f09940251b0 con 0x5628b2554470 2023-10-10T17:49:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5628b2546b20 con 0x5628b2554470 2023-10-10T17:49:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f0987fff700 1 -- 172.21.15.78:0/1784947580 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0994025370 con 0x5628b2554470 2023-10-10T17:49:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 10 client.16884 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:21.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5628b22bbc60 con 0x5628b2554470 2023-10-10T17:49:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f0987fff700 1 -- 172.21.15.78:0/1784947580 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5628b2546b20 con 0x5628b2554470 2023-10-10T17:49:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f0987fff700 1 -- 172.21.15.78:0/1784947580 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f099402b020 con 0x5628b2554470 2023-10-10T17:49:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 10 client.16884 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:21.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 20 client.16884 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b25c21c0 0x5628b25c45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 4 client.16884 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:49:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5628b22f3f70 con 0x5628b25c21c0 2023-10-10T17:49:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.050+0000 7f09acfde700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b25c21c0 0x5628b25c45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:21.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f0987fff700 10 client.16884 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09acfde700 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b25c21c0 0x5628b25c45d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d792f5df044f9d13 server_cookie=2ddd8faba290fc6b in_seq=0 out_seq=0 2023-10-10T17:49:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f0987fff700 1 -- 172.21.15.78:0/1784947580 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5628b22f3f70 con 0x5628b25c21c0 2023-10-10T17:49:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f0987fff700 10 client.16884 handle_command_reply: tid=0 2023-10-10T17:49:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 client.16884 shutdown 2023-10-10T17:49:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2554470 msgr2=0x5628b255bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2554470 0x5628b255bfb0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f09a810d5d0 tx=0x7f0994005d20 comp rx=0 tx=0).stop 2023-10-10T17:49:21.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 shutdown_connections 2023-10-10T17:49:21.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5628b2553b60 0x5628b255ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5628b25c21c0 0x5628b25c45d0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5628b25588b0 0x5628b255c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f0987fff700 0 client.16884 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 --2- 172.21.15.78:0/1784947580 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5628b2554470 0x5628b255bfb0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 >> 172.21.15.78:0/1784947580 conn(0x5628b2549360 msgr2=0x5628b255a570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 shutdown_connections 2023-10-10T17:49:21.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 1 -- 172.21.15.78:0/1784947580 wait complete. 2023-10-10T17:49:21.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.051+0000 7f09bb3ceb80 20 client.16884 trim_cache size 0 max 16384 2023-10-10T17:49:21.087 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:49:21.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f09980a12c0 msgr2=0x7f09980a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f09980a12c0 0x7f09980a3770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0994005950 tx=0x7f09940058e0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 msgr2=0x7f09a819d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a819d3d0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f099c004730 tx=0x7f099c005e80 comp rx=0 tx=0).stop 2023-10-10T17:49:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 shutdown_connections 2023-10-10T17:49:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f09980a12c0 0x7f09980a3770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f09a81055f0 0x7f09a8198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09a81005b0 0x7f09a819d3d0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 --2- 172.21.15.78:0/3507452863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f09a8100f50 0x7f09a819d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:21.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 >> 172.21.15.78:0/3507452863 conn(0x7f09a8077170 msgr2=0x7f09a8103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:21.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 shutdown_connections 2023-10-10T17:49:21.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:21.052+0000 7f09af242700 1 -- 172.21.15.78:0/3507452863 wait complete. 2023-10-10T17:49:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:20 smithi078 ceph-mon[99494]: pgmap v209: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 KiB/s wr, 113 op/s 2023-10-10T17:49:21.091 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:20 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3776097421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:49:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:20 smithi190 ceph-mon[101220]: pgmap v209: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 KiB/s wr, 113 op/s 2023-10-10T17:49:21.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:20 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3776097421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:49:21.227 INFO:tasks.workunit.client.0.smithi078.stderr:+ mkdir -p dir 2023-10-10T17:49:21.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:20 smithi157 ceph-mon[100896]: pgmap v209: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 KiB/s wr, 113 op/s 2023-10-10T17:49:21.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:20 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3776097421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:49:21.229 INFO:tasks.workunit.client.0.smithi078.stderr:+ getfattr -n ceph.dir.pin dir 2023-10-10T17:49:21.229 INFO:tasks.workunit.client.0.smithi078.stderr:+ grep 'ceph.dir.pin="-1"' 2023-10-10T17:49:21.230 INFO:tasks.workunit.client.0.smithi078.stdout:ceph.dir.pin="-1" 2023-10-10T17:49:21.231 INFO:tasks.workunit.client.0.smithi078.stderr:+ setfattr -n ceph.dir.pin dir 2023-10-10T17:49:21.231 INFO:tasks.workunit.client.0.smithi078.stderr:+ grep 'setfattr: dir: Invalid argument' 2023-10-10T17:49:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T17:49:21.233 INFO:tasks.workunit:Stopping ['fs/misc'] on client.0... 2023-10-10T17:49:21.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-10T17:49:22.092 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:49:22.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:49:22.162 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-10-10T17:49:22.265 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9a3ebbdf078c4d02844107561eb782e4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_chrisphoffman_ceph_48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh' 2023-10-10T17:49:22.268 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2023-10-10T17:49:22.280 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:49:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 -- 172.21.15.78:0/3190421124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37281063f0 msgr2=0x7f37281067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 --2- 172.21.15.78:0/3190421124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37281063f0 0x7f37281067c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3718009990 tx=0x7f3718009ca0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 -- 172.21.15.78:0/3190421124 shutdown_connections 2023-10-10T17:49:22.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 --2- 172.21.15.78:0/3190421124 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3728101400 0x7f372810d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 --2- 172.21.15.78:0/3190421124 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f37281063f0 0x7f37281067c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 --2- 172.21.15.78:0/3190421124 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728100a50 0x7f3728100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 -- 172.21.15.78:0/3190421124 >> 172.21.15.78:0/3190421124 conn(0x7f3728077590 msgr2=0x7f3728077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 -- 172.21.15.78:0/3190421124 shutdown_connections 2023-10-10T17:49:22.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 -- 172.21.15.78:0/3190421124 wait complete. 2023-10-10T17:49:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 Processor -- start 2023-10-10T17:49:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.221+0000 7f372f2f8700 1 -- start start 2023-10-10T17:49:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728100a50 0x7f3728196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3728101400 0x7f3728196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37281063f0 0x7f372819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372819b410 con 0x7f3728101400 2023-10-10T17:49:22.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372819b580 con 0x7f37281063f0 2023-10-10T17:49:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372819b740 con 0x7f3728100a50 2023-10-10T17:49:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3728101400 0x7f3728196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3728101400 0x7f3728196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47582/0 (socket says 172.21.15.78:47582) 2023-10-10T17:49:22.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 -- 172.21.15.78:0/46602396 learned_addr learned my addr 172.21.15.78:0/46602396 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372d094700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728100a50 0x7f3728196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372d895700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37281063f0 0x7f372819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 -- 172.21.15.78:0/46602396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728100a50 msgr2=0x7f3728196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728100a50 0x7f3728196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 -- 172.21.15.78:0/46602396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37281063f0 msgr2=0x7f372819add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37281063f0 0x7f372819add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 -- 172.21.15.78:0/46602396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37180095f0 con 0x7f3728101400 2023-10-10T17:49:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372c893700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3728101400 0x7f3728196970 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f372400d950 tx=0x7f372400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f371e7fc700 1 -- 172.21.15.78:0/46602396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724009940 con 0x7f3728101400 2023-10-10T17:49:22.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f371e7fc700 1 -- 172.21.15.78:0/46602396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3724010460 con 0x7f3728101400 2023-10-10T17:49:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3728110380 con 0x7f3728101400 2023-10-10T17:49:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f371e7fc700 1 -- 172.21.15.78:0/46602396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372400f620 con 0x7f3728101400 2023-10-10T17:49:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.222+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37281101f0 con 0x7f3728101400 2023-10-10T17:49:22.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.224+0000 7f371e7fc700 1 -- 172.21.15.78:0/46602396 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f37240108d0 con 0x7f3728101400 2023-10-10T17:49:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.225+0000 7f371e7fc700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37140a12c0 0x7f37140a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.225+0000 7f372d094700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37140a12c0 0x7f37140a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.225+0000 7f371e7fc700 1 -- 172.21.15.78:0/46602396 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f37240ccf30 con 0x7f3728101400 2023-10-10T17:49:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.226+0000 7f372d094700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37140a12c0 0x7f37140a3770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f371800b5c0 tx=0x7f3718005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.226+0000 7f373b484b80 1 Processor -- start 2023-10-10T17:49:22.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.226+0000 7f373b484b80 1 -- start start 2023-10-10T17:49:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f373b484b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a580efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f373b484b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a58133f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f373b484b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a5813930 0x5614a5813d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f373b484b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a57140e0 con 0x5614a580f4f0 2023-10-10T17:49:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f373b484b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a5766920 con 0x5614a5813930 2023-10-10T17:49:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f373b484b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a575fdb0 con 0x5614a580ebe0 2023-10-10T17:49:22.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a58133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a58133f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47590/0 (socket says 172.21.15.78:47590) 2023-10-10T17:49:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 -- 172.21.15.78:0/2358570657 learned_addr learned my addr 172.21.15.78:0/2358570657 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372d094700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a580efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372d895700 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a5813930 0x5614a5813d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 -- 172.21.15.78:0/2358570657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 msgr2=0x5614a580efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a580efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 -- 172.21.15.78:0/2358570657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a5813930 msgr2=0x5614a5813d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a5813930 0x5614a5813d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 -- 172.21.15.78:0/2358570657 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614a55e16b0 con 0x5614a580f4f0 2023-10-10T17:49:22.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372d895700 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a5813930 0x5614a5813d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f372c893700 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a58133f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3724099c00 tx=0x7f37240996a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5447a03d063fbe server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f37137fe700 1 -- 172.21.15.78:0/2358570657 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37240a20c0 con 0x5614a580f4f0 2023-10-10T17:49:22.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f37137fe700 1 -- 172.21.15.78:0/2358570657 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3724096380 con 0x5614a580f4f0 2023-10-10T17:49:22.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.227+0000 7f37137fe700 1 -- 172.21.15.78:0/2358570657 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37240a1c30 con 0x5614a580f4f0 2023-10-10T17:49:22.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 -- 172.21.15.78:0/2358570657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 msgr2=0x5614a58133f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a58133f0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f3724099c00 tx=0x7f37240996a0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 -- 172.21.15.78:0/2358570657 shutdown_connections 2023-10-10T17:49:22.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a580efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a5813930 0x5614a5813d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 --2- 172.21.15.78:0/2358570657 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a58133f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 -- 172.21.15.78:0/2358570657 >> 172.21.15.78:0/2358570657 conn(0x5614a58043e0 msgr2=0x5614a5809bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 -- 172.21.15.78:0/2358570657 shutdown_connections 2023-10-10T17:49:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 -- 172.21.15.78:0/2358570657 wait complete. 2023-10-10T17:49:22.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.228+0000 7f373b484b80 1 -- start start 2023-10-10T17:49:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 1 Processor -- start 2023-10-10T17:49:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 0x5614a581b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580f4f0 0x5614a58183d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a5818ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a5701960 con 0x5614a5813930 2023-10-10T17:49:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a5701600 con 0x5614a580ebe0 2023-10-10T17:49:22.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a55e39a0 con 0x5614a580f4f0 2023-10-10T17:49:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f373b484b80 10 client.0 _ll_register_callbacks cb 0x5614a571f800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a5818ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a5818ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47598/0 (socket says 172.21.15.78:47598) 2023-10-10T17:49:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 -- 172.21.15.78:0/2008799667 learned_addr learned my addr 172.21.15.78:0/2008799667 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372c893700 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580f4f0 0x5614a58183d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d094700 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 0x5614a581b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 -- 172.21.15.78:0/2008799667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580f4f0 msgr2=0x5614a58183d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580f4f0 0x5614a58183d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 -- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 msgr2=0x5614a581b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 0x5614a581b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.229+0000 7f372d895700 1 -- 172.21.15.78:0/2008799667 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37281101f0 con 0x5614a5813930 2023-10-10T17:49:22.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f372d895700 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a5818ac0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3720007920 tx=0x7f3720007c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a953fc8b6c851db4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f3710ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f3710ff9700 1 -- 172.21.15.78:0/2008799667 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720007e70 con 0x5614a5813930 2023-10-10T17:49:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f3710ff9700 1 -- 172.21.15.78:0/2008799667 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f3720010860 con 0x5614a5813930 2023-10-10T17:49:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5614a5801ba0 con 0x5614a5813930 2023-10-10T17:49:22.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f3710ff9700 1 -- 172.21.15.78:0/2008799667 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3720010a20 con 0x5614a5813930 2023-10-10T17:49:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 10 client.16896 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5614a5726210 con 0x5614a5813930 2023-10-10T17:49:22.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f3710ff9700 1 -- 172.21.15.78:0/2008799667 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5614a5801ba0 con 0x5614a5813930 2023-10-10T17:49:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f3710ff9700 1 -- 172.21.15.78:0/2008799667 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3720015cb0 con 0x5614a5813930 2023-10-10T17:49:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 10 client.16896 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 10 client.16896 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:22.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 20 client.16896 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a58289c0 0x5614a582add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 4 client.16896 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:22.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.230+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5614a5722120 con 0x5614a58289c0 2023-10-10T17:49:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.231+0000 7f372d094700 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a58289c0 0x5614a582add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.231+0000 7f3710ff9700 10 client.16896 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.231+0000 7f372d094700 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a58289c0 0x5614a582add0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9318e2514103827d server_cookie=44e2bacbef22b6e in_seq=0 out_seq=0 2023-10-10T17:49:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f3710ff9700 1 -- 172.21.15.78:0/2008799667 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5614a5722120 con 0x5614a58289c0 2023-10-10T17:49:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f3710ff9700 10 client.16896 handle_command_reply: tid=0 2023-10-10T17:49:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 client.16896 shutdown 2023-10-10T17:49:22.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 msgr2=0x5614a5818ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a5818ac0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3720007920 tx=0x7f3720007c30 comp rx=0 tx=0).stop 2023-10-10T17:49:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 shutdown_connections 2023-10-10T17:49:22.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a58289c0 0x5614a582add0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a5818ac0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f3710ff9700 0 client.16896 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 0x5614a581b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 --2- 172.21.15.78:0/2008799667 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580f4f0 0x5614a58183d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 >> 172.21.15.78:0/2008799667 conn(0x5614a58043e0 msgr2=0x5614a5814de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 shutdown_connections 2023-10-10T17:49:22.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 1 -- 172.21.15.78:0/2008799667 wait complete. 2023-10-10T17:49:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.232+0000 7f373b484b80 20 client.16896 trim_cache size 0 max 16384 2023-10-10T17:49:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 Processor -- start 2023-10-10T17:49:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 -- start start 2023-10-10T17:49:22.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 0x5614a5809c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a581acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a5813930 0x5614a5817be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a5803ff0 con 0x5614a580f4f0 2023-10-10T17:49:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a58190d0 con 0x5614a580ebe0 2023-10-10T17:49:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f373b484b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a5816980 con 0x5614a5813930 2023-10-10T17:49:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a581acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a581acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47604/0 (socket says 172.21.15.78:47604) 2023-10-10T17:49:22.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 -- 172.21.15.78:0/518909073 learned_addr learned my addr 172.21.15.78:0/518909073 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 -- 172.21.15.78:0/518909073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a5813930 msgr2=0x5614a5817be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 --2- 172.21.15.78:0/518909073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a5813930 0x5614a5817be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 -- 172.21.15.78:0/518909073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 msgr2=0x5614a5809c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 --2- 172.21.15.78:0/518909073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 0x5614a5809c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 -- 172.21.15.78:0/518909073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5614a5805b80 con 0x5614a580f4f0 2023-10-10T17:49:22.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f372c893700 1 --2- 172.21.15.78:0/518909073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a581acf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x5614a5814380 tx=0x7f3724000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2b89d44a1f808 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.249+0000 7f3713fff700 1 -- 172.21.15.78:0/518909073 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37240b8070 con 0x5614a580f4f0 2023-10-10T17:49:22.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f3713fff700 1 -- 172.21.15.78:0/518909073 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f37240964f0 con 0x5614a580f4f0 2023-10-10T17:49:22.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f3713fff700 1 -- 172.21.15.78:0/518909073 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3724099160 con 0x5614a580f4f0 2023-10-10T17:49:22.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 -- 172.21.15.78:0/518909073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 msgr2=0x5614a581acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 --2- 172.21.15.78:0/518909073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a581acf0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x5614a5814380 tx=0x7f3724000bc0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 -- 172.21.15.78:0/518909073 shutdown_connections 2023-10-10T17:49:22.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 --2- 172.21.15.78:0/518909073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a5813930 0x5614a5817be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 --2- 172.21.15.78:0/518909073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580ebe0 0x5614a5809c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 --2- 172.21.15.78:0/518909073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a580f4f0 0x5614a581acf0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 -- 172.21.15.78:0/518909073 >> 172.21.15.78:0/518909073 conn(0x5614a58043e0 msgr2=0x5614a5824d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 -- 172.21.15.78:0/518909073 shutdown_connections 2023-10-10T17:49:22.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 -- 172.21.15.78:0/518909073 wait complete. 2023-10-10T17:49:22.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.250+0000 7f373b484b80 1 -- start start 2023-10-10T17:49:22.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 1 Processor -- start 2023-10-10T17:49:22.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a581f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580f4f0 0x5614a581fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a581db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a581e260 con 0x5614a5813930 2023-10-10T17:49:22.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a581e3d0 con 0x5614a580f4f0 2023-10-10T17:49:22.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614a581e590 con 0x5614a580ebe0 2023-10-10T17:49:22.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f373b484b80 10 client.0 _ll_register_callbacks cb 0x7f3720007500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:22.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a581db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a581db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47612/0 (socket says 172.21.15.78:47612) 2023-10-10T17:49:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d094700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a581f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 -- 172.21.15.78:0/1536774759 learned_addr learned my addr 172.21.15.78:0/1536774759 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372c893700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580f4f0 0x5614a581fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 -- 172.21.15.78:0/1536774759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 msgr2=0x5614a581f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a581f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 -- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580f4f0 msgr2=0x5614a581fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580f4f0 0x5614a581fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 -- 172.21.15.78:0/1536774759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5614a55e16b0 con 0x5614a5813930 2023-10-10T17:49:22.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d094700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a581f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:22.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372c893700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580f4f0 0x5614a581fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:22.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f372d895700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a581db50 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f3720002230 tx=0x7f3720007f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511ff3b44cf3297d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.251+0000 7f3710ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:22.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f3710ff9700 1 -- 172.21.15.78:0/1536774759 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37200093b0 con 0x5614a5813930 2023-10-10T17:49:22.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f3710ff9700 1 -- 172.21.15.78:0/1536774759 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f372001d9a0 con 0x5614a5813930 2023-10-10T17:49:22.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3724096380 con 0x5614a5813930 2023-10-10T17:49:22.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f3710ff9700 1 -- 172.21.15.78:0/1536774759 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f372001d820 con 0x5614a5813930 2023-10-10T17:49:22.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 10 client.16902 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:22.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5614a558de60 con 0x5614a5813930 2023-10-10T17:49:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f3710ff9700 1 -- 172.21.15.78:0/1536774759 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3724096380 con 0x5614a5813930 2023-10-10T17:49:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f3710ff9700 1 -- 172.21.15.78:0/1536774759 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f372002e020 con 0x5614a5813930 2023-10-10T17:49:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 10 client.16902 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:22.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 10 client.16902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:22.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 20 client.16902 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:22.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a587dc20 0x5614a5880030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 4 client.16902 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:49:22.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5614a572e6e0 con 0x5614a587dc20 2023-10-10T17:49:22.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.252+0000 7f372d094700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a587dc20 0x5614a5880030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.253+0000 7f3710ff9700 10 client.16902 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.253+0000 7f372d094700 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a587dc20 0x5614a5880030 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71110ccde87cd9c5 server_cookie=47870ff74a37525d in_seq=0 out_seq=0 2023-10-10T17:49:22.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f3710ff9700 1 -- 172.21.15.78:0/1536774759 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5614a572e6e0 con 0x5614a587dc20 2023-10-10T17:49:22.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f3710ff9700 10 client.16902 handle_command_reply: tid=0 2023-10-10T17:49:22.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 client.16902 shutdown 2023-10-10T17:49:22.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 msgr2=0x5614a581db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a581db50 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f3720002230 tx=0x7f3720007f20 comp rx=0 tx=0).stop 2023-10-10T17:49:22.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 shutdown_connections 2023-10-10T17:49:22.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5614a580ebe0 0x5614a581f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5614a587dc20 0x5614a5880030 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5614a5813930 0x5614a581db50 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 --2- 172.21.15.78:0/1536774759 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5614a580f4f0 0x5614a581fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 >> 172.21.15.78:0/1536774759 conn(0x5614a58043e0 msgr2=0x5614a580d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f3710ff9700 0 client.16902 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 shutdown_connections 2023-10-10T17:49:22.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 1 -- 172.21.15.78:0/1536774759 wait complete. 2023-10-10T17:49:22.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.254+0000 7f373b484b80 20 client.16902 trim_cache size 0 max 16384 2023-10-10T17:49:22.380 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:49:22.381 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:49:22.381 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "d9710b7c-77e1-4308-8db0-5390ff61e8e8", 2023-10-10T17:49:22.381 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:49:22.381 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:49:22.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37140a12c0 msgr2=0x7f37140a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37140a12c0 0x7f37140a3770 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f371800b5c0 tx=0x7f3718005fd0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3728101400 msgr2=0x7f3728196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3728101400 0x7f3728196970 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f372400d950 tx=0x7f372400dc60 comp rx=0 tx=0).stop 2023-10-10T17:49:22.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 shutdown_connections 2023-10-10T17:49:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3728100a50 0x7f3728196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f37140a12c0 0x7f37140a3770 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f37281063f0 0x7f372819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 --2- 172.21.15.78:0/46602396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3728101400 0x7f3728196970 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 >> 172.21.15.78:0/46602396 conn(0x7f3728077590 msgr2=0x7f37280ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 shutdown_connections 2023-10-10T17:49:22.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.255+0000 7f372f2f8700 1 -- 172.21.15.78:0/46602396 wait complete. 2023-10-10T17:49:22.387 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:49:22.388 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2023-10-10T17:49:22.398 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2023-10-10T17:49:22.398 INFO:teuthology.task.exec:Executing custom commands... 2023-10-10T17:49:22.399 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:49:22.399 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2023-10-10T17:49:22.491 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2023-10-10T17:49:22.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 -- 172.21.15.78:0/3437737219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 msgr2=0x7efdec10f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 --2- 172.21.15.78:0/3437737219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec10f080 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7efde400b0a0 tx=0x7efde400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 -- 172.21.15.78:0/3437737219 shutdown_connections 2023-10-10T17:49:22.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 --2- 172.21.15.78:0/3437737219 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec0ff0f0 0x7efdec0ff4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 --2- 172.21.15.78:0/3437737219 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec10f080 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 --2- 172.21.15.78:0/3437737219 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdec0ffa00 0x7efdec10c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 -- 172.21.15.78:0/3437737219 >> 172.21.15.78:0/3437737219 conn(0x7efdec0770b0 msgr2=0x7efdec0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 -- 172.21.15.78:0/3437737219 shutdown_connections 2023-10-10T17:49:22.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.549+0000 7efdf1977700 1 -- 172.21.15.78:0/3437737219 wait complete. 2023-10-10T17:49:22.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 Processor -- start 2023-10-10T17:49:22.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 -- start start 2023-10-10T17:49:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdec0ff0f0 0x7efdec1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec0ffa00 0x7efdec198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec19bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19c5c0 con 0x7efdec10cca0 2023-10-10T17:49:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19c730 con 0x7efdec0ff0f0 2023-10-10T17:49:22.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdf1977700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdec19c8f0 con 0x7efdec0ffa00 2023-10-10T17:49:22.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdeb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec19bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdeb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec19bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47624/0 (socket says 172.21.15.78:47624) 2023-10-10T17:49:22.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdeb7fe700 1 -- 172.21.15.78:0/1492748636 learned_addr learned my addr 172.21.15.78:0/1492748636 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.550+0000 7efdeaffd700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdec0ff0f0 0x7efdec1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdea7fc700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec0ffa00 0x7efdec198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdeb7fe700 1 -- 172.21.15.78:0/1492748636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec0ffa00 msgr2=0x7efdec198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec0ffa00 0x7efdec198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdeb7fe700 1 -- 172.21.15.78:0/1492748636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdec0ff0f0 msgr2=0x7efdec1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdec0ff0f0 0x7efdec1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdeb7fe700 1 -- 172.21.15.78:0/1492748636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde4009d00 con 0x7efdec10cca0 2023-10-10T17:49:22.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec19bfc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7efde40062a0 tx=0x7efde40079b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdf0975700 1 -- 172.21.15.78:0/1492748636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde400e040 con 0x7efdec10cca0 2023-10-10T17:49:22.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdf0975700 1 -- 172.21.15.78:0/1492748636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efde4004d60 con 0x7efdec10cca0 2023-10-10T17:49:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdf0975700 1 -- 172.21.15.78:0/1492748636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde401d480 con 0x7efdec10cca0 2023-10-10T17:49:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec112520 con 0x7efdec10cca0 2023-10-10T17:49:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.551+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdec112390 con 0x7efdec10cca0 2023-10-10T17:49:22.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.554+0000 7efdf0975700 1 -- 172.21.15.78:0/1492748636 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7efde40040b0 con 0x7efdec10cca0 2023-10-10T17:49:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdf0975700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efdd40a12c0 0x7efdd40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdf0975700 1 -- 172.21.15.78:0/1492748636 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7efde40ce7c0 con 0x7efdec10cca0 2023-10-10T17:49:22.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdeaffd700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efdd40a12c0 0x7efdd40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 Processor -- start 2023-10-10T17:49:22.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 -- start start 2023-10-10T17:49:22.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1db60 0x55a690a1df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1e470 0x55a690a22370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a228b0 0x55a690a22d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690923060 con 0x55a690a1db60 2023-10-10T17:49:22.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6909758a0 con 0x55a690a1e470 2023-10-10T17:49:22.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.555+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a69096ed30 con 0x55a690a228b0 2023-10-10T17:49:22.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1db60 0x55a690a1df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1db60 0x55a690a1df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47628/0 (socket says 172.21.15.78:47628) 2023-10-10T17:49:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 -- 172.21.15.78:0/3694842742 learned_addr learned my addr 172.21.15.78:0/3694842742 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efdd40a12c0 0x7efdd40a3770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efddc005950 tx=0x7efddc00b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdea7fc700 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1e470 0x55a690a22370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 -- 172.21.15.78:0/3694842742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a228b0 msgr2=0x55a690a22d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a228b0 0x55a690a22d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 -- 172.21.15.78:0/3694842742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1e470 msgr2=0x55a690a22370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1e470 0x55a690a22370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 -- 172.21.15.78:0/3694842742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6907f0630 con 0x55a690a1db60 2023-10-10T17:49:22.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdea7fc700 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1e470 0x55a690a22370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdeaffd700 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1db60 0x55a690a1df30 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7efddc010fa0 tx=0x7efddc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8508459430012cc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdd1ffb700 1 -- 172.21.15.78:0/3694842742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc011e60 con 0x55a690a1db60 2023-10-10T17:49:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdd1ffb700 1 -- 172.21.15.78:0/3694842742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc015070 con 0x55a690a1db60 2023-10-10T17:49:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.556+0000 7efdd1ffb700 1 -- 172.21.15.78:0/3694842742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc0266f0 con 0x55a690a1db60 2023-10-10T17:49:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 -- 172.21.15.78:0/3694842742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1db60 msgr2=0x55a690a1df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1db60 0x55a690a1df30 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7efddc010fa0 tx=0x7efddc013000 comp rx=0 tx=0).stop 2023-10-10T17:49:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 -- 172.21.15.78:0/3694842742 shutdown_connections 2023-10-10T17:49:22.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a228b0 0x55a690a22d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1db60 0x55a690a1df30 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/3694842742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1e470 0x55a690a22370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 -- 172.21.15.78:0/3694842742 >> 172.21.15.78:0/3694842742 conn(0x55a690a13360 msgr2=0x55a690a18b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 -- 172.21.15.78:0/3694842742 shutdown_connections 2023-10-10T17:49:22.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.557+0000 7efdfdb03b80 1 -- 172.21.15.78:0/3694842742 wait complete. 2023-10-10T17:49:22.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 -- start start 2023-10-10T17:49:22.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 Processor -- start 2023-10-10T17:49:22.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a27310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a228b0 0x55a690a27a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6909108e0 con 0x55a690a228b0 2023-10-10T17:49:22.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690910580 con 0x55a690a1db60 2023-10-10T17:49:22.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6907f2920 con 0x55a690a1e470 2023-10-10T17:49:22.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdfdb03b80 10 client.0 _ll_register_callbacks cb 0x55a69092e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:22.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdeb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a228b0 0x55a690a27a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdeb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a228b0 0x55a690a27a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47638/0 (socket says 172.21.15.78:47638) 2023-10-10T17:49:22.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdeb7fe700 1 -- 172.21.15.78:0/514650413 learned_addr learned my addr 172.21.15.78:0/514650413 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdea7fc700 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a27310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.558+0000 7efdeaffd700 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdeb7fe700 1 -- 172.21.15.78:0/514650413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 msgr2=0x55a690a27310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a27310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdeb7fe700 1 -- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 msgr2=0x55a690a2a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdeb7fe700 1 -- 172.21.15.78:0/514650413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efddc010c50 con 0x55a690a228b0 2023-10-10T17:49:22.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a228b0 0x55a690a27a50 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efde4097a90 tx=0x7efde40972a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4e3ab50ca7cbf27 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdcaffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:22.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdcaffd700 1 -- 172.21.15.78:0/514650413 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde40ce610 con 0x55a690a228b0 2023-10-10T17:49:22.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdcaffd700 1 -- 172.21.15.78:0/514650413 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7efde4024360 con 0x55a690a228b0 2023-10-10T17:49:22.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a690a10b20 con 0x55a690a228b0 2023-10-10T17:49:22.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdcaffd700 1 -- 172.21.15.78:0/514650413 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7efde409aef0 con 0x55a690a228b0 2023-10-10T17:49:22.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdfdb03b80 10 client.16914 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:22.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a690935190 con 0x55a690a228b0 2023-10-10T17:49:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.559+0000 7efdcaffd700 1 -- 172.21.15.78:0/514650413 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a690a10b20 con 0x55a690a228b0 2023-10-10T17:49:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdcaffd700 1 -- 172.21.15.78:0/514650413 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7efde409aa80 con 0x55a690a228b0 2023-10-10T17:49:22.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdfdb03b80 10 client.16914 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdfdb03b80 10 client.16914 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdfdb03b80 20 client.16914 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a37730 0x55a690a39b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdfdb03b80 4 client.16914 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:22.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6909310a0 con 0x55a690a37730 2023-10-10T17:49:22.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdeaffd700 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a37730 0x55a690a39b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdcaffd700 10 client.16914 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.560+0000 7efdeaffd700 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a37730 0x55a690a39b40 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a675057dffb945b server_cookie=a142c5b6c87c3cdb in_seq=0 out_seq=0 2023-10-10T17:49:22.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdcaffd700 1 -- 172.21.15.78:0/514650413 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a6909310a0 con 0x55a690a37730 2023-10-10T17:49:22.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdcaffd700 10 client.16914 handle_command_reply: tid=0 2023-10-10T17:49:22.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 client.16914 shutdown 2023-10-10T17:49:22.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a228b0 msgr2=0x55a690a27a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a228b0 0x55a690a27a50 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7efde4097a90 tx=0x7efde40972a0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 shutdown_connections 2023-10-10T17:49:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a37730 0x55a690a39b40 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a228b0 0x55a690a27a50 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdcaffd700 0 client.16914 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/514650413 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a27310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 >> 172.21.15.78:0/514650413 conn(0x55a690a13360 msgr2=0x55a690a17de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 shutdown_connections 2023-10-10T17:49:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 1 -- 172.21.15.78:0/514650413 wait complete. 2023-10-10T17:49:22.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.561+0000 7efdfdb03b80 20 client.16914 trim_cache size 0 max 16384 2023-10-10T17:49:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 Processor -- start 2023-10-10T17:49:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 -- start start 2023-10-10T17:49:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a13360 0x55a690a2e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a29c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a2a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690a1ebc0 con 0x55a690a13360 2023-10-10T17:49:22.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690a2eef0 con 0x55a690a1db60 2023-10-10T17:49:22.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690a275a0 con 0x55a690a1e470 2023-10-10T17:49:22.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a13360 0x55a690a2e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdeaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a13360 0x55a690a2e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47642/0 (socket says 172.21.15.78:47642) 2023-10-10T17:49:22.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdeaffd700 1 -- 172.21.15.78:0/4006895148 learned_addr learned my addr 172.21.15.78:0/4006895148 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdea7fc700 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a29c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.578+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a2a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdeaffd700 1 -- 172.21.15.78:0/4006895148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 msgr2=0x55a690a2a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdeaffd700 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a2a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdeaffd700 1 -- 172.21.15.78:0/4006895148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 msgr2=0x55a690a29c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdeaffd700 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a29c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdeaffd700 1 -- 172.21.15.78:0/4006895148 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a690934ee0 con 0x55a690a13360 2023-10-10T17:49:22.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdeaffd700 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a13360 0x55a690a2e9b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efddc01ee60 tx=0x7efddc015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6589ffe477d795f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdd27fc700 1 -- 172.21.15.78:0/4006895148 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc02c070 con 0x55a690a13360 2023-10-10T17:49:22.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdd27fc700 1 -- 172.21.15.78:0/4006895148 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc02d040 con 0x55a690a13360 2023-10-10T17:49:22.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.579+0000 7efdd27fc700 1 -- 172.21.15.78:0/4006895148 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efddc004690 con 0x55a690a13360 2023-10-10T17:49:22.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- 172.21.15.78:0/4006895148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a13360 msgr2=0x55a690a2e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a13360 0x55a690a2e9b0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7efddc01ee60 tx=0x7efddc015000 comp rx=0 tx=0).stop 2023-10-10T17:49:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- 172.21.15.78:0/4006895148 shutdown_connections 2023-10-10T17:49:22.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a1e470 0x55a690a2a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a13360 0x55a690a2e9b0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/4006895148 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a29c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- 172.21.15.78:0/4006895148 >> 172.21.15.78:0/4006895148 conn(0x55a690a27bc0 msgr2=0x55a690a219e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- 172.21.15.78:0/4006895148 shutdown_connections 2023-10-10T17:49:22.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- 172.21.15.78:0/4006895148 wait complete. 2023-10-10T17:49:22.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- start start 2023-10-10T17:49:22.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 Processor -- start 2023-10-10T17:49:22.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a13360 0x55a690a2cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1e470 0x55a690a2d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690a2ed90 con 0x55a690a1e470 2023-10-10T17:49:22.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690a8b600 con 0x55a690a1db60 2023-10-10T17:49:22.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a690a8b770 con 0x55a690a13360 2023-10-10T17:49:22.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.580+0000 7efdfdb03b80 10 client.0 _ll_register_callbacks cb 0x55a69092e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:22.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1e470 0x55a690a2d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1e470 0x55a690a2d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47652/0 (socket says 172.21.15.78:47652) 2023-10-10T17:49:22.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 -- 172.21.15.78:0/2184029736 learned_addr learned my addr 172.21.15.78:0/2184029736 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:22.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeaffd700 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a13360 0x55a690a2cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdea7fc700 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 -- 172.21.15.78:0/2184029736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a13360 msgr2=0x55a690a2cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a13360 0x55a690a2cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 -- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 msgr2=0x55a690a2d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:22.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 -- 172.21.15.78:0/2184029736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6907f0630 con 0x55a690a1e470 2023-10-10T17:49:22.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdeb7fe700 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1e470 0x55a690a2d7c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7efde40cc180 tx=0x7efde409acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989548d0ff6c6778 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:22.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdcaffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:22.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdcaffd700 1 -- 172.21.15.78:0/2184029736 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde40ce610 con 0x55a690a1e470 2023-10-10T17:49:22.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdcaffd700 1 -- 172.21.15.78:0/2184029736 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7efde4024360 con 0x55a690a1e470 2023-10-10T17:49:22.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efddc015070 con 0x55a690a1e470 2023-10-10T17:49:22.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdcaffd700 1 -- 172.21.15.78:0/2184029736 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7efde409aef0 con 0x55a690a1e470 2023-10-10T17:49:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdfdb03b80 10 client.16920 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a690785c60 con 0x55a690a1e470 2023-10-10T17:49:22.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.581+0000 7efdcaffd700 1 -- 172.21.15.78:0/2184029736 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efddc015070 con 0x55a690a1e470 2023-10-10T17:49:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdcaffd700 1 -- 172.21.15.78:0/2184029736 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7efde409f120 con 0x55a690a1e470 2023-10-10T17:49:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdfdb03b80 10 client.16920 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:22.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdfdb03b80 10 client.16920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdfdb03b80 20 client.16920 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a228b0 0x55a690a8e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:22.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdfdb03b80 4 client.16920 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:49:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a69093d660 con 0x55a690a228b0 2023-10-10T17:49:22.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdeaffd700 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a228b0 0x55a690a8e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:22.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdcaffd700 10 client.16920 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdeaffd700 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a228b0 0x55a690a8e9c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3160427707d6d158 server_cookie=d7dd55faa8fdb978 in_seq=0 out_seq=0 2023-10-10T17:49:22.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdcaffd700 1 -- 172.21.15.78:0/2184029736 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a69093d660 con 0x55a690a228b0 2023-10-10T17:49:22.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdcaffd700 10 client.16920 handle_command_reply: tid=0 2023-10-10T17:49:22.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.582+0000 7efdfdb03b80 1 client.16920 shutdown 2023-10-10T17:49:22.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1e470 msgr2=0x55a690a2d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1e470 0x55a690a2d7c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7efde40cc180 tx=0x7efde409acb0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 shutdown_connections 2023-10-10T17:49:22.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a690a13360 0x55a690a2cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a690a228b0 0x55a690a8e9c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a690a1e470 0x55a690a2d7c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 --2- 172.21.15.78:0/2184029736 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a690a1db60 0x55a690a2d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 >> 172.21.15.78:0/2184029736 conn(0x55a690a27bc0 msgr2=0x55a690a1c960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdcaffd700 0 client.16920 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:22.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 shutdown_connections 2023-10-10T17:49:22.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 1 -- 172.21.15.78:0/2184029736 wait complete. 2023-10-10T17:49:22.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdfdb03b80 20 client.16920 trim_cache size 0 max 16384 2023-10-10T17:49:22.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:49:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:49:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:49:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: "d9710b7c-77e1-4308-8db0-5390ff61e8e8": { 2023-10-10T17:49:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:49:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "d9710b7c-77e1-4308-8db0-5390ff61e8e8", 2023-10-10T17:49:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:49:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:49:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:49:22.634 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:49:22.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efdd40a12c0 msgr2=0x7efdd40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efdd40a12c0 0x7efdd40a3770 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efddc005950 tx=0x7efddc00b410 comp rx=0 tx=0).stop 2023-10-10T17:49:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 msgr2=0x7efdec19bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:22.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec19bfc0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7efde40062a0 tx=0x7efde40079b0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 shutdown_connections 2023-10-10T17:49:22.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efdd40a12c0 0x7efdd40a3770 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efdec10cca0 0x7efdec19bfc0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efdec0ff0f0 0x7efdec1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 --2- 172.21.15.78:0/1492748636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efdec0ffa00 0x7efdec198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:22.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 >> 172.21.15.78:0/1492748636 conn(0x7efdec0770b0 msgr2=0x7efdec10df40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:22.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 shutdown_connections 2023-10-10T17:49:22.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:22.583+0000 7efdf1977700 1 -- 172.21.15.78:0/1492748636 wait complete. 2023-10-10T17:49:22.642 INFO:tasks.cephfs.filesystem:scrub status for tag:d9710b7c-77e1-4308-8db0-5390ff61e8e8 - {'path': '/', 'tag': 'd9710b7c-77e1-4308-8db0-5390ff61e8e8', 'options': 'recursive,force'} 2023-10-10T17:49:23.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:22 smithi078 ceph-mon[99494]: pgmap v210: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 KiB/s wr, 87 op/s 2023-10-10T17:49:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:22 smithi190 ceph-mon[101220]: pgmap v210: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 KiB/s wr, 87 op/s 2023-10-10T17:49:23.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:22 smithi157 ceph-mon[100896]: pgmap v210: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 12 KiB/s wr, 87 op/s 2023-10-10T17:49:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:23 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:49:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:23 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:23 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:49:24.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:23 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:24.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:23 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:49:24.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:23 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:24.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:23 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:49:24.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:23 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:24.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:23 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:49:24.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:23 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:24.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:23 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:49:24.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:23 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:25.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:24 smithi078 ceph-mon[99494]: pgmap v211: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2023-10-10T17:49:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:24 smithi190 ceph-mon[101220]: pgmap v211: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2023-10-10T17:49:25.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:24 smithi157 ceph-mon[100896]: pgmap v211: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2023-10-10T17:49:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:26 smithi078 ceph-mon[99494]: pgmap v212: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2023-10-10T17:49:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:26 smithi190 ceph-mon[101220]: pgmap v212: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2023-10-10T17:49:27.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:26 smithi157 ceph-mon[100896]: pgmap v212: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 8.0 KiB/s wr, 9 op/s 2023-10-10T17:49:28.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:27 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:49:28.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:27 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:49:28.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:27 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:49:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:28 smithi078 ceph-mon[99494]: pgmap v213: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 9 op/s 2023-10-10T17:49:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:28 smithi190 ceph-mon[101220]: pgmap v213: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 9 op/s 2023-10-10T17:49:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:28 smithi157 ceph-mon[100896]: pgmap v213: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 9 op/s 2023-10-10T17:49:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:30 smithi078 ceph-mon[99494]: pgmap v214: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2023-10-10T17:49:31.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:30 smithi190 ceph-mon[101220]: pgmap v214: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2023-10-10T17:49:31.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:30 smithi157 ceph-mon[100896]: pgmap v214: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2023-10-10T17:49:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:32 smithi078 ceph-mon[99494]: pgmap v215: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.0 KiB/s wr, 9 op/s 2023-10-10T17:49:33.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:32 smithi190 ceph-mon[101220]: pgmap v215: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.0 KiB/s wr, 9 op/s 2023-10-10T17:49:33.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:32 smithi157 ceph-mon[100896]: pgmap v215: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.0 KiB/s wr, 9 op/s 2023-10-10T17:49:35.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:34 smithi078 ceph-mon[99494]: pgmap v216: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2023-10-10T17:49:35.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:34 smithi190 ceph-mon[101220]: pgmap v216: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2023-10-10T17:49:35.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:34 smithi157 ceph-mon[100896]: pgmap v216: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.2 KiB/s wr, 10 op/s 2023-10-10T17:49:36.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:35 smithi078 ceph-mon[99494]: pgmap v217: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-10-10T17:49:36.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:35 smithi190 ceph-mon[101220]: pgmap v217: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-10-10T17:49:36.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:35 smithi157 ceph-mon[100896]: pgmap v217: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-10-10T17:49:38.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:38 smithi190 ceph-mon[101220]: pgmap v218: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2023-10-10T17:49:38.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:38 smithi157 ceph-mon[100896]: pgmap v218: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2023-10-10T17:49:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:38 smithi078 ceph-mon[99494]: pgmap v218: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.7 KiB/s wr, 9 op/s 2023-10-10T17:49:40.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:40 smithi157 ceph-mon[100896]: pgmap v219: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-10-10T17:49:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:40 smithi078 ceph-mon[99494]: pgmap v219: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-10-10T17:49:41.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:40 smithi190 ceph-mon[101220]: pgmap v219: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-10-10T17:49:42.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:42 smithi157 ceph-mon[100896]: pgmap v220: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-10-10T17:49:43.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:42 smithi078 ceph-mon[99494]: pgmap v220: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-10-10T17:49:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:42 smithi190 ceph-mon[101220]: pgmap v220: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-10-10T17:49:44.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:44 smithi157 ceph-mon[100896]: pgmap v221: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-10-10T17:49:45.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:44 smithi078 ceph-mon[99494]: pgmap v221: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-10-10T17:49:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:44 smithi190 ceph-mon[101220]: pgmap v221: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-10-10T17:49:46.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:46 smithi157 ceph-mon[100896]: pgmap v222: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-10-10T17:49:47.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:46 smithi078 ceph-mon[99494]: pgmap v222: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-10-10T17:49:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:46 smithi190 ceph-mon[101220]: pgmap v222: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-10-10T17:49:48.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:48 smithi157 ceph-mon[100896]: pgmap v223: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-10T17:49:49.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:48 smithi078 ceph-mon[99494]: pgmap v223: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-10T17:49:49.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:48 smithi190 ceph-mon[101220]: pgmap v223: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 7 op/s 2023-10-10T17:49:50.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:50 smithi157 ceph-mon[100896]: pgmap v224: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:49:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:50 smithi078 ceph-mon[99494]: pgmap v224: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:49:51.208 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:50 smithi190 ceph-mon[101220]: pgmap v224: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:49:52.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:49:52.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 -- 172.21.15.78:0/2251833698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981055f0 msgr2=0x7f299810d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 --2- 172.21.15.78:0/2251833698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981055f0 0x7f299810d1c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f299000b0d0 tx=0x7f299000b3e0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 -- 172.21.15.78:0/2251833698 shutdown_connections 2023-10-10T17:49:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 --2- 172.21.15.78:0/2251833698 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981055f0 0x7f299810d1c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 --2- 172.21.15.78:0/2251833698 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29981005b0 0x7f2998100980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 --2- 172.21.15.78:0/2251833698 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2998100f50 0x7f2998104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 -- 172.21.15.78:0/2251833698 >> 172.21.15.78:0/2251833698 conn(0x7f2998077170 msgr2=0x7f2998077570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 -- 172.21.15.78:0/2251833698 shutdown_connections 2023-10-10T17:49:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.787+0000 7f299e912700 1 -- 172.21.15.78:0/2251833698 wait complete. 2023-10-10T17:49:52.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 Processor -- start 2023-10-10T17:49:52.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 -- start start 2023-10-10T17:49:52.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981005b0 0x7f2998196500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2998100f50 0x7f2998196a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29981055f0 0x7f299819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299819b4e0 con 0x7f29981005b0 2023-10-10T17:49:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299819b650 con 0x7f29981055f0 2023-10-10T17:49:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.788+0000 7f299e912700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299819b810 con 0x7f2998100f50 2023-10-10T17:49:52.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981005b0 0x7f2998196500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981005b0 0x7f2998196500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47662/0 (socket says 172.21.15.78:47662) 2023-10-10T17:49:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 -- 172.21.15.78:0/2010791252 learned_addr learned my addr 172.21.15.78:0/2010791252 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:52.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f29977fe700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2998100f50 0x7f2998196a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f299ceaf700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29981055f0 0x7f299819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 -- 172.21.15.78:0/2010791252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2998100f50 msgr2=0x7f2998196a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2998100f50 0x7f2998196a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 -- 172.21.15.78:0/2010791252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29981055f0 msgr2=0x7f299819aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29981055f0 0x7f299819aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 -- 172.21.15.78:0/2010791252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2990009d00 con 0x7f29981005b0 2023-10-10T17:49:52.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f299ceaf700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29981055f0 0x7f299819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f2997fff700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981005b0 0x7f2998196500 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f298c00d950 tx=0x7f298c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f29957fa700 1 -- 172.21.15.78:0/2010791252 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c009940 con 0x7f29981005b0 2023-10-10T17:49:52.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29981103d0 con 0x7f29981005b0 2023-10-10T17:49:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f29957fa700 1 -- 172.21.15.78:0/2010791252 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f298c010460 con 0x7f29981005b0 2023-10-10T17:49:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f29957fa700 1 -- 172.21.15.78:0/2010791252 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c00f620 con 0x7f29981005b0 2023-10-10T17:49:52.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.789+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2998110240 con 0x7f29981005b0 2023-10-10T17:49:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.791+0000 7f29957fa700 1 -- 172.21.15.78:0/2010791252 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f298c0108d0 con 0x7f29981005b0 2023-10-10T17:49:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.792+0000 7f29957fa700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29800a12c0 0x7f29800a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.792+0000 7f29977fe700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29800a12c0 0x7f29800a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.792+0000 7f29957fa700 1 -- 172.21.15.78:0/2010791252 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f298c0ccf30 con 0x7f29981005b0 2023-10-10T17:49:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29977fe700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29800a12c0 0x7f29800a3770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2998197b10 tx=0x7f2988009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 Processor -- start 2023-10-10T17:49:52.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 -- start start 2023-10-10T17:49:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a15b60 0x561a11a15f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a1a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a1a8b0 0x561a11a1ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a1191b060 con 0x561a11a15b60 2023-10-10T17:49:52.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a1196d8a0 con 0x561a11a16470 2023-10-10T17:49:52.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.793+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11966d30 con 0x561a11a1a8b0 2023-10-10T17:49:52.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a15b60 0x561a11a15f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a15b60 0x561a11a15f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47670/0 (socket says 172.21.15.78:47670) 2023-10-10T17:49:52.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 -- 172.21.15.78:0/2066841833 learned_addr learned my addr 172.21.15.78:0/2066841833 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:52.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f29977fe700 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a1a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f299ceaf700 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a1a8b0 0x561a11a1ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 -- 172.21.15.78:0/2066841833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a1a8b0 msgr2=0x561a11a1ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a1a8b0 0x561a11a1ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 -- 172.21.15.78:0/2066841833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 msgr2=0x561a11a1a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a1a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 -- 172.21.15.78:0/2066841833 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a117e8630 con 0x561a11a15b60 2023-10-10T17:49:52.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f299ceaf700 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a1a8b0 0x561a11a1ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:52.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f2997fff700 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a15b60 0x561a11a15f30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f298c0a1b90 tx=0x7f298c099c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c16f303dd36d94 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:52.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f297e7fc700 1 -- 172.21.15.78:0/2066841833 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c096380 con 0x561a11a15b60 2023-10-10T17:49:52.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f297e7fc700 1 -- 172.21.15.78:0/2066841833 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f298c0964e0 con 0x561a11a15b60 2023-10-10T17:49:52.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.794+0000 7f297e7fc700 1 -- 172.21.15.78:0/2066841833 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298c0a1e90 con 0x561a11a15b60 2023-10-10T17:49:52.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/2066841833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a15b60 msgr2=0x561a11a15f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a15b60 0x561a11a15f30 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f298c0a1b90 tx=0x7f298c099c70 comp rx=0 tx=0).stop 2023-10-10T17:49:52.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/2066841833 shutdown_connections 2023-10-10T17:49:52.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a1a8b0 0x561a11a1ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a15b60 0x561a11a15f30 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/2066841833 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a1a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/2066841833 >> 172.21.15.78:0/2066841833 conn(0x561a11a0b360 msgr2=0x561a11a10b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/2066841833 shutdown_connections 2023-10-10T17:49:52.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/2066841833 wait complete. 2023-10-10T17:49:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.795+0000 7f29aaa9eb80 1 -- start start 2023-10-10T17:49:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 1 Processor -- start 2023-10-10T17:49:52.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a1f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a1f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a1fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a119088e0 con 0x561a11a1a8b0 2023-10-10T17:49:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11908580 con 0x561a11a15b60 2023-10-10T17:49:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a117ea920 con 0x561a11a16470 2023-10-10T17:49:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29aaa9eb80 10 client.0 _ll_register_callbacks cb 0x561a11926780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f299ceaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a1fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f299ceaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a1fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47680/0 (socket says 172.21.15.78:47680) 2023-10-10T17:49:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f29977fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a1f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f299ceaf700 1 -- 172.21.15.78:0/3996254052 learned_addr learned my addr 172.21.15.78:0/3996254052 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f2997fff700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a1f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f299ceaf700 1 -- 172.21.15.78:0/3996254052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 msgr2=0x561a11a1f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f299ceaf700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a1f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f299ceaf700 1 -- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 msgr2=0x561a11a1f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.796+0000 7f299ceaf700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a1f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f299ceaf700 1 -- 172.21.15.78:0/3996254052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2998110240 con 0x561a11a1a8b0 2023-10-10T17:49:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f2997fff700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a1f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29977fe700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a1f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:49:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f299ceaf700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a1fe10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f299000bdf0 tx=0x7f299000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349248830d64b817 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29777fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29777fe700 1 -- 172.21.15.78:0/3996254052 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299000e040 con 0x561a11a1a8b0 2023-10-10T17:49:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29777fe700 1 -- 172.21.15.78:0/3996254052 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f2990004d60 con 0x561a11a1a8b0 2023-10-10T17:49:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a11a08b20 con 0x561a11a1a8b0 2023-10-10T17:49:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29777fe700 1 -- 172.21.15.78:0/3996254052 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29900074e0 con 0x561a11a1a8b0 2023-10-10T17:49:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 10 client.16932 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561a1192d190 con 0x561a11a1a8b0 2023-10-10T17:49:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29777fe700 1 -- 172.21.15.78:0/3996254052 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561a11a08b20 con 0x561a11a1a8b0 2023-10-10T17:49:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29777fe700 1 -- 172.21.15.78:0/3996254052 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f2990019070 con 0x561a11a1a8b0 2023-10-10T17:49:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 10 client.16932 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 10 client.16932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 20 client.16932 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a2f8f0 0x561a11a31d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 4 client.16932 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.797+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a119290a0 con 0x561a11a2f8f0 2023-10-10T17:49:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.798+0000 7f2997fff700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a2f8f0 0x561a11a31d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.798+0000 7f2997fff700 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a2f8f0 0x561a11a31d00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d82bd29ce977ba7 server_cookie=8d81a86d40acd333 in_seq=0 out_seq=0 2023-10-10T17:49:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.798+0000 7f29777fe700 10 client.16932 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29777fe700 1 -- 172.21.15.78:0/3996254052 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561a119290a0 con 0x561a11a2f8f0 2023-10-10T17:49:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29777fe700 10 client.16932 handle_command_reply: tid=0 2023-10-10T17:49:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 client.16932 shutdown 2023-10-10T17:49:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 msgr2=0x561a11a1fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a1fe10 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f299000bdf0 tx=0x7f299000be20 comp rx=0 tx=0).stop 2023-10-10T17:49:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 shutdown_connections 2023-10-10T17:49:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a2f8f0 0x561a11a31d00 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a1fe10 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29777fe700 0 client.16932 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a1f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/3996254052 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a1f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 >> 172.21.15.78:0/3996254052 conn(0x561a11a0b360 msgr2=0x561a11a1bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 shutdown_connections 2023-10-10T17:49:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.799+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/3996254052 wait complete. 2023-10-10T17:49:52.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.800+0000 7f29aaa9eb80 20 client.16932 trim_cache size 0 max 16384 2023-10-10T17:49:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 Processor -- start 2023-10-10T17:49:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 -- start start 2023-10-10T17:49:52.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a26940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a19930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a19e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11a2cf50 con 0x561a11a1a8b0 2023-10-10T17:49:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11a16bc0 con 0x561a11a15b60 2023-10-10T17:49:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.816+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11a2c300 con 0x561a11a16470 2023-10-10T17:49:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a19e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a19e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47688/0 (socket says 172.21.15.78:47688) 2023-10-10T17:49:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 -- 172.21.15.78:0/1051890087 learned_addr learned my addr 172.21.15.78:0/1051890087 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f2997fff700 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a26940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 -- 172.21.15.78:0/1051890087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 msgr2=0x561a11a19930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f29977fe700 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a19930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a19930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 -- 172.21.15.78:0/1051890087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 msgr2=0x561a11a26940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a26940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 -- 172.21.15.78:0/1051890087 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a11a0cb00 con 0x561a11a1a8b0 2023-10-10T17:49:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f299ceaf700 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a19e70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2990009fd0 tx=0x7f299001cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a3dfb0bed79788 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f297effd700 1 -- 172.21.15.78:0/1051890087 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299000e040 con 0x561a11a1a8b0 2023-10-10T17:49:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f297effd700 1 -- 172.21.15.78:0/1051890087 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2990007960 con 0x561a11a1a8b0 2023-10-10T17:49:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.817+0000 7f297effd700 1 -- 172.21.15.78:0/1051890087 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299001c9e0 con 0x561a11a1a8b0 2023-10-10T17:49:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1051890087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 msgr2=0x561a11a19e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a19e70 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2990009fd0 tx=0x7f299001cee0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1051890087 shutdown_connections 2023-10-10T17:49:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a19e70 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a15b60 0x561a11a26940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1051890087 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a16470 0x561a11a19930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1051890087 >> 172.21.15.78:0/1051890087 conn(0x561a11a0b360 msgr2=0x561a11a18a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1051890087 shutdown_connections 2023-10-10T17:49:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1051890087 wait complete. 2023-10-10T17:49:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 -- start start 2023-10-10T17:49:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.818+0000 7f29aaa9eb80 1 Processor -- start 2023-10-10T17:49:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a15b60 0x561a11a22920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a22e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29aaa9eb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a233a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11a26f40 con 0x561a11a1a8b0 2023-10-10T17:49:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11a827a0 con 0x561a11a16470 2023-10-10T17:49:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29aaa9eb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a11a82910 con 0x561a11a15b60 2023-10-10T17:49:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29aaa9eb80 10 client.0 _ll_register_callbacks cb 0x7f2988005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a233a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a233a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47694/0 (socket says 172.21.15.78:47694) 2023-10-10T17:49:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 -- 172.21.15.78:0/1792248131 learned_addr learned my addr 172.21.15.78:0/1792248131 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f2997fff700 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a15b60 0x561a11a22920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29977fe700 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a22e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 -- 172.21.15.78:0/1792248131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a15b60 msgr2=0x561a11a22920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a15b60 0x561a11a22920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 -- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 msgr2=0x561a11a22e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a22e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 -- 172.21.15.78:0/1792248131 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a1192d190 con 0x561a11a1a8b0 2023-10-10T17:49:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f299ceaf700 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a233a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f299000bae0 tx=0x7f2990005200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cf77b1e63abd7a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.819+0000 7f29777fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29777fe700 1 -- 172.21.15.78:0/1792248131 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299000e040 con 0x561a11a1a8b0 2023-10-10T17:49:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29777fe700 1 -- 172.21.15.78:0/1792248131 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f29900076b0 con 0x561a11a1a8b0 2023-10-10T17:49:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f298c0964e0 con 0x561a11a1a8b0 2023-10-10T17:49:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29777fe700 1 -- 172.21.15.78:0/1792248131 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f299001c6a0 con 0x561a11a1a8b0 2023-10-10T17:49:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 10 client.16938 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561a11794de0 con 0x561a11a1a8b0 2023-10-10T17:49:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29777fe700 1 -- 172.21.15.78:0/1792248131 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f298c0964e0 con 0x561a11a1a8b0 2023-10-10T17:49:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29777fe700 1 -- 172.21.15.78:0/1792248131 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f2990019070 con 0x561a11a1a8b0 2023-10-10T17:49:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 10 client.16938 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 10 client.16938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 20 client.16938 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a83b60 0x561a11a85f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 4 client.16938 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:49:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561a11935660 con 0x561a11a83b60 2023-10-10T17:49:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.820+0000 7f2997fff700 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a83b60 0x561a11a85f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29777fe700 10 client.16938 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f2997fff700 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a83b60 0x561a11a85f40 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be81d80a0318617e server_cookie=b62c4c99540378ed in_seq=0 out_seq=0 2023-10-10T17:49:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29777fe700 1 -- 172.21.15.78:0/1792248131 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561a11935660 con 0x561a11a83b60 2023-10-10T17:49:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29777fe700 10 client.16938 handle_command_reply: tid=0 2023-10-10T17:49:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 client.16938 shutdown 2023-10-10T17:49:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 msgr2=0x561a11a233a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a233a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f299000bae0 tx=0x7f2990005200 comp rx=0 tx=0).stop 2023-10-10T17:49:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 shutdown_connections 2023-10-10T17:49:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a11a15b60 0x561a11a22920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a11a83b60 0x561a11a85f40 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a11a1a8b0 0x561a11a233a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 --2- 172.21.15.78:0/1792248131 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a11a16470 0x561a11a22e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29777fe700 0 client.16938 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 >> 172.21.15.78:0/1792248131 conn(0x561a11a0b360 msgr2=0x561a11a1f950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:52.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 shutdown_connections 2023-10-10T17:49:52.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 1 -- 172.21.15.78:0/1792248131 wait complete. 2023-10-10T17:49:52.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.821+0000 7f29aaa9eb80 20 client.16938 trim_cache size 0 max 16384 2023-10-10T17:49:52.868 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:49:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:49:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:49:52.868 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:49:52.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29800a12c0 msgr2=0x7f29800a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29800a12c0 0x7f29800a3770 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2998197b10 tx=0x7f2988009260 comp rx=0 tx=0).stop 2023-10-10T17:49:52.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981005b0 msgr2=0x7f2998196500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981005b0 0x7f2998196500 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f298c00d950 tx=0x7f298c00dc60 comp rx=0 tx=0).stop 2023-10-10T17:49:52.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 shutdown_connections 2023-10-10T17:49:52.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29800a12c0 0x7f29800a3770 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f29981055f0 0x7f299819aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f29981005b0 0x7f2998196500 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 --2- 172.21.15.78:0/2010791252 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2998100f50 0x7f2998196a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 >> 172.21.15.78:0/2010791252 conn(0x7f2998077170 msgr2=0x7f29980fdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:52.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 shutdown_connections 2023-10-10T17:49:52.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.822+0000 7f299e912700 1 -- 172.21.15.78:0/2010791252 wait complete. 2023-10-10T17:49:52.873 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:49:52.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:49:52.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:52 smithi157 ceph-mon[100896]: pgmap v225: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:49:52.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 -- 172.21.15.78:0/3936535731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 msgr2=0x7f2290103d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:52.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 --2- 172.21.15.78:0/3936535731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f2290103d80 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f22800099c0 tx=0x7f2280009cd0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 -- 172.21.15.78:0/3936535731 shutdown_connections 2023-10-10T17:49:52.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 --2- 172.21.15.78:0/3936535731 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2290104460 0x7f229010c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 --2- 172.21.15.78:0/3936535731 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22900ff420 0x7f22900ff7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 --2- 172.21.15.78:0/3936535731 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f2290103d80 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:52.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 -- 172.21.15.78:0/3936535731 >> 172.21.15.78:0/3936535731 conn(0x7f2290076310 msgr2=0x7f2290076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:52.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 -- 172.21.15.78:0/3936535731 shutdown_connections 2023-10-10T17:49:52.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.994+0000 7f2295f88700 1 -- 172.21.15.78:0/3936535731 wait complete. 2023-10-10T17:49:53.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 Processor -- start 2023-10-10T17:49:53.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 -- start start 2023-10-10T17:49:53.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22900ff420 0x7f2290195380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f22901958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2290104460 0x7f2290199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229019a2c0 con 0x7f22900ffdc0 2023-10-10T17:49:53.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229019a430 con 0x7f22900ff420 2023-10-10T17:49:53.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.995+0000 7f2295f88700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f229019a5f0 con 0x7f2290104460 2023-10-10T17:49:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f22901958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f22901958c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47706/0 (socket says 172.21.15.78:47706) 2023-10-10T17:49:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 -- 172.21.15.78:0/188333636 learned_addr learned my addr 172.21.15.78:0/188333636 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228f7fe700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22900ff420 0x7f2290195380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 -- 172.21.15.78:0/188333636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2290104460 msgr2=0x7f2290199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2290104460 0x7f2290199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 -- 172.21.15.78:0/188333636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22900ff420 msgr2=0x7f2290195380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22900ff420 0x7f2290195380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 -- 172.21.15.78:0/188333636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800095f0 con 0x7f22900ffdc0 2023-10-10T17:49:53.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228f7fe700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22900ff420 0x7f2290195380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:53.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228effd700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f22901958c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2280009990 tx=0x7f2280009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228cff9700 1 -- 172.21.15.78:0/188333636 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228001f070 con 0x7f22900ffdc0 2023-10-10T17:49:53.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f229010f230 con 0x7f22900ffdc0 2023-10-10T17:49:53.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228cff9700 1 -- 172.21.15.78:0/188333636 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f228001ab50 con 0x7f22900ffdc0 2023-10-10T17:49:53.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f228cff9700 1 -- 172.21.15.78:0/188333636 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2280004150 con 0x7f22900ffdc0 2023-10-10T17:49:53.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.996+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f229010f0a0 con 0x7f22900ffdc0 2023-10-10T17:49:53.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.997+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f229010f3c0 con 0x7f22900ffdc0 2023-10-10T17:49:53.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.998+0000 7f228cff9700 1 -- 172.21.15.78:0/188333636 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f2280023070 con 0x7f22900ffdc0 2023-10-10T17:49:53.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.999+0000 7f228cff9700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f22780a12c0 0x7f22780a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.999+0000 7f228f7fe700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f22780a12c0 0x7f22780a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.999+0000 7f228cff9700 1 -- 172.21.15.78:0/188333636 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f22800d0590 con 0x7f22900ffdc0 2023-10-10T17:49:53.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:52.999+0000 7f228f7fe700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f22780a12c0 0x7f22780a3770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2284005fd0 tx=0x7f2284005dc0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.002+0000 7f228cff9700 1 -- 172.21.15.78:0/188333636 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2280099640 con 0x7f22900ffdc0 2023-10-10T17:49:53.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:52 smithi078 ceph-mon[99494]: pgmap v225: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:49:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:52 smithi190 ceph-mon[101220]: pgmap v225: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:49:53.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.237+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2290067a60 con 0x7f22900ffdc0 2023-10-10T17:49:53.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.238+0000 7f228cff9700 1 -- 172.21.15.78:0/188333636 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10817 (secure 0 0 0) 0x7f228009cc60 con 0x7f22900ffdc0 2023-10-10T17:49:53.240 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:49:53.240 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:46:23.648656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:49:53.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.239+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f22780a12c0 msgr2=0x7f22780a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f22780a12c0 0x7f22780a3770 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2284005fd0 tx=0x7f2284005dc0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 msgr2=0x7f22901958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f22901958c0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f2280009990 tx=0x7f2280009580 comp rx=0 tx=0).stop 2023-10-10T17:49:53.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 shutdown_connections 2023-10-10T17:49:53.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2290104460 0x7f2290199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f22780a12c0 0x7f22780a3770 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22900ff420 0x7f2290195380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 --2- 172.21.15.78:0/188333636 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22900ffdc0 0x7f22901958c0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 >> 172.21.15.78:0/188333636 conn(0x7f2290076310 msgr2=0x7f22900fca60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 shutdown_connections 2023-10-10T17:49:53.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.240+0000 7f2295f88700 1 -- 172.21.15.78:0/188333636 wait complete. 2023-10-10T17:49:53.247 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2023-10-10T17:49:53.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:49:53.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 -- 172.21.15.78:0/2586391610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e81005d0 msgr2=0x7ff5e8103e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 --2- 172.21.15.78:0/2586391610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e81005d0 0x7ff5e8103e60 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40099c0 tx=0x7ff5e4009cd0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 -- 172.21.15.78:0/2586391610 shutdown_connections 2023-10-10T17:49:53.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 --2- 172.21.15.78:0/2586391610 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5e8104590 0x7ff5e810c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 --2- 172.21.15.78:0/2586391610 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e80ffc30 0x7ff5e8100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 --2- 172.21.15.78:0/2586391610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e81005d0 0x7ff5e8103e60 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 -- 172.21.15.78:0/2586391610 >> 172.21.15.78:0/2586391610 conn(0x7ff5e8075470 msgr2=0x7ff5e8075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 -- 172.21.15.78:0/2586391610 shutdown_connections 2023-10-10T17:49:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.375+0000 7ff5f075f700 1 -- 172.21.15.78:0/2586391610 wait complete. 2023-10-10T17:49:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 Processor -- start 2023-10-10T17:49:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 -- start start 2023-10-10T17:49:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e80ffc30 0x7ff5e8195370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e81005d0 0x7ff5e81958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5e8104590 0x7ff5e8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e819a350 con 0x7ff5e80ffc30 2023-10-10T17:49:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e819a4c0 con 0x7ff5e81005d0 2023-10-10T17:49:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5f075f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e819a680 con 0x7ff5e8104590 2023-10-10T17:49:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5ee4fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e80ffc30 0x7ff5e8195370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5ee4fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e80ffc30 0x7ff5e8195370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47718/0 (socket says 172.21.15.78:47718) 2023-10-10T17:49:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/140172650 learned_addr learned my addr 172.21.15.78:0/140172650 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e81005d0 0x7ff5e81958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.376+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5e8104590 0x7ff5e8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/140172650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5e8104590 msgr2=0x7ff5e8199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5e8104590 0x7ff5e8199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/140172650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e81005d0 msgr2=0x7ff5e81958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e81005d0 0x7ff5e81958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/140172650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e40095f0 con 0x7ff5e80ffc30 2023-10-10T17:49:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e80ffc30 0x7ff5e8195370 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c920 tx=0x7ff5d800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5df7fe700 1 -- 172.21.15.78:0/140172650 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8004a30 con 0x7ff5e80ffc30 2023-10-10T17:49:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5df7fe700 1 -- 172.21.15.78:0/140172650 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5d80054e0 con 0x7ff5e80ffc30 2023-10-10T17:49:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e810f310 con 0x7ff5e80ffc30 2023-10-10T17:49:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5df7fe700 1 -- 172.21.15.78:0/140172650 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8007bd0 con 0x7ff5e80ffc30 2023-10-10T17:49:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.377+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e810f180 con 0x7ff5e80ffc30 2023-10-10T17:49:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.379+0000 7ff5df7fe700 1 -- 172.21.15.78:0/140172650 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff5d8005000 con 0x7ff5e80ffc30 2023-10-10T17:49:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.380+0000 7ff5df7fe700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff5d40a1340 0x7ff5d40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.380+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff5d40a1340 0x7ff5d40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.380+0000 7ff5df7fe700 1 -- 172.21.15.78:0/140172650 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7ff5d80ccb40 con 0x7ff5e80ffc30 2023-10-10T17:49:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.380+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff5d40a1340 0x7ff5d40a37f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40098e0 tx=0x7ff5e4009f10 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 Processor -- start 2023-10-10T17:49:53.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 -- start start 2023-10-10T17:49:53.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e835bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835c470 0x5556e8360370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8360d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e8261060 con 0x5556e835c470 2023-10-10T17:49:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e82b38a0 con 0x5556e835bb60 2023-10-10T17:49:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e82acd30 con 0x5556e83608b0 2023-10-10T17:49:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.381+0000 7ff5edcfa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835c470 0x5556e8360370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835c470 0x5556e8360370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47728/0 (socket says 172.21.15.78:47728) 2023-10-10T17:49:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 -- 172.21.15.78:0/194551839 learned_addr learned my addr 172.21.15.78:0/194551839 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e835bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8360d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 -- 172.21.15.78:0/194551839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 msgr2=0x5556e8360d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8360d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 -- 172.21.15.78:0/194551839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 msgr2=0x5556e835bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e835bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 -- 172.21.15.78:0/194551839 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556e812e630 con 0x5556e835c470 2023-10-10T17:49:53.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835c470 0x5556e8360370 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40099c0 tx=0x7ff5e4020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f62cd74c1b3bbac server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5dcff9700 1 -- 172.21.15.78:0/194551839 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e400bdf0 con 0x5556e835c470 2023-10-10T17:49:53.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5dcff9700 1 -- 172.21.15.78:0/194551839 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5e401e040 con 0x5556e835c470 2023-10-10T17:49:53.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.382+0000 7ff5dcff9700 1 -- 172.21.15.78:0/194551839 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e4004760 con 0x5556e835c470 2023-10-10T17:49:53.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/194551839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835c470 msgr2=0x5556e8360370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835c470 0x5556e8360370 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40099c0 tx=0x7ff5e4020000 comp rx=0 tx=0).stop 2023-10-10T17:49:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/194551839 shutdown_connections 2023-10-10T17:49:53.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8360d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e835bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/194551839 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835c470 0x5556e8360370 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/194551839 >> 172.21.15.78:0/194551839 conn(0x5556e8351360 msgr2=0x5556e8356b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/194551839 shutdown_connections 2023-10-10T17:49:53.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/194551839 wait complete. 2023-10-10T17:49:53.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 -- start start 2023-10-10T17:49:53.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.383+0000 7ff5fc8ebb80 1 Processor -- start 2023-10-10T17:49:53.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e83680f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e8365300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e83608b0 0x5556e8365a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e824e8e0 con 0x5556e83608b0 2023-10-10T17:49:53.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e824e580 con 0x5556e835bb60 2023-10-10T17:49:53.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e8130920 con 0x5556e835c470 2023-10-10T17:49:53.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5fc8ebb80 10 client.0 _ll_register_callbacks cb 0x5556e826c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:53.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e83608b0 0x5556e8365a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e83608b0 0x5556e8365a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47736/0 (socket says 172.21.15.78:47736) 2023-10-10T17:49:53.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 -- 172.21.15.78:0/1030836374 learned_addr learned my addr 172.21.15.78:0/1030836374 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e8365300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e83680f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 -- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 msgr2=0x5556e8365300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e8365300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 -- 172.21.15.78:0/1030836374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 msgr2=0x5556e83680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e83680f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 -- 172.21.15.78:0/1030836374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e810c950 con 0x5556e83608b0 2023-10-10T17:49:53.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e8365300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:53.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e83608b0 0x5556e8365a40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009f40 tx=0x7ff5e00043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7a00605378e0a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5d1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.384+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/1030836374 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e00077d0 con 0x5556e83608b0 2023-10-10T17:49:53.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/1030836374 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7ff5e0007930 con 0x5556e83608b0 2023-10-10T17:49:53.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556e834eb20 con 0x5556e83608b0 2023-10-10T17:49:53.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/1030836374 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5e0007af0 con 0x5556e83608b0 2023-10-10T17:49:53.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 10 client.16956 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:53.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5556e8273190 con 0x5556e83608b0 2023-10-10T17:49:53.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/1030836374 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5556e834eb20 con 0x5556e83608b0 2023-10-10T17:49:53.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/1030836374 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff5e0014070 con 0x5556e83608b0 2023-10-10T17:49:53.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 10 client.16956 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:53.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 10 client.16956 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:49:53.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 20 client.16956 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:53.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83759f0 0x5556e8377e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 4 client.16956 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:53.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556e826f0a0 con 0x5556e83759f0 2023-10-10T17:49:53.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.385+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83759f0 0x5556e8377e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.386+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83759f0 0x5556e8377e00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f32b5a3b79dfd2 server_cookie=e9dfcfb6a2003049 in_seq=0 out_seq=0 2023-10-10T17:49:53.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.386+0000 7ff5d1ffb700 10 client.16956 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:53.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/1030836374 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5556e826f0a0 con 0x5556e83759f0 2023-10-10T17:49:53.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5d1ffb700 10 client.16956 handle_command_reply: tid=0 2023-10-10T17:49:53.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 client.16956 shutdown 2023-10-10T17:49:53.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e83608b0 msgr2=0x5556e8365a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e83608b0 0x5556e8365a40 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff5e0009f40 tx=0x7ff5e00043d0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 shutdown_connections 2023-10-10T17:49:53.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e83608b0 0x5556e8365a40 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835bb60 0x5556e83680f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83759f0 0x5556e8377e00 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/1030836374 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e8365300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5d1ffb700 0 client.16956 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:53.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 >> 172.21.15.78:0/1030836374 conn(0x5556e8351360 msgr2=0x5556e8355de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 shutdown_connections 2023-10-10T17:49:53.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/1030836374 wait complete. 2023-10-10T17:49:53.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.387+0000 7ff5fc8ebb80 20 client.16956 trim_cache size 0 max 16384 2023-10-10T17:49:53.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 Processor -- start 2023-10-10T17:49:53.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 -- start start 2023-10-10T17:49:53.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8362dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e836c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e83608b0 0x5556e835a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e834ea10 con 0x5556e835bb60 2023-10-10T17:49:53.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e8372f50 con 0x5556e83608b0 2023-10-10T17:49:53.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e8363300 con 0x5556e835c470 2023-10-10T17:49:53.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8362dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8362dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47740/0 (socket says 172.21.15.78:47740) 2023-10-10T17:49:53.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/4140681897 learned_addr learned my addr 172.21.15.78:0/4140681897 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e836c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/4140681897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 msgr2=0x5556e836c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e836c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/4140681897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e83608b0 msgr2=0x5556e835a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e83608b0 0x5556e835a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e83608b0 0x5556e835a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/4140681897 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556e8364e70 con 0x5556e835bb60 2023-10-10T17:49:53.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e83608b0 0x5556e835a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:53.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8362dc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c8f0 tx=0x7ff5d80cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64254d13093aaf76 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.400+0000 7ff5dd7fa700 1 -- 172.21.15.78:0/4140681897 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d809d8b0 con 0x5556e835bb60 2023-10-10T17:49:53.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5dd7fa700 1 -- 172.21.15.78:0/4140681897 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5d80a1cd0 con 0x5556e835bb60 2023-10-10T17:49:53.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5dd7fa700 1 -- 172.21.15.78:0/4140681897 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8095f90 con 0x5556e835bb60 2023-10-10T17:49:53.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/4140681897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 msgr2=0x5556e8362dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8362dc0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c8f0 tx=0x7ff5d80cc040 comp rx=0 tx=0).stop 2023-10-10T17:49:53.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/4140681897 shutdown_connections 2023-10-10T17:49:53.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e83608b0 0x5556e835a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8362dc0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/4140681897 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e835c470 0x5556e836c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/4140681897 >> 172.21.15.78:0/4140681897 conn(0x5556e8351360 msgr2=0x5556e835e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/4140681897 shutdown_connections 2023-10-10T17:49:53.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/4140681897 wait complete. 2023-10-10T17:49:53.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 -- start start 2023-10-10T17:49:53.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.401+0000 7ff5fc8ebb80 1 Processor -- start 2023-10-10T17:49:53.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8355550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835c470 0x5556e8355a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8355fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e83566a0 con 0x5556e835bb60 2023-10-10T17:49:53.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e8356810 con 0x5556e835c470 2023-10-10T17:49:53.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556e83569d0 con 0x5556e83608b0 2023-10-10T17:49:53.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 10 client.0 _ll_register_callbacks cb 0x5556e826c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:53.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8355550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8355550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47746/0 (socket says 172.21.15.78:47746) 2023-10-10T17:49:53.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/3485388349 learned_addr learned my addr 172.21.15.78:0/3485388349 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835c470 0x5556e8355a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8355fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 msgr2=0x5556e8355fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8355fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/3485388349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835c470 msgr2=0x5556e8355a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835c470 0x5556e8355a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 -- 172.21.15.78:0/3485388349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e810f180 con 0x5556e835bb60 2023-10-10T17:49:53.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5eecfc700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8355fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:53.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5ee4fb700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8355550 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80a17a0 tx=0x7ff5d8097e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff370d8cf1dce9ae server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5d1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:53.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/3485388349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8007d50 con 0x5556e835bb60 2023-10-10T17:49:53.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/3485388349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7ff5d8095f90 con 0x5556e835bb60 2023-10-10T17:49:53.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5e401e040 con 0x5556e835bb60 2023-10-10T17:49:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.402+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/3485388349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5d80b7a10 con 0x5556e835bb60 2023-10-10T17:49:53.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 10 client.16962 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:53.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5556e80dade0 con 0x5556e835bb60 2023-10-10T17:49:53.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/3485388349 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff5e401e040 con 0x5556e835bb60 2023-10-10T17:49:53.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/3485388349 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ff5d8099c00 con 0x5556e835bb60 2023-10-10T17:49:53.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 10 client.16962 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:53.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 10 client.16962 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:49:53.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 20 client.16962 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:53.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83ca240 0x5556e83cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 4 client.16962 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:49:53.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5556e827b660 con 0x5556e83ca240 2023-10-10T17:49:53.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83ca240 0x5556e83cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5d1ffb700 10 client.16962 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.403+0000 7ff5edcfa700 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83ca240 0x5556e83cc650 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e51bc8f77204d82 server_cookie=ce64dbf045b6c792 in_seq=0 out_seq=0 2023-10-10T17:49:53.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5d1ffb700 1 -- 172.21.15.78:0/3485388349 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5556e827b660 con 0x5556e83ca240 2023-10-10T17:49:53.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5d1ffb700 10 client.16962 handle_command_reply: tid=0 2023-10-10T17:49:53.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 client.16962 shutdown 2023-10-10T17:49:53.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 msgr2=0x5556e8355550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8355550 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff5d80a17a0 tx=0x7ff5d8097e30 comp rx=0 tx=0).stop 2023-10-10T17:49:53.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 shutdown_connections 2023-10-10T17:49:53.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5556e83ca240 0x5556e83cc650 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5556e83608b0 0x5556e8355fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5556e835bb60 0x5556e8355550 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5d1ffb700 0 client.16962 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:49:53.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 --2- 172.21.15.78:0/3485388349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5556e835c470 0x5556e8355a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 >> 172.21.15.78:0/3485388349 conn(0x5556e8351360 msgr2=0x5556e835a960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 shutdown_connections 2023-10-10T17:49:53.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 1 -- 172.21.15.78:0/3485388349 wait complete. 2023-10-10T17:49:53.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5fc8ebb80 20 client.16962 trim_cache size 0 max 16384 2023-10-10T17:49:53.463 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:49:53.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff5d40a1340 msgr2=0x7ff5d40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff5d40a1340 0x7ff5d40a37f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40098e0 tx=0x7ff5e4009f10 comp rx=0 tx=0).stop 2023-10-10T17:49:53.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e80ffc30 msgr2=0x7ff5e8195370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e80ffc30 0x7ff5e8195370 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c920 tx=0x7ff5d800cc30 comp rx=0 tx=0).stop 2023-10-10T17:49:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 shutdown_connections 2023-10-10T17:49:53.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff5e8104590 0x7ff5e8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff5d40a1340 0x7ff5d40a37f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5e80ffc30 0x7ff5e8195370 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 --2- 172.21.15.78:0/140172650 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff5e81005d0 0x7ff5e81958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 >> 172.21.15.78:0/140172650 conn(0x7ff5e8075470 msgr2=0x7ff5e80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 shutdown_connections 2023-10-10T17:49:53.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.404+0000 7ff5f075f700 1 -- 172.21.15.78:0/140172650 wait complete. 2023-10-10T17:49:53.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:49:53.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.591+0000 7fa0a3336700 1 -- 172.21.15.78:0/2668967789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101ca0 msgr2=0x7fa09c10a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.591+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2668967789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101ca0 0x7fa09c10a370 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fa08c009990 tx=0x7fa08c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- 172.21.15.78:0/2668967789 shutdown_connections 2023-10-10T17:49:53.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2668967789 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c10a8b0 0x7fa09c10ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2668967789 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa09c101390 0x7fa09c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2668967789 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101ca0 0x7fa09c10a370 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- 172.21.15.78:0/2668967789 >> 172.21.15.78:0/2668967789 conn(0x7fa09c0770b0 msgr2=0x7fa09c0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- 172.21.15.78:0/2668967789 shutdown_connections 2023-10-10T17:49:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- 172.21.15.78:0/2668967789 wait complete. 2023-10-10T17:49:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 Processor -- start 2023-10-10T17:49:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- start start 2023-10-10T17:49:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101390 0x7fa09c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa09c101ca0 0x7fa09c1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c10a8b0 0x7fa09c19ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c19b390 con 0x7fa09c101390 2023-10-10T17:49:53.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c19b500 con 0x7fa09c101ca0 2023-10-10T17:49:53.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.592+0000 7fa0a3336700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09c19b6c0 con 0x7fa09c10a8b0 2023-10-10T17:49:53.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101390 0x7fa09c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101390 0x7fa09c1963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47760/0 (socket says 172.21.15.78:47760) 2023-10-10T17:49:53.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 -- 172.21.15.78:0/2294042096 learned_addr learned my addr 172.21.15.78:0/2294042096 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c10a8b0 0x7fa09c19ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa09c101ca0 0x7fa09c1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 -- 172.21.15.78:0/2294042096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c10a8b0 msgr2=0x7fa09c19ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c10a8b0 0x7fa09c19ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 -- 172.21.15.78:0/2294042096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa09c101ca0 msgr2=0x7fa09c1968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa09c101ca0 0x7fa09c1968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 -- 172.21.15.78:0/2294042096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c0095f0 con 0x7fa09c101390 2023-10-10T17:49:53.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa09c101ca0 0x7fa09c1968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:53.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101390 0x7fa09c1963b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa09800d950 tx=0x7fa09800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0927fc700 1 -- 172.21.15.78:0/2294042096 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098009940 con 0x7fa09c101390 2023-10-10T17:49:53.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa09c110340 con 0x7fa09c101390 2023-10-10T17:49:53.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0927fc700 1 -- 172.21.15.78:0/2294042096 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa098010460 con 0x7fa09c101390 2023-10-10T17:49:53.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0927fc700 1 -- 172.21.15.78:0/2294042096 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09800f620 con 0x7fa09c101390 2023-10-10T17:49:53.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.593+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c1101b0 con 0x7fa09c101390 2023-10-10T17:49:53.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.596+0000 7fa0927fc700 1 -- 172.21.15.78:0/2294042096 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa0980108d0 con 0x7fa09c101390 2023-10-10T17:49:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.596+0000 7fa0927fc700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa0880a12c0 0x7fa0880a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.597+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa0880a12c0 0x7fa0880a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.597+0000 7fa0927fc700 1 -- 172.21.15.78:0/2294042096 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa0980ccf20 con 0x7fa09c101390 2023-10-10T17:49:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.597+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa0880a12c0 0x7fa0880a3770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa08c006010 tx=0x7fa08c00b540 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.597+0000 7fa0af4c2b80 1 Processor -- start 2023-10-10T17:49:53.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.597+0000 7fa0af4c2b80 1 -- start start 2023-10-10T17:49:53.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140344f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x565140349370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x565140349d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56514024a060 con 0x5651403498b0 2023-10-10T17:49:53.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56514029c8a0 con 0x565140345470 2023-10-10T17:49:53.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x565140295d30 con 0x565140344b60 2023-10-10T17:49:53.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x565140349d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x565140349d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47772/0 (socket says 172.21.15.78:47772) 2023-10-10T17:49:53.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 -- 172.21.15.78:0/2703002045 learned_addr learned my addr 172.21.15.78:0/2703002045 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x565140349370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140344f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 -- 172.21.15.78:0/2703002045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 msgr2=0x565140344f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140344f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 -- 172.21.15.78:0/2703002045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 msgr2=0x565140349370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x565140349370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 -- 172.21.15.78:0/2703002045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565140117630 con 0x5651403498b0 2023-10-10T17:49:53.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x565140349370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:53.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x565140349d00 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa094003f40 tx=0x7fa094009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c22b129a2e483d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0877fe700 1 -- 172.21.15.78:0/2703002045 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094019070 con 0x5651403498b0 2023-10-10T17:49:53.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0877fe700 1 -- 172.21.15.78:0/2703002045 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa09400f040 con 0x5651403498b0 2023-10-10T17:49:53.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.598+0000 7fa0877fe700 1 -- 172.21.15.78:0/2703002045 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094007ad0 con 0x5651403498b0 2023-10-10T17:49:53.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/2703002045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 msgr2=0x565140349d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x565140349d00 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fa094003f40 tx=0x7fa094009ba0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/2703002045 shutdown_connections 2023-10-10T17:49:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140344f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x565140349d00 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/2703002045 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x565140349370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/2703002045 >> 172.21.15.78:0/2703002045 conn(0x56514033a360 msgr2=0x56514033fb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/2703002045 shutdown_connections 2023-10-10T17:49:53.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/2703002045 wait complete. 2023-10-10T17:49:53.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 -- start start 2023-10-10T17:49:53.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.599+0000 7fa0af4c2b80 1 Processor -- start 2023-10-10T17:49:53.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140351040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565140345470 0x56514034e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5651403498b0 0x56514034e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651402378e0 con 0x565140345470 2023-10-10T17:49:53.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x565140237580 con 0x5651403498b0 2023-10-10T17:49:53.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x565140119920 con 0x565140344b60 2023-10-10T17:49:53.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0af4c2b80 10 client.0 _ll_register_callbacks cb 0x565140255780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:53.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a08d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565140345470 0x56514034e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a10d2700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140351040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a08d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565140345470 0x56514034e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47776/0 (socket says 172.21.15.78:47776) 2023-10-10T17:49:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a08d1700 1 -- 172.21.15.78:0/1181127783 learned_addr learned my addr 172.21.15.78:0/1181127783 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5651403498b0 0x56514034e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a10d2700 1 -- 172.21.15.78:0/1181127783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5651403498b0 msgr2=0x56514034e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5651403498b0 0x56514034e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a10d2700 1 -- 172.21.15.78:0/1181127783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565140345470 msgr2=0x56514034e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565140345470 0x56514034e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a10d2700 1 -- 172.21.15.78:0/1181127783 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa094003b70 con 0x565140344b60 2023-10-10T17:49:53.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5651403498b0 0x56514034e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.600+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565140345470 0x56514034e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:49:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140351040 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0980ca2e0 tx=0x7fa098099d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=414fefbdc8ca8ede server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa084ff9700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:49:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa084ff9700 1 -- 172.21.15.78:0/1181127783 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0980b2400 con 0x565140344b60 2023-10-10T17:49:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565140337b20 con 0x565140344b60 2023-10-10T17:49:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa084ff9700 1 -- 172.21.15.78:0/1181127783 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa098099ec0 con 0x565140344b60 2023-10-10T17:49:53.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa084ff9700 1 -- 172.21.15.78:0/1181127783 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa09809a080 con 0x565140344b60 2023-10-10T17:49:53.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 10 client.26386 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:53.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56514025c190 con 0x565140344b60 2023-10-10T17:49:53.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa084ff9700 1 -- 172.21.15.78:0/1181127783 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x565140337b20 con 0x565140344b60 2023-10-10T17:49:53.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa084ff9700 1 -- 172.21.15.78:0/1181127783 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa0980a1cb0 con 0x565140344b60 2023-10-10T17:49:53.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 10 client.26386 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:53.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 10 client.26386 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:49:53.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 20 client.26386 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:53.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x565140360c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 4 client.26386 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:53.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.601+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5651402580a0 con 0x56514035e810 2023-10-10T17:49:53.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.602+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x565140360c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.602+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x565140360c20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3f9240fc3788c2ff server_cookie=39683bd8eec529c in_seq=0 out_seq=0 2023-10-10T17:49:53.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.602+0000 7fa084ff9700 10 client.26386 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:53.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa084ff9700 1 -- 172.21.15.78:0/1181127783 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5651402580a0 con 0x56514035e810 2023-10-10T17:49:53.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa084ff9700 10 client.26386 handle_command_reply: tid=0 2023-10-10T17:49:53.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 client.26386 shutdown 2023-10-10T17:49:53.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 msgr2=0x565140351040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140351040 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0980ca2e0 tx=0x7fa098099d90 comp rx=0 tx=0).stop 2023-10-10T17:49:53.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 shutdown_connections 2023-10-10T17:49:53.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x565140360c20 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x565140351040 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa084ff9700 0 client.26386 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:53.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5651403498b0 0x56514034e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1181127783 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x565140345470 0x56514034e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 >> 172.21.15.78:0/1181127783 conn(0x56514033a360 msgr2=0x56514034ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 shutdown_connections 2023-10-10T17:49:53.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1181127783 wait complete. 2023-10-10T17:49:53.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.603+0000 7fa0af4c2b80 20 client.26386 trim_cache size 0 max 16384 2023-10-10T17:49:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.616+0000 7fa0af4c2b80 1 Processor -- start 2023-10-10T17:49:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.616+0000 7fa0af4c2b80 1 -- start start 2023-10-10T17:49:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140344b60 0x565140350b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140345470 0x56514033f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x56514033faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x565140339ee0 con 0x5651403498b0 2023-10-10T17:49:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56514034c7d0 con 0x565140344b60 2023-10-10T17:49:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651403483c0 con 0x565140345470 2023-10-10T17:49:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x56514033faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x56514033faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47786/0 (socket says 172.21.15.78:47786) 2023-10-10T17:49:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 -- 172.21.15.78:0/396779723 learned_addr learned my addr 172.21.15.78:0/396779723 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140344b60 0x565140350b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140345470 0x56514033f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 -- 172.21.15.78:0/396779723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140345470 msgr2=0x56514033f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140345470 0x56514033f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 -- 172.21.15.78:0/396779723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140344b60 msgr2=0x565140350b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140344b60 0x565140350b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 -- 172.21.15.78:0/396779723 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56514033bb00 con 0x5651403498b0 2023-10-10T17:49:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140344b60 0x565140350b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:49:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140345470 0x56514033f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:49:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x56514033faa0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa094000f80 tx=0x7fa0940095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8add94d3875f178 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa087fff700 1 -- 172.21.15.78:0/396779723 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094019040 con 0x5651403498b0 2023-10-10T17:49:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa087fff700 1 -- 172.21.15.78:0/396779723 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa0940058a0 con 0x5651403498b0 2023-10-10T17:49:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.617+0000 7fa087fff700 1 -- 172.21.15.78:0/396779723 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0940156e0 con 0x5651403498b0 2023-10-10T17:49:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/396779723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 msgr2=0x56514033faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x56514033faa0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fa094000f80 tx=0x7fa0940095c0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/396779723 shutdown_connections 2023-10-10T17:49:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x56514033faa0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140344b60 0x565140350b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/396779723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140345470 0x56514033f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/396779723 >> 172.21.15.78:0/396779723 conn(0x56514033a360 msgr2=0x5651403478b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/396779723 shutdown_connections 2023-10-10T17:49:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/396779723 wait complete. 2023-10-10T17:49:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 -- start start 2023-10-10T17:49:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.618+0000 7fa0af4c2b80 1 Processor -- start 2023-10-10T17:49:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x5651403aa640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x5651403aab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0af4c2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x5651403b56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651403ab310 con 0x5651403498b0 2023-10-10T17:49:53.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651403ab480 con 0x565140345470 2023-10-10T17:49:53.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0af4c2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5651403b5c00 con 0x565140344b60 2023-10-10T17:49:53.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0af4c2b80 10 client.0 _ll_register_callbacks cb 0x7fa0980997a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:53.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x5651403b56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x5651403b56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47792/0 (socket says 172.21.15.78:47792) 2023-10-10T17:49:53.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 -- 172.21.15.78:0/1893689346 learned_addr learned my addr 172.21.15.78:0/1893689346 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x5651403aa640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a08d1700 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x5651403aab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 -- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 msgr2=0x5651403aa640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x5651403aa640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 -- 172.21.15.78:0/1893689346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 msgr2=0x5651403aab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x5651403aab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 -- 172.21.15.78:0/1893689346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa09c10d990 con 0x5651403498b0 2023-10-10T17:49:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.619+0000 7fa0a18d3700 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x5651403b56c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa094007ab0 tx=0x7fa094007ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c719198c6557a4d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa084ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:53.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa084ff9700 1 -- 172.21.15.78:0/1893689346 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa094019040 con 0x5651403498b0 2023-10-10T17:49:53.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa084ff9700 1 -- 172.21.15.78:0/1893689346 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa094015690 con 0x5651403498b0 2023-10-10T17:49:53.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa09400f040 con 0x5651403498b0 2023-10-10T17:49:53.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa084ff9700 1 -- 172.21.15.78:0/1893689346 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa094005a10 con 0x5651403498b0 2023-10-10T17:49:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 10 client.16980 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x565140117630 con 0x5651403498b0 2023-10-10T17:49:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa084ff9700 1 -- 172.21.15.78:0/1893689346 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa09400f040 con 0x5651403498b0 2023-10-10T17:49:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa084ff9700 1 -- 172.21.15.78:0/1893689346 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa09401eb20 con 0x5651403498b0 2023-10-10T17:49:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 10 client.16980 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 10 client.16980 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:49:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 20 client.16980 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x5651403b8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 4 client.16980 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:49:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565140264660 con 0x56514035e810 2023-10-10T17:49:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.620+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x5651403b8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0a10d2700 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x5651403b8eb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7069c3cc9dc5fa2a server_cookie=5be31339427b5ff9 in_seq=0 out_seq=0 2023-10-10T17:49:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa084ff9700 10 client.16980 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa084ff9700 1 -- 172.21.15.78:0/1893689346 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565140264660 con 0x56514035e810 2023-10-10T17:49:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa084ff9700 10 client.16980 handle_command_reply: tid=0 2023-10-10T17:49:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 client.16980 shutdown 2023-10-10T17:49:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 msgr2=0x5651403b56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x5651403b56c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fa094007ab0 tx=0x7fa094007ae0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 shutdown_connections 2023-10-10T17:49:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x56514035e810 0x5651403b8eb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x565140344b60 0x5651403aa640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5651403498b0 0x5651403b56c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 --2- 172.21.15.78:0/1893689346 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x565140345470 0x5651403aab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 >> 172.21.15.78:0/1893689346 conn(0x56514033a360 msgr2=0x565140343960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa084ff9700 0 client.16980 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:49:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 shutdown_connections 2023-10-10T17:49:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 1 -- 172.21.15.78:0/1893689346 wait complete. 2023-10-10T17:49:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.621+0000 7fa0af4c2b80 20 client.16980 trim_cache size 0 max 16384 2023-10-10T17:49:53.684 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:49:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa0880a12c0 msgr2=0x7fa0880a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa0880a12c0 0x7fa0880a3770 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa08c006010 tx=0x7fa08c00b540 comp rx=0 tx=0).stop 2023-10-10T17:49:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101390 msgr2=0x7fa09c1963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101390 0x7fa09c1963b0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa09800d950 tx=0x7fa09800dc60 comp rx=0 tx=0).stop 2023-10-10T17:49:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 shutdown_connections 2023-10-10T17:49:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa09c10a8b0 0x7fa09c19ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa0880a12c0 0x7fa0880a3770 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa09c101390 0x7fa09c1963b0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 --2- 172.21.15.78:0/2294042096 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa09c101ca0 0x7fa09c1968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 >> 172.21.15.78:0/2294042096 conn(0x7fa09c0770b0 msgr2=0x7fa09c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 shutdown_connections 2023-10-10T17:49:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.622+0000 7fa0a3336700 1 -- 172.21.15.78:0/2294042096 wait complete. 2023-10-10T17:49:53.689 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:49:53.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 -- 172.21.15.78:0/1325984101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 msgr2=0x7f5bac104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/1325984101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac104560 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009990 tx=0x7f5b9c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 -- 172.21.15.78:0/1325984101 shutdown_connections 2023-10-10T17:49:53.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/1325984101 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bac100370 0x7f5bac10c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/1325984101 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac104560 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/1325984101 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bac0ff9c0 0x7f5bac0ffe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 -- 172.21.15.78:0/1325984101 >> 172.21.15.78:0/1325984101 conn(0x7f5bac075470 msgr2=0x7f5bac075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 -- 172.21.15.78:0/1325984101 shutdown_connections 2023-10-10T17:49:53.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 -- 172.21.15.78:0/1325984101 wait complete. 2023-10-10T17:49:53.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.813+0000 7f5bb16a9700 1 Processor -- start 2023-10-10T17:49:53.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bb16a9700 1 -- start start 2023-10-10T17:49:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bb16a9700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bac0ff9c0 0x7f5bac195350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bb16a9700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bac100370 0x7f5bac195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bb16a9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bb16a9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac19a330 con 0x7f5bac104190 2023-10-10T17:49:53.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bb16a9700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac19a4a0 con 0x7f5bac100370 2023-10-10T17:49:53.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bb16a9700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bac19a660 con 0x7f5bac0ff9c0 2023-10-10T17:49:53.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47806/0 (socket says 172.21.15.78:47806) 2023-10-10T17:49:53.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 -- 172.21.15.78:0/2851782151 learned_addr learned my addr 172.21.15.78:0/2851782151 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5baaffd700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bac0ff9c0 0x7f5bac195350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bac100370 0x7f5bac195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 -- 172.21.15.78:0/2851782151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bac0ff9c0 msgr2=0x7f5bac195350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bac0ff9c0 0x7f5bac195350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 -- 172.21.15.78:0/2851782151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bac100370 msgr2=0x7f5bac195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bac100370 0x7f5bac195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5bab7fe700 1 -- 172.21.15.78:0/2851782151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9c0095f0 con 0x7f5bac104190 2023-10-10T17:49:53.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.814+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bac100370 0x7f5bac195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:53.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.815+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac199cf0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400b970 tx=0x7f5ba400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.815+0000 7f5b93fff700 1 -- 172.21.15.78:0/2851782151 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba400c9b0 con 0x7f5bac104190 2023-10-10T17:49:53.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.815+0000 7f5b93fff700 1 -- 172.21.15.78:0/2851782151 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5ba4014440 con 0x7f5bac104190 2023-10-10T17:49:53.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.815+0000 7f5b93fff700 1 -- 172.21.15.78:0/2851782151 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba40075d0 con 0x7f5bac104190 2023-10-10T17:49:53.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.815+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bac10f400 con 0x7f5bac104190 2023-10-10T17:49:53.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.815+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bac10f270 con 0x7f5bac104190 2023-10-10T17:49:53.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.817+0000 7f5b93fff700 1 -- 172.21.15.78:0/2851782151 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5ba40145b0 con 0x7f5bac104190 2023-10-10T17:49:53.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.817+0000 7f5b93fff700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5b940a12c0 0x7f5b940a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.817+0000 7f5baaffd700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5b940a12c0 0x7f5b940a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.817+0000 7f5b93fff700 1 -- 172.21.15.78:0/2851782151 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f5ba4006e40 con 0x7f5bac104190 2023-10-10T17:49:53.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5baaffd700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5b940a12c0 0x7f5b940a3770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c006010 tx=0x7f5b9c00b540 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 Processor -- start 2023-10-10T17:49:53.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 -- start start 2023-10-10T17:49:53.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824de8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de9280 0x55d824df6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824df7530 0x55d824df9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824cee060 con 0x55d824de8970 2023-10-10T17:49:53.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824d408a0 con 0x55d824df7530 2023-10-10T17:49:53.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.818+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824d39d30 con 0x55d824de9280 2023-10-10T17:49:53.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824de8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824de8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47810/0 (socket says 172.21.15.78:47810) 2023-10-10T17:49:53.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 -- 172.21.15.78:0/781735425 learned_addr learned my addr 172.21.15.78:0/781735425 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824df7530 0x55d824df9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de9280 0x55d824df6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 -- 172.21.15.78:0/781735425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de9280 msgr2=0x55d824df6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de9280 0x55d824df6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 -- 172.21.15.78:0/781735425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824df7530 msgr2=0x55d824df9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824df7530 0x55d824df9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 -- 172.21.15.78:0/781735425 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d824bbb630 con 0x55d824de8970 2023-10-10T17:49:53.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824df7530 0x55d824df9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:53.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5baaffd700 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824de8d40 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00ba30 tx=0x7f5b9c01f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9d7f81d1c7b8d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5b917fa700 1 -- 172.21.15.78:0/781735425 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c02b070 con 0x55d824de8970 2023-10-10T17:49:53.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5b917fa700 1 -- 172.21.15.78:0/781735425 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b9c01e040 con 0x55d824de8970 2023-10-10T17:49:53.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.819+0000 7f5b917fa700 1 -- 172.21.15.78:0/781735425 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00f460 con 0x55d824de8970 2023-10-10T17:49:53.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 -- 172.21.15.78:0/781735425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 msgr2=0x55d824de8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824de8d40 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c00ba30 tx=0x7f5b9c01f000 comp rx=0 tx=0).stop 2023-10-10T17:49:53.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 -- 172.21.15.78:0/781735425 shutdown_connections 2023-10-10T17:49:53.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824df7530 0x55d824df9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824de8d40 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/781735425 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de9280 0x55d824df6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 -- 172.21.15.78:0/781735425 >> 172.21.15.78:0/781735425 conn(0x55d824dde360 msgr2=0x55d824de3b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 -- 172.21.15.78:0/781735425 shutdown_connections 2023-10-10T17:49:53.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 -- 172.21.15.78:0/781735425 wait complete. 2023-10-10T17:49:53.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 -- start start 2023-10-10T17:49:53.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.820+0000 7f5bbd835b80 1 Processor -- start 2023-10-10T17:49:53.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de8970 0x55d824df7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824df7530 0x55d824df8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824cdb8e0 con 0x55d824df7530 2023-10-10T17:49:53.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824cdb580 con 0x55d824de9280 2023-10-10T17:49:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824bbd920 con 0x55d824de8970 2023-10-10T17:49:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bbd835b80 10 client.0 _ll_register_callbacks cb 0x55d824cf9780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824df7530 0x55d824df8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824df7530 0x55d824df8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47820/0 (socket says 172.21.15.78:47820) 2023-10-10T17:49:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 -- 172.21.15.78:0/262729624 learned_addr learned my addr 172.21.15.78:0/262729624 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5baaffd700 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de8970 0x55d824df7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 -- 172.21.15.78:0/262729624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de8970 msgr2=0x55d824df7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de8970 0x55d824df7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 -- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 msgr2=0x55d824df7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 -- 172.21.15.78:0/262729624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bac10c990 con 0x55d824df7530 2023-10-10T17:49:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824df7530 0x55d824df8380 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40992b0 tx=0x7f5ba40995b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc26c783db519a0e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5b8eff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:53.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5b8eff5700 1 -- 172.21.15.78:0/262729624 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba40842b0 con 0x55d824df7530 2023-10-10T17:49:53.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.821+0000 7f5b8eff5700 1 -- 172.21.15.78:0/262729624 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f5ba40a1a00 con 0x55d824df7530 2023-10-10T17:49:53.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d824ddbb20 con 0x55d824df7530 2023-10-10T17:49:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5b8eff5700 1 -- 172.21.15.78:0/262729624 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ba40b90c0 con 0x55d824df7530 2023-10-10T17:49:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 10 client.16992 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55d824d00190 con 0x55d824df7530 2023-10-10T17:49:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5b8eff5700 1 -- 172.21.15.78:0/262729624 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55d824ddbb20 con 0x55d824df7530 2023-10-10T17:49:53.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5b8eff5700 1 -- 172.21.15.78:0/262729624 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5ba40ae060 con 0x55d824df7530 2023-10-10T17:49:53.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 10 client.16992 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:53.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 10 client.16992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:53.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 20 client.16992 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e08fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 4 client.16992 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:53.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d824cfc0a0 con 0x55d824e06bc0 2023-10-10T17:49:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.822+0000 7f5baaffd700 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e08fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5b8eff5700 10 client.16992 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5baaffd700 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e08fd0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87f0b0c17608736b server_cookie=d43e64e147acbce in_seq=0 out_seq=0 2023-10-10T17:49:53.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5b8eff5700 1 -- 172.21.15.78:0/262729624 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55d824cfc0a0 con 0x55d824e06bc0 2023-10-10T17:49:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5b8eff5700 10 client.16992 handle_command_reply: tid=0 2023-10-10T17:49:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5bbd835b80 1 client.16992 shutdown 2023-10-10T17:49:53.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824df7530 msgr2=0x55d824df8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824df7530 0x55d824df8380 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40992b0 tx=0x7f5ba40995b0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 shutdown_connections 2023-10-10T17:49:53.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824de8970 0x55d824df7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.823+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e08fd0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.824+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824df7530 0x55d824df8380 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.824+0000 7f5b8eff5700 0 client.16992 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.824+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/262729624 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.824+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 >> 172.21.15.78:0/262729624 conn(0x55d824dde360 msgr2=0x55d824de2de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.824+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 shutdown_connections 2023-10-10T17:49:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.824+0000 7f5bbd835b80 1 -- 172.21.15.78:0/262729624 wait complete. 2023-10-10T17:49:53.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.824+0000 7f5bbd835b80 20 client.16992 trim_cache size 0 max 16384 2023-10-10T17:49:53.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 Processor -- start 2023-10-10T17:49:53.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 -- start start 2023-10-10T17:49:53.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824dfc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824e020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824de3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824df6c90 con 0x55d824de8970 2023-10-10T17:49:53.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824dea3e0 con 0x55d824de9280 2023-10-10T17:49:53.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824df4bf0 con 0x55d824df7530 2023-10-10T17:49:53.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824dfc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824dfc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47824/0 (socket says 172.21.15.78:47824) 2023-10-10T17:49:53.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 -- 172.21.15.78:0/842354837 learned_addr learned my addr 172.21.15.78:0/842354837 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824de3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824e020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 -- 172.21.15.78:0/842354837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 msgr2=0x55d824de3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824de3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 -- 172.21.15.78:0/842354837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 msgr2=0x55d824e020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824e020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 -- 172.21.15.78:0/842354837 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d824cffee0 con 0x55d824de8970 2023-10-10T17:49:53.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5bab7fe700 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824de3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:53.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.837+0000 7f5baaffd700 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824dfc900 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009ce0 tx=0x7f5b9c00bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f3ca3436f20e65 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5b91ffb700 1 -- 172.21.15.78:0/842354837 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c02b040 con 0x55d824de8970 2023-10-10T17:49:53.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5b91ffb700 1 -- 172.21.15.78:0/842354837 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b9c004bd0 con 0x55d824de8970 2023-10-10T17:49:53.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5b91ffb700 1 -- 172.21.15.78:0/842354837 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c00f9a0 con 0x55d824de8970 2023-10-10T17:49:53.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 -- 172.21.15.78:0/842354837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 msgr2=0x55d824dfc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824dfc900 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009ce0 tx=0x7f5b9c00bb10 comp rx=0 tx=0).stop 2023-10-10T17:49:53.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 -- 172.21.15.78:0/842354837 shutdown_connections 2023-10-10T17:49:53.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824de3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824dfc900 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/842354837 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824e020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 -- 172.21.15.78:0/842354837 >> 172.21.15.78:0/842354837 conn(0x55d824dde360 msgr2=0x55d824df0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 -- 172.21.15.78:0/842354837 shutdown_connections 2023-10-10T17:49:53.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.838+0000 7f5bbd835b80 1 -- 172.21.15.78:0/842354837 wait complete. 2023-10-10T17:49:53.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 -- start start 2023-10-10T17:49:53.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 Processor -- start 2023-10-10T17:49:53.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824df8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df9010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824e53ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824e53ff0 con 0x55d824de8970 2023-10-10T17:49:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824e54160 con 0x55d824de9280 2023-10-10T17:49:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d824e54320 con 0x55d824df7530 2023-10-10T17:49:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5bbd835b80 10 client.0 _ll_register_callbacks cb 0x7f5ba40990f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824df8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824df8ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47830/0 (socket says 172.21.15.78:47830) 2023-10-10T17:49:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 -- 172.21.15.78:0/3703748829 learned_addr learned my addr 172.21.15.78:0/3703748829 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 -- 172.21.15.78:0/3703748829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 msgr2=0x55d824e53ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824e53ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 -- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 msgr2=0x55d824df9010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:53.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df9010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 -- 172.21.15.78:0/3703748829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d824bbb630 con 0x55d824de8970 2023-10-10T17:49:53.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:53.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5baaffd700 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824df8ad0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009990 tx=0x7f5b9c0384a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d5a90a0f43a606 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5b8eff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.839+0000 7f5b8eff5700 1 -- 172.21.15.78:0/3703748829 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9c02b070 con 0x55d824de8970 2023-10-10T17:49:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5b8eff5700 1 -- 172.21.15.78:0/3703748829 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f5b9c004e40 con 0x55d824de8970 2023-10-10T17:49:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b9c01e040 con 0x55d824de8970 2023-10-10T17:49:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5b8eff5700 1 -- 172.21.15.78:0/3703748829 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b9c004910 con 0x55d824de8970 2023-10-10T17:49:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 10 client.16998 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55d824b50c60 con 0x55d824de8970 2023-10-10T17:49:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5b8eff5700 1 -- 172.21.15.78:0/3703748829 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b9c01e040 con 0x55d824de8970 2023-10-10T17:49:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5b8eff5700 1 -- 172.21.15.78:0/3703748829 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5b9c03f030 con 0x55d824de8970 2023-10-10T17:49:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 10 client.16998 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 10 client.16998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 20 client.16998 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:53.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e5d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:53.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 4 client.16998 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:49:53.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d824d08660 con 0x55d824e06bc0 2023-10-10T17:49:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.840+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e5d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5b8eff5700 10 client.16998 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5baa7fc700 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e5d5a0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1128fdcfefbb2161 server_cookie=796e6c3e9b514c04 in_seq=0 out_seq=0 2023-10-10T17:49:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5b8eff5700 1 -- 172.21.15.78:0/3703748829 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d824d08660 con 0x55d824e06bc0 2023-10-10T17:49:53.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5b8eff5700 10 client.16998 handle_command_reply: tid=0 2023-10-10T17:49:53.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 client.16998 shutdown 2023-10-10T17:49:53.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 msgr2=0x55d824df8ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824df8ad0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c009990 tx=0x7f5b9c0384a0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 shutdown_connections 2023-10-10T17:49:53.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55d824df7530 0x55d824e53ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55d824e06bc0 0x55d824e5d5a0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55d824de8970 0x55d824df8ad0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5b8eff5700 0 client.16998 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:53.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 --2- 172.21.15.78:0/3703748829 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55d824de9280 0x55d824df9010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 >> 172.21.15.78:0/3703748829 conn(0x55d824dde360 msgr2=0x55d824df7900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 shutdown_connections 2023-10-10T17:49:53.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 1 -- 172.21.15.78:0/3703748829 wait complete. 2023-10-10T17:49:53.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.841+0000 7f5bbd835b80 20 client.16998 trim_cache size 0 max 16384 2023-10-10T17:49:53.887 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:49:53.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5b940a12c0 msgr2=0x7f5b940a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5b940a12c0 0x7f5b940a3770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c006010 tx=0x7f5b9c00b540 comp rx=0 tx=0).stop 2023-10-10T17:49:53.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 msgr2=0x7f5bac199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:53.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac199cf0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400b970 tx=0x7f5ba400bd30 comp rx=0 tx=0).stop 2023-10-10T17:49:53.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 shutdown_connections 2023-10-10T17:49:53.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5bac0ff9c0 0x7f5bac195350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5b940a12c0 0x7f5b940a3770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5bac104190 0x7f5bac199cf0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 --2- 172.21.15.78:0/2851782151 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5bac100370 0x7f5bac195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:53.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 >> 172.21.15.78:0/2851782151 conn(0x7f5bac075470 msgr2=0x7f5bac0fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:53.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 shutdown_connections 2023-10-10T17:49:53.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:53.842+0000 7f5bb16a9700 1 -- 172.21.15.78:0/2851782151 wait complete. 2023-10-10T17:49:54.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:53 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/188333636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:49:54.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:53 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/188333636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:49:54.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:53 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/188333636' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:49:54.892 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:49:54.893 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:49:55.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 -- 172.21.15.78:0/2608241774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 msgr2=0x7f0ca010c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/2608241774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca010c130 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800b0d0 tx=0x7f0c9800b3e0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 -- 172.21.15.78:0/2608241774 shutdown_connections 2023-10-10T17:49:55.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/2608241774 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ca00ff520 0x7f0ca00ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/2608241774 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca010c130 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/2608241774 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ca00ffec0 0x7f0ca0103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 -- 172.21.15.78:0/2608241774 >> 172.21.15.78:0/2608241774 conn(0x7f0ca00760d0 msgr2=0x7f0ca00764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 -- 172.21.15.78:0/2608241774 shutdown_connections 2023-10-10T17:49:55.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 -- 172.21.15.78:0/2608241774 wait complete. 2023-10-10T17:49:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 Processor -- start 2023-10-10T17:49:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.030+0000 7f0ca67d3700 1 -- start start 2023-10-10T17:49:55.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca67d3700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ca00ff520 0x7f0ca01953d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca67d3700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ca00ffec0 0x7f0ca0195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca67d3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca67d3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca019a3b0 con 0x7f0ca0104560 2023-10-10T17:49:55.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca67d3700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca019a520 con 0x7f0ca00ffec0 2023-10-10T17:49:55.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca67d3700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca019a6e0 con 0x7f0ca00ff520 2023-10-10T17:49:55.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca0199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47848/0 (socket says 172.21.15.78:47848) 2023-10-10T17:49:55.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 -- 172.21.15.78:0/3686322730 learned_addr learned my addr 172.21.15.78:0/3686322730 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ca00ff520 0x7f0ca01953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 -- 172.21.15.78:0/3686322730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ca00ff520 msgr2=0x7f0ca01953d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ca00ffec0 0x7f0ca0195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ca00ff520 0x7f0ca01953d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 -- 172.21.15.78:0/3686322730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ca00ffec0 msgr2=0x7f0ca0195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ca00ffec0 0x7f0ca0195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 -- 172.21.15.78:0/3686322730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c98009d00 con 0x7f0ca0104560 2023-10-10T17:49:55.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.031+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca0199d70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0c98007870 tx=0x7f0c980078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.032+0000 7f0c9d7fa700 1 -- 172.21.15.78:0/3686322730 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9800e040 con 0x7f0ca0104560 2023-10-10T17:49:55.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.032+0000 7f0c9d7fa700 1 -- 172.21.15.78:0/3686322730 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c98003e10 con 0x7f0ca0104560 2023-10-10T17:49:55.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.032+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca010f340 con 0x7f0ca0104560 2023-10-10T17:49:55.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.032+0000 7f0c9d7fa700 1 -- 172.21.15.78:0/3686322730 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9801b980 con 0x7f0ca0104560 2023-10-10T17:49:55.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.032+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca010f1b0 con 0x7f0ca0104560 2023-10-10T17:49:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.034+0000 7f0c9d7fa700 1 -- 172.21.15.78:0/3686322730 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0c98024410 con 0x7f0ca0104560 2023-10-10T17:49:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.035+0000 7f0c9d7fa700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0c880a12c0 0x7f0c880a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.035+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0c880a12c0 0x7f0c880a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.035+0000 7f0c9d7fa700 1 -- 172.21.15.78:0/3686322730 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f0c980ce6f0 con 0x7f0ca0104560 2023-10-10T17:49:55.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.035+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0c880a12c0 0x7f0c880a3770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0c90005950 tx=0x7f0c9000b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.035+0000 7f0cb295fb80 1 Processor -- start 2023-10-10T17:49:55.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.035+0000 7f0cb295fb80 1 -- start start 2023-10-10T17:49:55.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d291c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d2d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d2db40 0x558428d2df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428c2c0e0 con 0x558428d2db40 2023-10-10T17:49:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428c7e920 con 0x558428d28df0 2023-10-10T17:49:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428c77db0 con 0x558428d29700 2023-10-10T17:49:55.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d2db40 0x558428d2df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d2db40 0x558428d2df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47856/0 (socket says 172.21.15.78:47856) 2023-10-10T17:49:55.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 -- 172.21.15.78:0/4079535838 learned_addr learned my addr 172.21.15.78:0/4079535838 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d2d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d291c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 -- 172.21.15.78:0/4079535838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 msgr2=0x558428d2d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d2d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 -- 172.21.15.78:0/4079535838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 msgr2=0x558428d291c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d291c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 -- 172.21.15.78:0/4079535838 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558428af96b0 con 0x558428d2db40 2023-10-10T17:49:55.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d291c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:55.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d2db40 0x558428d2df90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0c980979c0 tx=0x7f0c980978d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a322c163f4802a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0c867fc700 1 -- 172.21.15.78:0/4079535838 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9809b4c0 con 0x558428d2db40 2023-10-10T17:49:55.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0c867fc700 1 -- 172.21.15.78:0/4079535838 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c9809f4f0 con 0x558428d2db40 2023-10-10T17:49:55.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.036+0000 7f0c867fc700 1 -- 172.21.15.78:0/4079535838 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98097bd0 con 0x558428d2db40 2023-10-10T17:49:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 -- 172.21.15.78:0/4079535838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d2db40 msgr2=0x558428d2df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d2db40 0x558428d2df90 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0c980979c0 tx=0x7f0c980978d0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 -- 172.21.15.78:0/4079535838 shutdown_connections 2023-10-10T17:49:55.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d2db40 0x558428d2df90 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d291c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/4079535838 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d2d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 -- 172.21.15.78:0/4079535838 >> 172.21.15.78:0/4079535838 conn(0x558428d1c3e0 msgr2=0x558428d21bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 -- 172.21.15.78:0/4079535838 shutdown_connections 2023-10-10T17:49:55.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 -- 172.21.15.78:0/4079535838 wait complete. 2023-10-10T17:49:55.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.037+0000 7f0cb295fb80 1 -- start start 2023-10-10T17:49:55.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 1 Processor -- start 2023-10-10T17:49:55.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d28df0 0x558428d352c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d32540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428c19960 con 0x558428d28df0 2023-10-10T17:49:55.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428c19600 con 0x558428d2db40 2023-10-10T17:49:55.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428afb9a0 con 0x558428d29700 2023-10-10T17:49:55.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0c9ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d28df0 0x558428d352c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0cb295fb80 10 client.0 _ll_register_callbacks cb 0x558428c37800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:55.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0c9ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d28df0 0x558428d352c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47858/0 (socket says 172.21.15.78:47858) 2023-10-10T17:49:55.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0c9ffff700 1 -- 172.21.15.78:0/2696710308 learned_addr learned my addr 172.21.15.78:0/2696710308 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.038+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d32540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c9ffff700 1 -- 172.21.15.78:0/2696710308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 msgr2=0x558428d32540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d32540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c9ffff700 1 -- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 msgr2=0x558428d32ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d32ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c9ffff700 1 -- 172.21.15.78:0/2696710308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca010f1b0 con 0x558428d28df0 2023-10-10T17:49:55.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d28df0 0x558428d352c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0c900116c0 tx=0x7f0c900047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2e6a7b25ebcf03 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c7f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:55.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/2696710308 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9001e830 con 0x558428d28df0 2023-10-10T17:49:55.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/2696710308 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f0c90017070 con 0x558428d28df0 2023-10-10T17:49:55.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558428d19ba0 con 0x558428d28df0 2023-10-10T17:49:55.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/2696710308 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c9001e990 con 0x558428d28df0 2023-10-10T17:49:55.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 10 client.17010 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:55.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x558428c3e210 con 0x558428d28df0 2023-10-10T17:49:55.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/2696710308 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558428d19ba0 con 0x558428d28df0 2023-10-10T17:49:55.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/2696710308 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0c90027c00 con 0x558428d28df0 2023-10-10T17:49:55.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 10 client.17010 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:55.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 10 client.17010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:55.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 20 client.17010 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:55.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d42900 0x558428d44d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 4 client.17010 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:55.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.039+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558428c3a120 con 0x558428d42900 2023-10-10T17:49:55.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.040+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d42900 0x558428d44d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.040+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d42900 0x558428d44d10 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=464799d6746c8a31 server_cookie=f8b11f9b866386ae in_seq=0 out_seq=0 2023-10-10T17:49:55.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.040+0000 7f0c7f7fe700 10 client.17010 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/2696710308 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x558428c3a120 con 0x558428d42900 2023-10-10T17:49:55.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0c7f7fe700 10 client.17010 handle_command_reply: tid=0 2023-10-10T17:49:55.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 client.17010 shutdown 2023-10-10T17:49:55.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d28df0 msgr2=0x558428d352c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d28df0 0x558428d352c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0c900116c0 tx=0x7f0c900047c0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 shutdown_connections 2023-10-10T17:49:55.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d42900 0x558428d44d10 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d32ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d28df0 0x558428d352c0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/2696710308 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d29700 0x558428d32540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0c7f7fe700 0 client.17010 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 >> 172.21.15.78:0/2696710308 conn(0x558428d1c3e0 msgr2=0x558428d2eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 shutdown_connections 2023-10-10T17:49:55.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 1 -- 172.21.15.78:0/2696710308 wait complete. 2023-10-10T17:49:55.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.041+0000 7f0cb295fb80 20 client.17010 trim_cache size 0 max 16384 2023-10-10T17:49:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 Processor -- start 2023-10-10T17:49:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 -- start start 2023-10-10T17:49:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d30090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d39a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d2db40 0x558428d2d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428d23030 con 0x558428d29700 2023-10-10T17:49:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428d29e80 con 0x558428d28df0 2023-10-10T17:49:55.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428d19ac0 con 0x558428d2db40 2023-10-10T17:49:55.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0c9f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d39a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0c9f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d39a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47868/0 (socket says 172.21.15.78:47868) 2023-10-10T17:49:55.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.058+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/3884642141 learned_addr learned my addr 172.21.15.78:0/3884642141 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d30090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/3884642141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d2db40 msgr2=0x558428d2d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d2db40 0x558428d2d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/3884642141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 msgr2=0x558428d30090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d30090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/3884642141 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558428d1db80 con 0x558428d29700 2023-10-10T17:49:55.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d39a40 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0ca01969e0 tx=0x7f0c9400e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d39e3e5727cfd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c86ffd700 1 -- 172.21.15.78:0/3884642141 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94017070 con 0x558428d29700 2023-10-10T17:49:55.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c86ffd700 1 -- 172.21.15.78:0/3884642141 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c94018040 con 0x558428d29700 2023-10-10T17:49:55.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0c86ffd700 1 -- 172.21.15.78:0/3884642141 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94021660 con 0x558428d29700 2023-10-10T17:49:55.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0cb295fb80 1 -- 172.21.15.78:0/3884642141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 msgr2=0x558428d39a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d39a40 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0ca01969e0 tx=0x7f0c9400e640 comp rx=0 tx=0).stop 2023-10-10T17:49:55.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0cb295fb80 1 -- 172.21.15.78:0/3884642141 shutdown_connections 2023-10-10T17:49:55.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d2db40 0x558428d2d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d28df0 0x558428d30090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/3884642141 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d39a40 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.059+0000 7f0cb295fb80 1 -- 172.21.15.78:0/3884642141 >> 172.21.15.78:0/3884642141 conn(0x558428d1c3e0 msgr2=0x558428d2cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 -- 172.21.15.78:0/3884642141 shutdown_connections 2023-10-10T17:49:55.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 -- 172.21.15.78:0/3884642141 wait complete. 2023-10-10T17:49:55.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 -- start start 2023-10-10T17:49:55.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 Processor -- start 2023-10-10T17:49:55.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d28df0 0x558428d31a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d31fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d90750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428d90c90 con 0x558428d29700 2023-10-10T17:49:55.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428d90e00 con 0x558428d2db40 2023-10-10T17:49:55.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x558428d90fc0 con 0x558428d28df0 2023-10-10T17:49:55.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0cb295fb80 10 client.0 _ll_register_callbacks cb 0x7f0c90010d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:55.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0c9f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d31fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0c9f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d31fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47874/0 (socket says 172.21.15.78:47874) 2023-10-10T17:49:55.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/1112109033 learned_addr learned my addr 172.21.15.78:0/1112109033 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.060+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d28df0 0x558428d31a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d90750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/1112109033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d28df0 msgr2=0x558428d31a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d28df0 0x558428d31a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 msgr2=0x558428d90750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d90750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c9f7fe700 1 -- 172.21.15.78:0/1112109033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9400ecf0 con 0x558428d29700 2023-10-10T17:49:55.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0ca4d70700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d90750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:55.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c9f7fe700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d31fa0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0c94005920 tx=0x7f0c940194c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2190b8dd426f2c9c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c7f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:55.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/1112109033 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94017050 con 0x558428d29700 2023-10-10T17:49:55.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/1112109033 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f0c94019c10 con 0x558428d29700 2023-10-10T17:49:55.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c9809f4f0 con 0x558428d29700 2023-10-10T17:49:55.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/1112109033 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c94019dd0 con 0x558428d29700 2023-10-10T17:49:55.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0cb295fb80 10 client.17016 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:55.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x558428aa5e60 con 0x558428d29700 2023-10-10T17:49:55.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/1112109033 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0c9809f4f0 con 0x558428d29700 2023-10-10T17:49:55.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/1112109033 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f0c9401f020 con 0x558428d29700 2023-10-10T17:49:55.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.061+0000 7f0cb295fb80 10 client.17016 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:55.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0cb295fb80 10 client.17016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:55.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0cb295fb80 20 client.17016 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:55.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d98700 0x558428d9ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0cb295fb80 4 client.17016 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:49:55.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558428c466e0 con 0x558428d98700 2023-10-10T17:49:55.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d98700 0x558428d9ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0c7f7fe700 10 client.17016 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.062+0000 7f0c9ffff700 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d98700 0x558428d9ab10 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd32b743e474390a server_cookie=8764b03da94cc0b3 in_seq=0 out_seq=0 2023-10-10T17:49:55.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.063+0000 7f0c7f7fe700 1 -- 172.21.15.78:0/1112109033 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558428c466e0 con 0x558428d98700 2023-10-10T17:49:55.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.063+0000 7f0c7f7fe700 10 client.17016 handle_command_reply: tid=0 2023-10-10T17:49:55.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.063+0000 7f0cb295fb80 1 client.17016 shutdown 2023-10-10T17:49:55.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.063+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 msgr2=0x558428d31fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.063+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d31fa0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f0c94005920 tx=0x7f0c940194c0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 shutdown_connections 2023-10-10T17:49:55.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x558428d28df0 0x558428d31a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x558428d98700 0x558428d9ab10 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x558428d2db40 0x558428d90750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0c7f7fe700 0 client.17016 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 --2- 172.21.15.78:0/1112109033 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x558428d29700 0x558428d31fa0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 >> 172.21.15.78:0/1112109033 conn(0x558428d1c3e0 msgr2=0x558428d2ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 shutdown_connections 2023-10-10T17:49:55.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 1 -- 172.21.15.78:0/1112109033 wait complete. 2023-10-10T17:49:55.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0cb295fb80 20 client.17016 trim_cache size 0 max 16384 2023-10-10T17:49:55.109 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:49:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:49:55.109 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "77b121a5-cb25-4a45-ae7a-b03b0ad8d640", 2023-10-10T17:49:55.110 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:49:55.110 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:49:55.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0c880a12c0 msgr2=0x7f0c880a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0c880a12c0 0x7f0c880a3770 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0c90005950 tx=0x7f0c9000b410 comp rx=0 tx=0).stop 2023-10-10T17:49:55.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 msgr2=0x7f0ca0199d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca0199d70 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0c98007870 tx=0x7f0c980078a0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 shutdown_connections 2023-10-10T17:49:55.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ca00ff520 0x7f0ca01953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0c880a12c0 0x7f0c880a3770 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ca0104560 0x7f0ca0199d70 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 --2- 172.21.15.78:0/3686322730 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ca00ffec0 0x7f0ca0195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 >> 172.21.15.78:0/3686322730 conn(0x7f0ca00760d0 msgr2=0x7f0ca00fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 shutdown_connections 2023-10-10T17:49:55.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.064+0000 7f0ca67d3700 1 -- 172.21.15.78:0/3686322730 wait complete. 2023-10-10T17:49:55.114 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:54 smithi078 ceph-mon[99494]: pgmap v226: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:49:55.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:49:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:54 smithi190 ceph-mon[101220]: pgmap v226: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:49:55.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:54 smithi157 ceph-mon[100896]: pgmap v226: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:49:55.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 -- 172.21.15.78:0/1311021234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc100530 msgr2=0x7f9cbc10c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/1311021234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc100530 0x7f9cbc10c040 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400b0a0 tx=0x7f9cb400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 -- 172.21.15.78:0/1311021234 shutdown_connections 2023-10-10T17:49:55.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/1311021234 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc105210 0x7f9cbc1055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/1311021234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc100530 0x7f9cbc10c040 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/1311021234 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cbc0ffba0 0x7f9cbc0ffff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 -- 172.21.15.78:0/1311021234 >> 172.21.15.78:0/1311021234 conn(0x7f9cbc0748f0 msgr2=0x7f9cbc074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 -- 172.21.15.78:0/1311021234 shutdown_connections 2023-10-10T17:49:55.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.237+0000 7f9cc3f50700 1 -- 172.21.15.78:0/1311021234 wait complete. 2023-10-10T17:49:55.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 Processor -- start 2023-10-10T17:49:55.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 -- start start 2023-10-10T17:49:55.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc0ffba0 0x7f9cbc197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cbc100530 0x7f9cbc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc105210 0x7f9cbc1a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc1982e0 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc1a2a20 con 0x7f9cbc100530 2023-10-10T17:49:55.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc3f50700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cbc1a2be0 con 0x7f9cbc105210 2023-10-10T17:49:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc0ffba0 0x7f9cbc197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc0ffba0 0x7f9cbc197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47886/0 (socket says 172.21.15.78:47886) 2023-10-10T17:49:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 -- 172.21.15.78:0/612207287 learned_addr learned my addr 172.21.15.78:0/612207287 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc105210 0x7f9cbc1a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cbc100530 0x7f9cbc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 -- 172.21.15.78:0/612207287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc105210 msgr2=0x7f9cbc1a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc105210 0x7f9cbc1a24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 -- 172.21.15.78:0/612207287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cbc100530 msgr2=0x7f9cbc1979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cbc100530 0x7f9cbc1979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.238+0000 7f9cc1cec700 1 -- 172.21.15.78:0/612207287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4009d00 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.239+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc0ffba0 0x7f9cbc197460 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00d950 tx=0x7f9cac00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.239+0000 7f9cb2ffd700 1 -- 172.21.15.78:0/612207287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac009940 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.239+0000 7f9cb2ffd700 1 -- 172.21.15.78:0/612207287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cac010460 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.239+0000 7f9cb2ffd700 1 -- 172.21.15.78:0/612207287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac00f620 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.239+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cbc10f210 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.239+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc10f080 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.241+0000 7f9cb2ffd700 1 -- 172.21.15.78:0/612207287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f9cac0108d0 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.241+0000 7f9cb2ffd700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9ca80a1340 0x7f9ca80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.241+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9ca80a1340 0x7f9ca80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.242+0000 7f9cb2ffd700 1 -- 172.21.15.78:0/612207287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f9cac0cce00 con 0x7f9cbc0ffba0 2023-10-10T17:49:55.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.242+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9ca80a1340 0x7f9ca80a37f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0ff360 tx=0x7f9cb8005ed0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.242+0000 7f9cd00dcb80 1 Processor -- start 2023-10-10T17:49:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.242+0000 7f9cd00dcb80 1 -- start start 2023-10-10T17:49:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.242+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b91522340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b91526780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b91527110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b91423060 con 0x555b91521f70 2023-10-10T17:49:55.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b914758a0 con 0x555b91522880 2023-10-10T17:49:55.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b9146ed30 con 0x555b91526cc0 2023-10-10T17:49:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b91522340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b91522340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47894/0 (socket says 172.21.15.78:47894) 2023-10-10T17:49:55.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 -- 172.21.15.78:0/1876831594 learned_addr learned my addr 172.21.15.78:0/1876831594 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b91527110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 -- 172.21.15.78:0/1876831594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 msgr2=0x555b91527110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b91527110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 -- 172.21.15.78:0/1876831594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 msgr2=0x555b91526780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b91526780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 -- 172.21.15.78:0/1876831594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b912f0630 con 0x555b91521f70 2023-10-10T17:49:55.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b91522340 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0a1a60 tx=0x7f9cac099570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d1d7ee4e2a4136 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9ca7fff700 1 -- 172.21.15.78:0/1876831594 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac0a1f90 con 0x555b91521f70 2023-10-10T17:49:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9ca7fff700 1 -- 172.21.15.78:0/1876831594 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cac096250 con 0x555b91521f70 2023-10-10T17:49:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.243+0000 7f9ca7fff700 1 -- 172.21.15.78:0/1876831594 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac0a1b00 con 0x555b91521f70 2023-10-10T17:49:55.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1876831594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 msgr2=0x555b91522340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b91522340 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f9cac0a1a60 tx=0x7f9cac099570 comp rx=0 tx=0).stop 2023-10-10T17:49:55.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1876831594 shutdown_connections 2023-10-10T17:49:55.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b91527110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b91522340 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1876831594 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b91526780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1876831594 >> 172.21.15.78:0/1876831594 conn(0x555b91513360 msgr2=0x555b91518b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1876831594 shutdown_connections 2023-10-10T17:49:55.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1876831594 wait complete. 2023-10-10T17:49:55.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.244+0000 7f9cd00dcb80 1 -- start start 2023-10-10T17:49:55.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 1 Processor -- start 2023-10-10T17:49:55.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b9152e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b9152b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9152bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b914108e0 con 0x555b91521f70 2023-10-10T17:49:55.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b91410580 con 0x555b91522880 2023-10-10T17:49:55.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b912f2920 con 0x555b91526cc0 2023-10-10T17:49:55.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cd00dcb80 10 client.0 _ll_register_callbacks cb 0x555b9142e780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:55.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b9152e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b9152e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47900/0 (socket says 172.21.15.78:47900) 2023-10-10T17:49:55.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 -- 172.21.15.78:0/3675876284 learned_addr learned my addr 172.21.15.78:0/3675876284 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9152bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b9152b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 -- 172.21.15.78:0/3675876284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 msgr2=0x555b9152bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9152bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 -- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 msgr2=0x555b9152b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b9152b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 -- 172.21.15.78:0/3675876284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc10f080 con 0x555b91521f70 2023-10-10T17:49:55.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b9152b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:49:55.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b9152e400 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9cac096040 tx=0x7f9cac010cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212ffc0ba5d9632a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.245+0000 7f9ca57fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:55.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9ca57fa700 1 -- 172.21.15.78:0/3675876284 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cac0a1f90 con 0x555b91521f70 2023-10-10T17:49:55.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9ca57fa700 1 -- 172.21.15.78:0/3675876284 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f9cac099030 con 0x555b91521f70 2023-10-10T17:49:55.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555b91510b20 con 0x555b91521f70 2023-10-10T17:49:55.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9ca57fa700 1 -- 172.21.15.78:0/3675876284 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cac0991f0 con 0x555b91521f70 2023-10-10T17:49:55.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 10 client.17028 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:55.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x555b91435190 con 0x555b91521f70 2023-10-10T17:49:55.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9ca57fa700 1 -- 172.21.15.78:0/3675876284 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555b91510b20 con 0x555b91521f70 2023-10-10T17:49:55.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9ca57fa700 1 -- 172.21.15.78:0/3675876284 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f9cac098bc0 con 0x555b91521f70 2023-10-10T17:49:55.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 10 client.17028 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:55.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 10 client.17028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:55.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 20 client.17028 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:55.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b9153b6d0 0x555b9153dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 4 client.17028 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:49:55.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b914310a0 con 0x555b9153b6d0 2023-10-10T17:49:55.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.246+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b9153b6d0 0x555b9153dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.247+0000 7f9ca57fa700 10 client.17028 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.247+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b9153b6d0 0x555b9153dae0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c460db50544fa91 server_cookie=3b30c4bb10c75c5 in_seq=0 out_seq=0 2023-10-10T17:49:55.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.247+0000 7f9ca57fa700 1 -- 172.21.15.78:0/3675876284 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555b914310a0 con 0x555b9153b6d0 2023-10-10T17:49:55.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.247+0000 7f9ca57fa700 10 client.17028 handle_command_reply: tid=0 2023-10-10T17:49:55.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.247+0000 7f9cd00dcb80 1 client.17028 shutdown 2023-10-10T17:49:55.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 msgr2=0x555b9152e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b9152e400 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f9cac096040 tx=0x7f9cac010cd0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 shutdown_connections 2023-10-10T17:49:55.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9152bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b9153b6d0 0x555b9153dae0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91521f70 0x555b9152e400 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3675876284 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91522880 0x555b9152b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9ca57fa700 0 client.17028 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 >> 172.21.15.78:0/3675876284 conn(0x555b91513360 msgr2=0x555b91528170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 shutdown_connections 2023-10-10T17:49:55.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3675876284 wait complete. 2023-10-10T17:49:55.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.248+0000 7f9cd00dcb80 20 client.17028 trim_cache size 0 max 16384 2023-10-10T17:49:55.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 Processor -- start 2023-10-10T17:49:55.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 -- start start 2023-10-10T17:49:55.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b91532c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91522880 0x555b9152dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91526cc0 0x555b91524d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b9152ea10 con 0x555b91526cc0 2023-10-10T17:49:55.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b91512ee0 con 0x555b91521f70 2023-10-10T17:49:55.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.264+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b915198b0 con 0x555b91522880 2023-10-10T17:49:55.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91526cc0 0x555b91524d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91526cc0 0x555b91524d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47912/0 (socket says 172.21.15.78:47912) 2023-10-10T17:49:55.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 -- 172.21.15.78:0/3503972204 learned_addr learned my addr 172.21.15.78:0/3503972204 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b91532c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 -- 172.21.15.78:0/3503972204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91522880 msgr2=0x555b9152dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91522880 0x555b9152dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 -- 172.21.15.78:0/3503972204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 msgr2=0x555b91532c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b91532c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 -- 172.21.15.78:0/3503972204 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b91514b00 con 0x555b91526cc0 2023-10-10T17:49:55.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91526cc0 0x555b91524d40 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009360 tx=0x7f9cb4009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61608f6c8f4fa6fe server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cb0ff9700 1 -- 172.21.15.78:0/3503972204 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400e040 con 0x555b91526cc0 2023-10-10T17:49:55.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cb0ff9700 1 -- 172.21.15.78:0/3503972204 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cb4022070 con 0x555b91526cc0 2023-10-10T17:49:55.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.265+0000 7f9cb0ff9700 1 -- 172.21.15.78:0/3503972204 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4027a10 con 0x555b91526cc0 2023-10-10T17:49:55.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3503972204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91526cc0 msgr2=0x555b91524d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91526cc0 0x555b91524d40 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4009360 tx=0x7f9cb4009390 comp rx=0 tx=0).stop 2023-10-10T17:49:55.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3503972204 shutdown_connections 2023-10-10T17:49:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91526cc0 0x555b91524d40 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b91532c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/3503972204 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91522880 0x555b9152dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3503972204 >> 172.21.15.78:0/3503972204 conn(0x555b91513360 msgr2=0x555b91537eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3503972204 shutdown_connections 2023-10-10T17:49:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/3503972204 wait complete. 2023-10-10T17:49:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 -- start start 2023-10-10T17:49:55.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.266+0000 7f9cd00dcb80 1 Processor -- start 2023-10-10T17:49:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b915834e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91522880 0x555b91583a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9158e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b915842a0 con 0x555b91522880 2023-10-10T17:49:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b9158eaa0 con 0x555b91521f70 2023-10-10T17:49:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b9158ebe0 con 0x555b91526cc0 2023-10-10T17:49:55.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 10 client.0 _ll_register_callbacks cb 0x7f9cb4009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:49:55.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91522880 0x555b91583a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91522880 0x555b91583a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47916/0 (socket says 172.21.15.78:47916) 2023-10-10T17:49:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 -- 172.21.15.78:0/1405850285 learned_addr learned my addr 172.21.15.78:0/1405850285 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:49:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b915834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9158e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 -- 172.21.15.78:0/1405850285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 msgr2=0x555b9158e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9158e560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 -- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 msgr2=0x555b915834e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:49:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b915834e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 -- 172.21.15.78:0/1405850285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cbc10c8b0 con 0x555b91522880 2023-10-10T17:49:55.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc24ed700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9158e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:49:55.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cc14eb700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91522880 0x555b91583a20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8005950 tx=0x7f9cb800ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8e6d0f2efa779af server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:49:55.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9ca57fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:49:55.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9ca57fa700 1 -- 172.21.15.78:0/1405850285 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80235c0 con 0x555b91522880 2023-10-10T17:49:55.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9ca57fa700 1 -- 172.21.15.78:0/1405850285 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f9cb80237f0 con 0x555b91522880 2023-10-10T17:49:55.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9cac096250 con 0x555b91522880 2023-10-10T17:49:55.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9ca57fa700 1 -- 172.21.15.78:0/1405850285 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9cb80239b0 con 0x555b91522880 2023-10-10T17:49:55.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 10 client.17034 fetch_fsmap learned FSMap version 32 2023-10-10T17:49:55.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.267+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x555b9129cde0 con 0x555b91522880 2023-10-10T17:49:55.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9ca57fa700 1 -- 172.21.15.78:0/1405850285 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9cac096250 con 0x555b91522880 2023-10-10T17:49:55.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9ca57fa700 1 -- 172.21.15.78:0/1405850285 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f9cb8029020 con 0x555b91522880 2023-10-10T17:49:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cd00dcb80 10 client.17034 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:49:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cd00dcb80 10 client.17034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:49:55.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cd00dcb80 20 client.17034 populate_metadata read hostname 'smithi078' 2023-10-10T17:49:55.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b91590330 0x555b91592740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:49:55.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cd00dcb80 4 client.17034 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:49:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555b9143d660 con 0x555b91590330 2023-10-10T17:49:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b91590330 0x555b91592740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:49:55.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9ca57fa700 10 client.17034 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.268+0000 7f9cc1cec700 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b91590330 0x555b91592740 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17aed4df4f78410d server_cookie=7c28149be62390b1 in_seq=0 out_seq=0 2023-10-10T17:49:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9ca57fa700 1 -- 172.21.15.78:0/1405850285 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555b9143d660 con 0x555b91590330 2023-10-10T17:49:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9ca57fa700 10 client.17034 handle_command_reply: tid=0 2023-10-10T17:49:55.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 client.17034 shutdown 2023-10-10T17:49:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91522880 msgr2=0x555b91583a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91522880 0x555b91583a20 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8005950 tx=0x7f9cb800ee30 comp rx=0 tx=0).stop 2023-10-10T17:49:55.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 shutdown_connections 2023-10-10T17:49:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555b91526cc0 0x555b9158e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555b91590330 0x555b91592740 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555b91521f70 0x555b915834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9ca57fa700 0 client.17034 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:49:55.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 --2- 172.21.15.78:0/1405850285 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555b91522880 0x555b91583a20 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 >> 172.21.15.78:0/1405850285 conn(0x555b91513360 msgr2=0x555b91524d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 shutdown_connections 2023-10-10T17:49:55.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 1 -- 172.21.15.78:0/1405850285 wait complete. 2023-10-10T17:49:55.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cd00dcb80 20 client.17034 trim_cache size 0 max 16384 2023-10-10T17:49:55.314 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:49:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:49:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:49:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "77b121a5-cb25-4a45-ae7a-b03b0ad8d640": { 2023-10-10T17:49:55.315 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:49:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "77b121a5-cb25-4a45-ae7a-b03b0ad8d640", 2023-10-10T17:49:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:49:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:49:55.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:49:55.317 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:49:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9ca80a1340 msgr2=0x7f9ca80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9ca80a1340 0x7f9ca80a37f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0ff360 tx=0x7f9cb8005ed0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc0ffba0 msgr2=0x7f9cbc197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:49:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc0ffba0 0x7f9cbc197460 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9cac00d950 tx=0x7f9cac00dc60 comp rx=0 tx=0).stop 2023-10-10T17:49:55.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 shutdown_connections 2023-10-10T17:49:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9cbc105210 0x7f9cbc1a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9ca80a1340 0x7f9ca80a37f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9cbc0ffba0 0x7f9cbc197460 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.269+0000 7f9cc3f50700 1 --2- 172.21.15.78:0/612207287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9cbc100530 0x7f9cbc1979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:49:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.270+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 >> 172.21.15.78:0/612207287 conn(0x7f9cbc0748f0 msgr2=0x7f9cbc0fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:49:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.270+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 shutdown_connections 2023-10-10T17:49:55.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:49:55.270+0000 7f9cc3f50700 1 -- 172.21.15.78:0/612207287 wait complete. 2023-10-10T17:49:55.322 INFO:tasks.cephfs.filesystem:scrub status for tag:77b121a5-cb25-4a45-ae7a-b03b0ad8d640 - {'path': '/', 'tag': '77b121a5-cb25-4a45-ae7a-b03b0ad8d640', 'options': 'recursive,force'} 2023-10-10T17:49:56.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:55 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:49:56.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:55 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:56.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:55 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:49:56.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:55 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:56.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:55 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:49:56.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:55 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:56.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:55 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:49:56.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:55 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:56.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:55 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:49:56.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:55 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:56.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:55 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:49:56.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:55 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:49:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:56 smithi078 ceph-mon[99494]: pgmap v227: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:49:57.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:56 smithi190 ceph-mon[101220]: pgmap v227: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:49:57.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:56 smithi157 ceph-mon[100896]: pgmap v227: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:49:58.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:57 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:49:58.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:57 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:49:58.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:57 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:49:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:58 smithi078 ceph-mon[99494]: pgmap v228: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:49:59.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:58 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:49:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:58 smithi190 ceph-mon[101220]: pgmap v228: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:49:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:58 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:49:59.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:58 smithi157 ceph-mon[100896]: pgmap v228: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:49:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:58 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:50:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:50:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:50:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:59 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:50:00.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:50:00.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:59 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:50:00.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:59 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:50:00.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:49:59 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:50:00.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:50:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:50:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:59 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:50:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:50:00.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:59 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:50:00.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:59 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:50:00.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:49:59 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:50:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:50:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:50:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:59 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:50:00.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:50:00.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:59 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:50:00.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:59 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:50:00.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:49:59 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:50:01.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:00 smithi078 ceph-mon[99494]: pgmap v229: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:01.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:00 smithi078 ceph-mon[99494]: overall HEALTH_OK 2023-10-10T17:50:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:00 smithi190 ceph-mon[101220]: pgmap v229: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:00 smithi190 ceph-mon[101220]: overall HEALTH_OK 2023-10-10T17:50:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:00 smithi157 ceph-mon[100896]: pgmap v229: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:00 smithi157 ceph-mon[100896]: overall HEALTH_OK 2023-10-10T17:50:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:02 smithi078 ceph-mon[99494]: pgmap v230: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:50:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:02 smithi190 ceph-mon[101220]: pgmap v230: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:50:03.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:02 smithi157 ceph-mon[100896]: pgmap v230: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:50:05.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:04 smithi078 ceph-mon[99494]: pgmap v231: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:50:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:04 smithi190 ceph-mon[101220]: pgmap v231: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:50:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:04 smithi157 ceph-mon[100896]: pgmap v231: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:50:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:06 smithi078 ceph-mon[99494]: pgmap v232: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:50:07.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:06 smithi190 ceph-mon[101220]: pgmap v232: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:50:07.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:06 smithi157 ceph-mon[100896]: pgmap v232: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:50:08.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:07 smithi078 ceph-mon[99494]: osdmap e97: 12 total, 12 up, 12 in 2023-10-10T17:50:08.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:07 smithi190 ceph-mon[101220]: osdmap e97: 12 total, 12 up, 12 in 2023-10-10T17:50:08.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:07 smithi157 ceph-mon[100896]: osdmap e97: 12 total, 12 up, 12 in 2023-10-10T17:50:09.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:08 smithi078 ceph-mon[99494]: pgmap v234: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:09.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:08 smithi190 ceph-mon[101220]: pgmap v234: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:09.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:08 smithi157 ceph-mon[100896]: pgmap v234: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:11.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:10 smithi078 ceph-mon[99494]: pgmap v235: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:11.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:10 smithi190 ceph-mon[101220]: pgmap v235: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:11.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:10 smithi157 ceph-mon[100896]: pgmap v235: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:13.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:12 smithi078 ceph-mon[99494]: pgmap v236: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:13.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:12 smithi190 ceph-mon[101220]: pgmap v236: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:13.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:12 smithi157 ceph-mon[100896]: pgmap v236: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:50:15.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:14 smithi078 ceph-mon[99494]: pgmap v237: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:50:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:14 smithi190 ceph-mon[101220]: pgmap v237: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:50:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:14 smithi157 ceph-mon[100896]: pgmap v237: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:50:17.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:16 smithi078 ceph-mon[99494]: pgmap v238: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:50:17.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:16 smithi190 ceph-mon[101220]: pgmap v238: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:50:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:16 smithi157 ceph-mon[100896]: pgmap v238: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:50:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:18 smithi078 ceph-mon[99494]: pgmap v239: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2023-10-10T17:50:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:18 smithi078 ceph-mon[99494]: osdmap e98: 12 total, 12 up, 12 in 2023-10-10T17:50:19.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:18 smithi190 ceph-mon[101220]: pgmap v239: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2023-10-10T17:50:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:18 smithi190 ceph-mon[101220]: osdmap e98: 12 total, 12 up, 12 in 2023-10-10T17:50:19.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:18 smithi157 ceph-mon[100896]: pgmap v239: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 97 B/s wr, 5 op/s 2023-10-10T17:50:19.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:18 smithi157 ceph-mon[100896]: osdmap e98: 12 total, 12 up, 12 in 2023-10-10T17:50:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:20 smithi078 ceph-mon[99494]: pgmap v241: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:50:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:20 smithi190 ceph-mon[101220]: pgmap v241: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:50:21.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:20 smithi157 ceph-mon[100896]: pgmap v241: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:50:22.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:21 smithi078 ceph-mon[99494]: pgmap v242: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:50:22.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:21 smithi190 ceph-mon[101220]: pgmap v242: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:50:22.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:21 smithi157 ceph-mon[100896]: pgmap v242: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:50:24.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:24 smithi157 ceph-mon[100896]: pgmap v243: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:25.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:24 smithi078 ceph-mon[99494]: pgmap v243: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:24 smithi190 ceph-mon[101220]: pgmap v243: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:25.323 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:50:25.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 -- 172.21.15.78:0/1926699642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd30104490 msgr2=0x7efd3010c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 --2- 172.21.15.78:0/1926699642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd30104490 0x7efd3010c060 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7efd28009fc0 tx=0x7efd2800b330 comp rx=0 tx=0).stop 2023-10-10T17:50:25.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 -- 172.21.15.78:0/1926699642 shutdown_connections 2023-10-10T17:50:25.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 --2- 172.21.15.78:0/1926699642 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd30104490 0x7efd3010c060 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 --2- 172.21.15.78:0/1926699642 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd300ff450 0x7efd300ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 --2- 172.21.15.78:0/1926699642 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd300ffdf0 0x7efd30103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 -- 172.21.15.78:0/1926699642 >> 172.21.15.78:0/1926699642 conn(0x7efd300770b0 msgr2=0x7efd300774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 -- 172.21.15.78:0/1926699642 shutdown_connections 2023-10-10T17:50:25.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.466+0000 7efd34ded700 1 -- 172.21.15.78:0/1926699642 wait complete. 2023-10-10T17:50:25.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.467+0000 7efd34ded700 1 Processor -- start 2023-10-10T17:50:25.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.467+0000 7efd34ded700 1 -- start start 2023-10-10T17:50:25.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd300ff450 0x7efd301954a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd300ffdf0 0x7efd301959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd30104490 0x7efd30199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3019a480 con 0x7efd300ffdf0 2023-10-10T17:50:25.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3019a5f0 con 0x7efd30104490 2023-10-10T17:50:25.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3019a7b0 con 0x7efd300ff450 2023-10-10T17:50:25.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd300ffdf0 0x7efd301959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd300ffdf0 0x7efd301959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47930/0 (socket says 172.21.15.78:47930) 2023-10-10T17:50:25.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2599339873 learned_addr learned my addr 172.21.15.78:0/2599339873 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:25.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2e59c700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd300ff450 0x7efd301954a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd30104490 0x7efd30199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2599339873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd300ff450 msgr2=0x7efd301954a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd300ff450 0x7efd301954a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2599339873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd30104490 msgr2=0x7efd30199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd30104490 0x7efd30199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2599339873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd28009c40 con 0x7efd300ffdf0 2023-10-10T17:50:25.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd30104490 0x7efd30199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:25.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd300ffdf0 0x7efd301959e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7efd2000d950 tx=0x7efd2000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd1f7fe700 1 -- 172.21.15.78:0/2599339873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2000bb60 con 0x7efd300ffdf0 2023-10-10T17:50:25.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd1f7fe700 1 -- 172.21.15.78:0/2599339873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd2000de30 con 0x7efd300ffdf0 2023-10-10T17:50:25.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd3010f260 con 0x7efd300ffdf0 2023-10-10T17:50:25.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd1f7fe700 1 -- 172.21.15.78:0/2599339873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd200055a0 con 0x7efd300ffdf0 2023-10-10T17:50:25.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.468+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3010f0d0 con 0x7efd300ffdf0 2023-10-10T17:50:25.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.470+0000 7efd1f7fe700 1 -- 172.21.15.78:0/2599339873 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7efd20004750 con 0x7efd300ffdf0 2023-10-10T17:50:25.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.471+0000 7efd1f7fe700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efd180a12c0 0x7efd180a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.471+0000 7efd2e59c700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efd180a12c0 0x7efd180a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.471+0000 7efd1f7fe700 1 -- 172.21.15.78:0/2599339873 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7efd200ccc90 con 0x7efd300ffdf0 2023-10-10T17:50:25.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.471+0000 7efd2e59c700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efd180a12c0 0x7efd180a3770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efd24005fd0 tx=0x7efd24009310 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 Processor -- start 2023-10-10T17:50:25.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 -- start start 2023-10-10T17:50:25.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc22b60 0x55749dc22f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc27370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc27d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749db28060 con 0x55749dc22b60 2023-10-10T17:50:25.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749db7a8a0 con 0x55749dc23470 2023-10-10T17:50:25.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.472+0000 7efd40f79b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749db73d30 con 0x55749dc278b0 2023-10-10T17:50:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc22b60 0x55749dc22f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc22b60 0x55749dc22f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47936/0 (socket says 172.21.15.78:47936) 2023-10-10T17:50:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 -- 172.21.15.78:0/214030264 learned_addr learned my addr 172.21.15.78:0/214030264 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:25.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc27370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 -- 172.21.15.78:0/214030264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 msgr2=0x55749dc27d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc27d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 -- 172.21.15.78:0/214030264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 msgr2=0x55749dc27370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc27370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 -- 172.21.15.78:0/214030264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55749d9f5630 con 0x55749dc22b60 2023-10-10T17:50:25.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd2e59c700 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc22b60 0x55749dc22f30 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7efd24010c40 tx=0x7efd24013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78174ddea39762e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd1cff9700 1 -- 172.21.15.78:0/214030264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd24009810 con 0x55749dc22b60 2023-10-10T17:50:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd1cff9700 1 -- 172.21.15.78:0/214030264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd2401f450 con 0x55749dc22b60 2023-10-10T17:50:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.473+0000 7efd1cff9700 1 -- 172.21.15.78:0/214030264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2401e610 con 0x55749dc22b60 2023-10-10T17:50:25.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 -- 172.21.15.78:0/214030264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc22b60 msgr2=0x55749dc22f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc22b60 0x55749dc22f30 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7efd24010c40 tx=0x7efd24013000 comp rx=0 tx=0).stop 2023-10-10T17:50:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 -- 172.21.15.78:0/214030264 shutdown_connections 2023-10-10T17:50:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc27d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc22b60 0x55749dc22f30 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 --2- 172.21.15.78:0/214030264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc27370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 -- 172.21.15.78:0/214030264 >> 172.21.15.78:0/214030264 conn(0x55749dc18360 msgr2=0x55749dc1db40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 -- 172.21.15.78:0/214030264 shutdown_connections 2023-10-10T17:50:25.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 -- 172.21.15.78:0/214030264 wait complete. 2023-10-10T17:50:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 -- start start 2023-10-10T17:50:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.474+0000 7efd40f79b80 1 Processor -- start 2023-10-10T17:50:25.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc2f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc2c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc278b0 0x55749dc2caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749db158e0 con 0x55749dc23470 2023-10-10T17:50:25.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749db15580 con 0x55749dc278b0 2023-10-10T17:50:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749d9f7920 con 0x55749dc22b60 2023-10-10T17:50:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 10 client.0 _ll_register_callbacks cb 0x55749db33780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:25.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc2c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc2c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47944/0 (socket says 172.21.15.78:47944) 2023-10-10T17:50:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 -- 172.21.15.78:0/4229556073 learned_addr learned my addr 172.21.15.78:0/4229556073 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc278b0 0x55749dc2caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2e59c700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc2f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 -- 172.21.15.78:0/4229556073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 msgr2=0x55749dc2f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc2f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 -- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc278b0 msgr2=0x55749dc2caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc278b0 0x55749dc2caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 -- 172.21.15.78:0/4229556073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd24010920 con 0x55749dc23470 2023-10-10T17:50:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2e59c700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc2f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:25.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc2c3b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7efd20099670 tx=0x7efd20005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c674750ef8f390 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd15ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd15ffb700 1 -- 172.21.15.78:0/4229556073 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd200a1dc0 con 0x55749dc23470 2023-10-10T17:50:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd15ffb700 1 -- 172.21.15.78:0/4229556073 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7efd2000bcc0 con 0x55749dc23470 2023-10-10T17:50:25.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.475+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55749dc15b20 con 0x55749dc23470 2023-10-10T17:50:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd15ffb700 1 -- 172.21.15.78:0/4229556073 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd20098e60 con 0x55749dc23470 2023-10-10T17:50:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 10 client.17046 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:25.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55749db3a190 con 0x55749dc23470 2023-10-10T17:50:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd15ffb700 1 -- 172.21.15.78:0/4229556073 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55749dc15b20 con 0x55749dc23470 2023-10-10T17:50:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd15ffb700 1 -- 172.21.15.78:0/4229556073 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7efd20021bc0 con 0x55749dc23470 2023-10-10T17:50:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 10 client.17046 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:25.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 10 client.17046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 20 client.17046 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc3c810 0x55749dc3ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 4 client.17046 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55749db360a0 con 0x55749dc3c810 2023-10-10T17:50:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.476+0000 7efd2e59c700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc3c810 0x55749dc3ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd15ffb700 10 client.17046 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:25.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd2e59c700 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc3c810 0x55749dc3ec20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b04d40927e73926 server_cookie=f6f1b0c08f536895 in_seq=0 out_seq=0 2023-10-10T17:50:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd15ffb700 1 -- 172.21.15.78:0/4229556073 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55749db360a0 con 0x55749dc3c810 2023-10-10T17:50:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd15ffb700 10 client.17046 handle_command_reply: tid=0 2023-10-10T17:50:25.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd40f79b80 1 client.17046 shutdown 2023-10-10T17:50:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 msgr2=0x55749dc2c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd40f79b80 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc2c3b0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7efd20099670 tx=0x7efd20005700 comp rx=0 tx=0).stop 2023-10-10T17:50:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 shutdown_connections 2023-10-10T17:50:25.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd40f79b80 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc2f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd40f79b80 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc3c810 0x55749dc3ec20 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.477+0000 7efd40f79b80 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc278b0 0x55749dc2caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.478+0000 7efd15ffb700 0 client.17046 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.478+0000 7efd40f79b80 1 --2- 172.21.15.78:0/4229556073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc2c3b0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.478+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 >> 172.21.15.78:0/4229556073 conn(0x55749dc18360 msgr2=0x55749dc1cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.478+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 shutdown_connections 2023-10-10T17:50:25.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.478+0000 7efd40f79b80 1 -- 172.21.15.78:0/4229556073 wait complete. 2023-10-10T17:50:25.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.478+0000 7efd40f79b80 20 client.17046 trim_cache size 0 max 16384 2023-10-10T17:50:25.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.494+0000 7efd40f79b80 1 Processor -- start 2023-10-10T17:50:25.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.494+0000 7efd40f79b80 1 -- start start 2023-10-10T17:50:25.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.494+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc261a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc1d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc278b0 0x55749dc1daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd40f79b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749dc2f7c0 con 0x55749dc278b0 2023-10-10T17:50:25.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd40f79b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749dc1e740 con 0x55749dc23470 2023-10-10T17:50:25.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd40f79b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749dc2a910 con 0x55749dc22b60 2023-10-10T17:50:25.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc278b0 0x55749dc1daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc278b0 0x55749dc1daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47954/0 (socket says 172.21.15.78:47954) 2023-10-10T17:50:25.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 -- 172.21.15.78:0/2558596475 learned_addr learned my addr 172.21.15.78:0/2558596475 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:25.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc1d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2e59c700 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc261a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 -- 172.21.15.78:0/2558596475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 msgr2=0x55749dc261a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc261a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 -- 172.21.15.78:0/2558596475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 msgr2=0x55749dc1d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc1d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 -- 172.21.15.78:0/2558596475 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55749dc19b00 con 0x55749dc278b0 2023-10-10T17:50:25.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc278b0 0x55749dc1daa0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7efd280093c0 tx=0x7efd280093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e6b9fb8fe0d1b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd1d7fa700 1 -- 172.21.15.78:0/2558596475 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800e040 con 0x55749dc278b0 2023-10-10T17:50:25.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd1d7fa700 1 -- 172.21.15.78:0/2558596475 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd28007cd0 con 0x55749dc278b0 2023-10-10T17:50:25.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.495+0000 7efd1d7fa700 1 -- 172.21.15.78:0/2558596475 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd2800bc90 con 0x55749dc278b0 2023-10-10T17:50:25.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 -- 172.21.15.78:0/2558596475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc278b0 msgr2=0x55749dc1daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc278b0 0x55749dc1daa0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7efd280093c0 tx=0x7efd280093f0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 -- 172.21.15.78:0/2558596475 shutdown_connections 2023-10-10T17:50:25.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc22b60 0x55749dc261a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc278b0 0x55749dc1daa0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2558596475 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc23470 0x55749dc1d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 -- 172.21.15.78:0/2558596475 >> 172.21.15.78:0/2558596475 conn(0x55749dc18360 msgr2=0x55749dc26ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 -- 172.21.15.78:0/2558596475 shutdown_connections 2023-10-10T17:50:25.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 -- 172.21.15.78:0/2558596475 wait complete. 2023-10-10T17:50:25.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 -- start start 2023-10-10T17:50:25.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.496+0000 7efd40f79b80 1 Processor -- start 2023-10-10T17:50:25.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc22b60 0x55749dc24380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc248c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc8b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749dc8bbf0 con 0x55749dc23470 2023-10-10T17:50:25.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749dc8bd60 con 0x55749dc22b60 2023-10-10T17:50:25.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55749dc8bf20 con 0x55749dc278b0 2023-10-10T17:50:25.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 10 client.0 _ll_register_callbacks cb 0x7efd20099310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:25.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc248c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc248c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47958/0 (socket says 172.21.15.78:47958) 2023-10-10T17:50:25.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2392272071 learned_addr learned my addr 172.21.15.78:0/2392272071 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:25.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2e59c700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc22b60 0x55749dc24380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc8b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2392272071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 msgr2=0x55749dc8b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc8b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc22b60 msgr2=0x55749dc24380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:25.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc22b60 0x55749dc24380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 -- 172.21.15.78:0/2392272071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd3010c8a0 con 0x55749dc23470 2023-10-10T17:50:25.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2ed9d700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc8b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:25.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd2dd9b700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc248c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7efd20099900 tx=0x7efd20095cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7baa777efa6e4ece server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd15ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:25.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd15ffb700 1 -- 172.21.15.78:0/2392272071 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd20004c80 con 0x55749dc23470 2023-10-10T17:50:25.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd15ffb700 1 -- 172.21.15.78:0/2392272071 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7efd2000bcc0 con 0x55749dc23470 2023-10-10T17:50:25.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.497+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd2401f450 con 0x55749dc23470 2023-10-10T17:50:25.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd15ffb700 1 -- 172.21.15.78:0/2392272071 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd20099740 con 0x55749dc23470 2023-10-10T17:50:25.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 10 client.17052 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:25.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55749d9a1de0 con 0x55749dc23470 2023-10-10T17:50:25.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd15ffb700 1 -- 172.21.15.78:0/2392272071 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efd2401f450 con 0x55749dc23470 2023-10-10T17:50:25.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd15ffb700 1 -- 172.21.15.78:0/2392272071 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7efd200988d0 con 0x55749dc23470 2023-10-10T17:50:25.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 10 client.17052 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:25.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 10 client.17052 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:25.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 20 client.17052 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:25.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc93370 0x55749dc95780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 4 client.17052 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:50:25.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55749db42660 con 0x55749dc93370 2023-10-10T17:50:25.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd2e59c700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc93370 0x55749dc95780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd15ffb700 10 client.17052 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:25.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.498+0000 7efd2e59c700 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc93370 0x55749dc95780 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad0340db2d238ef6 server_cookie=dc57f6d8b4c2dee in_seq=0 out_seq=0 2023-10-10T17:50:25.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd15ffb700 1 -- 172.21.15.78:0/2392272071 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55749db42660 con 0x55749dc93370 2023-10-10T17:50:25.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd15ffb700 10 client.17052 handle_command_reply: tid=0 2023-10-10T17:50:25.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 client.17052 shutdown 2023-10-10T17:50:25.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 msgr2=0x55749dc248c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc248c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7efd20099900 tx=0x7efd20095cd0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 shutdown_connections 2023-10-10T17:50:25.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55749dc278b0 0x55749dc8b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55749dc93370 0x55749dc95780 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55749dc22b60 0x55749dc24380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd15ffb700 0 client.17052 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 --2- 172.21.15.78:0/2392272071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55749dc23470 0x55749dc248c0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 >> 172.21.15.78:0/2392272071 conn(0x55749dc18360 msgr2=0x55749dc21960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 shutdown_connections 2023-10-10T17:50:25.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 1 -- 172.21.15.78:0/2392272071 wait complete. 2023-10-10T17:50:25.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd40f79b80 20 client.17052 trim_cache size 0 max 16384 2023-10-10T17:50:25.540 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:50:25.541 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:50:25.541 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:50:25.541 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:50:25.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efd180a12c0 msgr2=0x7efd180a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd34ded700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efd180a12c0 0x7efd180a3770 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efd24005fd0 tx=0x7efd24009310 comp rx=0 tx=0).stop 2023-10-10T17:50:25.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd300ffdf0 msgr2=0x7efd301959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.499+0000 7efd34ded700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd300ffdf0 0x7efd301959e0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7efd2000d950 tx=0x7efd2000dd10 comp rx=0 tx=0).stop 2023-10-10T17:50:25.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 shutdown_connections 2023-10-10T17:50:25.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efd300ff450 0x7efd301954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efd180a12c0 0x7efd180a3770 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efd30104490 0x7efd30199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 --2- 172.21.15.78:0/2599339873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd300ffdf0 0x7efd301959e0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 >> 172.21.15.78:0/2599339873 conn(0x7efd300770b0 msgr2=0x7efd300fcc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 shutdown_connections 2023-10-10T17:50:25.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.500+0000 7efd34ded700 1 -- 172.21.15.78:0/2599339873 wait complete. 2023-10-10T17:50:25.547 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:50:25.547 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:50:25.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 -- 172.21.15.78:0/67239357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 msgr2=0x7f2338103d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 --2- 172.21.15.78:0/67239357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338103d60 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f23340099c0 tx=0x7f2334009cd0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 -- 172.21.15.78:0/67239357 shutdown_connections 2023-10-10T17:50:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 --2- 172.21.15.78:0/67239357 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380ff450 0x7f23380ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 --2- 172.21.15.78:0/67239357 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2338104440 0x7f233810c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 --2- 172.21.15.78:0/67239357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338103d60 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 -- 172.21.15.78:0/67239357 >> 172.21.15.78:0/67239357 conn(0x7f2338076100 msgr2=0x7f2338076500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 -- 172.21.15.78:0/67239357 shutdown_connections 2023-10-10T17:50:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.671+0000 7f233f2d1700 1 -- 172.21.15.78:0/67239357 wait complete. 2023-10-10T17:50:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 Processor -- start 2023-10-10T17:50:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 -- start start 2023-10-10T17:50:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380ff450 0x7f2338195300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2338104440 0x7f2338199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233819a2e0 con 0x7f23380ffdf0 2023-10-10T17:50:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233819a450 con 0x7f2338104440 2023-10-10T17:50:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233f2d1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233819a610 con 0x7f23380ff450 2023-10-10T17:50:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233c86c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233c86c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338195840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47972/0 (socket says 172.21.15.78:47972) 2023-10-10T17:50:25.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233c86c700 1 -- 172.21.15.78:0/927063011 learned_addr learned my addr 172.21.15.78:0/927063011 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:25.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.672+0000 7f233d06d700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380ff450 0x7f2338195300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233d86e700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2338104440 0x7f2338199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233c86c700 1 -- 172.21.15.78:0/927063011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380ff450 msgr2=0x7f2338195300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233c86c700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380ff450 0x7f2338195300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233c86c700 1 -- 172.21.15.78:0/927063011 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2338104440 msgr2=0x7f2338199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233c86c700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2338104440 0x7f2338199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233c86c700 1 -- 172.21.15.78:0/927063011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23340095f0 con 0x7f23380ffdf0 2023-10-10T17:50:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233c86c700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338195840 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2334009990 tx=0x7f2334004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f232e7fc700 1 -- 172.21.15.78:0/927063011 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f233401d070 con 0x7f23380ffdf0 2023-10-10T17:50:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f232e7fc700 1 -- 172.21.15.78:0/927063011 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2334003f90 con 0x7f23380ffdf0 2023-10-10T17:50:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f232e7fc700 1 -- 172.21.15.78:0/927063011 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2334022b70 con 0x7f23380ffdf0 2023-10-10T17:50:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f233810f1e0 con 0x7f23380ffdf0 2023-10-10T17:50:25.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.673+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233810f050 con 0x7f23380ffdf0 2023-10-10T17:50:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.674+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233810f370 con 0x7f23380ffdf0 2023-10-10T17:50:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.675+0000 7f232e7fc700 1 -- 172.21.15.78:0/927063011 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f2334003ad0 con 0x7f23380ffdf0 2023-10-10T17:50:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.676+0000 7f232e7fc700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f23240a12c0 0x7f23240a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.676+0000 7f233d06d700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f23240a12c0 0x7f23240a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.676+0000 7f232e7fc700 1 -- 172.21.15.78:0/927063011 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f23340ce4c0 con 0x7f23380ffdf0 2023-10-10T17:50:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.676+0000 7f233d06d700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f23240a12c0 0x7f23240a3770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2328004630 tx=0x7f232800b3c0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.680+0000 7f232e7fc700 1 -- 172.21.15.78:0/927063011 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f23340975d0 con 0x7f23380ffdf0 2023-10-10T17:50:25.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.911+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23380671d0 con 0x7f23380ffdf0 2023-10-10T17:50:25.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.912+0000 7f232e7fc700 1 -- 172.21.15.78:0/927063011 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10817 (secure 0 0 0) 0x7f233409abf0 con 0x7f23380ffdf0 2023-10-10T17:50:25.913 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:50:25.913 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:46:23.648656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:50:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f23240a12c0 msgr2=0x7f23240a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f23240a12c0 0x7f23240a3770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2328004630 tx=0x7f232800b3c0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 msgr2=0x7f2338195840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338195840 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2334009990 tx=0x7f2334004730 comp rx=0 tx=0).stop 2023-10-10T17:50:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 shutdown_connections 2023-10-10T17:50:25.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f23380ff450 0x7f2338195300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f23240a12c0 0x7f23240a3770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2338104440 0x7f2338199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 --2- 172.21.15.78:0/927063011 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f23380ffdf0 0x7f2338195840 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:25.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 >> 172.21.15.78:0/927063011 conn(0x7f2338076100 msgr2=0x7f23380fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:25.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 shutdown_connections 2023-10-10T17:50:25.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:25.914+0000 7f233f2d1700 1 -- 172.21.15.78:0/927063011 wait complete. 2023-10-10T17:50:25.918 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2023-10-10T17:50:25.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:50:26.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 -- 172.21.15.78:0/37059135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8241063f0 msgr2=0x7fb8241067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 --2- 172.21.15.78:0/37059135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8241063f0 0x7fb8241067c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fb818009990 tx=0x7fb818009ca0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 -- 172.21.15.78:0/37059135 shutdown_connections 2023-10-10T17:50:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 --2- 172.21.15.78:0/37059135 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb824101400 0x7fb82410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 --2- 172.21.15.78:0/37059135 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8241063f0 0x7fb8241067c0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 --2- 172.21.15.78:0/37059135 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb824100a50 0x7fb824100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 -- 172.21.15.78:0/37059135 >> 172.21.15.78:0/37059135 conn(0x7fb824077590 msgr2=0x7fb824077990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 -- 172.21.15.78:0/37059135 shutdown_connections 2023-10-10T17:50:26.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.050+0000 7fb82ad99700 1 -- 172.21.15.78:0/37059135 wait complete. 2023-10-10T17:50:26.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 Processor -- start 2023-10-10T17:50:26.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 -- start start 2023-10-10T17:50:26.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb824100a50 0x7fb8241964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb824101400 0x7fb8241969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8241063f0 0x7fb82419ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82419b440 con 0x7fb824100a50 2023-10-10T17:50:26.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82419b5b0 con 0x7fb824101400 2023-10-10T17:50:26.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb82ad99700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82419b770 con 0x7fb8241063f0 2023-10-10T17:50:26.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb824100a50 0x7fb8241964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb824100a50 0x7fb8241964b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47984/0 (socket says 172.21.15.78:47984) 2023-10-10T17:50:26.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 -- 172.21.15.78:0/3906433913 learned_addr learned my addr 172.21.15.78:0/3906433913 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb829336700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8241063f0 0x7fb82419ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb823fff700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb824101400 0x7fb8241969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 -- 172.21.15.78:0/3906433913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8241063f0 msgr2=0x7fb82419ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8241063f0 0x7fb82419ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 -- 172.21.15.78:0/3906433913 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb824101400 msgr2=0x7fb8241969f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb824101400 0x7fb8241969f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb828b35700 1 -- 172.21.15.78:0/3906433913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8180095f0 con 0x7fb824100a50 2023-10-10T17:50:26.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.051+0000 7fb823fff700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb824101400 0x7fb8241969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.052+0000 7fb828b35700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb824100a50 0x7fb8241964b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb818006010 tx=0x7fb81800f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.052+0000 7fb821ffb700 1 -- 172.21.15.78:0/3906433913 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81801d070 con 0x7fb824100a50 2023-10-10T17:50:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.052+0000 7fb821ffb700 1 -- 172.21.15.78:0/3906433913 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb81800fb80 con 0x7fb824100a50 2023-10-10T17:50:26.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.052+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb824110450 con 0x7fb824100a50 2023-10-10T17:50:26.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.052+0000 7fb821ffb700 1 -- 172.21.15.78:0/3906433913 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818017650 con 0x7fb824100a50 2023-10-10T17:50:26.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.052+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8241102c0 con 0x7fb824100a50 2023-10-10T17:50:26.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.054+0000 7fb821ffb700 1 -- 172.21.15.78:0/3906433913 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fb81800fcf0 con 0x7fb824100a50 2023-10-10T17:50:26.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.055+0000 7fb821ffb700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8140a12c0 0x7fb8140a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.055+0000 7fb821ffb700 1 -- 172.21.15.78:0/3906433913 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb8180ce080 con 0x7fb824100a50 2023-10-10T17:50:26.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.055+0000 7fb823fff700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8140a12c0 0x7fb8140a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.055+0000 7fb823fff700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8140a12c0 0x7fb8140a3770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb824197a70 tx=0x7fb810008040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 Processor -- start 2023-10-10T17:50:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 -- start start 2023-10-10T17:50:26.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b8470 0x556ef03bc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03bc8b0 0x556ef03bcd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef02bd060 con 0x556ef03b8470 2023-10-10T17:50:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef030f8a0 con 0x556ef03bc8b0 2023-10-10T17:50:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb836f25b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef0308d30 con 0x556ef03b7b60 2023-10-10T17:50:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b8470 0x556ef03bc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b8470 0x556ef03bc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47994/0 (socket says 172.21.15.78:47994) 2023-10-10T17:50:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 -- 172.21.15.78:0/168547773 learned_addr learned my addr 172.21.15.78:0/168547773 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb828b35700 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 -- 172.21.15.78:0/168547773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 msgr2=0x556ef03b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 -- 172.21.15.78:0/168547773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03bc8b0 msgr2=0x556ef03bcd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03bc8b0 0x556ef03bcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 -- 172.21.15.78:0/168547773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ef018a630 con 0x556ef03b8470 2023-10-10T17:50:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb823fff700 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b8470 0x556ef03bc370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb81000c910 tx=0x7fb81000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7ed3d6311003178 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.056+0000 7fb80effd700 1 -- 172.21.15.78:0/168547773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb810007e10 con 0x556ef03b8470 2023-10-10T17:50:26.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb80effd700 1 -- 172.21.15.78:0/168547773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb810013df0 con 0x556ef03b8470 2023-10-10T17:50:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb80effd700 1 -- 172.21.15.78:0/168547773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8100113d0 con 0x556ef03b8470 2023-10-10T17:50:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 -- 172.21.15.78:0/168547773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b8470 msgr2=0x556ef03bc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b8470 0x556ef03bc370 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fb81000c910 tx=0x7fb81000f000 comp rx=0 tx=0).stop 2023-10-10T17:50:26.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 -- 172.21.15.78:0/168547773 shutdown_connections 2023-10-10T17:50:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03bc8b0 0x556ef03bcd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 --2- 172.21.15.78:0/168547773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b8470 0x556ef03bc370 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 -- 172.21.15.78:0/168547773 >> 172.21.15.78:0/168547773 conn(0x556ef03ad360 msgr2=0x556ef03b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 -- 172.21.15.78:0/168547773 shutdown_connections 2023-10-10T17:50:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.057+0000 7fb836f25b80 1 -- 172.21.15.78:0/168547773 wait complete. 2023-10-10T17:50:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 -- start start 2023-10-10T17:50:26.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 Processor -- start 2023-10-10T17:50:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b7b60 0x556ef03c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03bc8b0 0x556ef03c1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef02aa8e0 con 0x556ef03b7b60 2023-10-10T17:50:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef02aa580 con 0x556ef03b8470 2023-10-10T17:50:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef018c920 con 0x556ef03bc8b0 2023-10-10T17:50:26.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb836f25b80 10 client.0 _ll_register_callbacks cb 0x556ef02c8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb828b35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b7b60 0x556ef03c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb828b35700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b7b60 0x556ef03c1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:47998/0 (socket says 172.21.15.78:47998) 2023-10-10T17:50:26.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb828b35700 1 -- 172.21.15.78:0/1598685221 learned_addr learned my addr 172.21.15.78:0/1598685221 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb829336700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03bc8b0 0x556ef03c1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb828b35700 1 -- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03bc8b0 msgr2=0x556ef03c1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb828b35700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03bc8b0 0x556ef03c1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb823fff700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb828b35700 1 -- 172.21.15.78:0/1598685221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 msgr2=0x556ef03c18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.058+0000 7fb828b35700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb828b35700 1 -- 172.21.15.78:0/1598685221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81000c680 con 0x556ef03b7b60 2023-10-10T17:50:26.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb829336700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03bc8b0 0x556ef03c1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:26.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb828b35700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b7b60 0x556ef03c1390 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb818000c00 tx=0x7fb81809ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeb903022e07a4fd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb803fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:26.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb803fff700 1 -- 172.21.15.78:0/1598685221 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81809b320 con 0x556ef03b7b60 2023-10-10T17:50:26.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb803fff700 1 -- 172.21.15.78:0/1598685221 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb81809a4e0 con 0x556ef03b7b60 2023-10-10T17:50:26.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ef03aab20 con 0x556ef03b7b60 2023-10-10T17:50:26.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb803fff700 1 -- 172.21.15.78:0/1598685221 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8180b9080 con 0x556ef03b7b60 2023-10-10T17:50:26.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 10 client.17070 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x556ef02cf190 con 0x556ef03b7b60 2023-10-10T17:50:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb803fff700 1 -- 172.21.15.78:0/1598685221 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8180c9020 con 0x556ef03b7b60 2023-10-10T17:50:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb803fff700 1 -- 172.21.15.78:0/1598685221 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb818096be0 con 0x556ef03b7b60 2023-10-10T17:50:26.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 10 client.17070 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 10 client.17070 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:50:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 20 client.17070 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:26.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef03d1640 0x556ef03d3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 4 client.17070 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.059+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ef02cb0a0 con 0x556ef03d1640 2023-10-10T17:50:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.060+0000 7fb823fff700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef03d1640 0x556ef03d3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.060+0000 7fb803fff700 10 client.17070 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:26.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.060+0000 7fb823fff700 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef03d1640 0x556ef03d3a50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1cf51be8cc22abb5 server_cookie=ef5dde86ba04ee0b in_seq=0 out_seq=0 2023-10-10T17:50:26.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb803fff700 1 -- 172.21.15.78:0/1598685221 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x556ef02cb0a0 con 0x556ef03d1640 2023-10-10T17:50:26.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb803fff700 10 client.17070 handle_command_reply: tid=0 2023-10-10T17:50:26.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 client.17070 shutdown 2023-10-10T17:50:26.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b7b60 msgr2=0x556ef03c1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b7b60 0x556ef03c1390 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fb818000c00 tx=0x7fb81809ee20 comp rx=0 tx=0).stop 2023-10-10T17:50:26.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 shutdown_connections 2023-10-10T17:50:26.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef03d1640 0x556ef03d3a50 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03bc8b0 0x556ef03c1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03b7b60 0x556ef03c1390 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1598685221 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb803fff700 0 client.17070 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:26.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 >> 172.21.15.78:0/1598685221 conn(0x556ef03ad360 msgr2=0x556ef03b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 shutdown_connections 2023-10-10T17:50:26.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 1 -- 172.21.15.78:0/1598685221 wait complete. 2023-10-10T17:50:26.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.061+0000 7fb836f25b80 20 client.17070 trim_cache size 0 max 16384 2023-10-10T17:50:26.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 Processor -- start 2023-10-10T17:50:26.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 -- start start 2023-10-10T17:50:26.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03ba1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c89a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef03bb520 con 0x556ef03bc8b0 2023-10-10T17:50:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef03c2820 con 0x556ef03b8470 2023-10-10T17:50:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb836f25b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef03c8ee0 con 0x556ef03b7b60 2023-10-10T17:50:26.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48010/0 (socket says 172.21.15.78:48010) 2023-10-10T17:50:26.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 -- 172.21.15.78:0/360912694 learned_addr learned my addr 172.21.15.78:0/360912694 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb828b35700 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03ba1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb823fff700 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 -- 172.21.15.78:0/360912694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 msgr2=0x556ef03ba1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03ba1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 -- 172.21.15.78:0/360912694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 msgr2=0x556ef03c89a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c89a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.074+0000 7fb829336700 1 -- 172.21.15.78:0/360912694 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ef03aeb00 con 0x556ef03bc8b0 2023-10-10T17:50:26.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb829336700 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c1c90 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb81c008740 tx=0x7fb81c008a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34499216430676f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb80f7fe700 1 -- 172.21.15.78:0/360912694 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c015610 con 0x556ef03bc8b0 2023-10-10T17:50:26.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb80f7fe700 1 -- 172.21.15.78:0/360912694 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb81c010050 con 0x556ef03bc8b0 2023-10-10T17:50:26.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb80f7fe700 1 -- 172.21.15.78:0/360912694 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c014c40 con 0x556ef03bc8b0 2023-10-10T17:50:26.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 -- 172.21.15.78:0/360912694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 msgr2=0x556ef03c1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c1c90 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb81c008740 tx=0x7fb81c008a50 comp rx=0 tx=0).stop 2023-10-10T17:50:26.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 -- 172.21.15.78:0/360912694 shutdown_connections 2023-10-10T17:50:26.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b7b60 0x556ef03ba1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c1c90 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 --2- 172.21.15.78:0/360912694 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b8470 0x556ef03c89a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 -- 172.21.15.78:0/360912694 >> 172.21.15.78:0/360912694 conn(0x556ef03ad360 msgr2=0x556ef03cdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 -- 172.21.15.78:0/360912694 shutdown_connections 2023-10-10T17:50:26.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.075+0000 7fb836f25b80 1 -- 172.21.15.78:0/360912694 wait complete. 2023-10-10T17:50:26.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 -- start start 2023-10-10T17:50:26.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 Processor -- start 2023-10-10T17:50:26.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b7b60 0x556ef03b2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b8470 0x556ef03c4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef03c56d0 con 0x556ef03bc8b0 2023-10-10T17:50:26.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef03c5840 con 0x556ef03b7b60 2023-10-10T17:50:26.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ef0424570 con 0x556ef03b8470 2023-10-10T17:50:26.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb836f25b80 10 client.0 _ll_register_callbacks cb 0x7fb818097130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:26.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb829336700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb829336700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48016/0 (socket says 172.21.15.78:48016) 2023-10-10T17:50:26.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb829336700 1 -- 172.21.15.78:0/1933842329 learned_addr learned my addr 172.21.15.78:0/1933842329 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb828b35700 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b7b60 0x556ef03b2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb823fff700 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b8470 0x556ef03c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb829336700 1 -- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b8470 msgr2=0x556ef03c4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb829336700 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b8470 0x556ef03c4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb829336700 1 -- 172.21.15.78:0/1933842329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b7b60 msgr2=0x556ef03b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.076+0000 7fb829336700 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b7b60 0x556ef03b2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb829336700 1 -- 172.21.15.78:0/1933842329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c0083f0 con 0x556ef03bc8b0 2023-10-10T17:50:26.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb829336700 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c4f90 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb81c006090 tx=0x7fb81c014f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ea644aa113490e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb803fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:26.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb803fff700 1 -- 172.21.15.78:0/1933842329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c014370 con 0x556ef03bc8b0 2023-10-10T17:50:26.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb803fff700 1 -- 172.21.15.78:0/1933842329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb81c003a00 con 0x556ef03bc8b0 2023-10-10T17:50:26.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb810013df0 con 0x556ef03bc8b0 2023-10-10T17:50:26.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb803fff700 1 -- 172.21.15.78:0/1933842329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb81c003bc0 con 0x556ef03bc8b0 2023-10-10T17:50:26.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 10 client.17076 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:26.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x556ef0136de0 con 0x556ef03bc8b0 2023-10-10T17:50:26.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb803fff700 1 -- 172.21.15.78:0/1933842329 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb810013df0 con 0x556ef03bc8b0 2023-10-10T17:50:26.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb803fff700 1 -- 172.21.15.78:0/1933842329 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fb81c015530 con 0x556ef03bc8b0 2023-10-10T17:50:26.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 10 client.17076 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:26.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 10 client.17076 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:50:26.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 20 client.17076 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:26.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef0425c10 0x556ef0428020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 4 client.17076 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:50:26.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.077+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556ef02d7660 con 0x556ef0425c10 2023-10-10T17:50:26.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb828b35700 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef0425c10 0x556ef0428020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb803fff700 10 client.17076 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:26.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb828b35700 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef0425c10 0x556ef0428020 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b5c265970432fad server_cookie=3099706490dfe80 in_seq=0 out_seq=0 2023-10-10T17:50:26.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb803fff700 1 -- 172.21.15.78:0/1933842329 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556ef02d7660 con 0x556ef0425c10 2023-10-10T17:50:26.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb803fff700 10 client.17076 handle_command_reply: tid=0 2023-10-10T17:50:26.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 client.17076 shutdown 2023-10-10T17:50:26.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 msgr2=0x556ef03c4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c4f90 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb81c006090 tx=0x7fb81c014f50 comp rx=0 tx=0).stop 2023-10-10T17:50:26.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 shutdown_connections 2023-10-10T17:50:26.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556ef03bc8b0 0x556ef03c4f90 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556ef03b7b60 0x556ef03b2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x556ef0425c10 0x556ef0428020 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 --2- 172.21.15.78:0/1933842329 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556ef03b8470 0x556ef03c4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb803fff700 0 client.17076 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:26.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.078+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 >> 172.21.15.78:0/1933842329 conn(0x556ef03ad360 msgr2=0x556ef03bec90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 shutdown_connections 2023-10-10T17:50:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb836f25b80 1 -- 172.21.15.78:0/1933842329 wait complete. 2023-10-10T17:50:26.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb836f25b80 20 client.17076 trim_cache size 0 max 16384 2023-10-10T17:50:26.124 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:50:26.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8140a12c0 msgr2=0x7fb8140a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8140a12c0 0x7fb8140a3770 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb824197a70 tx=0x7fb810008040 comp rx=0 tx=0).stop 2023-10-10T17:50:26.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb824100a50 msgr2=0x7fb8241964b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb824100a50 0x7fb8241964b0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb818006010 tx=0x7fb81800f690 comp rx=0 tx=0).stop 2023-10-10T17:50:26.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 shutdown_connections 2023-10-10T17:50:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb8241063f0 0x7fb82419ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb8140a12c0 0x7fb8140a3770 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb824100a50 0x7fb8241964b0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 --2- 172.21.15.78:0/3906433913 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb824101400 0x7fb8241969f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 >> 172.21.15.78:0/3906433913 conn(0x7fb824077590 msgr2=0x7fb8240ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 shutdown_connections 2023-10-10T17:50:26.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.079+0000 7fb82ad99700 1 -- 172.21.15.78:0/3906433913 wait complete. 2023-10-10T17:50:26.129 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:50:26.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 -- 172.21.15.78:0/3623708812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb80ffec0 msgr2=0x7f8eb8103e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 --2- 172.21.15.78:0/3623708812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb80ffec0 0x7f8eb8103e80 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f8eac0099c0 tx=0x7f8eac009cd0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 -- 172.21.15.78:0/3623708812 shutdown_connections 2023-10-10T17:50:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 --2- 172.21.15.78:0/3623708812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eb80ff520 0x7f8eb80ff8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 --2- 172.21.15.78:0/3623708812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8eb8104560 0x7f8eb810c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 --2- 172.21.15.78:0/3623708812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb80ffec0 0x7f8eb8103e80 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 -- 172.21.15.78:0/3623708812 >> 172.21.15.78:0/3623708812 conn(0x7f8eb80760d0 msgr2=0x7f8eb80764d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 -- 172.21.15.78:0/3623708812 shutdown_connections 2023-10-10T17:50:26.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.251+0000 7f8ec0019700 1 -- 172.21.15.78:0/3623708812 wait complete. 2023-10-10T17:50:26.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 Processor -- start 2023-10-10T17:50:26.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 -- start start 2023-10-10T17:50:26.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eb80ff520 0x7f8eb8195320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8eb80ffec0 0x7f8eb8195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb8104560 0x7f8eb8199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb819a300 con 0x7f8eb8104560 2023-10-10T17:50:26.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb819a470 con 0x7f8eb80ffec0 2023-10-10T17:50:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ec0019700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb819a630 con 0x7f8eb80ff520 2023-10-10T17:50:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ebe5b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb8104560 0x7f8eb8199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.252+0000 7f8ebe5b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb8104560 0x7f8eb8199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48030/0 (socket says 172.21.15.78:48030) 2023-10-10T17:50:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/210799691 learned_addr learned my addr 172.21.15.78:0/210799691 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eb80ff520 0x7f8eb8195320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8eb80ffec0 0x7f8eb8195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/210799691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eb80ff520 msgr2=0x7f8eb8195320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eb80ff520 0x7f8eb8195320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/210799691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8eb80ffec0 msgr2=0x7f8eb8195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8eb80ffec0 0x7f8eb8195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/210799691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eac0095f0 con 0x7f8eb8104560 2023-10-10T17:50:26.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eb80ff520 0x7f8eb8195320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:26.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8eb80ffec0 0x7f8eb8195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:26.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb8104560 0x7f8eb8199cc0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800e940 tx=0x7f8ea800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8eb6ffd700 1 -- 172.21.15.78:0/210799691 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80097b0 con 0x7f8eb8104560 2023-10-10T17:50:26.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8eb6ffd700 1 -- 172.21.15.78:0/210799691 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ea8004cf0 con 0x7f8eb8104560 2023-10-10T17:50:26.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb810f2e0 con 0x7f8eb8104560 2023-10-10T17:50:26.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8eb6ffd700 1 -- 172.21.15.78:0/210799691 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80055d0 con 0x7f8eb8104560 2023-10-10T17:50:26.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.253+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb810f150 con 0x7f8eb8104560 2023-10-10T17:50:26.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.255+0000 7f8eb6ffd700 1 -- 172.21.15.78:0/210799691 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f8ea800cc60 con 0x7f8eb8104560 2023-10-10T17:50:26.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.256+0000 7f8eb6ffd700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ea00a12c0 0x7f8ea00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.256+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ea00a12c0 0x7f8ea00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.256+0000 7f8eb6ffd700 1 -- 172.21.15.78:0/210799691 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f8ea8014070 con 0x7f8eb8104560 2023-10-10T17:50:26.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.256+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ea00a12c0 0x7f8ea00a3770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009dd0 tx=0x7f8eb0009040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.256+0000 7f8ecc1a5b80 1 Processor -- start 2023-10-10T17:50:26.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ecc1a5b80 1 -- start start 2023-10-10T17:50:26.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e369f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e36e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e36ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e26f060 con 0x55a06e36e8b0 2023-10-10T17:50:26.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e2c18a0 con 0x55a06e369b60 2023-10-10T17:50:26.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e2bad30 con 0x55a06e36a470 2023-10-10T17:50:26.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e36ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e36ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48038/0 (socket says 172.21.15.78:48038) 2023-10-10T17:50:26.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/313939588 learned_addr learned my addr 172.21.15.78:0/313939588 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e36e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e369f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/313939588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 msgr2=0x55a06e36e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e36e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/313939588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 msgr2=0x55a06e369f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e369f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/313939588 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a06e13c630 con 0x55a06e36e8b0 2023-10-10T17:50:26.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.257+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e36ed00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8096b00 tx=0x7f8ea8099ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee1917e5aa8aec server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8e9ffff700 1 -- 172.21.15.78:0/313939588 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80b1070 con 0x55a06e36e8b0 2023-10-10T17:50:26.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8e9ffff700 1 -- 172.21.15.78:0/313939588 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ea809a090 con 0x55a06e36e8b0 2023-10-10T17:50:26.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8e9ffff700 1 -- 172.21.15.78:0/313939588 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea809a2f0 con 0x55a06e36e8b0 2023-10-10T17:50:26.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/313939588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 msgr2=0x55a06e36ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e36ed00 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8ea8096b00 tx=0x7f8ea8099ca0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/313939588 shutdown_connections 2023-10-10T17:50:26.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e36ed00 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e369f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/313939588 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e36e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/313939588 >> 172.21.15.78:0/313939588 conn(0x55a06e35f360 msgr2=0x55a06e364b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/313939588 shutdown_connections 2023-10-10T17:50:26.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.258+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/313939588 wait complete. 2023-10-10T17:50:26.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 -- start start 2023-10-10T17:50:26.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 Processor -- start 2023-10-10T17:50:26.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e3760a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36a470 0x55a06e373320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 0x55a06e373a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e25c8e0 con 0x55a06e36a470 2023-10-10T17:50:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e25c580 con 0x55a06e369b60 2023-10-10T17:50:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e13e920 con 0x55a06e36e8b0 2023-10-10T17:50:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ecc1a5b80 10 client.0 _ll_register_callbacks cb 0x55a06e27a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ebd5b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36a470 0x55a06e373320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ebd5b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36a470 0x55a06e373320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48042/0 (socket says 172.21.15.78:48042) 2023-10-10T17:50:26.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ebd5b4700 1 -- 172.21.15.78:0/2065968348 learned_addr learned my addr 172.21.15.78:0/2065968348 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.259+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e3760a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 0x55a06e373a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ebd5b4700 1 -- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 msgr2=0x55a06e373a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 0x55a06e373a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ebd5b4700 1 -- 172.21.15.78:0/2065968348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 msgr2=0x55a06e3760a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e3760a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ebd5b4700 1 -- 172.21.15.78:0/2065968348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb810f150 con 0x55a06e36a470 2023-10-10T17:50:26.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36a470 0x55a06e373320 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8100290 tx=0x7f8eac004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e275ff7ed7dd507e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8e9d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/2065968348 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eac027070 con 0x55a06e36a470 2023-10-10T17:50:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/2065968348 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f8eac0053d0 con 0x55a06e36a470 2023-10-10T17:50:26.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a06e35cb20 con 0x55a06e36a470 2023-10-10T17:50:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/2065968348 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8eac01d050 con 0x55a06e36a470 2023-10-10T17:50:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ecc1a5b80 10 client.17088 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a06e281190 con 0x55a06e36a470 2023-10-10T17:50:26.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.260+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/2065968348 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a06e35cb20 con 0x55a06e36a470 2023-10-10T17:50:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/2065968348 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8eac02a020 con 0x55a06e36a470 2023-10-10T17:50:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ecc1a5b80 10 client.17088 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:26.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ecc1a5b80 10 client.17088 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:50:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ecc1a5b80 20 client.17088 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3839d0 0x55a06e385de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ecc1a5b80 4 client.17088 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:26.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a06e27d0a0 con 0x55a06e3839d0 2023-10-10T17:50:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3839d0 0x55a06e385de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8e9d7fa700 10 client.17088 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:26.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.261+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3839d0 0x55a06e385de0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa0c8d733406d083 server_cookie=95df096c744da43 in_seq=0 out_seq=0 2023-10-10T17:50:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/2065968348 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a06e27d0a0 con 0x55a06e3839d0 2023-10-10T17:50:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8e9d7fa700 10 client.17088 handle_command_reply: tid=0 2023-10-10T17:50:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 client.17088 shutdown 2023-10-10T17:50:26.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36a470 msgr2=0x55a06e373320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36a470 0x55a06e373320 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8100290 tx=0x7f8eac004f90 comp rx=0 tx=0).stop 2023-10-10T17:50:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 shutdown_connections 2023-10-10T17:50:26.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3839d0 0x55a06e385de0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 0x55a06e373a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8e9d7fa700 0 client.17088 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e3760a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/2065968348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36a470 0x55a06e373320 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 >> 172.21.15.78:0/2065968348 conn(0x55a06e35f360 msgr2=0x55a06e36fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 shutdown_connections 2023-10-10T17:50:26.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/2065968348 wait complete. 2023-10-10T17:50:26.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.262+0000 7f8ecc1a5b80 20 client.17088 trim_cache size 0 max 16384 2023-10-10T17:50:26.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 Processor -- start 2023-10-10T17:50:26.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 -- start start 2023-10-10T17:50:26.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e369b60 0x55a06e37a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e36a470 0x55a06e36c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 0x55a06e36ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e36a9b0 con 0x55a06e369b60 2023-10-10T17:50:26.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e370390 con 0x55a06e36a470 2023-10-10T17:50:26.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e380fe0 con 0x55a06e36e8b0 2023-10-10T17:50:26.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.275+0000 7f8ebddb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e369b60 0x55a06e37a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e369b60 0x55a06e37a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48048/0 (socket says 172.21.15.78:48048) 2023-10-10T17:50:26.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 -- 172.21.15.78:0/1458629540 learned_addr learned my addr 172.21.15.78:0/1458629540 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e36a470 0x55a06e36c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 -- 172.21.15.78:0/1458629540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 msgr2=0x55a06e36ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 0x55a06e36ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 -- 172.21.15.78:0/1458629540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e36a470 msgr2=0x55a06e36c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e36a470 0x55a06e36c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 -- 172.21.15.78:0/1458629540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a06e280ee0 con 0x55a06e369b60 2023-10-10T17:50:26.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e36a470 0x55a06e36c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:26.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e369b60 0x55a06e37a9a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0007cf0 tx=0x7f8eb0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdddb36ef4eb0ebe server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8eb4ff9700 1 -- 172.21.15.78:0/1458629540 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0016df0 con 0x55a06e369b60 2023-10-10T17:50:26.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8eb4ff9700 1 -- 172.21.15.78:0/1458629540 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8eb0004020 con 0x55a06e369b60 2023-10-10T17:50:26.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.276+0000 7f8eb4ff9700 1 -- 172.21.15.78:0/1458629540 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb0031680 con 0x55a06e369b60 2023-10-10T17:50:26.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/1458629540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e369b60 msgr2=0x55a06e37a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e369b60 0x55a06e37a9a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0007cf0 tx=0x7f8eb0015000 comp rx=0 tx=0).stop 2023-10-10T17:50:26.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/1458629540 shutdown_connections 2023-10-10T17:50:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36e8b0 0x55a06e36ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e369b60 0x55a06e37a9a0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/1458629540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e36a470 0x55a06e36c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/1458629540 >> 172.21.15.78:0/1458629540 conn(0x55a06e35f360 msgr2=0x55a06e37fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/1458629540 shutdown_connections 2023-10-10T17:50:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/1458629540 wait complete. 2023-10-10T17:50:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- start start 2023-10-10T17:50:26.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 Processor -- start 2023-10-10T17:50:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e372c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e373170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e371ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e372010 con 0x55a06e36e8b0 2023-10-10T17:50:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e372180 con 0x55a06e369b60 2023-10-10T17:50:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.277+0000 7f8ecc1a5b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a06e372340 con 0x55a06e36a470 2023-10-10T17:50:26.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ecc1a5b80 10 client.0 _ll_register_callbacks cb 0x7f8ea8005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e371ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e371ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48058/0 (socket says 172.21.15.78:48058) 2023-10-10T17:50:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/575691283 learned_addr learned my addr 172.21.15.78:0/575691283 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e372c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 msgr2=0x55a06e373170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebd5b4700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e373170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e373170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/575691283 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 msgr2=0x55a06e372c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e372c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 -- 172.21.15.78:0/575691283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb001e040 con 0x55a06e36e8b0 2023-10-10T17:50:26.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e372c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ebe5b6700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e371ad0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80983e0 tx=0x7f8ea8005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce62acb026e8e90 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8e9d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/575691283 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea80b1070 con 0x55a06e36e8b0 2023-10-10T17:50:26.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/575691283 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f8ea809e0c0 con 0x55a06e36e8b0 2023-10-10T17:50:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ea809a090 con 0x55a06e36e8b0 2023-10-10T17:50:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/575691283 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ea8099790 con 0x55a06e36e8b0 2023-10-10T17:50:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ecc1a5b80 10 client.17094 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:26.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55a06e0d1c60 con 0x55a06e36e8b0 2023-10-10T17:50:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.278+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/575691283 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8ea809a090 con 0x55a06e36e8b0 2023-10-10T17:50:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/575691283 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f8ea8095e80 con 0x55a06e36e8b0 2023-10-10T17:50:26.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ecc1a5b80 10 client.17094 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ecc1a5b80 10 client.17094 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:50:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ecc1a5b80 20 client.17094 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3da440 0x55a06e3dc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ecc1a5b80 4 client.17094 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:50:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a06e289660 con 0x55a06e3da440 2023-10-10T17:50:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3da440 0x55a06e3dc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8ebddb5700 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3da440 0x55a06e3dc850 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d8f8bcdb008e175a server_cookie=f14b495cc6519587 in_seq=0 out_seq=0 2023-10-10T17:50:26.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.279+0000 7f8e9d7fa700 10 client.17094 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8e9d7fa700 1 -- 172.21.15.78:0/575691283 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a06e289660 con 0x55a06e3da440 2023-10-10T17:50:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8e9d7fa700 10 client.17094 handle_command_reply: tid=0 2023-10-10T17:50:26.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 client.17094 shutdown 2023-10-10T17:50:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 msgr2=0x55a06e371ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e371ad0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8ea80983e0 tx=0x7f8ea8005730 comp rx=0 tx=0).stop 2023-10-10T17:50:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 shutdown_connections 2023-10-10T17:50:26.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a06e36e8b0 0x55a06e371ad0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a06e369b60 0x55a06e372c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55a06e3da440 0x55a06e3dc850 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 --2- 172.21.15.78:0/575691283 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a06e36a470 0x55a06e373170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8e9d7fa700 0 client.17094 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 >> 172.21.15.78:0/575691283 conn(0x55a06e35f360 msgr2=0x55a06e36c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 shutdown_connections 2023-10-10T17:50:26.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 1 -- 172.21.15.78:0/575691283 wait complete. 2023-10-10T17:50:26.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ecc1a5b80 20 client.17094 trim_cache size 0 max 16384 2023-10-10T17:50:26.322 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:50:26.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ea00a12c0 msgr2=0x7f8ea00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ea00a12c0 0x7f8ea00a3770 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8eb0009dd0 tx=0x7f8eb0009040 comp rx=0 tx=0).stop 2023-10-10T17:50:26.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb8104560 msgr2=0x7f8eb8199cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb8104560 0x7f8eb8199cc0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8ea800e940 tx=0x7f8ea800ed00 comp rx=0 tx=0).stop 2023-10-10T17:50:26.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 shutdown_connections 2023-10-10T17:50:26.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8eb80ff520 0x7f8eb8195320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f8ea00a12c0 0x7f8ea00a3770 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.280+0000 7f8ec0019700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8eb8104560 0x7f8eb8199cc0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.281+0000 7f8ec0019700 1 --2- 172.21.15.78:0/210799691 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8eb80ffec0 0x7f8eb8195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.281+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 >> 172.21.15.78:0/210799691 conn(0x7f8eb80760d0 msgr2=0x7f8eb80fcdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.281+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 shutdown_connections 2023-10-10T17:50:26.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.281+0000 7f8ec0019700 1 -- 172.21.15.78:0/210799691 wait complete. 2023-10-10T17:50:26.327 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:50:26.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 -- 172.21.15.78:0/2632834850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4100ffe20 msgr2=0x7fa41010a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 --2- 172.21.15.78:0/2632834850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4100ffe20 0x7fa41010a890 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009990 tx=0x7fa3f8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 -- 172.21.15.78:0/2632834850 shutdown_connections 2023-10-10T17:50:26.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 --2- 172.21.15.78:0/2632834850 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa41010add0 0x7fa41010d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 --2- 172.21.15.78:0/2632834850 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4100ff510 0x7fa4100ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 --2- 172.21.15.78:0/2632834850 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4100ffe20 0x7fa41010a890 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 -- 172.21.15.78:0/2632834850 >> 172.21.15.78:0/2632834850 conn(0x7fa410074b00 msgr2=0x7fa410074f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 -- 172.21.15.78:0/2632834850 shutdown_connections 2023-10-10T17:50:26.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.450+0000 7fa415d06700 1 -- 172.21.15.78:0/2632834850 wait complete. 2023-10-10T17:50:26.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.451+0000 7fa415d06700 1 Processor -- start 2023-10-10T17:50:26.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.451+0000 7fa415d06700 1 -- start start 2023-10-10T17:50:26.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.451+0000 7fa415d06700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4100ff510 0x7fa4101963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa415d06700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4100ffe20 0x7fa4101968f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa415d06700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa41010add0 0x7fa41019ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa415d06700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41019b390 con 0x7fa41010add0 2023-10-10T17:50:26.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa415d06700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41019b500 con 0x7fa4100ffe20 2023-10-10T17:50:26.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa415d06700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa41019b6c0 con 0x7fa4100ff510 2023-10-10T17:50:26.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa41010add0 0x7fa41019ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa41010add0 0x7fa41019ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48072/0 (socket says 172.21.15.78:48072) 2023-10-10T17:50:26.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 -- 172.21.15.78:0/1581714751 learned_addr learned my addr 172.21.15.78:0/1581714751 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4100ff510 0x7fa4101963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40effd700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4100ffe20 0x7fa4101968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 -- 172.21.15.78:0/1581714751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4100ff510 msgr2=0x7fa4101963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4100ff510 0x7fa4101963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 -- 172.21.15.78:0/1581714751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4100ffe20 msgr2=0x7fa4101968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4100ffe20 0x7fa4101968f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 -- 172.21.15.78:0/1581714751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f80095f0 con 0x7fa41010add0 2023-10-10T17:50:26.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40ffff700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa41010add0 0x7fa41019ad50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa408007ee0 tx=0x7fa40800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40cff9700 1 -- 172.21.15.78:0/1581714751 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408004db0 con 0x7fa41010add0 2023-10-10T17:50:26.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4101103b0 con 0x7fa41010add0 2023-10-10T17:50:26.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40cff9700 1 -- 172.21.15.78:0/1581714751 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40800f040 con 0x7fa41010add0 2023-10-10T17:50:26.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa40cff9700 1 -- 172.21.15.78:0/1581714751 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408005980 con 0x7fa41010add0 2023-10-10T17:50:26.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.452+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa410110220 con 0x7fa41010add0 2023-10-10T17:50:26.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.455+0000 7fa40cff9700 1 -- 172.21.15.78:0/1581714751 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa408005020 con 0x7fa41010add0 2023-10-10T17:50:26.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa40cff9700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa3fc0a12c0 0x7fa3fc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa3fc0a12c0 0x7fa3fc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa40cff9700 1 -- 172.21.15.78:0/1581714751 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa4080cc9f0 con 0x7fa41010add0 2023-10-10T17:50:26.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa3fc0a12c0 0x7fa3fc0a3770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa400005950 tx=0x7fa40000b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 Processor -- start 2023-10-10T17:50:26.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 -- start start 2023-10-10T17:50:26.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5675f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a557b060 con 0x5621a567a8b0 2023-10-10T17:50:26.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a55cd8a0 con 0x5621a5675b60 2023-10-10T17:50:26.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.456+0000 7fa421e92b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a55c6d30 con 0x5621a5676470 2023-10-10T17:50:26.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48080/0 (socket says 172.21.15.78:48080) 2023-10-10T17:50:26.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 -- 172.21.15.78:0/1952955483 learned_addr learned my addr 172.21.15.78:0/1952955483 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5675f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40effd700 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 -- 172.21.15.78:0/1952955483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 msgr2=0x5621a567a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 -- 172.21.15.78:0/1952955483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 msgr2=0x5621a5675f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5675f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 -- 172.21.15.78:0/1952955483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621a5448630 con 0x5621a567a8b0 2023-10-10T17:50:26.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40effd700 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:26.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa40ffff700 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567ad00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa408095c30 tx=0x7fa408099580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9e1cea2c83e190 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa405ffb700 1 -- 172.21.15.78:0/1952955483 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408099730 con 0x5621a567a8b0 2023-10-10T17:50:26.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa405ffb700 1 -- 172.21.15.78:0/1952955483 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40809d760 con 0x5621a567a8b0 2023-10-10T17:50:26.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.457+0000 7fa405ffb700 1 -- 172.21.15.78:0/1952955483 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408098c20 con 0x5621a567a8b0 2023-10-10T17:50:26.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 -- 172.21.15.78:0/1952955483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 msgr2=0x5621a567ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567ad00 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fa408095c30 tx=0x7fa408099580 comp rx=0 tx=0).stop 2023-10-10T17:50:26.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 -- 172.21.15.78:0/1952955483 shutdown_connections 2023-10-10T17:50:26.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567ad00 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5675f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 --2- 172.21.15.78:0/1952955483 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 -- 172.21.15.78:0/1952955483 >> 172.21.15.78:0/1952955483 conn(0x5621a566b360 msgr2=0x5621a5670b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 -- 172.21.15.78:0/1952955483 shutdown_connections 2023-10-10T17:50:26.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 -- 172.21.15.78:0/1952955483 wait complete. 2023-10-10T17:50:26.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 -- start start 2023-10-10T17:50:26.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.458+0000 7fa421e92b80 1 Processor -- start 2023-10-10T17:50:26.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a567f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a55688e0 con 0x5621a567a8b0 2023-10-10T17:50:26.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a5568580 con 0x5621a5675b60 2023-10-10T17:50:26.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a544a920 con 0x5621a5676470 2023-10-10T17:50:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 10 client.0 _ll_register_callbacks cb 0x5621a5586780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48086/0 (socket says 172.21.15.78:48086) 2023-10-10T17:50:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 -- 172.21.15.78:0/3646332264 learned_addr learned my addr 172.21.15.78:0/3646332264 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40effd700 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a567f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 -- 172.21.15.78:0/3646332264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 msgr2=0x5621a567f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 -- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 msgr2=0x5621a567f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a567f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 -- 172.21.15.78:0/3646332264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa410110220 con 0x5621a567a8b0 2023-10-10T17:50:26.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa40ffff700 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567fe20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa4080053d0 tx=0x7fa408009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3ef96483d5426c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa3eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:26.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa3eeffd700 1 -- 172.21.15.78:0/3646332264 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4080a1b80 con 0x5621a567a8b0 2023-10-10T17:50:26.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa3eeffd700 1 -- 172.21.15.78:0/3646332264 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa408099ba0 con 0x5621a567a8b0 2023-10-10T17:50:26.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.459+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5621a5668b20 con 0x5621a567a8b0 2023-10-10T17:50:26.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa3eeffd700 1 -- 172.21.15.78:0/3646332264 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4080a16f0 con 0x5621a567a8b0 2023-10-10T17:50:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 10 client.17106 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5621a558d190 con 0x5621a567a8b0 2023-10-10T17:50:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa3eeffd700 1 -- 172.21.15.78:0/3646332264 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5621a5668b20 con 0x5621a567a8b0 2023-10-10T17:50:26.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa3eeffd700 1 -- 172.21.15.78:0/3646332264 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa408084040 con 0x5621a567a8b0 2023-10-10T17:50:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 10 client.17106 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 10 client.17106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:26.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 20 client.17106 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a568f6e0 0x5621a5691af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 4 client.17106 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621a55890a0 con 0x5621a568f6e0 2023-10-10T17:50:26.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.460+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a568f6e0 0x5621a5691af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.461+0000 7fa3eeffd700 10 client.17106 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:26.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.461+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a568f6e0 0x5621a5691af0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5af1820fdf99a46 server_cookie=8010cec11eaad5b2 in_seq=0 out_seq=0 2023-10-10T17:50:26.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.461+0000 7fa3eeffd700 1 -- 172.21.15.78:0/3646332264 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5621a55890a0 con 0x5621a568f6e0 2023-10-10T17:50:26.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.461+0000 7fa3eeffd700 10 client.17106 handle_command_reply: tid=0 2023-10-10T17:50:26.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.461+0000 7fa421e92b80 1 client.17106 shutdown 2023-10-10T17:50:26.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 msgr2=0x5621a567fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567fe20 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fa4080053d0 tx=0x7fa408009f30 comp rx=0 tx=0).stop 2023-10-10T17:50:26.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 shutdown_connections 2023-10-10T17:50:26.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a568f6e0 0x5621a5691af0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a567fe20 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a567f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa3eeffd700 0 client.17106 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:26.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3646332264 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a567f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 >> 172.21.15.78:0/3646332264 conn(0x5621a566b360 msgr2=0x5621a567bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 shutdown_connections 2023-10-10T17:50:26.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 1 -- 172.21.15.78:0/3646332264 wait complete. 2023-10-10T17:50:26.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.462+0000 7fa421e92b80 20 client.17106 trim_cache size 0 max 16384 2023-10-10T17:50:26.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.474+0000 7fa421e92b80 1 Processor -- start 2023-10-10T17:50:26.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.474+0000 7fa421e92b80 1 -- start start 2023-10-10T17:50:26.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5675b60 0x5621a5670be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a5676470 0x5621a56869a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a567a8b0 0x5621a567fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa421e92b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a5676bc0 con 0x5621a5676470 2023-10-10T17:50:26.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa421e92b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a56718b0 con 0x5621a567a8b0 2023-10-10T17:50:26.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa421e92b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a5676840 con 0x5621a5675b60 2023-10-10T17:50:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a5676470 0x5621a56869a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a5676470 0x5621a56869a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48092/0 (socket says 172.21.15.78:48092) 2023-10-10T17:50:26.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 -- 172.21.15.78:0/3942318545 learned_addr learned my addr 172.21.15.78:0/3942318545 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5675b60 0x5621a5670be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40ffff700 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a567a8b0 0x5621a567fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 -- 172.21.15.78:0/3942318545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5675b60 msgr2=0x5621a5670be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5675b60 0x5621a5670be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 -- 172.21.15.78:0/3942318545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a567a8b0 msgr2=0x5621a567fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a567a8b0 0x5621a567fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 -- 172.21.15.78:0/3942318545 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621a558cee0 con 0x5621a5676470 2023-10-10T17:50:26.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5675b60 0x5621a5670be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40ffff700 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a567a8b0 0x5621a567fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa40effd700 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a5676470 0x5621a56869a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009990 tx=0x7fa3f8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ad4fc95bf0fefd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa4067fc700 1 -- 172.21.15.78:0/3942318545 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f801d030 con 0x5621a5676470 2023-10-10T17:50:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa4067fc700 1 -- 172.21.15.78:0/3942318545 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3f801e040 con 0x5621a5676470 2023-10-10T17:50:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.475+0000 7fa4067fc700 1 -- 172.21.15.78:0/3942318545 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8003db0 con 0x5621a5676470 2023-10-10T17:50:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 -- 172.21.15.78:0/3942318545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a5676470 msgr2=0x5621a56869a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a5676470 0x5621a56869a0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8009990 tx=0x7fa3f8005fd0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 -- 172.21.15.78:0/3942318545 shutdown_connections 2023-10-10T17:50:26.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5675b60 0x5621a5670be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a567a8b0 0x5621a567fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 --2- 172.21.15.78:0/3942318545 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a5676470 0x5621a56869a0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 -- 172.21.15.78:0/3942318545 >> 172.21.15.78:0/3942318545 conn(0x5621a566b360 msgr2=0x5621a567ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 -- 172.21.15.78:0/3942318545 shutdown_connections 2023-10-10T17:50:26.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 -- 172.21.15.78:0/3942318545 wait complete. 2023-10-10T17:50:26.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.476+0000 7fa421e92b80 1 -- start start 2023-10-10T17:50:26.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 1 Processor -- start 2023-10-10T17:50:26.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5684990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a5684ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a5685410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a56e26e0 con 0x5621a567a8b0 2023-10-10T17:50:26.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a56e2850 con 0x5621a5675b60 2023-10-10T17:50:26.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621a56e6e20 con 0x5621a5676470 2023-10-10T17:50:26.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa421e92b80 10 client.0 _ll_register_callbacks cb 0x7fa40000b990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:26.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a5685410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a5685410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48100/0 (socket says 172.21.15.78:48100) 2023-10-10T17:50:26.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 -- 172.21.15.78:0/693456255 learned_addr learned my addr 172.21.15.78:0/693456255 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:26.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5684990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40effd700 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a5684ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 -- 172.21.15.78:0/693456255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 msgr2=0x5621a5684ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a5684ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 -- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 msgr2=0x5621a5684990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:26.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5684990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 -- 172.21.15.78:0/693456255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa41010da70 con 0x5621a567a8b0 2023-10-10T17:50:26.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa40ffff700 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a5685410 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa408095c30 tx=0x7fa408009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbfd68c443f50e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:26.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.477+0000 7fa3eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:26.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa3eeffd700 1 -- 172.21.15.78:0/693456255 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa408099270 con 0x5621a567a8b0 2023-10-10T17:50:26.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa3eeffd700 1 -- 172.21.15.78:0/693456255 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fa4080a1b80 con 0x5621a567a8b0 2023-10-10T17:50:26.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa40809d760 con 0x5621a567a8b0 2023-10-10T17:50:26.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa3eeffd700 1 -- 172.21.15.78:0/693456255 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa408099500 con 0x5621a567a8b0 2023-10-10T17:50:26.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 10 client.17112 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:26.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5621a53ddc60 con 0x5621a567a8b0 2023-10-10T17:50:26.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa3eeffd700 1 -- 172.21.15.78:0/693456255 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa40809d760 con 0x5621a567a8b0 2023-10-10T17:50:26.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa3eeffd700 1 -- 172.21.15.78:0/693456255 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fa408095520 con 0x5621a567a8b0 2023-10-10T17:50:26.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 10 client.17112 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:26.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:26.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 20 client.17112 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:26.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a56e7fe0 0x5621a56ea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:26.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 4 client.17112 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:50:26.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5621a5595660 con 0x5621a56e7fe0 2023-10-10T17:50:26.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.478+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a56e7fe0 0x5621a56ea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:26.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa3eeffd700 10 client.17112 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:26.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa40f7fe700 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a56e7fe0 0x5621a56ea3c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=848d4122992bcefb server_cookie=78672a9898a01d54 in_seq=0 out_seq=0 2023-10-10T17:50:26.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa3eeffd700 1 -- 172.21.15.78:0/693456255 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5621a5595660 con 0x5621a56e7fe0 2023-10-10T17:50:26.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa3eeffd700 10 client.17112 handle_command_reply: tid=0 2023-10-10T17:50:26.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 client.17112 shutdown 2023-10-10T17:50:26.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 msgr2=0x5621a5685410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a5685410 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa408095c30 tx=0x7fa408009540 comp rx=0 tx=0).stop 2023-10-10T17:50:26.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 shutdown_connections 2023-10-10T17:50:26.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5621a56e7fe0 0x5621a56ea3c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5621a567a8b0 0x5621a5685410 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5621a5675b60 0x5621a5684990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 --2- 172.21.15.78:0/693456255 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5621a5676470 0x5621a5684ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 >> 172.21.15.78:0/693456255 conn(0x5621a566b360 msgr2=0x5621a56798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa3eeffd700 0 client.17112 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:26.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 shutdown_connections 2023-10-10T17:50:26.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 1 -- 172.21.15.78:0/693456255 wait complete. 2023-10-10T17:50:26.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.479+0000 7fa421e92b80 20 client.17112 trim_cache size 0 max 16384 2023-10-10T17:50:26.525 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:50:26.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa3fc0a12c0 msgr2=0x7fa3fc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa3fc0a12c0 0x7fa3fc0a3770 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa400005950 tx=0x7fa40000b410 comp rx=0 tx=0).stop 2023-10-10T17:50:26.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa41010add0 msgr2=0x7fa41019ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:26.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa41010add0 0x7fa41019ad50 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa408007ee0 tx=0x7fa40800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 shutdown_connections 2023-10-10T17:50:26.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa4100ff510 0x7fa4101963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa3fc0a12c0 0x7fa3fc0a3770 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa41010add0 0x7fa41019ad50 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 --2- 172.21.15.78:0/1581714751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa4100ffe20 0x7fa4101968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:26.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 >> 172.21.15.78:0/1581714751 conn(0x7fa410074b00 msgr2=0x7fa410102830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:26.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 shutdown_connections 2023-10-10T17:50:26.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:26.480+0000 7fa415d06700 1 -- 172.21.15.78:0/1581714751 wait complete. 2023-10-10T17:50:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:26 smithi157 ceph-mon[100896]: pgmap v244: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:26.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:26 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/927063011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:50:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:26 smithi078 ceph-mon[99494]: pgmap v244: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:26 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/927063011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:50:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:26 smithi190 ceph-mon[101220]: pgmap v244: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:27.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:26 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/927063011' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:50:27.531 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:50:27.531 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:50:27.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 -- 172.21.15.78:0/1242816304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 msgr2=0x7f08cc103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 --2- 172.21.15.78:0/1242816304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc103db0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0099c0 tx=0x7f08bc009cd0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 -- 172.21.15.78:0/1242816304 shutdown_connections 2023-10-10T17:50:27.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 --2- 172.21.15.78:0/1242816304 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08cc0ff450 0x7f08cc0ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 --2- 172.21.15.78:0/1242816304 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08cc104490 0x7f08cc10c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 --2- 172.21.15.78:0/1242816304 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc103db0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 -- 172.21.15.78:0/1242816304 >> 172.21.15.78:0/1242816304 conn(0x7f08cc0770b0 msgr2=0x7f08cc0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 -- 172.21.15.78:0/1242816304 shutdown_connections 2023-10-10T17:50:27.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.672+0000 7f08d0a96700 1 -- 172.21.15.78:0/1242816304 wait complete. 2023-10-10T17:50:27.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 Processor -- start 2023-10-10T17:50:27.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 -- start start 2023-10-10T17:50:27.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08cc0ff450 0x7f08cc197250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08cc104490 0x7f08cc1a22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc1a2810 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc1a2950 con 0x7f08cc0ff450 2023-10-10T17:50:27.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08d0a96700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc1a2a90 con 0x7f08cc104490 2023-10-10T17:50:27.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc197790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48112/0 (socket says 172.21.15.78:48112) 2023-10-10T17:50:27.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 -- 172.21.15.78:0/351139390 learned_addr learned my addr 172.21.15.78:0/351139390 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08cad9d700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08cc104490 0x7f08cc1a22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08ca59c700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08cc0ff450 0x7f08cc197250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 -- 172.21.15.78:0/351139390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08cc104490 msgr2=0x7f08cc1a22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08cc104490 0x7f08cc1a22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 -- 172.21.15.78:0/351139390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08cc0ff450 msgr2=0x7f08cc197250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08cc0ff450 0x7f08cc197250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.673+0000 7f08c9d9b700 1 -- 172.21.15.78:0/351139390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0095f0 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.674+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc197790 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0098e0 tx=0x7f08bc005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.674+0000 7f08bb7fe700 1 -- 172.21.15.78:0/351139390 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc01d070 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.674+0000 7f08bb7fe700 1 -- 172.21.15.78:0/351139390 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08bc004ca0 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.674+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc10f2b0 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.674+0000 7f08bb7fe700 1 -- 172.21.15.78:0/351139390 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08bc005290 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.674+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc10f120 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.676+0000 7f08bb7fe700 1 -- 172.21.15.78:0/351139390 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f08bc003c60 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.676+0000 7f08bb7fe700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08b40a12c0 0x7f08b40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.677+0000 7f08ca59c700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08b40a12c0 0x7f08b40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.677+0000 7f08bb7fe700 1 -- 172.21.15.78:0/351139390 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f08bc0cdf80 con 0x7f08cc0ffdf0 2023-10-10T17:50:27.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.677+0000 7f08ca59c700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08b40a12c0 0x7f08b40a3770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f08c0005950 tx=0x7f08c00058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.677+0000 7f08dcc22b80 1 Processor -- start 2023-10-10T17:50:27.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08dcc22b80 1 -- start start 2023-10-10T17:50:27.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028ad7ff0 0x556028ad83c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028adcd40 0x556028add190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560289d90e0 con 0x556028ad7ff0 2023-10-10T17:50:27.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028a2b920 con 0x556028adcd40 2023-10-10T17:50:27.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028a24db0 con 0x556028ad8900 2023-10-10T17:50:27.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028ad7ff0 0x556028ad83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028ad7ff0 0x556028ad83c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48118/0 (socket says 172.21.15.78:48118) 2023-10-10T17:50:27.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 -- 172.21.15.78:0/2719286772 learned_addr learned my addr 172.21.15.78:0/2719286772 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 -- 172.21.15.78:0/2719286772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 msgr2=0x556028adc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 -- 172.21.15.78:0/2719286772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028adcd40 msgr2=0x556028add190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028adcd40 0x556028add190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028adcd40 0x556028add190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 -- 172.21.15.78:0/2719286772 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560288a66b0 con 0x556028ad7ff0 2023-10-10T17:50:27.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028adcd40 0x556028add190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:27.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.678+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028ad7ff0 0x556028ad83c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f08c000fd70 tx=0x7f08c0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b2ac1561b521e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08b8ff9700 1 -- 172.21.15.78:0/2719286772 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c0011c60 con 0x556028ad7ff0 2023-10-10T17:50:27.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08b8ff9700 1 -- 172.21.15.78:0/2719286772 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08c0011dc0 con 0x556028ad7ff0 2023-10-10T17:50:27.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08b8ff9700 1 -- 172.21.15.78:0/2719286772 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c001d7e0 con 0x556028ad7ff0 2023-10-10T17:50:27.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2719286772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028ad7ff0 msgr2=0x556028ad83c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028ad7ff0 0x556028ad83c0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f08c000fd70 tx=0x7f08c0012000 comp rx=0 tx=0).stop 2023-10-10T17:50:27.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2719286772 shutdown_connections 2023-10-10T17:50:27.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028adcd40 0x556028add190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028ad7ff0 0x556028ad83c0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2719286772 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2719286772 >> 172.21.15.78:0/2719286772 conn(0x556028ac93e0 msgr2=0x556028acebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.679+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2719286772 shutdown_connections 2023-10-10T17:50:27.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2719286772 wait complete. 2023-10-10T17:50:27.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 -- start start 2023-10-10T17:50:27.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 Processor -- start 2023-10-10T17:50:27.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ae1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560289c6960 con 0x556028adcd40 2023-10-10T17:50:27.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560289c6600 con 0x556028ad8900 2023-10-10T17:50:27.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560288a89a0 con 0x556028ad7ff0 2023-10-10T17:50:27.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08dcc22b80 10 client.0 _ll_register_callbacks cb 0x5560289e4800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:27.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08cad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ae1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08cad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ae1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48128/0 (socket says 172.21.15.78:48128) 2023-10-10T17:50:27.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08cad9d700 1 -- 172.21.15.78:0/2643414701 learned_addr learned my addr 172.21.15.78:0/2643414701 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.680+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08cad9d700 1 -- 172.21.15.78:0/2643414701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 msgr2=0x556028ae44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08cad9d700 1 -- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 msgr2=0x556028ae17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08cad9d700 1 -- 172.21.15.78:0/2643414701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c000fa20 con 0x556028adcd40 2023-10-10T17:50:27.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ae1e80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009b70 tx=0x7f08c4009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885617aff7044044 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08b1ffb700 1 -- 172.21.15.78:0/2643414701 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4007bb0 con 0x556028adcd40 2023-10-10T17:50:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08b1ffb700 1 -- 172.21.15.78:0/2643414701 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f08c4006e40 con 0x556028adcd40 2023-10-10T17:50:27.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556028ac6ba0 con 0x556028adcd40 2023-10-10T17:50:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08b1ffb700 1 -- 172.21.15.78:0/2643414701 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08c400aa50 con 0x556028adcd40 2023-10-10T17:50:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 10 client.17124 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5560289eb210 con 0x556028adcd40 2023-10-10T17:50:27.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08b1ffb700 1 -- 172.21.15.78:0/2643414701 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556028ac6ba0 con 0x556028adcd40 2023-10-10T17:50:27.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08b1ffb700 1 -- 172.21.15.78:0/2643414701 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f08c4013070 con 0x556028adcd40 2023-10-10T17:50:27.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 10 client.17124 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:27.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 10 client.17124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:27.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 20 client.17124 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:27.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028af1e50 0x556028af4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 4 client.17124 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:27.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.681+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560289e7120 con 0x556028af1e50 2023-10-10T17:50:27.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.682+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028af1e50 0x556028af4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.682+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028af1e50 0x556028af4260 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f81934f69d6a573 server_cookie=4fda676ad80efe70 in_seq=0 out_seq=0 2023-10-10T17:50:27.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.682+0000 7f08b1ffb700 10 client.17124 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08b1ffb700 1 -- 172.21.15.78:0/2643414701 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5560289e7120 con 0x556028af1e50 2023-10-10T17:50:27.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08b1ffb700 10 client.17124 handle_command_reply: tid=0 2023-10-10T17:50:27.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 client.17124 shutdown 2023-10-10T17:50:27.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 msgr2=0x556028ae1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ae1e80 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009b70 tx=0x7f08c4009c50 comp rx=0 tx=0).stop 2023-10-10T17:50:27.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 shutdown_connections 2023-10-10T17:50:27.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028af1e50 0x556028af4260 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ae1e80 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2643414701 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08b1ffb700 0 client.17124 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 >> 172.21.15.78:0/2643414701 conn(0x556028ac93e0 msgr2=0x556028ade1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 shutdown_connections 2023-10-10T17:50:27.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2643414701 wait complete. 2023-10-10T17:50:27.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.683+0000 7f08dcc22b80 20 client.17124 trim_cache size 0 max 16384 2023-10-10T17:50:27.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 Processor -- start 2023-10-10T17:50:27.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 -- start start 2023-10-10T17:50:27.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad7ff0 0x556028ae8e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ade120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028aef400 con 0x556028adcd40 2023-10-10T17:50:27.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028adfd40 con 0x556028ad7ff0 2023-10-10T17:50:27.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028adb970 con 0x556028ad8900 2023-10-10T17:50:27.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08cad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ade120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08cad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ade120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48134/0 (socket says 172.21.15.78:48134) 2023-10-10T17:50:27.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.700+0000 7f08cad9d700 1 -- 172.21.15.78:0/2862556572 learned_addr learned my addr 172.21.15.78:0/2862556572 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad7ff0 0x556028ae8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08cad9d700 1 -- 172.21.15.78:0/2862556572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 msgr2=0x556028adc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08cad9d700 1 -- 172.21.15.78:0/2862556572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad7ff0 msgr2=0x556028ae8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad7ff0 0x556028ae8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08cad9d700 1 -- 172.21.15.78:0/2862556572 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560289eaf60 con 0x556028adcd40 2023-10-10T17:50:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08ca59c700 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad7ff0 0x556028ae8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08cad9d700 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ade120 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f08c4002230 tx=0x7f08c401d5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b930abe4f00aaa58 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08b97fa700 1 -- 172.21.15.78:0/2862556572 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4006e40 con 0x556028adcd40 2023-10-10T17:50:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08b97fa700 1 -- 172.21.15.78:0/2862556572 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08c4010040 con 0x556028adcd40 2023-10-10T17:50:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.701+0000 7f08b97fa700 1 -- 172.21.15.78:0/2862556572 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c401dcc0 con 0x556028adcd40 2023-10-10T17:50:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2862556572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 msgr2=0x556028ade120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ade120 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f08c4002230 tx=0x7f08c401d5d0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2862556572 shutdown_connections 2023-10-10T17:50:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028ade120 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad7ff0 0x556028ae8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/2862556572 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad8900 0x556028adc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2862556572 >> 172.21.15.78:0/2862556572 conn(0x556028ac93e0 msgr2=0x556028adbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2862556572 shutdown_connections 2023-10-10T17:50:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- 172.21.15.78:0/2862556572 wait complete. 2023-10-10T17:50:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- start start 2023-10-10T17:50:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 Processor -- start 2023-10-10T17:50:27.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028b45d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028b46250 con 0x556028adcd40 2023-10-10T17:50:27.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028b46390 con 0x556028ad8900 2023-10-10T17:50:27.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x556028b46520 con 0x556028ad7ff0 2023-10-10T17:50:27.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.702+0000 7f08dcc22b80 10 client.0 _ll_register_callbacks cb 0x7f08c40093b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:27.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028b45d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028b45d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48142/0 (socket says 172.21.15.78:48142) 2023-10-10T17:50:27.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 -- 172.21.15.78:0/3480542422 learned_addr learned my addr 172.21.15.78:0/3480542422 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 -- 172.21.15.78:0/3480542422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 msgr2=0x556028ae0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08ca59c700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 -- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 msgr2=0x556028ae1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 -- 172.21.15.78:0/3480542422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5560289eb210 con 0x556028adcd40 2023-10-10T17:50:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08c9d9b700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08cad9d700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028b45d10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009c90 tx=0x7f08c4006e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7ad532349d49b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08b1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08b1ffb700 1 -- 172.21.15.78:0/3480542422 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4007bb0 con 0x556028adcd40 2023-10-10T17:50:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08b1ffb700 1 -- 172.21.15.78:0/3480542422 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f08c4007d10 con 0x556028adcd40 2023-10-10T17:50:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f08c0011dc0 con 0x556028adcd40 2023-10-10T17:50:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08b1ffb700 1 -- 172.21.15.78:0/3480542422 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08c4007ed0 con 0x556028adcd40 2023-10-10T17:50:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08dcc22b80 10 client.17130 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55602883bce0 con 0x556028adcd40 2023-10-10T17:50:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.703+0000 7f08b1ffb700 1 -- 172.21.15.78:0/3480542422 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f08c0011dc0 con 0x556028adcd40 2023-10-10T17:50:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08b1ffb700 1 -- 172.21.15.78:0/3480542422 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f08c4008040 con 0x556028adcd40 2023-10-10T17:50:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08dcc22b80 10 client.17130 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08dcc22b80 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08dcc22b80 20 client.17130 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028b47950 0x556028b49d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08dcc22b80 4 client.17130 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:50:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5560289f36e0 con 0x556028b47950 2023-10-10T17:50:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08ca59c700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028b47950 0x556028b49d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08ca59c700 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028b47950 0x556028b49d60 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9ce0311012257e0 server_cookie=573f7b60347d6b7d in_seq=0 out_seq=0 2023-10-10T17:50:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.704+0000 7f08b1ffb700 10 client.17130 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.705+0000 7f08b1ffb700 1 -- 172.21.15.78:0/3480542422 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5560289f36e0 con 0x556028b47950 2023-10-10T17:50:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.705+0000 7f08b1ffb700 10 client.17130 handle_command_reply: tid=0 2023-10-10T17:50:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.705+0000 7f08dcc22b80 1 client.17130 shutdown 2023-10-10T17:50:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.705+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 msgr2=0x556028b45d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.705+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028b45d10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f08c4009c90 tx=0x7f08c4006e40 comp rx=0 tx=0).stop 2023-10-10T17:50:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 shutdown_connections 2023-10-10T17:50:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x556028ad7ff0 0x556028ae0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x556028b47950 0x556028b49d60 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556028adcd40 0x556028b45d10 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 --2- 172.21.15.78:0/3480542422 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x556028ad8900 0x556028ae1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 >> 172.21.15.78:0/3480542422 conn(0x556028ac93e0 msgr2=0x556028adc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08b1ffb700 0 client.17130 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 shutdown_connections 2023-10-10T17:50:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 1 -- 172.21.15.78:0/3480542422 wait complete. 2023-10-10T17:50:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08dcc22b80 20 client.17130 trim_cache size 0 max 16384 2023-10-10T17:50:27.746 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:50:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:50:27.746 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "81266012-992e-447a-b911-354e74fc8447", 2023-10-10T17:50:27.747 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:50:27.747 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:50:27.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08b40a12c0 msgr2=0x7f08b40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08b40a12c0 0x7f08b40a3770 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f08c0005950 tx=0x7f08c00058e0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 msgr2=0x7f08cc197790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc197790 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f08bc0098e0 tx=0x7f08bc005ef0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 shutdown_connections 2023-10-10T17:50:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f08cc104490 0x7f08cc1a22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f08b40a12c0 0x7f08b40a3770 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f08cc0ff450 0x7f08cc197250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 --2- 172.21.15.78:0/351139390 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f08cc0ffdf0 0x7f08cc197790 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 >> 172.21.15.78:0/351139390 conn(0x7f08cc0770b0 msgr2=0x7f08cc0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 shutdown_connections 2023-10-10T17:50:27.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.706+0000 7f08d0a96700 1 -- 172.21.15.78:0/351139390 wait complete. 2023-10-10T17:50:27.752 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:50:27.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 -- 172.21.15.78:0/3639134293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ffdf0 msgr2=0x7f8958103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 --2- 172.21.15.78:0/3639134293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ffdf0 0x7f8958103db0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f89540099c0 tx=0x7f8954009cd0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 -- 172.21.15.78:0/3639134293 shutdown_connections 2023-10-10T17:50:27.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 --2- 172.21.15.78:0/3639134293 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f8958104490 0x7f895810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 --2- 172.21.15.78:0/3639134293 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f89580ff450 0x7f89580ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 --2- 172.21.15.78:0/3639134293 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ffdf0 0x7f8958103db0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 -- 172.21.15.78:0/3639134293 >> 172.21.15.78:0/3639134293 conn(0x7f89580770b0 msgr2=0x7f89580774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 -- 172.21.15.78:0/3639134293 shutdown_connections 2023-10-10T17:50:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 -- 172.21.15.78:0/3639134293 wait complete. 2023-10-10T17:50:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.875+0000 7f89607fa700 1 Processor -- start 2023-10-10T17:50:27.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 -- start start 2023-10-10T17:50:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ff450 0x7f8958196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89580ffdf0 0x7f8958196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8958104490 0x7f895819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895819b410 con 0x7f89580ff450 2023-10-10T17:50:27.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895819b580 con 0x7f8958104490 2023-10-10T17:50:27.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895819b740 con 0x7f89580ffdf0 2023-10-10T17:50:27.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ff450 0x7f8958196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ff450 0x7f8958196430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48152/0 (socket says 172.21.15.78:48152) 2023-10-10T17:50:27.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 -- 172.21.15.78:0/3839621547 learned_addr learned my addr 172.21.15.78:0/3839621547 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895dd95700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89580ffdf0 0x7f8958196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895ed97700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8958104490 0x7f895819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 -- 172.21.15.78:0/3839621547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89580ffdf0 msgr2=0x7f8958196970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89580ffdf0 0x7f8958196970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 -- 172.21.15.78:0/3839621547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8958104490 msgr2=0x7f895819add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8958104490 0x7f895819add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 -- 172.21.15.78:0/3839621547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89540095f0 con 0x7f89580ff450 2023-10-10T17:50:27.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f895e596700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ff450 0x7f8958196430 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f894800c920 tx=0x7f894800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f894f7fe700 1 -- 172.21.15.78:0/3839621547 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948007b00 con 0x7f89580ff450 2023-10-10T17:50:27.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895810f230 con 0x7f89580ff450 2023-10-10T17:50:27.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f894f7fe700 1 -- 172.21.15.78:0/3839621547 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f894800ce70 con 0x7f89580ff450 2023-10-10T17:50:27.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f894f7fe700 1 -- 172.21.15.78:0/3839621547 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8948005460 con 0x7f89580ff450 2023-10-10T17:50:27.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.876+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895810f0a0 con 0x7f89580ff450 2023-10-10T17:50:27.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.878+0000 7f894f7fe700 1 -- 172.21.15.78:0/3839621547 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f8948004750 con 0x7f89580ff450 2023-10-10T17:50:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.879+0000 7f894f7fe700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f89440a12c0 0x7f89440a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.879+0000 7f895dd95700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f89440a12c0 0x7f89440a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.879+0000 7f894f7fe700 1 -- 172.21.15.78:0/3839621547 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f89480cccc0 con 0x7f89580ff450 2023-10-10T17:50:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f895dd95700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f89440a12c0 0x7f89440a3770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8954009f50 tx=0x7f895400b540 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 Processor -- start 2023-10-10T17:50:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 -- start start 2023-10-10T17:50:27.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef5cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef61370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef61d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ee62060 con 0x55bc0ef618b0 2023-10-10T17:50:27.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0eeb48a0 con 0x55bc0ef5d470 2023-10-10T17:50:27.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f896c986b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0eeadd30 con 0x55bc0ef5cb60 2023-10-10T17:50:27.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef61d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef61d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48164/0 (socket says 172.21.15.78:48164) 2023-10-10T17:50:27.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f895ed97700 1 -- 172.21.15.78:0/2601757858 learned_addr learned my addr 172.21.15.78:0/2601757858 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f895dd95700 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef61370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.880+0000 7f895e596700 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef5cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f895ed97700 1 -- 172.21.15.78:0/2601757858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 msgr2=0x55bc0ef5cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f895ed97700 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef5cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f895ed97700 1 -- 172.21.15.78:0/2601757858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 msgr2=0x55bc0ef61370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f895ed97700 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef61370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f895ed97700 1 -- 172.21.15.78:0/2601757858 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc0ed2f630 con 0x55bc0ef618b0 2023-10-10T17:50:27.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f895ed97700 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef61d00 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8950009c40 tx=0x7f8950009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f096e20ac14c7f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f894cff9700 1 -- 172.21.15.78:0/2601757858 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950014400 con 0x55bc0ef618b0 2023-10-10T17:50:27.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f894cff9700 1 -- 172.21.15.78:0/2601757858 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8950014a20 con 0x55bc0ef618b0 2023-10-10T17:50:27.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.881+0000 7f894cff9700 1 -- 172.21.15.78:0/2601757858 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950013990 con 0x55bc0ef618b0 2023-10-10T17:50:27.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- 172.21.15.78:0/2601757858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 msgr2=0x55bc0ef61d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef61d00 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f8950009c40 tx=0x7f8950009f50 comp rx=0 tx=0).stop 2023-10-10T17:50:27.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- 172.21.15.78:0/2601757858 shutdown_connections 2023-10-10T17:50:27.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef5cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef61d00 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 --2- 172.21.15.78:0/2601757858 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef61370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- 172.21.15.78:0/2601757858 >> 172.21.15.78:0/2601757858 conn(0x55bc0ef52360 msgr2=0x55bc0ef57b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- 172.21.15.78:0/2601757858 shutdown_connections 2023-10-10T17:50:27.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- 172.21.15.78:0/2601757858 wait complete. 2023-10-10T17:50:27.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- start start 2023-10-10T17:50:27.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 Processor -- start 2023-10-10T17:50:27.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef69080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef66290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef669d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ee4f8e0 con 0x55bc0ef618b0 2023-10-10T17:50:27.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ee4f580 con 0x55bc0ef5d470 2023-10-10T17:50:27.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ed31920 con 0x55bc0ef5cb60 2023-10-10T17:50:27.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.882+0000 7f896c986b80 10 client.0 _ll_register_callbacks cb 0x55bc0ee6d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:27.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef669d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef669d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48170/0 (socket says 172.21.15.78:48170) 2023-10-10T17:50:27.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 -- 172.21.15.78:0/3353560392 learned_addr learned my addr 172.21.15.78:0/3353560392 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895e596700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef69080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895dd95700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef66290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 -- 172.21.15.78:0/3353560392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 msgr2=0x55bc0ef69080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef69080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 -- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 msgr2=0x55bc0ef66290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef66290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 -- 172.21.15.78:0/3353560392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89500098a0 con 0x55bc0ef618b0 2023-10-10T17:50:27.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895dd95700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef66290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:27.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f895ed97700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef669d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f895000c040 tx=0x7f8950006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b37020243ee9c43 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f8941ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:27.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f8941ffb700 1 -- 172.21.15.78:0/3353560392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950003c00 con 0x55bc0ef618b0 2023-10-10T17:50:27.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f8941ffb700 1 -- 172.21.15.78:0/3353560392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f8950003d60 con 0x55bc0ef618b0 2023-10-10T17:50:27.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc0ef4fb20 con 0x55bc0ef618b0 2023-10-10T17:50:27.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f8941ffb700 1 -- 172.21.15.78:0/3353560392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8950014400 con 0x55bc0ef618b0 2023-10-10T17:50:27.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f896c986b80 10 client.17142 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:27.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.883+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55bc0ee74190 con 0x55bc0ef618b0 2023-10-10T17:50:27.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f8941ffb700 1 -- 172.21.15.78:0/3353560392 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bc0ef4fb20 con 0x55bc0ef618b0 2023-10-10T17:50:27.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f8941ffb700 1 -- 172.21.15.78:0/3353560392 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f89500047b0 con 0x55bc0ef618b0 2023-10-10T17:50:27.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f896c986b80 10 client.17142 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:27.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f896c986b80 10 client.17142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:27.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f896c986b80 20 client.17142 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:27.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f896c986b80 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0ef78bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f896c986b80 4 client.17142 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:27.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc0ee700a0 con 0x55bc0ef767c0 2023-10-10T17:50:27.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f895e596700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0ef78bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f8941ffb700 10 client.17142 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.884+0000 7f895e596700 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0ef78bd0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=409139847a6a74fb server_cookie=810c13987b86c1d9 in_seq=0 out_seq=0 2023-10-10T17:50:27.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f8941ffb700 1 -- 172.21.15.78:0/3353560392 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bc0ee700a0 con 0x55bc0ef767c0 2023-10-10T17:50:27.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f8941ffb700 10 client.17142 handle_command_reply: tid=0 2023-10-10T17:50:27.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 client.17142 shutdown 2023-10-10T17:50:27.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 msgr2=0x55bc0ef669d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef669d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f895000c040 tx=0x7f8950006000 comp rx=0 tx=0).stop 2023-10-10T17:50:27.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 shutdown_connections 2023-10-10T17:50:27.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef69080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0ef78bd0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef669d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f8941ffb700 0 client.17142 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 --2- 172.21.15.78:0/3353560392 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef66290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 >> 172.21.15.78:0/3353560392 conn(0x55bc0ef52360 msgr2=0x55bc0ef62d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 shutdown_connections 2023-10-10T17:50:27.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 1 -- 172.21.15.78:0/3353560392 wait complete. 2023-10-10T17:50:27.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.885+0000 7f896c986b80 20 client.17142 trim_cache size 0 max 16384 2023-10-10T17:50:27.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 Processor -- start 2023-10-10T17:50:27.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 -- start start 2023-10-10T17:50:27.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef57ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef6d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef72bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ef67050 con 0x55bc0ef618b0 2023-10-10T17:50:27.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ef603f0 con 0x55bc0ef5d470 2023-10-10T17:50:27.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f896c986b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ef60140 con 0x55bc0ef5cb60 2023-10-10T17:50:27.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef72bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef72bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48178/0 (socket says 172.21.15.78:48178) 2023-10-10T17:50:27.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 -- 172.21.15.78:0/351289892 learned_addr learned my addr 172.21.15.78:0/351289892 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895e596700 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef57ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 -- 172.21.15.78:0/351289892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 msgr2=0x55bc0ef57ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef57ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 -- 172.21.15.78:0/351289892 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 msgr2=0x55bc0ef6d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef6d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 -- 172.21.15.78:0/351289892 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc0ee73ee0 con 0x55bc0ef618b0 2023-10-10T17:50:27.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.902+0000 7f895ed97700 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef72bc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f8950013670 tx=0x7f8950013f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bcc1f45fd18cb8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f894d7fa700 1 -- 172.21.15.78:0/351289892 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89500077e0 con 0x55bc0ef618b0 2023-10-10T17:50:27.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f894d7fa700 1 -- 172.21.15.78:0/351289892 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8950013b20 con 0x55bc0ef618b0 2023-10-10T17:50:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f894d7fa700 1 -- 172.21.15.78:0/351289892 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950014660 con 0x55bc0ef618b0 2023-10-10T17:50:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 -- 172.21.15.78:0/351289892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 msgr2=0x55bc0ef72bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef72bc0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f8950013670 tx=0x7f8950013f10 comp rx=0 tx=0).stop 2023-10-10T17:50:27.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 -- 172.21.15.78:0/351289892 shutdown_connections 2023-10-10T17:50:27.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef57ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef72bc0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 --2- 172.21.15.78:0/351289892 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef6d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 -- 172.21.15.78:0/351289892 >> 172.21.15.78:0/351289892 conn(0x55bc0ef52360 msgr2=0x55bc0ef63c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 -- 172.21.15.78:0/351289892 shutdown_connections 2023-10-10T17:50:27.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.903+0000 7f896c986b80 1 -- 172.21.15.78:0/351289892 wait complete. 2023-10-10T17:50:27.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 -- start start 2023-10-10T17:50:27.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 Processor -- start 2023-10-10T17:50:27.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef6d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef6b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef6bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ef6df90 con 0x55bc0ef618b0 2023-10-10T17:50:27.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ef6c450 con 0x55bc0ef5d470 2023-10-10T17:50:27.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc0ef6c5e0 con 0x55bc0ef5cb60 2023-10-10T17:50:27.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f896c986b80 10 client.0 _ll_register_callbacks cb 0x7f8954003680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:27.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef6bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef6bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48184/0 (socket says 172.21.15.78:48184) 2023-10-10T17:50:27.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 -- 172.21.15.78:0/966461957 learned_addr learned my addr 172.21.15.78:0/966461957 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895e596700 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef6d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895dd95700 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef6b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 -- 172.21.15.78:0/966461957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 msgr2=0x55bc0ef6d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef6d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 -- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 msgr2=0x55bc0ef6b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef6b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.904+0000 7f895ed97700 1 -- 172.21.15.78:0/966461957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc0ee74190 con 0x55bc0ef618b0 2023-10-10T17:50:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f895ed97700 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef6bf10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8950006040 tx=0x7f8950013790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3053660aa59caa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f8941ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f8941ffb700 1 -- 172.21.15.78:0/966461957 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950014610 con 0x55bc0ef618b0 2023-10-10T17:50:27.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f8941ffb700 1 -- 172.21.15.78:0/966461957 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f8950014770 con 0x55bc0ef618b0 2023-10-10T17:50:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8950014a20 con 0x55bc0ef618b0 2023-10-10T17:50:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f8941ffb700 1 -- 172.21.15.78:0/966461957 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f895003b410 con 0x55bc0ef618b0 2023-10-10T17:50:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 10 client.17148 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:27.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55bc0ecc4c60 con 0x55bc0ef618b0 2023-10-10T17:50:27.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f8941ffb700 1 -- 172.21.15.78:0/966461957 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8950014a20 con 0x55bc0ef618b0 2023-10-10T17:50:27.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f8941ffb700 1 -- 172.21.15.78:0/966461957 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f895001e530 con 0x55bc0ef618b0 2023-10-10T17:50:27.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 10 client.17148 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:27.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 10 client.17148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:27.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 20 client.17148 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:27.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0efcd530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:27.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 4 client.17148 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:50:27.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bc0ee7c660 con 0x55bc0ef767c0 2023-10-10T17:50:27.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.905+0000 7f895e596700 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0efcd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:27.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f8941ffb700 10 client.17148 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f895e596700 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0efcd530 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bb8466cef4fd93f server_cookie=b613d50647c0af56 in_seq=0 out_seq=0 2023-10-10T17:50:27.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f8941ffb700 1 -- 172.21.15.78:0/966461957 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55bc0ee7c660 con 0x55bc0ef767c0 2023-10-10T17:50:27.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f8941ffb700 10 client.17148 handle_command_reply: tid=0 2023-10-10T17:50:27.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 client.17148 shutdown 2023-10-10T17:50:27.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 msgr2=0x55bc0ef6bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef6bf10 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8950006040 tx=0x7f8950013790 comp rx=0 tx=0).stop 2023-10-10T17:50:27.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 shutdown_connections 2023-10-10T17:50:27.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc0ef5cb60 0x55bc0ef6d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bc0ef767c0 0x55bc0efcd530 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc0ef618b0 0x55bc0ef6bf10 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f8941ffb700 0 client.17148 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:27.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 --2- 172.21.15.78:0/966461957 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc0ef5d470 0x55bc0ef6b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 >> 172.21.15.78:0/966461957 conn(0x55bc0ef52360 msgr2=0x55bc0ef72bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 shutdown_connections 2023-10-10T17:50:27.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 1 -- 172.21.15.78:0/966461957 wait complete. 2023-10-10T17:50:27.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.906+0000 7f896c986b80 20 client.17148 trim_cache size 0 max 16384 2023-10-10T17:50:27.950 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:50:27.950 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:50:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:50:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "81266012-992e-447a-b911-354e74fc8447": { 2023-10-10T17:50:27.951 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:50:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "81266012-992e-447a-b911-354e74fc8447", 2023-10-10T17:50:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:50:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:50:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:50:27.953 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:50:27.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f89440a12c0 msgr2=0x7f89440a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f89440a12c0 0x7f89440a3770 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8954009f50 tx=0x7f895400b540 comp rx=0 tx=0).stop 2023-10-10T17:50:27.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ff450 msgr2=0x7f8958196430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:27.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ff450 0x7f8958196430 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f894800c920 tx=0x7f894800cc30 comp rx=0 tx=0).stop 2023-10-10T17:50:27.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 shutdown_connections 2023-10-10T17:50:27.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f89440a12c0 0x7f89440a3770 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f8958104490 0x7f895819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89580ff450 0x7f8958196430 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 --2- 172.21.15.78:0/3839621547 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f89580ffdf0 0x7f8958196970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 >> 172.21.15.78:0/3839621547 conn(0x7f89580770b0 msgr2=0x7f8958067eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 shutdown_connections 2023-10-10T17:50:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:27.907+0000 7f89607fa700 1 -- 172.21.15.78:0/3839621547 wait complete. 2023-10-10T17:50:27.957 INFO:tasks.cephfs.filesystem:scrub status for tag:81266012-992e-447a-b911-354e74fc8447 - {'path': '/', 'tag': '81266012-992e-447a-b911-354e74fc8447', 'options': 'recursive,force'} 2023-10-10T17:50:28.977 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:28 smithi157 ceph-mon[100896]: pgmap v245: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:28.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:28 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:50:28.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:28 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:50:28.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:28 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:50:28.979 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:28 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:50:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:28 smithi078 ceph-mon[99494]: pgmap v245: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:28 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:50:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:28 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:50:29.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:28 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:50:29.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:28 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:50:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:28 smithi190 ceph-mon[101220]: pgmap v245: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:28 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:50:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:28 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:50:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:28 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:50:29.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:28 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:50:30.978 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:30 smithi157 ceph-mon[100896]: pgmap v246: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-10T17:50:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:30 smithi078 ceph-mon[99494]: pgmap v246: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-10T17:50:31.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:30 smithi190 ceph-mon[101220]: pgmap v246: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-10-10T17:50:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:32 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:50:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:32 smithi078 ceph-mon[99494]: pgmap v247: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:50:33.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:32 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:50:33.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:32 smithi190 ceph-mon[101220]: pgmap v247: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:50:33.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:32 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:50:33.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:32 smithi157 ceph-mon[100896]: pgmap v247: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-10-10T17:50:35.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:34 smithi078 ceph-mon[99494]: pgmap v248: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-10T17:50:35.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:34 smithi190 ceph-mon[101220]: pgmap v248: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-10T17:50:35.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:34 smithi157 ceph-mon[100896]: pgmap v248: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-10T17:50:37.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:36 smithi078 ceph-mon[99494]: pgmap v249: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:37.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:36 smithi190 ceph-mon[101220]: pgmap v249: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:37.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:36 smithi157 ceph-mon[100896]: pgmap v249: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:39.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:38 smithi078 ceph-mon[99494]: pgmap v250: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:50:39.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:38 smithi190 ceph-mon[101220]: pgmap v250: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:50:39.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:38 smithi157 ceph-mon[100896]: pgmap v250: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:50:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:40 smithi078 ceph-mon[99494]: pgmap v251: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:41.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:40 smithi190 ceph-mon[101220]: pgmap v251: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:41.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:40 smithi157 ceph-mon[100896]: pgmap v251: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:50:43.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:42 smithi078 ceph-mon[99494]: pgmap v252: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-10T17:50:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:42 smithi190 ceph-mon[101220]: pgmap v252: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-10T17:50:43.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:42 smithi157 ceph-mon[100896]: pgmap v252: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-10T17:50:45.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:44 smithi078 ceph-mon[99494]: pgmap v253: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:44 smithi190 ceph-mon[101220]: pgmap v253: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:45.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:44 smithi157 ceph-mon[100896]: pgmap v253: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-10-10T17:50:47.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:46 smithi078 ceph-mon[99494]: pgmap v254: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:46 smithi190 ceph-mon[101220]: pgmap v254: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:47.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:46 smithi157 ceph-mon[100896]: pgmap v254: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:49.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:48 smithi078 ceph-mon[99494]: pgmap v255: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:50:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:48 smithi190 ceph-mon[101220]: pgmap v255: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:50:49.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:48 smithi157 ceph-mon[100896]: pgmap v255: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:50:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:50 smithi078 ceph-mon[99494]: pgmap v256: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:50 smithi190 ceph-mon[101220]: pgmap v256: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:51.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:50 smithi157 ceph-mon[100896]: pgmap v256: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:53.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:52 smithi078 ceph-mon[99494]: pgmap v257: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:50:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:52 smithi190 ceph-mon[101220]: pgmap v257: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:50:53.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:52 smithi157 ceph-mon[100896]: pgmap v257: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:50:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:54 smithi078 ceph-mon[99494]: pgmap v258: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:50:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:54 smithi190 ceph-mon[101220]: pgmap v258: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:50:55.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:54 smithi157 ceph-mon[100896]: pgmap v258: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:50:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:56 smithi078 ceph-mon[99494]: pgmap v259: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:57.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:56 smithi190 ceph-mon[101220]: pgmap v259: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:57.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:56 smithi157 ceph-mon[100896]: pgmap v259: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:50:57.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:50:58.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 -- 172.21.15.78:0/2278357625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ffe20 msgr2=0x7f11fc10a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 --2- 172.21.15.78:0/2278357625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ffe20 0x7f11fc10a890 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f11f00099c0 tx=0x7f11f0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 -- 172.21.15.78:0/2278357625 shutdown_connections 2023-10-10T17:50:58.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 --2- 172.21.15.78:0/2278357625 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11fc10add0 0x7f11fc10d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 --2- 172.21.15.78:0/2278357625 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11fc0ff510 0x7f11fc0ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 --2- 172.21.15.78:0/2278357625 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ffe20 0x7f11fc10a890 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 -- 172.21.15.78:0/2278357625 >> 172.21.15.78:0/2278357625 conn(0x7f11fc074b00 msgr2=0x7f11fc074f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 -- 172.21.15.78:0/2278357625 shutdown_connections 2023-10-10T17:50:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.101+0000 7f120210f700 1 -- 172.21.15.78:0/2278357625 wait complete. 2023-10-10T17:50:58.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.102+0000 7f120210f700 1 Processor -- start 2023-10-10T17:50:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.102+0000 7f120210f700 1 -- start start 2023-10-10T17:50:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.102+0000 7f120210f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ff510 0x7f11fc1963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.102+0000 7f120210f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11fc0ffe20 0x7f11fc1968e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f120210f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11fc10add0 0x7f11fc19ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f120210f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc19b380 con 0x7f11fc0ff510 2023-10-10T17:50:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f120210f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc19b4f0 con 0x7f11fc10add0 2023-10-10T17:50:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f120210f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc19b6b0 con 0x7f11fc0ffe20 2023-10-10T17:50:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ff510 0x7f11fc1963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ff510 0x7f11fc1963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48190/0 (socket says 172.21.15.78:48190) 2023-10-10T17:50:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 -- 172.21.15.78:0/1820553758 learned_addr learned my addr 172.21.15.78:0/1820553758 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11faffd700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11fc0ffe20 0x7f11fc1968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fbfff700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11fc10add0 0x7f11fc19ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 -- 172.21.15.78:0/1820553758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11fc0ffe20 msgr2=0x7f11fc1968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11fc0ffe20 0x7f11fc1968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 -- 172.21.15.78:0/1820553758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11fc10add0 msgr2=0x7f11fc19ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11fc10add0 0x7f11fc19ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 -- 172.21.15.78:0/1820553758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f00095f0 con 0x7f11fc0ff510 2023-10-10T17:50:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ff510 0x7f11fc1963a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f11ec00d950 tx=0x7f11ec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11f8ff9700 1 -- 172.21.15.78:0/1820553758 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec009940 con 0x7f11fc0ff510 2023-10-10T17:50:58.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11f8ff9700 1 -- 172.21.15.78:0/1820553758 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11ec010460 con 0x7f11fc0ff510 2023-10-10T17:50:58.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc110390 con 0x7f11fc0ff510 2023-10-10T17:50:58.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f11f8ff9700 1 -- 172.21.15.78:0/1820553758 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec00f620 con 0x7f11fc0ff510 2023-10-10T17:50:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.103+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc110200 con 0x7f11fc0ff510 2023-10-10T17:50:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.106+0000 7f11f8ff9700 1 -- 172.21.15.78:0/1820553758 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f11ec0108d0 con 0x7f11fc0ff510 2023-10-10T17:50:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.106+0000 7f11f8ff9700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f11e40a12c0 0x7f11e40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.106+0000 7f11faffd700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f11e40a12c0 0x7f11e40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.106+0000 7f11f8ff9700 1 -- 172.21.15.78:0/1820553758 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f11ec0ccf20 con 0x7f11fc0ff510 2023-10-10T17:50:58.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f11faffd700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f11e40a12c0 0x7f11e40a3770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f11f0009990 tx=0x7f11f0005fb0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 Processor -- start 2023-10-10T17:50:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 -- start start 2023-10-10T17:50:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fcffbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fd000370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd000d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcf01060 con 0x55b0fcffbb60 2023-10-10T17:50:58.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcf538a0 con 0x55b0fd0008b0 2023-10-10T17:50:58.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f120e29bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcf4cd30 con 0x55b0fcffc470 2023-10-10T17:50:58.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f11fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fcffbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f11fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fcffbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48198/0 (socket says 172.21.15.78:48198) 2023-10-10T17:50:58.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.107+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2626349349 learned_addr learned my addr 172.21.15.78:0/2626349349 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fbfff700 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd000d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11faffd700 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fd000370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2626349349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 msgr2=0x55b0fd000370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fd000370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2626349349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 msgr2=0x55b0fd000d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd000d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2626349349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0fcdce630 con 0x55b0fcffbb60 2023-10-10T17:50:58.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11faffd700 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fd000370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:58.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fbfff700 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd000d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:58.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fcffbf30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f11ec099bf0 tx=0x7f11ec099690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8412931db03fa9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11e1ffb700 1 -- 172.21.15.78:0/2626349349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec096370 con 0x55b0fcffbb60 2023-10-10T17:50:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11e1ffb700 1 -- 172.21.15.78:0/2626349349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11ec0964d0 con 0x55b0fcffbb60 2023-10-10T17:50:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.108+0000 7f11e1ffb700 1 -- 172.21.15.78:0/2626349349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0a1e80 con 0x55b0fcffbb60 2023-10-10T17:50:58.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 -- 172.21.15.78:0/2626349349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 msgr2=0x55b0fcffbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fcffbf30 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f11ec099bf0 tx=0x7f11ec099690 comp rx=0 tx=0).stop 2023-10-10T17:50:58.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 -- 172.21.15.78:0/2626349349 shutdown_connections 2023-10-10T17:50:58.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd000d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fcffbf30 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2626349349 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fd000370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 -- 172.21.15.78:0/2626349349 >> 172.21.15.78:0/2626349349 conn(0x55b0fcff1360 msgr2=0x55b0fcff6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 -- 172.21.15.78:0/2626349349 shutdown_connections 2023-10-10T17:50:58.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 -- 172.21.15.78:0/2626349349 wait complete. 2023-10-10T17:50:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 -- start start 2023-10-10T17:50:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.109+0000 7f120e29bb80 1 Processor -- start 2023-10-10T17:50:58.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffbb60 0x55b0fd005390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffc470 0x55b0fd0058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd005e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fceee8e0 con 0x55b0fcffc470 2023-10-10T17:50:58.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fceee580 con 0x55b0fd0008b0 2023-10-10T17:50:58.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcdd0920 con 0x55b0fcffbb60 2023-10-10T17:50:58.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 10 client.0 _ll_register_callbacks cb 0x55b0fcf0c780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:58.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffc470 0x55b0fd0058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffc470 0x55b0fd0058d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48206/0 (socket says 172.21.15.78:48206) 2023-10-10T17:50:58.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 -- 172.21.15.78:0/59563968 learned_addr learned my addr 172.21.15.78:0/59563968 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffbb60 0x55b0fd005390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11fbfff700 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd005e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 -- 172.21.15.78:0/59563968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffbb60 msgr2=0x55b0fd005390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffbb60 0x55b0fd005390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 -- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 msgr2=0x55b0fd005e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd005e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 -- 172.21.15.78:0/59563968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc110200 con 0x55b0fcffc470 2023-10-10T17:50:58.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11faffd700 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffc470 0x55b0fd0058d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f11f00036c0 tx=0x7f11f0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e249451ceb3df6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11daffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:58.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11daffd700 1 -- 172.21.15.78:0/59563968 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f002b070 con 0x55b0fcffc470 2023-10-10T17:50:58.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0fcfeeb20 con 0x55b0fcffc470 2023-10-10T17:50:58.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.110+0000 7f11daffd700 1 -- 172.21.15.78:0/59563968 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f11f0004260 con 0x55b0fcffc470 2023-10-10T17:50:58.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f11daffd700 1 -- 172.21.15.78:0/59563968 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f00038c0 con 0x55b0fcffc470 2023-10-10T17:50:58.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 10 client.17160 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:58.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55b0fcf13190 con 0x55b0fcffc470 2023-10-10T17:50:58.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f11daffd700 1 -- 172.21.15.78:0/59563968 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b0fcfeeb20 con 0x55b0fcffc470 2023-10-10T17:50:58.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f11daffd700 1 -- 172.21.15.78:0/59563968 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f11f000fae0 con 0x55b0fcffc470 2023-10-10T17:50:58.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 10 client.17160 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:58.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 10 client.17160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:58.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 20 client.17160 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:58.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd015940 0x55b0fd017d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 4 client.17160 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:58.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0fcf0f0a0 con 0x55b0fd015940 2023-10-10T17:50:58.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.111+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd015940 0x55b0fd017d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.112+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd015940 0x55b0fd017d50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4fcb9df842dcf88 server_cookie=9560729f888b255f in_seq=0 out_seq=0 2023-10-10T17:50:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.112+0000 7f11daffd700 10 client.17160 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.112+0000 7f11daffd700 1 -- 172.21.15.78:0/59563968 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55b0fcf0f0a0 con 0x55b0fd015940 2023-10-10T17:50:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.112+0000 7f11daffd700 10 client.17160 handle_command_reply: tid=0 2023-10-10T17:50:58.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.112+0000 7f120e29bb80 1 client.17160 shutdown 2023-10-10T17:50:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffc470 msgr2=0x55b0fd0058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffc470 0x55b0fd0058d0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f11f00036c0 tx=0x7f11f0009310 comp rx=0 tx=0).stop 2023-10-10T17:50:58.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 shutdown_connections 2023-10-10T17:50:58.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffbb60 0x55b0fd005390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd015940 0x55b0fd017d50 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fd0008b0 0x55b0fd005e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f11daffd700 0 client.17160 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:58.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 --2- 172.21.15.78:0/59563968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffc470 0x55b0fd0058d0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 >> 172.21.15.78:0/59563968 conn(0x55b0fcff1360 msgr2=0x55b0fd001d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 shutdown_connections 2023-10-10T17:50:58.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 1 -- 172.21.15.78:0/59563968 wait complete. 2023-10-10T17:50:58.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.113+0000 7f120e29bb80 20 client.17160 trim_cache size 0 max 16384 2023-10-10T17:50:58.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.129+0000 7f120e29bb80 1 Processor -- start 2023-10-10T17:50:58.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f120e29bb80 1 -- start start 2023-10-10T17:50:58.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fd00c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffc470 0x55b0fcfff930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fd0008b0 0x55b0fcfffe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f120e29bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcfff520 con 0x55b0fcffbb60 2023-10-10T17:50:58.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f120e29bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fd0122a0 con 0x55b0fcffc470 2023-10-10T17:50:58.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f120e29bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcffc900 con 0x55b0fd0008b0 2023-10-10T17:50:58.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fd00c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fd00c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48212/0 (socket says 172.21.15.78:48212) 2023-10-10T17:50:58.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2072404792 learned_addr learned my addr 172.21.15.78:0/2072404792 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11faffd700 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffc470 0x55b0fcfff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fbfff700 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fd0008b0 0x55b0fcfffe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2072404792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fd0008b0 msgr2=0x55b0fcfffe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fd0008b0 0x55b0fcfffe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2072404792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffc470 msgr2=0x55b0fcfff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffc470 0x55b0fcfff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 -- 172.21.15.78:0/2072404792 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0fcff2b00 con 0x55b0fcffbb60 2023-10-10T17:50:58.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fbfff700 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fd0008b0 0x55b0fcfffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:58.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11faffd700 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffc470 0x55b0fcfff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:58.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fd00c940 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0cd200 tx=0x7f11ec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d32133d5052ca59 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.130+0000 7f11e27fc700 1 -- 172.21.15.78:0/2072404792 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec099a30 con 0x55b0fcffbb60 2023-10-10T17:50:58.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f11e27fc700 1 -- 172.21.15.78:0/2072404792 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11ec099ec0 con 0x55b0fcffbb60 2023-10-10T17:50:58.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f11e27fc700 1 -- 172.21.15.78:0/2072404792 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ec0a1e80 con 0x55b0fcffbb60 2023-10-10T17:50:58.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 -- 172.21.15.78:0/2072404792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 msgr2=0x55b0fd00c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fd00c940 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f11ec0cd200 tx=0x7f11ec000bc0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 -- 172.21.15.78:0/2072404792 shutdown_connections 2023-10-10T17:50:58.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fd0008b0 0x55b0fcfffe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fcffbb60 0x55b0fd00c940 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 --2- 172.21.15.78:0/2072404792 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffc470 0x55b0fcfff930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 -- 172.21.15.78:0/2072404792 >> 172.21.15.78:0/2072404792 conn(0x55b0fcff1360 msgr2=0x55b0fcffea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 -- 172.21.15.78:0/2072404792 shutdown_connections 2023-10-10T17:50:58.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 -- 172.21.15.78:0/2072404792 wait complete. 2023-10-10T17:50:58.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.131+0000 7f120e29bb80 1 -- start start 2023-10-10T17:50:58.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 1 Processor -- start 2023-10-10T17:50:58.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffbb60 0x55b0fcffd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fcffdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fd0008b0 0x55b0fd004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcffe250 con 0x55b0fd0008b0 2023-10-10T17:50:58.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcffe390 con 0x55b0fcffbb60 2023-10-10T17:50:58.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0fcffe550 con 0x55b0fcffc470 2023-10-10T17:50:58.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f120e29bb80 10 client.0 _ll_register_callbacks cb 0x7f11f0003d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:58.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fd0008b0 0x55b0fd004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fd0008b0 0x55b0fd004c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48222/0 (socket says 172.21.15.78:48222) 2023-10-10T17:50:58.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 -- 172.21.15.78:0/667598675 learned_addr learned my addr 172.21.15.78:0/667598675 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffbb60 0x55b0fcffd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11faffd700 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fcffdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 -- 172.21.15.78:0/667598675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 msgr2=0x55b0fcffdd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fcffdd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 -- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffbb60 msgr2=0x55b0fcffd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffbb60 0x55b0fcffd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 -- 172.21.15.78:0/667598675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0fcdce630 con 0x55b0fd0008b0 2023-10-10T17:50:58.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11fbfff700 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fd0008b0 0x55b0fd004c80 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f11f4007de0 tx=0x7f11f4007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9550c2c37d04bbb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.132+0000 7f11daffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:58.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f11daffd700 1 -- 172.21.15.78:0/667598675 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400e070 con 0x55b0fd0008b0 2023-10-10T17:50:58.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f11daffd700 1 -- 172.21.15.78:0/667598675 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f11f4014c60 con 0x55b0fd0008b0 2023-10-10T17:50:58.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11ec0964d0 con 0x55b0fd0008b0 2023-10-10T17:50:58.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f11daffd700 1 -- 172.21.15.78:0/667598675 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f4014e20 con 0x55b0fd0008b0 2023-10-10T17:50:58.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 10 client.17166 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:58.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55b0fcd7ade0 con 0x55b0fd0008b0 2023-10-10T17:50:58.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f11daffd700 1 -- 172.21.15.78:0/667598675 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f11ec0964d0 con 0x55b0fd0008b0 2023-10-10T17:50:58.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f11daffd700 1 -- 172.21.15.78:0/667598675 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f11f401f550 con 0x55b0fd0008b0 2023-10-10T17:50:58.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 10 client.17166 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:58.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 10 client.17166 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:58.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 20 client.17166 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:58.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd06a2e0 0x55b0fd06c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 4 client.17166 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:50:58.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b0fcf1b660 con 0x55b0fd06a2e0 2023-10-10T17:50:58.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.133+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd06a2e0 0x55b0fd06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f11daffd700 10 client.17166 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:58.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f11fb7fe700 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd06a2e0 0x55b0fd06c6f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c0525cf1bdb083 server_cookie=6f2a5fbd513f5f8d in_seq=0 out_seq=0 2023-10-10T17:50:58.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f11daffd700 1 -- 172.21.15.78:0/667598675 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b0fcf1b660 con 0x55b0fd06a2e0 2023-10-10T17:50:58.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f11daffd700 10 client.17166 handle_command_reply: tid=0 2023-10-10T17:50:58.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 client.17166 shutdown 2023-10-10T17:50:58.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fd0008b0 msgr2=0x55b0fd004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fd0008b0 0x55b0fd004c80 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f11f4007de0 tx=0x7f11f4007ec0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 shutdown_connections 2023-10-10T17:50:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55b0fd06a2e0 0x55b0fd06c6f0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55b0fd0008b0 0x55b0fd004c80 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f11daffd700 0 client.17166 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:58.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55b0fcffbb60 0x55b0fcffd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 --2- 172.21.15.78:0/667598675 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55b0fcffc470 0x55b0fcffdd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 >> 172.21.15.78:0/667598675 conn(0x55b0fcff1360 msgr2=0x55b0fcfff930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 shutdown_connections 2023-10-10T17:50:58.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 1 -- 172.21.15.78:0/667598675 wait complete. 2023-10-10T17:50:58.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.134+0000 7f120e29bb80 20 client.17166 trim_cache size 0 max 16384 2023-10-10T17:50:58.173 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:50:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:50:58.173 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:50:58.173 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:50:58.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f11e40a12c0 msgr2=0x7f11e40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f11e40a12c0 0x7f11e40a3770 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f11f0009990 tx=0x7f11f0005fb0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ff510 msgr2=0x7f11fc1963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ff510 0x7f11fc1963a0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f11ec00d950 tx=0x7f11ec00dc60 comp rx=0 tx=0).stop 2023-10-10T17:50:58.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 shutdown_connections 2023-10-10T17:50:58.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f11e40a12c0 0x7f11e40a3770 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f11fc10add0 0x7f11fc19ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f11fc0ff510 0x7f11fc1963a0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 --2- 172.21.15.78:0/1820553758 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f11fc0ffe20 0x7f11fc1968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 >> 172.21.15.78:0/1820553758 conn(0x7f11fc074b00 msgr2=0x7f11fc102830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 shutdown_connections 2023-10-10T17:50:58.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.135+0000 7f120210f700 1 -- 172.21.15.78:0/1820553758 wait complete. 2023-10-10T17:50:58.178 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:50:58.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:50:58.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 -- 172.21.15.78:0/1611108655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe6410a800 msgr2=0x7efe6410cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1611108655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe6410a800 0x7efe6410cc90 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7efe5c00b0a0 tx=0x7efe5c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 -- 172.21.15.78:0/1611108655 shutdown_connections 2023-10-10T17:50:58.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1611108655 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe641014c0 0x7efe64101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1611108655 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe6410a800 0x7efe6410cc90 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1611108655 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe64101dd0 0x7efe6410a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 -- 172.21.15.78:0/1611108655 >> 172.21.15.78:0/1611108655 conn(0x7efe64076310 msgr2=0x7efe64076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 -- 172.21.15.78:0/1611108655 shutdown_connections 2023-10-10T17:50:58.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 -- 172.21.15.78:0/1611108655 wait complete. 2023-10-10T17:50:58.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.299+0000 7efe6ab49700 1 Processor -- start 2023-10-10T17:50:58.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe6ab49700 1 -- start start 2023-10-10T17:50:58.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe6ab49700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe641014c0 0x7efe640ff110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe6ab49700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe64101dd0 0x7efe640ff650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe6ab49700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe6410a800 0x7efe641a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe6ab49700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe640ffe80 con 0x7efe641014c0 2023-10-10T17:50:58.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe6ab49700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe641a4b80 con 0x7efe64101dd0 2023-10-10T17:50:58.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe6ab49700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe641a4cc0 con 0x7efe6410a800 2023-10-10T17:50:58.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe641014c0 0x7efe640ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe641014c0 0x7efe640ff110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48232/0 (socket says 172.21.15.78:48232) 2023-10-10T17:50:58.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 -- 172.21.15.78:0/1758466348 learned_addr learned my addr 172.21.15.78:0/1758466348 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe690e6700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe6410a800 0x7efe641a4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe63fff700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe64101dd0 0x7efe640ff650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 -- 172.21.15.78:0/1758466348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe6410a800 msgr2=0x7efe641a4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe6410a800 0x7efe641a4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 -- 172.21.15.78:0/1758466348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe64101dd0 msgr2=0x7efe640ff650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe64101dd0 0x7efe640ff650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 -- 172.21.15.78:0/1758466348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c009d00 con 0x7efe641014c0 2023-10-10T17:50:58.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe63fff700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe64101dd0 0x7efe640ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:58.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe688e5700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe641014c0 0x7efe640ff110 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efe5800e9a0 tx=0x7efe5800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.300+0000 7efe61ffb700 1 -- 172.21.15.78:0/1758466348 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58009790 con 0x7efe641014c0 2023-10-10T17:50:58.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.301+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe641102c0 con 0x7efe641014c0 2023-10-10T17:50:58.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.301+0000 7efe61ffb700 1 -- 172.21.15.78:0/1758466348 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efe5800cbd0 con 0x7efe641014c0 2023-10-10T17:50:58.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.301+0000 7efe61ffb700 1 -- 172.21.15.78:0/1758466348 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58018680 con 0x7efe641014c0 2023-10-10T17:50:58.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.301+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe64110130 con 0x7efe641014c0 2023-10-10T17:50:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.302+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe64110450 con 0x7efe641014c0 2023-10-10T17:50:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.303+0000 7efe61ffb700 1 -- 172.21.15.78:0/1758466348 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7efe5800cd40 con 0x7efe641014c0 2023-10-10T17:50:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.303+0000 7efe61ffb700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efe540a1340 0x7efe540a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.303+0000 7efe61ffb700 1 -- 172.21.15.78:0/1758466348 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7efe58014070 con 0x7efe641014c0 2023-10-10T17:50:58.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.303+0000 7efe63fff700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efe540a1340 0x7efe540a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.304+0000 7efe63fff700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efe540a1340 0x7efe540a37f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efe64100900 tx=0x7efe50009550 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.307+0000 7efe61ffb700 1 -- 172.21.15.78:0/1758466348 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7efe580965d0 con 0x7efe641014c0 2023-10-10T17:50:58.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.539+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe64105120 con 0x7efe641014c0 2023-10-10T17:50:58.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.540+0000 7efe61ffb700 1 -- 172.21.15.78:0/1758466348 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10817 (secure 0 0 0) 0x7efe58026090 con 0x7efe641014c0 2023-10-10T17:50:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:50:58.542 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:46:23.648656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:50:58.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efe540a1340 msgr2=0x7efe540a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efe540a1340 0x7efe540a37f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efe64100900 tx=0x7efe50009550 comp rx=0 tx=0).stop 2023-10-10T17:50:58.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe641014c0 msgr2=0x7efe640ff110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe641014c0 0x7efe640ff110 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7efe5800e9a0 tx=0x7efe5800ecb0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 shutdown_connections 2023-10-10T17:50:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7efe6410a800 0x7efe641a4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7efe540a1340 0x7efe540a37f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efe641014c0 0x7efe640ff110 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 --2- 172.21.15.78:0/1758466348 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7efe64101dd0 0x7efe640ff650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 >> 172.21.15.78:0/1758466348 conn(0x7efe64076310 msgr2=0x7efe64102260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 shutdown_connections 2023-10-10T17:50:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.542+0000 7efe6ab49700 1 -- 172.21.15.78:0/1758466348 wait complete. 2023-10-10T17:50:58.547 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2023-10-10T17:50:58.556 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:50:58.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 -- 172.21.15.78:0/4071691032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c104180 msgr2=0x7f054c104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 --2- 172.21.15.78:0/4071691032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c104180 0x7f054c104550 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f05400099c0 tx=0x7f0540009cd0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 -- 172.21.15.78:0/4071691032 shutdown_connections 2023-10-10T17:50:58.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 --2- 172.21.15.78:0/4071691032 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f054c100340 0x7f054c10c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 --2- 172.21.15.78:0/4071691032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c104180 0x7f054c104550 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 --2- 172.21.15.78:0/4071691032 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054c0ff990 0x7f054c0ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 -- 172.21.15.78:0/4071691032 >> 172.21.15.78:0/4071691032 conn(0x7f054c075920 msgr2=0x7f054c075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 -- 172.21.15.78:0/4071691032 shutdown_connections 2023-10-10T17:50:58.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 -- 172.21.15.78:0/4071691032 wait complete. 2023-10-10T17:50:58.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 Processor -- start 2023-10-10T17:50:58.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.678+0000 7f05530e0700 1 -- start start 2023-10-10T17:50:58.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f05530e0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f054c0ff990 0x7f054c1985f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f05530e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c100340 0x7f054c198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f05530e0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054c104180 0x7f054c19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f05530e0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c19d5d0 con 0x7f054c100340 2023-10-10T17:50:58.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f05530e0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c19d740 con 0x7f054c104180 2023-10-10T17:50:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f05530e0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c19d900 con 0x7f054c0ff990 2023-10-10T17:50:58.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c100340 0x7f054c198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c100340 0x7f054c198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48248/0 (socket says 172.21.15.78:48248) 2023-10-10T17:50:58.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 -- 172.21.15.78:0/3964622857 learned_addr learned my addr 172.21.15.78:0/3964622857 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f055167d700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054c104180 0x7f054c19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f0550e7c700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f054c0ff990 0x7f054c1985f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 -- 172.21.15.78:0/3964622857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f054c0ff990 msgr2=0x7f054c1985f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f054c0ff990 0x7f054c1985f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 -- 172.21.15.78:0/3964622857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054c104180 msgr2=0x7f054c19cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054c104180 0x7f054c19cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 -- 172.21.15.78:0/3964622857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05400095f0 con 0x7f054c100340 2023-10-10T17:50:58.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f055167d700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054c104180 0x7f054c19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:58.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f0550e7c700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f054c0ff990 0x7f054c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:58.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f054bfff700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c100340 0x7f054c198b30 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f053800c920 tx=0x7f053800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f0549ffb700 1 -- 172.21.15.78:0/3964622857 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538007b00 con 0x7f054c100340 2023-10-10T17:50:58.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f0549ffb700 1 -- 172.21.15.78:0/3964622857 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0538004cf0 con 0x7f054c100340 2023-10-10T17:50:58.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c10f240 con 0x7f054c100340 2023-10-10T17:50:58.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.679+0000 7f0549ffb700 1 -- 172.21.15.78:0/3964622857 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0538005610 con 0x7f054c100340 2023-10-10T17:50:58.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.680+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c10f0b0 con 0x7f054c100340 2023-10-10T17:50:58.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.682+0000 7f0549ffb700 1 -- 172.21.15.78:0/3964622857 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0538004750 con 0x7f054c100340 2023-10-10T17:50:58.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.682+0000 7f0549ffb700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f053c0a1340 0x7f053c0a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.682+0000 7f0549ffb700 1 -- 172.21.15.78:0/3964622857 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f05380ccfe0 con 0x7f054c100340 2023-10-10T17:50:58.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.682+0000 7f0550e7c700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f053c0a1340 0x7f053c0a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f0550e7c700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f053c0a1340 0x7f053c0a37f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0540009fd0 tx=0x7f0540005fb0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 Processor -- start 2023-10-10T17:50:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 -- start start 2023-10-10T17:50:58.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x561863319f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x56186331e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186321f060 con 0x561863319b60 2023-10-10T17:50:58.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618632718a0 con 0x56186331e8b0 2023-10-10T17:50:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f055f26cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186326ad30 con 0x56186331a470 2023-10-10T17:50:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.683+0000 7f0550e7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x561863319f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x561863319f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48254/0 (socket says 172.21.15.78:48254) 2023-10-10T17:50:58.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 -- 172.21.15.78:0/598462298 learned_addr learned my addr 172.21.15.78:0/598462298 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f055167d700 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 -- 172.21.15.78:0/598462298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 msgr2=0x56186331e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f054bfff700 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x56186331e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x56186331e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 -- 172.21.15.78:0/598462298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 msgr2=0x56186331ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 -- 172.21.15.78:0/598462298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5618630ec630 con 0x561863319b60 2023-10-10T17:50:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f055167d700 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0550e7c700 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x561863319f30 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f05400047c0 tx=0x7f054001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7abbe8132214d647 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0536ffd700 1 -- 172.21.15.78:0/598462298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054002a070 con 0x561863319b60 2023-10-10T17:50:58.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0536ffd700 1 -- 172.21.15.78:0/598462298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054001d040 con 0x561863319b60 2023-10-10T17:50:58.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.684+0000 7f0536ffd700 1 -- 172.21.15.78:0/598462298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054000f6a0 con 0x561863319b60 2023-10-10T17:50:58.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 -- 172.21.15.78:0/598462298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 msgr2=0x561863319f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x561863319f30 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f05400047c0 tx=0x7f054001e000 comp rx=0 tx=0).stop 2023-10-10T17:50:58.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 -- 172.21.15.78:0/598462298 shutdown_connections 2023-10-10T17:50:58.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x561863319f30 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 --2- 172.21.15.78:0/598462298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x56186331e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 -- 172.21.15.78:0/598462298 >> 172.21.15.78:0/598462298 conn(0x56186330f360 msgr2=0x561863314b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 -- 172.21.15.78:0/598462298 shutdown_connections 2023-10-10T17:50:58.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 -- 172.21.15.78:0/598462298 wait complete. 2023-10-10T17:50:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 -- start start 2023-10-10T17:50:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.685+0000 7f055f26cb80 1 Processor -- start 2023-10-10T17:50:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561863319b60 0x561863326070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331a470 0x561863323360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863323a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186320c8e0 con 0x56186331e8b0 2023-10-10T17:50:58.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186320c580 con 0x56186331a470 2023-10-10T17:50:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618630ee920 con 0x561863319b60 2023-10-10T17:50:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 10 client.0 _ll_register_callbacks cb 0x56186322a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863323a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863323a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48264/0 (socket says 172.21.15.78:48264) 2023-10-10T17:50:58.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 -- 172.21.15.78:0/1602591175 learned_addr learned my addr 172.21.15.78:0/1602591175 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f0550e7c700 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561863319b60 0x561863326070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f054bfff700 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331a470 0x561863323360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 -- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561863319b60 msgr2=0x561863326070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561863319b60 0x561863326070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 -- 172.21.15.78:0/1602591175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331a470 msgr2=0x561863323360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331a470 0x561863323360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 -- 172.21.15.78:0/1602591175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c10c8a0 con 0x56186331e8b0 2023-10-10T17:50:58.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055167d700 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863323a00 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f0544008f50 tx=0x7f054400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9e94b27feeb428 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f052bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f052bfff700 1 -- 172.21.15.78:0/1602591175 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054400fcb0 con 0x56186331e8b0 2023-10-10T17:50:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f052bfff700 1 -- 172.21.15.78:0/1602591175 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f054400fe10 con 0x56186331e8b0 2023-10-10T17:50:58.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.686+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56186330cb20 con 0x56186331e8b0 2023-10-10T17:50:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f052bfff700 1 -- 172.21.15.78:0/1602591175 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0544007be0 con 0x56186331e8b0 2023-10-10T17:50:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 10 client.17184 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561863231190 con 0x56186331e8b0 2023-10-10T17:50:58.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f052bfff700 1 -- 172.21.15.78:0/1602591175 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56186330cb20 con 0x56186331e8b0 2023-10-10T17:50:58.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f052bfff700 1 -- 172.21.15.78:0/1602591175 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f054401e4b0 con 0x56186331e8b0 2023-10-10T17:50:58.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 10 client.17184 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:58.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 10 client.17184 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:50:58.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 20 client.17184 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:58.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5618633338a0 0x561863335cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 4 client.17184 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:58.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56186322d0a0 con 0x5618633338a0 2023-10-10T17:50:58.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.687+0000 7f0550e7c700 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5618633338a0 0x561863335cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.688+0000 7f0550e7c700 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5618633338a0 0x561863335cb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d9d3ba84cfa9695f server_cookie=ec8a6f90371eb464 in_seq=0 out_seq=0 2023-10-10T17:50:58.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.688+0000 7f052bfff700 10 client.17184 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:58.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.688+0000 7f052bfff700 1 -- 172.21.15.78:0/1602591175 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56186322d0a0 con 0x5618633338a0 2023-10-10T17:50:58.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.688+0000 7f052bfff700 10 client.17184 handle_command_reply: tid=0 2023-10-10T17:50:58.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 client.17184 shutdown 2023-10-10T17:50:58.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 msgr2=0x561863323a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863323a00 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f0544008f50 tx=0x7f054400f3c0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 shutdown_connections 2023-10-10T17:50:58.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x5618633338a0 0x561863335cb0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561863319b60 0x561863326070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f052bfff700 0 client.17184 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863323a00 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 --2- 172.21.15.78:0/1602591175 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331a470 0x561863323360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 >> 172.21.15.78:0/1602591175 conn(0x56186330f360 msgr2=0x56186331fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 shutdown_connections 2023-10-10T17:50:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 1 -- 172.21.15.78:0/1602591175 wait complete. 2023-10-10T17:50:58.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.689+0000 7f055f26cb80 20 client.17184 trim_cache size 0 max 16384 2023-10-10T17:50:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 Processor -- start 2023-10-10T17:50:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 -- start start 2023-10-10T17:50:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x56186331c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863320c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561863326470 con 0x561863319b60 2023-10-10T17:50:58.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561863321910 con 0x56186331e8b0 2023-10-10T17:50:58.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055f26cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186331d520 con 0x56186331a470 2023-10-10T17:50:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x56186331c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x56186331c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48268/0 (socket says 172.21.15.78:48268) 2023-10-10T17:50:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 -- 172.21.15.78:0/3434169843 learned_addr learned my addr 172.21.15.78:0/3434169843 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f054bfff700 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863320c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f055167d700 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 -- 172.21.15.78:0/3434169843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 msgr2=0x561863320c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863320c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 -- 172.21.15.78:0/3434169843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 msgr2=0x56186331ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 -- 172.21.15.78:0/3434169843 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561863230ee0 con 0x561863319b60 2023-10-10T17:50:58.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.702+0000 7f0550e7c700 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x56186331c1a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f05400040c0 tx=0x7f0540004050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859b39840d4f80bf server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f05377fe700 1 -- 172.21.15.78:0/3434169843 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054002a040 con 0x561863319b60 2023-10-10T17:50:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f05377fe700 1 -- 172.21.15.78:0/3434169843 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054000bd90 con 0x561863319b60 2023-10-10T17:50:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f05377fe700 1 -- 172.21.15.78:0/3434169843 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0540003bc0 con 0x561863319b60 2023-10-10T17:50:58.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 -- 172.21.15.78:0/3434169843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 msgr2=0x56186331c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x56186331c1a0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f05400040c0 tx=0x7f0540004050 comp rx=0 tx=0).stop 2023-10-10T17:50:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 -- 172.21.15.78:0/3434169843 shutdown_connections 2023-10-10T17:50:58.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56186331e8b0 0x56186331ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561863319b60 0x56186331c1a0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 --2- 172.21.15.78:0/3434169843 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863320c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 -- 172.21.15.78:0/3434169843 >> 172.21.15.78:0/3434169843 conn(0x56186330f360 msgr2=0x561863325ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 -- 172.21.15.78:0/3434169843 shutdown_connections 2023-10-10T17:50:58.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 -- 172.21.15.78:0/3434169843 wait complete. 2023-10-10T17:50:58.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.703+0000 7f055f26cb80 1 -- start start 2023-10-10T17:50:58.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 1 Processor -- start 2023-10-10T17:50:58.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561863319b60 0x5618633288d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863328e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863329350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56186332af10 con 0x56186331e8b0 2023-10-10T17:50:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561863386790 con 0x561863319b60 2023-10-10T17:50:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561863386950 con 0x56186331a470 2023-10-10T17:50:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055f26cb80 10 client.0 _ll_register_callbacks cb 0x7f0544008790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:58.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863329350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863329350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48278/0 (socket says 172.21.15.78:48278) 2023-10-10T17:50:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 -- 172.21.15.78:0/817811452 learned_addr learned my addr 172.21.15.78:0/817811452 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f054bfff700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863328e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f0550e7c700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561863319b60 0x5618633288d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 -- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 msgr2=0x561863328e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863328e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 -- 172.21.15.78:0/817811452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561863319b60 msgr2=0x5618633288d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561863319b60 0x5618633288d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f055167d700 1 -- 172.21.15.78:0/817811452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618630ec630 con 0x56186331e8b0 2023-10-10T17:50:58.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.704+0000 7f0550e7c700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561863319b60 0x5618633288d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:58.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055167d700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863329350 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f054400a010 tx=0x7f0544007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef7a2a953d1e55d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f052bfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:58.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f052bfff700 1 -- 172.21.15.78:0/817811452 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054401e5b0 con 0x56186331e8b0 2023-10-10T17:50:58.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f052bfff700 1 -- 172.21.15.78:0/817811452 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f054400fab0 con 0x56186331e8b0 2023-10-10T17:50:58.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f054001d040 con 0x56186331e8b0 2023-10-10T17:50:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f052bfff700 1 -- 172.21.15.78:0/817811452 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f054400fc70 con 0x56186331e8b0 2023-10-10T17:50:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 10 client.17190 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561863081c60 con 0x56186331e8b0 2023-10-10T17:50:58.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f052bfff700 1 -- 172.21.15.78:0/817811452 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054001d040 con 0x56186331e8b0 2023-10-10T17:50:58.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f052bfff700 1 -- 172.21.15.78:0/817811452 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f054402d020 con 0x56186331e8b0 2023-10-10T17:50:58.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 10 client.17190 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:58.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 10 client.17190 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:50:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 20 client.17190 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561863388110 0x56186338a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 4 client.17190 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:50:58.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.705+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561863239660 con 0x561863388110 2023-10-10T17:50:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f0550e7c700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561863388110 0x56186338a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f052bfff700 10 client.17190 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f0550e7c700 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561863388110 0x56186338a520 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=600b71561e4f6f9f server_cookie=aeac8c14c9917a3 in_seq=0 out_seq=0 2023-10-10T17:50:58.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f052bfff700 1 -- 172.21.15.78:0/817811452 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561863239660 con 0x561863388110 2023-10-10T17:50:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f052bfff700 10 client.17190 handle_command_reply: tid=0 2023-10-10T17:50:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 client.17190 shutdown 2023-10-10T17:50:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 msgr2=0x561863329350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863329350 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f054400a010 tx=0x7f0544007e60 comp rx=0 tx=0).stop 2023-10-10T17:50:58.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 shutdown_connections 2023-10-10T17:50:58.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56186331e8b0 0x561863329350 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561863319b60 0x5618633288d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x561863388110 0x56186338a520 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 --2- 172.21.15.78:0/817811452 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56186331a470 0x561863328e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 >> 172.21.15.78:0/817811452 conn(0x56186330f360 msgr2=0x56186332fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f052bfff700 0 client.17190 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:50:58.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 shutdown_connections 2023-10-10T17:50:58.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.706+0000 7f055f26cb80 1 -- 172.21.15.78:0/817811452 wait complete. 2023-10-10T17:50:58.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f055f26cb80 20 client.17190 trim_cache size 0 max 16384 2023-10-10T17:50:58.765 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:50:58.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f053c0a1340 msgr2=0x7f053c0a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f053c0a1340 0x7f053c0a37f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0540009fd0 tx=0x7f0540005fb0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c100340 msgr2=0x7f054c198b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c100340 0x7f054c198b30 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f053800c920 tx=0x7f053800cce0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 shutdown_connections 2023-10-10T17:50:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f054c0ff990 0x7f054c1985f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f053c0a1340 0x7f053c0a37f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f054c104180 0x7f054c19cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 --2- 172.21.15.78:0/3964622857 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f054c100340 0x7f054c198b30 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 >> 172.21.15.78:0/3964622857 conn(0x7f054c075920 msgr2=0x7f054c0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 shutdown_connections 2023-10-10T17:50:58.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.707+0000 7f05530e0700 1 -- 172.21.15.78:0/3964622857 wait complete. 2023-10-10T17:50:58.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:50:58.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:58 smithi078 ceph-mon[99494]: pgmap v260: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:50:58.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:58 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1758466348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:50:58.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 -- 172.21.15.78:0/4244757815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 msgr2=0x7f44a410f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 --2- 172.21.15.78:0/4244757815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a410f040 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f449c00b0a0 tx=0x7f449c00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 -- 172.21.15.78:0/4244757815 shutdown_connections 2023-10-10T17:50:58.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 --2- 172.21.15.78:0/4244757815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44a4101210 0x7f44a41015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 --2- 172.21.15.78:0/4244757815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a410f040 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 --2- 172.21.15.78:0/4244757815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44a4068e70 0x7f44a40692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 -- 172.21.15.78:0/4244757815 >> 172.21.15.78:0/4244757815 conn(0x7f44a4075470 msgr2=0x7f44a4075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 -- 172.21.15.78:0/4244757815 shutdown_connections 2023-10-10T17:50:58.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.892+0000 7f44ac564700 1 -- 172.21.15.78:0/4244757815 wait complete. 2023-10-10T17:50:58.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 Processor -- start 2023-10-10T17:50:58.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 -- start start 2023-10-10T17:50:58.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44a4068e70 0x7f44a4198630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a4198b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44a4101210 0x7f44a419cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a419d610 con 0x7f44a4069900 2023-10-10T17:50:58.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a419d780 con 0x7f44a4101210 2023-10-10T17:50:58.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44ac564700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a419d940 con 0x7f44a4068e70 2023-10-10T17:50:58.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44a9aff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a4198b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44a9aff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a4198b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48290/0 (socket says 172.21.15.78:48290) 2023-10-10T17:50:58.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44a9aff700 1 -- 172.21.15.78:0/758652493 learned_addr learned my addr 172.21.15.78:0/758652493 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44aa300700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44a4068e70 0x7f44a4198630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44aab01700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44a4101210 0x7f44a419cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44a9aff700 1 -- 172.21.15.78:0/758652493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44a4068e70 msgr2=0x7f44a4198630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44a9aff700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44a4068e70 0x7f44a4198630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44a9aff700 1 -- 172.21.15.78:0/758652493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44a4101210 msgr2=0x7f44a419cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.893+0000 7f44a9aff700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44a4101210 0x7f44a419cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.894+0000 7f44a9aff700 1 -- 172.21.15.78:0/758652493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c009d00 con 0x7f44a4069900 2023-10-10T17:50:58.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.894+0000 7f44a9aff700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a4198b70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f449400e9c0 tx=0x7f449400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.894+0000 7f449b7fe700 1 -- 172.21.15.78:0/758652493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44940096e0 con 0x7f44a4069900 2023-10-10T17:50:58.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.894+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a4112580 con 0x7f44a4069900 2023-10-10T17:50:58.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.894+0000 7f449b7fe700 1 -- 172.21.15.78:0/758652493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f449400c7c0 con 0x7f44a4069900 2023-10-10T17:50:58.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.894+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a41123f0 con 0x7f44a4069900 2023-10-10T17:50:58.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.894+0000 7f449b7fe700 1 -- 172.21.15.78:0/758652493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494003b80 con 0x7f44a4069900 2023-10-10T17:50:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.896+0000 7f449b7fe700 1 -- 172.21.15.78:0/758652493 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f449400c930 con 0x7f44a4069900 2023-10-10T17:50:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.897+0000 7f449b7fe700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44900a12c0 0x7f44900a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.897+0000 7f44aa300700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44900a12c0 0x7f44900a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.897+0000 7f449b7fe700 1 -- 172.21.15.78:0/758652493 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f4494014070 con 0x7f44a4069900 2023-10-10T17:50:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.897+0000 7f44aa300700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44900a12c0 0x7f44900a3770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009c60 tx=0x7f44a0009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.898+0000 7f44b86f0b80 1 Processor -- start 2023-10-10T17:50:58.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44b86f0b80 1 -- start start 2023-10-10T17:50:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f23b60 0x55ea04f23f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f28370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f288b0 0x55ea04f28d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04e29060 con 0x55ea04f23b60 2023-10-10T17:50:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04e7b8a0 con 0x55ea04f288b0 2023-10-10T17:50:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04e74d30 con 0x55ea04f24470 2023-10-10T17:50:58.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f23b60 0x55ea04f23f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f23b60 0x55ea04f23f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48296/0 (socket says 172.21.15.78:48296) 2023-10-10T17:50:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 -- 172.21.15.78:0/4041086658 learned_addr learned my addr 172.21.15.78:0/4041086658 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44a9aff700 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f28370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 -- 172.21.15.78:0/4041086658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 msgr2=0x55ea04f28370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f28370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 -- 172.21.15.78:0/4041086658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f288b0 msgr2=0x55ea04f28d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aab01700 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f288b0 0x55ea04f28d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f288b0 0x55ea04f28d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.899+0000 7f44aa300700 1 -- 172.21.15.78:0/4041086658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea04cf6630 con 0x55ea04f23b60 2023-10-10T17:50:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.900+0000 7f44aa300700 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f23b60 0x55ea04f23f30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f44a000e4e0 tx=0x7f44a0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916b1c855890efcc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.900+0000 7f4498ff9700 1 -- 172.21.15.78:0/4041086658 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a000ee00 con 0x55ea04f23b60 2023-10-10T17:50:58.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.900+0000 7f4498ff9700 1 -- 172.21.15.78:0/4041086658 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44a0021070 con 0x55ea04f23b60 2023-10-10T17:50:58.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.900+0000 7f4498ff9700 1 -- 172.21.15.78:0/4041086658 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a001d7b0 con 0x55ea04f23b60 2023-10-10T17:50:58.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 -- 172.21.15.78:0/4041086658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f23b60 msgr2=0x55ea04f23f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f23b60 0x55ea04f23f30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f44a000e4e0 tx=0x7f44a0014000 comp rx=0 tx=0).stop 2023-10-10T17:50:58.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 -- 172.21.15.78:0/4041086658 shutdown_connections 2023-10-10T17:50:58.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f288b0 0x55ea04f28d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f23b60 0x55ea04f23f30 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/4041086658 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f28370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 -- 172.21.15.78:0/4041086658 >> 172.21.15.78:0/4041086658 conn(0x55ea04f19360 msgr2=0x55ea04f1eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 -- 172.21.15.78:0/4041086658 shutdown_connections 2023-10-10T17:50:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 -- 172.21.15.78:0/4041086658 wait complete. 2023-10-10T17:50:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.901+0000 7f44b86f0b80 1 -- start start 2023-10-10T17:50:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 1 Processor -- start 2023-10-10T17:50:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f300c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f24470 0x55ea04f2d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f288b0 0x55ea04f2da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04e168e0 con 0x55ea04f24470 2023-10-10T17:50:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04e16580 con 0x55ea04f23b60 2023-10-10T17:50:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04cf8920 con 0x55ea04f288b0 2023-10-10T17:50:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44b86f0b80 10 client.0 _ll_register_callbacks cb 0x55ea04e34780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f24470 0x55ea04f2d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f24470 0x55ea04f2d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48304/0 (socket says 172.21.15.78:48304) 2023-10-10T17:50:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 -- 172.21.15.78:0/421904996 learned_addr learned my addr 172.21.15.78:0/421904996 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44aab01700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f288b0 0x55ea04f2da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44aa300700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f300c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 -- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f288b0 msgr2=0x55ea04f2da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f288b0 0x55ea04f2da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 -- 172.21.15.78:0/421904996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 msgr2=0x55ea04f300c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f300c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44a9aff700 1 -- 172.21.15.78:0/421904996 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a0017070 con 0x55ea04f24470 2023-10-10T17:50:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.902+0000 7f44aa300700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f44a9aff700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f24470 0x55ea04f2d340 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4494000c00 tx=0x7f4494099f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f4d7e5d7428bb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f448dffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:58.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f448dffb700 1 -- 172.21.15.78:0/421904996 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449409a130 con 0x55ea04f24470 2023-10-10T17:50:58.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f448dffb700 1 -- 172.21.15.78:0/421904996 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f449409a290 con 0x55ea04f24470 2023-10-10T17:50:58.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea04f16b20 con 0x55ea04f24470 2023-10-10T17:50:58.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f448dffb700 1 -- 172.21.15.78:0/421904996 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44940b82b0 con 0x55ea04f24470 2023-10-10T17:50:58.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f44b86f0b80 10 client.17202 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:58.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ea04e3b190 con 0x55ea04f24470 2023-10-10T17:50:58.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f448dffb700 1 -- 172.21.15.78:0/421904996 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f44940ca080 con 0x55ea04f24470 2023-10-10T17:50:58.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f448dffb700 1 -- 172.21.15.78:0/421904996 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f4494099450 con 0x55ea04f24470 2023-10-10T17:50:58.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f44b86f0b80 10 client.17202 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:58.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.903+0000 7f44b86f0b80 10 client.17202 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:50:58.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.904+0000 7f44b86f0b80 20 client.17202 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:58.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.904+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f3d850 0x55ea04f3fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.904+0000 7f44b86f0b80 4 client.17202 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:58.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.904+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea04e370a0 con 0x55ea04f3d850 2023-10-10T17:50:58.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.904+0000 7f44aa300700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f3d850 0x55ea04f3fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.904+0000 7f44aa300700 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f3d850 0x55ea04f3fc60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eec73d0279d3b149 server_cookie=79db98ff18c6bd99 in_seq=0 out_seq=0 2023-10-10T17:50:58.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.904+0000 7f448dffb700 10 client.17202 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:58.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f448dffb700 1 -- 172.21.15.78:0/421904996 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ea04e370a0 con 0x55ea04f3d850 2023-10-10T17:50:58.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f448dffb700 10 client.17202 handle_command_reply: tid=0 2023-10-10T17:50:58.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 client.17202 shutdown 2023-10-10T17:50:58.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f24470 msgr2=0x55ea04f2d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f24470 0x55ea04f2d340 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f4494000c00 tx=0x7f4494099f50 comp rx=0 tx=0).stop 2023-10-10T17:50:58.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 shutdown_connections 2023-10-10T17:50:58.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f3d850 0x55ea04f3fc60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f288b0 0x55ea04f2da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f300c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/421904996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f24470 0x55ea04f2d340 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 >> 172.21.15.78:0/421904996 conn(0x55ea04f19360 msgr2=0x55ea04f29d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f448dffb700 0 client.17202 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:58.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 shutdown_connections 2023-10-10T17:50:58.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 1 -- 172.21.15.78:0/421904996 wait complete. 2023-10-10T17:50:58.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.905+0000 7f44b86f0b80 20 client.17202 trim_cache size 0 max 16384 2023-10-10T17:50:58.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 Processor -- start 2023-10-10T17:50:58.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 -- start start 2023-10-10T17:50:58.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f261a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f2fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f26a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04f274c0 con 0x55ea04f288b0 2023-10-10T17:50:58.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04f3ff70 con 0x55ea04f23b60 2023-10-10T17:50:58.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04f3a260 con 0x55ea04f24470 2023-10-10T17:50:58.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f26a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f26a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48314/0 (socket says 172.21.15.78:48314) 2023-10-10T17:50:58.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 -- 172.21.15.78:0/368203768 learned_addr learned my addr 172.21.15.78:0/368203768 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aa300700 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f261a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44a9aff700 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f2fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 -- 172.21.15.78:0/368203768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 msgr2=0x55ea04f2fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f2fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 -- 172.21.15.78:0/368203768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 msgr2=0x55ea04f261a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f261a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 -- 172.21.15.78:0/368203768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea04e3aee0 con 0x55ea04f288b0 2023-10-10T17:50:58.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.919+0000 7f44aab01700 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f26a10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f449c0093c0 tx=0x7f449c0093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d02cc40116fbb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44997fa700 1 -- 172.21.15.78:0/368203768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c00e040 con 0x55ea04f288b0 2023-10-10T17:50:58.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44997fa700 1 -- 172.21.15.78:0/368203768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f449c009b80 con 0x55ea04f288b0 2023-10-10T17:50:58.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44997fa700 1 -- 172.21.15.78:0/368203768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c0052a0 con 0x55ea04f288b0 2023-10-10T17:50:58.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 -- 172.21.15.78:0/368203768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 msgr2=0x55ea04f26a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f26a10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f449c0093c0 tx=0x7f449c0093f0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 -- 172.21.15.78:0/368203768 shutdown_connections 2023-10-10T17:50:58.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f26a10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f261a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/368203768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f2fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 -- 172.21.15.78:0/368203768 >> 172.21.15.78:0/368203768 conn(0x55ea04f19360 msgr2=0x55ea04f3f860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 -- 172.21.15.78:0/368203768 shutdown_connections 2023-10-10T17:50:58.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.920+0000 7f44b86f0b80 1 -- 172.21.15.78:0/368203768 wait complete. 2023-10-10T17:50:58.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 -- start start 2023-10-10T17:50:58.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 Processor -- start 2023-10-10T17:50:58.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f32990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f32ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f33410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04f34fd0 con 0x55ea04f288b0 2023-10-10T17:50:58.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04f90740 con 0x55ea04f23b60 2023-10-10T17:50:58.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea04f90900 con 0x55ea04f24470 2023-10-10T17:50:58.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.921+0000 7f44b86f0b80 10 client.0 _ll_register_callbacks cb 0x7f4494099870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:58.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f33410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f33410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48318/0 (socket says 172.21.15.78:48318) 2023-10-10T17:50:58.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 -- 172.21.15.78:0/2093879568 learned_addr learned my addr 172.21.15.78:0/2093879568 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:58.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aa300700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f32990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44a9aff700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f32ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 -- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 msgr2=0x55ea04f32ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f32ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 -- 172.21.15.78:0/2093879568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 msgr2=0x55ea04f32990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:58.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f32990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 -- 172.21.15.78:0/2093879568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a410fc10 con 0x55ea04f288b0 2023-10-10T17:50:58.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44a9aff700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f32ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:50:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aa300700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f32990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:50:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44aab01700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f33410 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f449c006260 tx=0x7f449c0099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcc6c18e868351a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f448dffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:58.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f448dffb700 1 -- 172.21.15.78:0/2093879568 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c00e070 con 0x55ea04f288b0 2023-10-10T17:50:58.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f448dffb700 1 -- 172.21.15.78:0/2093879568 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f449c003e10 con 0x55ea04f288b0 2023-10-10T17:50:58.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea04e43660 con 0x55ea04f288b0 2023-10-10T17:50:58.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f448dffb700 1 -- 172.21.15.78:0/2093879568 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f449c009b80 con 0x55ea04f288b0 2023-10-10T17:50:58.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44b86f0b80 10 client.17208 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:58.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.922+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ea04c8bc60 con 0x55ea04f288b0 2023-10-10T17:50:58.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f448dffb700 1 -- 172.21.15.78:0/2093879568 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ea04e43660 con 0x55ea04f288b0 2023-10-10T17:50:58.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f448dffb700 1 -- 172.21.15.78:0/2093879568 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f449c023020 con 0x55ea04f288b0 2023-10-10T17:50:58.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f44b86f0b80 10 client.17208 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:58.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f44b86f0b80 10 client.17208 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:50:58.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f44b86f0b80 20 client.17208 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:58.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f91dc0 0x55ea04f941d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:58.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f44b86f0b80 4 client.17208 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:50:58.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ea04cc3f70 con 0x55ea04f91dc0 2023-10-10T17:50:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.923+0000 7f44aa300700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f91dc0 0x55ea04f941d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f448dffb700 10 client.17208 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44aa300700 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f91dc0 0x55ea04f941d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b4cb68fed9899d5a server_cookie=527399f3fd33b7b4 in_seq=0 out_seq=0 2023-10-10T17:50:58.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f448dffb700 1 -- 172.21.15.78:0/2093879568 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ea04cc3f70 con 0x55ea04f91dc0 2023-10-10T17:50:58.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f448dffb700 10 client.17208 handle_command_reply: tid=0 2023-10-10T17:50:58.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 client.17208 shutdown 2023-10-10T17:50:58.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 msgr2=0x55ea04f33410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f33410 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f449c006260 tx=0x7f449c0099e0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 shutdown_connections 2023-10-10T17:50:58.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ea04f288b0 0x55ea04f33410 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ea04f23b60 0x55ea04f32990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55ea04f91dc0 0x55ea04f941d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 --2- 172.21.15.78:0/2093879568 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ea04f24470 0x55ea04f32ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 >> 172.21.15.78:0/2093879568 conn(0x55ea04f19360 msgr2=0x55ea04f26a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f448dffb700 0 client.17208 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:50:58.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 shutdown_connections 2023-10-10T17:50:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 1 -- 172.21.15.78:0/2093879568 wait complete. 2023-10-10T17:50:58.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.924+0000 7f44b86f0b80 20 client.17208 trim_cache size 0 max 16384 2023-10-10T17:50:58.971 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:50:58.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44900a12c0 msgr2=0x7f44900a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44900a12c0 0x7f44900a3770 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f44a0009c60 tx=0x7f44a0009260 comp rx=0 tx=0).stop 2023-10-10T17:50:58.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 msgr2=0x7f44a4198b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:58.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a4198b70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f449400e9c0 tx=0x7f449400ecd0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 shutdown_connections 2023-10-10T17:50:58.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44a4068e70 0x7f44a4198630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44900a12c0 0x7f44900a3770 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44a4101210 0x7f44a419cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 --2- 172.21.15.78:0/758652493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44a4069900 0x7f44a4198b70 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:58.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 >> 172.21.15.78:0/758652493 conn(0x7f44a4075470 msgr2=0x7f44a40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:58.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 shutdown_connections 2023-10-10T17:50:58.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:58.925+0000 7f44ac564700 1 -- 172.21.15.78:0/758652493 wait complete. 2023-10-10T17:50:58.977 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:50:59.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 -- 172.21.15.78:0/1364321514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5314101ca0 msgr2=0x7f531410a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 --2- 172.21.15.78:0/1364321514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5314101ca0 0x7f531410a370 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f5308009990 tx=0x7f5308009ca0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 -- 172.21.15.78:0/1364321514 shutdown_connections 2023-10-10T17:50:59.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 --2- 172.21.15.78:0/1364321514 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f531410a8b0 0x7f531410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 --2- 172.21.15.78:0/1364321514 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5314101390 0x7f5314101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 --2- 172.21.15.78:0/1364321514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5314101ca0 0x7f531410a370 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 -- 172.21.15.78:0/1364321514 >> 172.21.15.78:0/1364321514 conn(0x7f53140770b0 msgr2=0x7f53140774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:59.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 -- 172.21.15.78:0/1364321514 shutdown_connections 2023-10-10T17:50:59.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.103+0000 7f5319d8d700 1 -- 172.21.15.78:0/1364321514 wait complete. 2023-10-10T17:50:59.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.104+0000 7f5319d8d700 1 Processor -- start 2023-10-10T17:50:59.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.104+0000 7f5319d8d700 1 -- start start 2023-10-10T17:50:59.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5314101390 0x7f5314196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5314101ca0 0x7f53141969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f531410a8b0 0x7f531419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531419b460 con 0x7f531410a8b0 2023-10-10T17:50:59.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531419b5d0 con 0x7f5314101ca0 2023-10-10T17:50:59.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531419b790 con 0x7f5314101390 2023-10-10T17:50:59.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f531410a8b0 0x7f531419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f531410a8b0 0x7f531419ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48334/0 (socket says 172.21.15.78:48334) 2023-10-10T17:50:59.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 -- 172.21.15.78:0/3452149359 learned_addr learned my addr 172.21.15.78:0/3452149359 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:59.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f53137fe700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5314101390 0x7f5314196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5312ffd700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5314101ca0 0x7f53141969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 -- 172.21.15.78:0/3452149359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5314101390 msgr2=0x7f5314196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5314101390 0x7f5314196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 -- 172.21.15.78:0/3452149359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5314101ca0 msgr2=0x7f53141969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5314101ca0 0x7f53141969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 -- 172.21.15.78:0/3452149359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53080095f0 con 0x7f531410a8b0 2023-10-10T17:50:59.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5313fff700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f531410a8b0 0x7f531419ae20 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f530c00ef90 tx=0x7f530c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:59.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5310ff9700 1 -- 172.21.15.78:0/3452149359 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c018070 con 0x7f531410a8b0 2023-10-10T17:50:59.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5310ff9700 1 -- 172.21.15.78:0/3452149359 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f530c00f040 con 0x7f531410a8b0 2023-10-10T17:50:59.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5314110390 con 0x7f531410a8b0 2023-10-10T17:50:59.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5310ff9700 1 -- 172.21.15.78:0/3452149359 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c00cda0 con 0x7f531410a8b0 2023-10-10T17:50:59.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.105+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5314110200 con 0x7f531410a8b0 2023-10-10T17:50:59.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.108+0000 7f5310ff9700 1 -- 172.21.15.78:0/3452149359 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f530c003a00 con 0x7f531410a8b0 2023-10-10T17:50:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.109+0000 7f5310ff9700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f52fc0a12c0 0x7f52fc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.109+0000 7f53137fe700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f52fc0a12c0 0x7f52fc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.109+0000 7f5310ff9700 1 -- 172.21.15.78:0/3452149359 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f530c0ccb50 con 0x7f531410a8b0 2023-10-10T17:50:59.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f53137fe700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f52fc0a12c0 0x7f52fc0a3770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5304009c60 tx=0x7f5304009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:59.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 Processor -- start 2023-10-10T17:50:59.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 -- start start 2023-10-10T17:50:59.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd24ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd254370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd254d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd155060 con 0x55f6bd250470 2023-10-10T17:50:59.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd1a78a0 con 0x55f6bd24fb60 2023-10-10T17:50:59.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5325f19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd1a0d30 con 0x55f6bd2548b0 2023-10-10T17:50:59.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd254370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5313fff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd254d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd254370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48340/0 (socket says 172.21.15.78:48340) 2023-10-10T17:50:59.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 -- 172.21.15.78:0/1310414486 learned_addr learned my addr 172.21.15.78:0/1310414486 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:59.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f53137fe700 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd24ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 -- 172.21.15.78:0/1310414486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 msgr2=0x55f6bd254d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd254d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 -- 172.21.15.78:0/1310414486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 msgr2=0x55f6bd24ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd24ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5312ffd700 1 -- 172.21.15.78:0/1310414486 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6bd022630 con 0x55f6bd250470 2023-10-10T17:50:59.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f53137fe700 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd24ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:50:59.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.110+0000 7f5313fff700 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd254d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:50:59.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.111+0000 7f5312ffd700 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd254370 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5308015000 tx=0x7f5308005ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7341ebb40cadc18 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:59.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.111+0000 7f52f9ffb700 1 -- 172.21.15.78:0/1310414486 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530801d070 con 0x55f6bd250470 2023-10-10T17:50:59.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.111+0000 7f52f9ffb700 1 -- 172.21.15.78:0/1310414486 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f530801e040 con 0x55f6bd250470 2023-10-10T17:50:59.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.111+0000 7f52f9ffb700 1 -- 172.21.15.78:0/1310414486 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5308004920 con 0x55f6bd250470 2023-10-10T17:50:59.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 -- 172.21.15.78:0/1310414486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 msgr2=0x55f6bd254370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd254370 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5308015000 tx=0x7f5308005ea0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 -- 172.21.15.78:0/1310414486 shutdown_connections 2023-10-10T17:50:59.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd254d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd24ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1310414486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd254370 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 -- 172.21.15.78:0/1310414486 >> 172.21.15.78:0/1310414486 conn(0x55f6bd245360 msgr2=0x55f6bd24ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:59.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 -- 172.21.15.78:0/1310414486 shutdown_connections 2023-10-10T17:50:59.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 -- 172.21.15.78:0/1310414486 wait complete. 2023-10-10T17:50:59.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 -- start start 2023-10-10T17:50:59.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.112+0000 7f5325f19b80 1 Processor -- start 2023-10-10T17:50:59.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd259260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd259870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5325f19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd1428e0 con 0x55f6bd250470 2023-10-10T17:50:59.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5325f19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd142580 con 0x55f6bd24fb60 2023-10-10T17:50:59.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5325f19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd024920 con 0x55f6bd2548b0 2023-10-10T17:50:59.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd259260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd259260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48346/0 (socket says 172.21.15.78:48346) 2023-10-10T17:50:59.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5325f19b80 10 client.0 _ll_register_callbacks cb 0x55f6bd160780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:59.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 -- 172.21.15.78:0/2367860488 learned_addr learned my addr 172.21.15.78:0/2367860488 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:59.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f53137fe700 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5313fff700 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd259870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 -- 172.21.15.78:0/2367860488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 msgr2=0x55f6bd259870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd259870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 -- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 msgr2=0x55f6bd25c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 -- 172.21.15.78:0/2367860488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531410d990 con 0x55f6bd250470 2023-10-10T17:50:59.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f5312ffd700 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd259260 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5308009ce0 tx=0x7f5308005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedc76f0fc9a85a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:59.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.113+0000 7f52f2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:59.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f52f2ffd700 1 -- 172.21.15.78:0/2367860488 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530801d040 con 0x55f6bd250470 2023-10-10T17:50:59.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f52f2ffd700 1 -- 172.21.15.78:0/2367860488 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f5308005050 con 0x55f6bd250470 2023-10-10T17:50:59.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f6bd242b20 con 0x55f6bd250470 2023-10-10T17:50:59.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f52f2ffd700 1 -- 172.21.15.78:0/2367860488 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5308005bc0 con 0x55f6bd250470 2023-10-10T17:50:59.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 10 client.17220 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:59.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55f6bd167190 con 0x55f6bd250470 2023-10-10T17:50:59.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f52f2ffd700 1 -- 172.21.15.78:0/2367860488 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f6bd242b20 con 0x55f6bd250470 2023-10-10T17:50:59.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f52f2ffd700 1 -- 172.21.15.78:0/2367860488 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5308004ab0 con 0x55f6bd250470 2023-10-10T17:50:59.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 10 client.17220 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 10 client.17220 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 20 client.17220 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd269490 0x55f6bd26b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 4 client.17220 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:50:59.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f6bd1630a0 con 0x55f6bd269490 2023-10-10T17:50:59.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.114+0000 7f53137fe700 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd269490 0x55f6bd26b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.115+0000 7f52f2ffd700 10 client.17220 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.115+0000 7f53137fe700 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd269490 0x55f6bd26b8a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc5302718a5359e server_cookie=b97f763d585c71be in_seq=0 out_seq=0 2023-10-10T17:50:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.115+0000 7f52f2ffd700 1 -- 172.21.15.78:0/2367860488 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55f6bd1630a0 con 0x55f6bd269490 2023-10-10T17:50:59.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.115+0000 7f52f2ffd700 10 client.17220 handle_command_reply: tid=0 2023-10-10T17:50:59.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 client.17220 shutdown 2023-10-10T17:50:59.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 msgr2=0x55f6bd259260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd259260 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f5308009ce0 tx=0x7f5308005f10 comp rx=0 tx=0).stop 2023-10-10T17:50:59.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 shutdown_connections 2023-10-10T17:50:59.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd2548b0 0x55f6bd259870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd269490 0x55f6bd26b8a0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f52f2ffd700 0 client.17220 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:59.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 --2- 172.21.15.78:0/2367860488 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd250470 0x55f6bd259260 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 >> 172.21.15.78:0/2367860488 conn(0x55f6bd245360 msgr2=0x55f6bd249de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:59.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 shutdown_connections 2023-10-10T17:50:59.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 1 -- 172.21.15.78:0/2367860488 wait complete. 2023-10-10T17:50:59.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.116+0000 7f5325f19b80 20 client.17220 trim_cache size 0 max 16384 2023-10-10T17:50:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:58 smithi190 ceph-mon[101220]: pgmap v260: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:50:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:58 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1758466348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:50:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 Processor -- start 2023-10-10T17:50:59.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 -- start start 2023-10-10T17:50:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd24fb60 0x55f6bd25bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd250470 0x55f6bd260940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd2548b0 0x55f6bd258b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd245170 con 0x55f6bd2548b0 2023-10-10T17:50:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd250bc0 con 0x55f6bd250470 2023-10-10T17:50:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5325f19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd259e70 con 0x55f6bd24fb60 2023-10-10T17:50:59.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5313fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd2548b0 0x55f6bd258b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.129+0000 7f5313fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd2548b0 0x55f6bd258b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48356/0 (socket says 172.21.15.78:48356) 2023-10-10T17:50:59.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5313fff700 1 -- 172.21.15.78:0/1366809056 learned_addr learned my addr 172.21.15.78:0/1366809056 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:59.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5312ffd700 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd250470 0x55f6bd260940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f53137fe700 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd24fb60 0x55f6bd25bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5313fff700 1 -- 172.21.15.78:0/1366809056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd24fb60 msgr2=0x55f6bd25bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5313fff700 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd24fb60 0x55f6bd25bb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.174 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5313fff700 1 -- 172.21.15.78:0/1366809056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd250470 msgr2=0x55f6bd260940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5313fff700 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd250470 0x55f6bd260940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5313fff700 1 -- 172.21.15.78:0/1366809056 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6bd246b00 con 0x55f6bd2548b0 2023-10-10T17:50:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5312ffd700 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd250470 0x55f6bd260940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:50:59.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f53137fe700 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd24fb60 0x55f6bd25bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:50:59.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f5313fff700 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd2548b0 0x55f6bd258b00 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f530c0c9f70 tx=0x7f530c095d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b1e81c5134b71a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:59.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f52fa7fc700 1 -- 172.21.15.78:0/1366809056 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c099280 con 0x55f6bd2548b0 2023-10-10T17:50:59.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f52fa7fc700 1 -- 172.21.15.78:0/1366809056 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f530c099920 con 0x55f6bd2548b0 2023-10-10T17:50:59.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.130+0000 7f52fa7fc700 1 -- 172.21.15.78:0/1366809056 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530c098e10 con 0x55f6bd2548b0 2023-10-10T17:50:59.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 -- 172.21.15.78:0/1366809056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd2548b0 msgr2=0x55f6bd258b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd2548b0 0x55f6bd258b00 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f530c0c9f70 tx=0x7f530c095d80 comp rx=0 tx=0).stop 2023-10-10T17:50:59.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 -- 172.21.15.78:0/1366809056 shutdown_connections 2023-10-10T17:50:59.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd24fb60 0x55f6bd25bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd2548b0 0x55f6bd258b00 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 --2- 172.21.15.78:0/1366809056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd250470 0x55f6bd260940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 -- 172.21.15.78:0/1366809056 >> 172.21.15.78:0/1366809056 conn(0x55f6bd245360 msgr2=0x55f6bd256c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:59.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 -- 172.21.15.78:0/1366809056 shutdown_connections 2023-10-10T17:50:59.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 -- 172.21.15.78:0/1366809056 wait complete. 2023-10-10T17:50:59.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 -- start start 2023-10-10T17:50:59.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.131+0000 7f5325f19b80 1 Processor -- start 2023-10-10T17:50:59.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd250470 0x55f6bd25ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd2548b0 0x55f6bd25d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd2bc3e0 con 0x55f6bd24fb60 2023-10-10T17:50:59.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd2bc520 con 0x55f6bd2548b0 2023-10-10T17:50:59.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6bd2bc6e0 con 0x55f6bd250470 2023-10-10T17:50:59.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 10 client.0 _ll_register_callbacks cb 0x7f53040096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:50:59.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48358/0 (socket says 172.21.15.78:48358) 2023-10-10T17:50:59.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 -- 172.21.15.78:0/649157812 learned_addr learned my addr 172.21.15.78:0/649157812 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:50:59.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5312ffd700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd250470 0x55f6bd25ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5313fff700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd2548b0 0x55f6bd25d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 -- 172.21.15.78:0/649157812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd250470 msgr2=0x55f6bd25ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd250470 0x55f6bd25ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 -- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd2548b0 msgr2=0x55f6bd25d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:50:59.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd2548b0 0x55f6bd25d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 -- 172.21.15.78:0/649157812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f6bd167190 con 0x55f6bd24fb60 2023-10-10T17:50:59.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5313fff700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd2548b0 0x55f6bd25d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:50:59.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f53137fe700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c920 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5304013010 tx=0x7f53040047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a3fa2cf66daa95 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:50:59.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f52f2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:50:59.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f52f2ffd700 1 -- 172.21.15.78:0/649157812 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304022040 con 0x55f6bd24fb60 2023-10-10T17:50:59.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f52f2ffd700 1 -- 172.21.15.78:0/649157812 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f530402b5a0 con 0x55f6bd24fb60 2023-10-10T17:50:59.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.132+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f530801e040 con 0x55f6bd24fb60 2023-10-10T17:50:59.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f52f2ffd700 1 -- 172.21.15.78:0/649157812 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5304003680 con 0x55f6bd24fb60 2023-10-10T17:50:59.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 10 client.17226 fetch_fsmap learned FSMap version 32 2023-10-10T17:50:59.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x7f5314110390 con 0x55f6bd24fb60 2023-10-10T17:50:59.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f52f2ffd700 1 -- 172.21.15.78:0/649157812 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f530801e040 con 0x55f6bd24fb60 2023-10-10T17:50:59.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f52f2ffd700 1 -- 172.21.15.78:0/649157812 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f5304034820 con 0x55f6bd24fb60 2023-10-10T17:50:59.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 10 client.17226 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:50:59.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 10 client.17226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:50:59.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 20 client.17226 populate_metadata read hostname 'smithi078' 2023-10-10T17:50:59.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd2bde90 0x55f6bd2c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:50:59.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 4 client.17226 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:50:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f6bd16f660 con 0x55f6bd2bde90 2023-10-10T17:50:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.133+0000 7f5312ffd700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd2bde90 0x55f6bd2c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:50:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f52f2ffd700 10 client.17226 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:59.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5312ffd700 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd2bde90 0x55f6bd2c02a0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c8306a6aa6eec0 server_cookie=457b86ee4bc218aa in_seq=0 out_seq=0 2023-10-10T17:50:59.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f52f2ffd700 1 -- 172.21.15.78:0/649157812 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f6bd16f660 con 0x55f6bd2bde90 2023-10-10T17:50:59.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f52f2ffd700 10 client.17226 handle_command_reply: tid=0 2023-10-10T17:50:59.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 client.17226 shutdown 2023-10-10T17:50:59.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd24fb60 msgr2=0x55f6bd25c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c920 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f5304013010 tx=0x7f53040047c0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 shutdown_connections 2023-10-10T17:50:59.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55f6bd2bde90 0x55f6bd2c02a0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55f6bd2548b0 0x55f6bd25d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55f6bd24fb60 0x55f6bd25c920 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f52f2ffd700 0 client.17226 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:50:59.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 --2- 172.21.15.78:0/649157812 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55f6bd250470 0x55f6bd25ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 >> 172.21.15.78:0/649157812 conn(0x55f6bd245360 msgr2=0x55f6bd24f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:59.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 shutdown_connections 2023-10-10T17:50:59.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 1 -- 172.21.15.78:0/649157812 wait complete. 2023-10-10T17:50:59.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.134+0000 7f5325f19b80 20 client.17226 trim_cache size 0 max 16384 2023-10-10T17:50:59.194 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:50:59.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f52fc0a12c0 msgr2=0x7f52fc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f52fc0a12c0 0x7f52fc0a3770 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5304009c60 tx=0x7f5304009260 comp rx=0 tx=0).stop 2023-10-10T17:50:59.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f531410a8b0 msgr2=0x7f531419ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:50:59.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f531410a8b0 0x7f531419ae20 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f530c00ef90 tx=0x7f530c00c3a0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 shutdown_connections 2023-10-10T17:50:59.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5314101390 0x7f5314196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f52fc0a12c0 0x7f52fc0a3770 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f531410a8b0 0x7f531419ae20 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 --2- 172.21.15.78:0/3452149359 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5314101ca0 0x7f53141969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:50:59.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 >> 172.21.15.78:0/3452149359 conn(0x7f53140770b0 msgr2=0x7f53140fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:50:59.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 shutdown_connections 2023-10-10T17:50:59.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:50:59.135+0000 7f5319d8d700 1 -- 172.21.15.78:0/3452149359 wait complete. 2023-10-10T17:50:59.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:58 smithi157 ceph-mon[100896]: pgmap v260: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:50:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:58 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1758466348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:51:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:51:00.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:51:00.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:51:00.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:50:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:51:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:51:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:51:00.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:51:00.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:50:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:51:00.199 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:51:00.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:51:00.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:51:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:51:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:51:00.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:50:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:51:00.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 -- 172.21.15.78:0/783887075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 msgr2=0x7fe138100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/783887075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138100980 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fe12c0099c0 tx=0x7fe12c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 -- 172.21.15.78:0/783887075 shutdown_connections 2023-10-10T17:51:00.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/783887075 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe1381055f0 0x7fe13810d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/783887075 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138100980 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/783887075 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe138100f50 0x7fe138104f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 -- 172.21.15.78:0/783887075 >> 172.21.15.78:0/783887075 conn(0x7fe138077170 msgr2=0x7fe138077570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 -- 172.21.15.78:0/783887075 shutdown_connections 2023-10-10T17:51:00.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 -- 172.21.15.78:0/783887075 wait complete. 2023-10-10T17:51:00.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.326+0000 7fe13dfc4700 1 Processor -- start 2023-10-10T17:51:00.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe13dfc4700 1 -- start start 2023-10-10T17:51:00.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe13dfc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe13dfc4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe138100f50 0x7fe138196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe13dfc4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1381055f0 0x7fe13819adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe13dfc4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13819b3f0 con 0x7fe1381005b0 2023-10-10T17:51:00.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe13dfc4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13819b560 con 0x7fe138100f50 2023-10-10T17:51:00.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe13dfc4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe13819b720 con 0x7fe1381055f0 2023-10-10T17:51:00.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138196410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48370/0 (socket says 172.21.15.78:48370) 2023-10-10T17:51:00.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 -- 172.21.15.78:0/2773309058 learned_addr learned my addr 172.21.15.78:0/2773309058 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe137fff700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1381055f0 0x7fe13819adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe136ffd700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe138100f50 0x7fe138196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 -- 172.21.15.78:0/2773309058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1381055f0 msgr2=0x7fe13819adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1381055f0 0x7fe13819adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 -- 172.21.15.78:0/2773309058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe138100f50 msgr2=0x7fe138196950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe138100f50 0x7fe138196950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe1377fe700 1 -- 172.21.15.78:0/2773309058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c0095f0 con 0x7fe1381005b0 2023-10-10T17:51:00.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe136ffd700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe138100f50 0x7fe138196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:00.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.327+0000 7fe137fff700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1381055f0 0x7fe13819adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:00.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.328+0000 7fe1377fe700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138196410 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009990 tx=0x7fe12c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.328+0000 7fe134ff9700 1 -- 172.21.15.78:0/2773309058 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c01d070 con 0x7fe1381005b0 2023-10-10T17:51:00.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.328+0000 7fe134ff9700 1 -- 172.21.15.78:0/2773309058 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe12c004010 con 0x7fe1381005b0 2023-10-10T17:51:00.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.328+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1381103b0 con 0x7fe1381005b0 2023-10-10T17:51:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.328+0000 7fe134ff9700 1 -- 172.21.15.78:0/2773309058 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c022d40 con 0x7fe1381005b0 2023-10-10T17:51:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.328+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe138110220 con 0x7fe1381005b0 2023-10-10T17:51:00.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.330+0000 7fe134ff9700 1 -- 172.21.15.78:0/2773309058 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fe12c022470 con 0x7fe1381005b0 2023-10-10T17:51:00.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.330+0000 7fe134ff9700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe1200a1340 0x7fe1200a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.330+0000 7fe136ffd700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe1200a1340 0x7fe1200a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.330+0000 7fe134ff9700 1 -- 172.21.15.78:0/2773309058 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fe12c0ce540 con 0x7fe1381005b0 2023-10-10T17:51:00.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe136ffd700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe1200a1340 0x7fe1200a37f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe128005fd0 tx=0x7fe128005ee0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 Processor -- start 2023-10-10T17:51:00.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 -- start start 2023-10-10T17:51:00.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4047dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 0x561a404823f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a40482d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a403830e0 con 0x561a4047dbe0 2023-10-10T17:51:00.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a403d5920 con 0x561a4047e4f0 2023-10-10T17:51:00.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.331+0000 7fe14a150b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a403cedb0 con 0x561a40482930 2023-10-10T17:51:00.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4047dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4047dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48378/0 (socket says 172.21.15.78:48378) 2023-10-10T17:51:00.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 -- 172.21.15.78:0/4184161102 learned_addr learned my addr 172.21.15.78:0/4184161102 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe137fff700 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a40482d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe136ffd700 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 0x561a404823f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 -- 172.21.15.78:0/4184161102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 msgr2=0x561a40482d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a40482d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 -- 172.21.15.78:0/4184161102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 msgr2=0x561a404823f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 0x561a404823f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 -- 172.21.15.78:0/4184161102 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a402506b0 con 0x561a4047dbe0 2023-10-10T17:51:00.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe1377fe700 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4047dfb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe12c097d20 tx=0x7fe12c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cfe8089bd82441 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe11dffb700 1 -- 172.21.15.78:0/4184161102 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c022ea0 con 0x561a4047dbe0 2023-10-10T17:51:00.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe11dffb700 1 -- 172.21.15.78:0/4184161102 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe12c09b2b0 con 0x561a4047dbe0 2023-10-10T17:51:00.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.332+0000 7fe11dffb700 1 -- 172.21.15.78:0/4184161102 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c09b510 con 0x561a4047dbe0 2023-10-10T17:51:00.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 -- 172.21.15.78:0/4184161102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 msgr2=0x561a4047dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4047dfb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fe12c097d20 tx=0x7fe12c009310 comp rx=0 tx=0).stop 2023-10-10T17:51:00.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 -- 172.21.15.78:0/4184161102 shutdown_connections 2023-10-10T17:51:00.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a40482d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4047dfb0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 --2- 172.21.15.78:0/4184161102 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 0x561a404823f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 -- 172.21.15.78:0/4184161102 >> 172.21.15.78:0/4184161102 conn(0x561a404733e0 msgr2=0x561a40478bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 -- 172.21.15.78:0/4184161102 shutdown_connections 2023-10-10T17:51:00.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 -- 172.21.15.78:0/4184161102 wait complete. 2023-10-10T17:51:00.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 -- start start 2023-10-10T17:51:00.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.333+0000 7fe14a150b80 1 Processor -- start 2023-10-10T17:51:00.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a40489f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4047e4f0 0x561a40487210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a40482930 0x561a40487750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a40370960 con 0x561a4047dbe0 2023-10-10T17:51:00.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a40370600 con 0x561a40482930 2023-10-10T17:51:00.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a402529a0 con 0x561a4047e4f0 2023-10-10T17:51:00.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 10 client.0 _ll_register_callbacks cb 0x561a4038e800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:00.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a40489f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a40489f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48384/0 (socket says 172.21.15.78:48384) 2023-10-10T17:51:00.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 -- 172.21.15.78:0/555243622 learned_addr learned my addr 172.21.15.78:0/555243622 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe136ffd700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4047e4f0 0x561a40487210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe137fff700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a40482930 0x561a40487750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 -- 172.21.15.78:0/555243622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4047e4f0 msgr2=0x561a40487210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4047e4f0 0x561a40487210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 -- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a40482930 msgr2=0x561a40487750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a40482930 0x561a40487750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 -- 172.21.15.78:0/555243622 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe138110220 con 0x561a4047dbe0 2023-10-10T17:51:00.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe137fff700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a40482930 0x561a40487750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:00.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe1377fe700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a40489f90 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005910 tx=0x7fe12c022720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8dd59c4d4eeded3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe116ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:00.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe116ffd700 1 -- 172.21.15.78:0/555243622 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c0b2070 con 0x561a4047dbe0 2023-10-10T17:51:00.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe116ffd700 1 -- 172.21.15.78:0/555243622 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fe12c09f2e0 con 0x561a4047dbe0 2023-10-10T17:51:00.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561a40470ba0 con 0x561a4047dbe0 2023-10-10T17:51:00.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe116ffd700 1 -- 172.21.15.78:0/555243622 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe12c022ea0 con 0x561a4047dbe0 2023-10-10T17:51:00.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 10 client.17238 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:00.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.334+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561a40395210 con 0x561a4047dbe0 2023-10-10T17:51:00.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe116ffd700 1 -- 172.21.15.78:0/555243622 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561a40470ba0 con 0x561a4047dbe0 2023-10-10T17:51:00.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe116ffd700 1 -- 172.21.15.78:0/555243622 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe12c09b660 con 0x561a4047dbe0 2023-10-10T17:51:00.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe14a150b80 10 client.17238 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:00.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe14a150b80 10 client.17238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:00.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe14a150b80 20 client.17238 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:00.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe14a150b80 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404970e0 0x561a404994f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe14a150b80 4 client.17238 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:00.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561a40391120 con 0x561a404970e0 2023-10-10T17:51:00.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe136ffd700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404970e0 0x561a404994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe116ffd700 10 client.17238 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.335+0000 7fe136ffd700 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404970e0 0x561a404994f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32a9f6a31e576b62 server_cookie=4f24b27427ee494f in_seq=0 out_seq=0 2023-10-10T17:51:00.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe116ffd700 1 -- 172.21.15.78:0/555243622 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x561a40391120 con 0x561a404970e0 2023-10-10T17:51:00.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe116ffd700 10 client.17238 handle_command_reply: tid=0 2023-10-10T17:51:00.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 client.17238 shutdown 2023-10-10T17:51:00.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 msgr2=0x561a40489f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a40489f90 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fe12c005910 tx=0x7fe12c022720 comp rx=0 tx=0).stop 2023-10-10T17:51:00.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 shutdown_connections 2023-10-10T17:51:00.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404970e0 0x561a404994f0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a40482930 0x561a40487750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe116ffd700 0 client.17238 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a40489f90 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 --2- 172.21.15.78:0/555243622 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a4047e4f0 0x561a40487210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.336+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 >> 172.21.15.78:0/555243622 conn(0x561a404733e0 msgr2=0x561a40483de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.337+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 shutdown_connections 2023-10-10T17:51:00.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.337+0000 7fe14a150b80 1 -- 172.21.15.78:0/555243622 wait complete. 2023-10-10T17:51:00.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.337+0000 7fe14a150b80 20 client.17238 trim_cache size 0 max 16384 2023-10-10T17:51:00.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 Processor -- start 2023-10-10T17:51:00.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 -- start start 2023-10-10T17:51:00.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4048e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 0x561a40489b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a40478670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a404811c0 con 0x561a4047dbe0 2023-10-10T17:51:00.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a404887f0 con 0x561a4047e4f0 2023-10-10T17:51:00.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe14a150b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a40478460 con 0x561a40482930 2023-10-10T17:51:00.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4048e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4048e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48392/0 (socket says 172.21.15.78:48392) 2023-10-10T17:51:00.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 -- 172.21.15.78:0/3559867884 learned_addr learned my addr 172.21.15.78:0/3559867884 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 -- 172.21.15.78:0/3559867884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 msgr2=0x561a40478670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 --2- 172.21.15.78:0/3559867884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a40478670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 -- 172.21.15.78:0/3559867884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 msgr2=0x561a40489b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 --2- 172.21.15.78:0/3559867884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 0x561a40489b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 -- 172.21.15.78:0/3559867884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561a40484fe0 con 0x561a4047dbe0 2023-10-10T17:51:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe1377fe700 1 --2- 172.21.15.78:0/3559867884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4048e7e0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe12c09b240 tx=0x7fe12c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e62cf7158b76c21 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe11e7fc700 1 -- 172.21.15.78:0/3559867884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c022ea0 con 0x561a4047dbe0 2023-10-10T17:51:00.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe11e7fc700 1 -- 172.21.15.78:0/3559867884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe12c0b8030 con 0x561a4047dbe0 2023-10-10T17:51:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.354+0000 7fe11e7fc700 1 -- 172.21.15.78:0/3559867884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12c09b510 con 0x561a4047dbe0 2023-10-10T17:51:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 -- 172.21.15.78:0/3559867884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 msgr2=0x561a4048e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 --2- 172.21.15.78:0/3559867884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4048e7e0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fe12c09b240 tx=0x7fe12c009310 comp rx=0 tx=0).stop 2023-10-10T17:51:00.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 -- 172.21.15.78:0/3559867884 shutdown_connections 2023-10-10T17:51:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 --2- 172.21.15.78:0/3559867884 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a40478670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 --2- 172.21.15.78:0/3559867884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047dbe0 0x561a4048e7e0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 --2- 172.21.15.78:0/3559867884 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047e4f0 0x561a40489b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 -- 172.21.15.78:0/3559867884 >> 172.21.15.78:0/3559867884 conn(0x561a404733e0 msgr2=0x561a40477d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 -- 172.21.15.78:0/3559867884 shutdown_connections 2023-10-10T17:51:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 -- 172.21.15.78:0/3559867884 wait complete. 2023-10-10T17:51:00.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 -- start start 2023-10-10T17:51:00.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 Processor -- start 2023-10-10T17:51:00.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047dbe0 0x561a404e4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.355+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047e4f0 0x561a404e4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a404e3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a404e3710 con 0x561a4047e4f0 2023-10-10T17:51:00.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a404e3880 con 0x561a4047dbe0 2023-10-10T17:51:00.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x561a404e3a40 con 0x561a40482930 2023-10-10T17:51:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 10 client.0 _ll_register_callbacks cb 0x7fe1300091d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047e4f0 0x561a404e4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047e4f0 0x561a404e4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48400/0 (socket says 172.21.15.78:48400) 2023-10-10T17:51:00.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 -- 172.21.15.78:0/1822884156 learned_addr learned my addr 172.21.15.78:0/1822884156 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe1377fe700 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047dbe0 0x561a404e4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 -- 172.21.15.78:0/1822884156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 msgr2=0x561a404e3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a404e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 -- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047dbe0 msgr2=0x561a404e4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047dbe0 0x561a404e4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 -- 172.21.15.78:0/1822884156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561a40395210 con 0x561a4047e4f0 2023-10-10T17:51:00.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe1377fe700 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047dbe0 0x561a404e4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe136ffd700 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047e4f0 0x561a404e4950 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe138197a20 tx=0x7fe128004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c03c9f949987b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe116ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe116ffd700 1 -- 172.21.15.78:0/1822884156 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe128004220 con 0x561a4047e4f0 2023-10-10T17:51:00.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe116ffd700 1 -- 172.21.15.78:0/1822884156 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fe128025040 con 0x561a4047e4f0 2023-10-10T17:51:00.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe12c09b2b0 con 0x561a4047e4f0 2023-10-10T17:51:00.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe116ffd700 1 -- 172.21.15.78:0/1822884156 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1280285b0 con 0x561a4047e4f0 2023-10-10T17:51:00.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 10 client.17244 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:00.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x561a401fce60 con 0x561a4047e4f0 2023-10-10T17:51:00.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.356+0000 7fe116ffd700 1 -- 172.21.15.78:0/1822884156 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe12c09b2b0 con 0x561a4047e4f0 2023-10-10T17:51:00.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe116ffd700 1 -- 172.21.15.78:0/1822884156 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fe128011470 con 0x561a4047e4f0 2023-10-10T17:51:00.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe14a150b80 10 client.17244 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe14a150b80 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe14a150b80 20 client.17244 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe14a150b80 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404ec570 0x561a404ee980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe14a150b80 4 client.17244 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:51:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561a4039d6e0 con 0x561a404ec570 2023-10-10T17:51:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe1377fe700 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404ec570 0x561a404ee980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe116ffd700 10 client.17244 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.357+0000 7fe1377fe700 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404ec570 0x561a404ee980 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69625843c99ac90 server_cookie=cc29a52ed3942e7f in_seq=0 out_seq=0 2023-10-10T17:51:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.358+0000 7fe116ffd700 1 -- 172.21.15.78:0/1822884156 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561a4039d6e0 con 0x561a404ec570 2023-10-10T17:51:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.358+0000 7fe116ffd700 10 client.17244 handle_command_reply: tid=0 2023-10-10T17:51:00.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.358+0000 7fe14a150b80 1 client.17244 shutdown 2023-10-10T17:51:00.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.358+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047e4f0 msgr2=0x561a404e4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.358+0000 7fe14a150b80 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047e4f0 0x561a404e4950 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fe138197a20 tx=0x7fe128004750 comp rx=0 tx=0).stop 2023-10-10T17:51:00.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 shutdown_connections 2023-10-10T17:51:00.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x561a40482930 0x561a404e3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x561a404ec570 0x561a404ee980 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x561a4047dbe0 0x561a404e4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 --2- 172.21.15.78:0/1822884156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x561a4047e4f0 0x561a404e4950 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe116ffd700 0 client.17244 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 >> 172.21.15.78:0/1822884156 conn(0x561a404733e0 msgr2=0x561a40478670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 shutdown_connections 2023-10-10T17:51:00.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 1 -- 172.21.15.78:0/1822884156 wait complete. 2023-10-10T17:51:00.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe14a150b80 20 client.17244 trim_cache size 0 max 16384 2023-10-10T17:51:00.407 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:51:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:51:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "714a6b05-72a7-41cf-b268-b1e63a5c4df7", 2023-10-10T17:51:00.408 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:51:00.409 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:51:00.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe1200a1340 msgr2=0x7fe1200a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe1200a1340 0x7fe1200a37f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe128005fd0 tx=0x7fe128005ee0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 msgr2=0x7fe138196410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138196410 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe12c009990 tx=0x7fe12c009580 comp rx=0 tx=0).stop 2023-10-10T17:51:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 shutdown_connections 2023-10-10T17:51:00.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe1381055f0 0x7fe13819adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe1200a1340 0x7fe1200a37f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe1381005b0 0x7fe138196410 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 --2- 172.21.15.78:0/2773309058 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe138100f50 0x7fe138196950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 >> 172.21.15.78:0/2773309058 conn(0x7fe138077170 msgr2=0x7fe138103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 shutdown_connections 2023-10-10T17:51:00.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.359+0000 7fe13dfc4700 1 -- 172.21.15.78:0/2773309058 wait complete. 2023-10-10T17:51:00.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:51:00.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 -- 172.21.15.78:0/3362839861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d40ffc30 msgr2=0x7fc6d4100000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 --2- 172.21.15.78:0/3362839861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d40ffc30 0x7fc6d4100000 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80099c0 tx=0x7fc6c8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 -- 172.21.15.78:0/3362839861 shutdown_connections 2023-10-10T17:51:00.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 --2- 172.21.15.78:0/3362839861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6d4104590 0x7fc6d410c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 --2- 172.21.15.78:0/3362839861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d40ffc30 0x7fc6d4100000 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 --2- 172.21.15.78:0/3362839861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d41005d0 0x7fc6d4103e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 -- 172.21.15.78:0/3362839861 >> 172.21.15.78:0/3362839861 conn(0x7fc6d4075470 msgr2=0x7fc6d4075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 -- 172.21.15.78:0/3362839861 shutdown_connections 2023-10-10T17:51:00.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.535+0000 7fc6d8811700 1 -- 172.21.15.78:0/3362839861 wait complete. 2023-10-10T17:51:00.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 Processor -- start 2023-10-10T17:51:00.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 -- start start 2023-10-10T17:51:00.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6d40ffc30 0x7fc6d4195360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d41005d0 0x7fc6d41958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d4104590 0x7fc6d4199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d419a340 con 0x7fc6d41005d0 2023-10-10T17:51:00.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d419a4b0 con 0x7fc6d40ffc30 2023-10-10T17:51:00.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d8811700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d419a670 con 0x7fc6d4104590 2023-10-10T17:51:00.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d41005d0 0x7fc6d41958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d41005d0 0x7fc6d41958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48414/0 (socket says 172.21.15.78:48414) 2023-10-10T17:51:00.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/750646924 learned_addr learned my addr 172.21.15.78:0/750646924 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d259c700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6d40ffc30 0x7fc6d4195360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d2d9d700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d4104590 0x7fc6d4199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/750646924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d4104590 msgr2=0x7fc6d4199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d4104590 0x7fc6d4199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/750646924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6d40ffc30 msgr2=0x7fc6d4195360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6d40ffc30 0x7fc6d4195360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/750646924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c80095f0 con 0x7fc6d41005d0 2023-10-10T17:51:00.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.536+0000 7fc6d259c700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6d40ffc30 0x7fc6d4195360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:00.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.537+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d41005d0 0x7fc6d41958a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400d950 tx=0x7fc6c400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.537+0000 7fc6c37fe700 1 -- 172.21.15.78:0/750646924 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4004220 con 0x7fc6d41005d0 2023-10-10T17:51:00.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.537+0000 7fc6c37fe700 1 -- 172.21.15.78:0/750646924 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6c40075c0 con 0x7fc6d41005d0 2023-10-10T17:51:00.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.537+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d410f300 con 0x7fc6d41005d0 2023-10-10T17:51:00.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.537+0000 7fc6c37fe700 1 -- 172.21.15.78:0/750646924 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4005590 con 0x7fc6d41005d0 2023-10-10T17:51:00.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.537+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d410f170 con 0x7fc6d41005d0 2023-10-10T17:51:00.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.539+0000 7fc6c37fe700 1 -- 172.21.15.78:0/750646924 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc6c4004750 con 0x7fc6d41005d0 2023-10-10T17:51:00.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.539+0000 7fc6c37fe700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc6bc0a12c0 0x7fc6bc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.539+0000 7fc6d259c700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc6bc0a12c0 0x7fc6bc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.539+0000 7fc6c37fe700 1 -- 172.21.15.78:0/750646924 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fc6c40ccf20 con 0x7fc6d41005d0 2023-10-10T17:51:00.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.540+0000 7fc6d259c700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc6bc0a12c0 0x7fc6bc0a3770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009fd0 tx=0x7fc6c801a040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.540+0000 7fc6e499db80 1 Processor -- start 2023-10-10T17:51:00.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.540+0000 7fc6e499db80 1 -- start start 2023-10-10T17:51:00.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9149f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb914e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb914ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6e499db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb904f060 con 0x555fb914a470 2023-10-10T17:51:00.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6e499db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb90a18a0 con 0x555fb914e8b0 2023-10-10T17:51:00.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6e499db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb909ad30 con 0x555fb9149b60 2023-10-10T17:51:00.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb914e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d259c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb914e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48422/0 (socket says 172.21.15.78:48422) 2023-10-10T17:51:00.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2100534676 learned_addr learned my addr 172.21.15.78:0/2100534676 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d2d9d700 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb914ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2100534676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 msgr2=0x555fb9149f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9149f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2100534676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 msgr2=0x555fb914ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb914ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2100534676 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fb8f1c630 con 0x555fb914a470 2023-10-10T17:51:00.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d259c700 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:51:00.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb914e370 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40a1b80 tx=0x7fc6c400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f39a41805c80ca server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.541+0000 7fc6c0ff9700 1 -- 172.21.15.78:0/2100534676 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40a20b0 con 0x555fb914a470 2023-10-10T17:51:00.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6c0ff9700 1 -- 172.21.15.78:0/2100534676 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6c40056f0 con 0x555fb914a470 2023-10-10T17:51:00.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6c0ff9700 1 -- 172.21.15.78:0/2100534676 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4099ec0 con 0x555fb914a470 2023-10-10T17:51:00.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 -- 172.21.15.78:0/2100534676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 msgr2=0x555fb914e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.583 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb914e370 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40a1b80 tx=0x7fc6c400f490 comp rx=0 tx=0).stop 2023-10-10T17:51:00.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 -- 172.21.15.78:0/2100534676 shutdown_connections 2023-10-10T17:51:00.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9149f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb914ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2100534676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb914e370 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 -- 172.21.15.78:0/2100534676 >> 172.21.15.78:0/2100534676 conn(0x555fb913f360 msgr2=0x555fb9144b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 -- 172.21.15.78:0/2100534676 shutdown_connections 2023-10-10T17:51:00.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.542+0000 7fc6e499db80 1 -- 172.21.15.78:0/2100534676 wait complete. 2023-10-10T17:51:00.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 -- start start 2023-10-10T17:51:00.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 Processor -- start 2023-10-10T17:51:00.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb9149b60 0x555fb91533f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914a470 0x555fb9153930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb9153e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb903c8e0 con 0x555fb9149b60 2023-10-10T17:51:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb903c580 con 0x555fb914e8b0 2023-10-10T17:51:00.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb8f1e920 con 0x555fb914a470 2023-10-10T17:51:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6e499db80 10 client.0 _ll_register_callbacks cb 0x555fb905a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb9149b60 0x555fb91533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb9149b60 0x555fb91533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48426/0 (socket says 172.21.15.78:48426) 2023-10-10T17:51:00.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 -- 172.21.15.78:0/2377667687 learned_addr learned my addr 172.21.15.78:0/2377667687 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914a470 0x555fb9153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d2d9d700 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb9153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 -- 172.21.15.78:0/2377667687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914a470 msgr2=0x555fb9153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914a470 0x555fb9153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 -- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 msgr2=0x555fb9153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb9153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 -- 172.21.15.78:0/2377667687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d410f170 con 0x555fb9149b60 2023-10-10T17:51:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6d259c700 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb9149b60 0x555fb91533f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80099c0 tx=0x7fc6c8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d470fad93925b0c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.543+0000 7fc6b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/2377667687 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c801d040 con 0x555fb9149b60 2023-10-10T17:51:00.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/2377667687 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fc6c8023960 con 0x555fb9149b60 2023-10-10T17:51:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555fb913cb20 con 0x555fb9149b60 2023-10-10T17:51:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/2377667687 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6c8023b60 con 0x555fb9149b60 2023-10-10T17:51:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 10 client.17256 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:00.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x555fb9061190 con 0x555fb9149b60 2023-10-10T17:51:00.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/2377667687 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555fb913cb20 con 0x555fb9149b60 2023-10-10T17:51:00.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/2377667687 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc6c80173f0 con 0x555fb9149b60 2023-10-10T17:51:00.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 10 client.17256 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:00.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:00.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 20 client.17256 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:00.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb9163a40 0x555fb9165e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 4 client.17256 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:00.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555fb905d0a0 con 0x555fb9163a40 2023-10-10T17:51:00.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.544+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb9163a40 0x555fb9165e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6b9ffb700 10 client.17256 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb9163a40 0x555fb9165e50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3598b1b2a13cbf6 server_cookie=7cbc2c34d905ae83 in_seq=0 out_seq=0 2023-10-10T17:51:00.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/2377667687 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555fb905d0a0 con 0x555fb9163a40 2023-10-10T17:51:00.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6b9ffb700 10 client.17256 handle_command_reply: tid=0 2023-10-10T17:51:00.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6e499db80 1 client.17256 shutdown 2023-10-10T17:51:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb9149b60 msgr2=0x555fb91533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb9149b60 0x555fb91533f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fc6c80099c0 tx=0x7fc6c8009310 comp rx=0 tx=0).stop 2023-10-10T17:51:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 shutdown_connections 2023-10-10T17:51:00.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.545+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb9163a40 0x555fb9165e50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914e8b0 0x555fb9153e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb9149b60 0x555fb91533f0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6b9ffb700 0 client.17256 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2377667687 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914a470 0x555fb9153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 >> 172.21.15.78:0/2377667687 conn(0x555fb913f360 msgr2=0x555fb9143de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 shutdown_connections 2023-10-10T17:51:00.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6e499db80 1 -- 172.21.15.78:0/2377667687 wait complete. 2023-10-10T17:51:00.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.546+0000 7fc6e499db80 20 client.17256 trim_cache size 0 max 16384 2023-10-10T17:51:00.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 Processor -- start 2023-10-10T17:51:00.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- start start 2023-10-10T17:51:00.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb9149b60 0x555fb915a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb9153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914e8b0 0x555fb9150a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb914d520 con 0x555fb914a470 2023-10-10T17:51:00.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb9156410 con 0x555fb9149b60 2023-10-10T17:51:00.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb915aee0 con 0x555fb914e8b0 2023-10-10T17:51:00.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb9153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb9153cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48436/0 (socket says 172.21.15.78:48436) 2023-10-10T17:51:00.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2570846679 learned_addr learned my addr 172.21.15.78:0/2570846679 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d2d9d700 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914e8b0 0x555fb9150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d259c700 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb9149b60 0x555fb915a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2570846679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914e8b0 msgr2=0x555fb9150a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914e8b0 0x555fb9150a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2570846679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb9149b60 msgr2=0x555fb915a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb9149b60 0x555fb915a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 -- 172.21.15.78:0/2570846679 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555fb914f1a0 con 0x555fb914a470 2023-10-10T17:51:00.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d259c700 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb9149b60 0x555fb915a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:00.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb9153cf0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40ca390 tx=0x7fc6c4004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=489a64697e045411 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6c17fa700 1 -- 172.21.15.78:0/2570846679 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40a2130 con 0x555fb914a470 2023-10-10T17:51:00.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6c17fa700 1 -- 172.21.15.78:0/2570846679 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6c4004380 con 0x555fb914a470 2023-10-10T17:51:00.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6c17fa700 1 -- 172.21.15.78:0/2570846679 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40993b0 con 0x555fb914a470 2023-10-10T17:51:00.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- 172.21.15.78:0/2570846679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 msgr2=0x555fb9153cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb9153cf0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40ca390 tx=0x7fc6c4004f10 comp rx=0 tx=0).stop 2023-10-10T17:51:00.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- 172.21.15.78:0/2570846679 shutdown_connections 2023-10-10T17:51:00.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb914e8b0 0x555fb9150a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb9149b60 0x555fb915a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 --2- 172.21.15.78:0/2570846679 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914a470 0x555fb9153cf0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- 172.21.15.78:0/2570846679 >> 172.21.15.78:0/2570846679 conn(0x555fb913f360 msgr2=0x555fb914c990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- 172.21.15.78:0/2570846679 shutdown_connections 2023-10-10T17:51:00.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.564+0000 7fc6e499db80 1 -- 172.21.15.78:0/2570846679 wait complete. 2023-10-10T17:51:00.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 -- start start 2023-10-10T17:51:00.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 Processor -- start 2023-10-10T17:51:00.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9143550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914a470 0x555fb9143a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914e8b0 0x555fb9143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb915ad50 con 0x555fb914e8b0 2023-10-10T17:51:00.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb9144510 con 0x555fb914a470 2023-10-10T17:51:00.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555fb91446d0 con 0x555fb9149b60 2023-10-10T17:51:00.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6e499db80 10 client.0 _ll_register_callbacks cb 0x555fb905a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:00.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914e8b0 0x555fb9143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914e8b0 0x555fb9143fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48444/0 (socket says 172.21.15.78:48444) 2023-10-10T17:51:00.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 -- 172.21.15.78:0/1684650180 learned_addr learned my addr 172.21.15.78:0/1684650180 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:00.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d259c700 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9143550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d1d9b700 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914a470 0x555fb9143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 -- 172.21.15.78:0/1684650180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 msgr2=0x555fb9143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9143550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 -- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914a470 msgr2=0x555fb9143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:00.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914a470 0x555fb9143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.565+0000 7fc6d2d9d700 1 -- 172.21.15.78:0/1684650180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555fb8f1c630 con 0x555fb914e8b0 2023-10-10T17:51:00.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6d2d9d700 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914e8b0 0x555fb9143fd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007f00 tx=0x7fc6cc009390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54f3f34b643cf9d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:00.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6b9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:00.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/1684650180 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00e040 con 0x555fb914e8b0 2023-10-10T17:51:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/1684650180 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fc6cc004db0 con 0x555fb914e8b0 2023-10-10T17:51:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6c40056f0 con 0x555fb914e8b0 2023-10-10T17:51:00.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/1684650180 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6cc0168d0 con 0x555fb914e8b0 2023-10-10T17:51:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 10 client.17262 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x555fb8ec8de0 con 0x555fb914e8b0 2023-10-10T17:51:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/1684650180 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc6c40056f0 con 0x555fb914e8b0 2023-10-10T17:51:00.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/1684650180 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7fc6cc020c00 con 0x555fb914e8b0 2023-10-10T17:51:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 10 client.17262 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 20 client.17262 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:00.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb91b84e0 0x555fb91ba8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 4 client.17262 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:51:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555fb9069660 con 0x555fb91b84e0 2023-10-10T17:51:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.566+0000 7fc6d259c700 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb91b84e0 0x555fb91ba8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:00.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6b9ffb700 10 client.17262 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6d259c700 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb91b84e0 0x555fb91ba8f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfd40c3f95936e86 server_cookie=732363a9e43c8b7d in_seq=0 out_seq=0 2023-10-10T17:51:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6b9ffb700 1 -- 172.21.15.78:0/1684650180 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555fb9069660 con 0x555fb91b84e0 2023-10-10T17:51:00.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6b9ffb700 10 client.17262 handle_command_reply: tid=0 2023-10-10T17:51:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 client.17262 shutdown 2023-10-10T17:51:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914e8b0 msgr2=0x555fb9143fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914e8b0 0x555fb9143fd0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc007f00 tx=0x7fc6cc009390 comp rx=0 tx=0).stop 2023-10-10T17:51:00.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 shutdown_connections 2023-10-10T17:51:00.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555fb9149b60 0x555fb9143550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x555fb91b84e0 0x555fb91ba8f0 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555fb914e8b0 0x555fb9143fd0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 --2- 172.21.15.78:0/1684650180 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555fb914a470 0x555fb9143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 >> 172.21.15.78:0/1684650180 conn(0x555fb913f360 msgr2=0x555fb914d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6b9ffb700 0 client.17262 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:00.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 shutdown_connections 2023-10-10T17:51:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 1 -- 172.21.15.78:0/1684650180 wait complete. 2023-10-10T17:51:00.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.567+0000 7fc6e499db80 20 client.17262 trim_cache size 0 max 16384 2023-10-10T17:51:00.623 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:51:00.624 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:51:00.624 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:51:00.624 INFO:teuthology.orchestra.run.smithi078.stdout: "714a6b05-72a7-41cf-b268-b1e63a5c4df7": { 2023-10-10T17:51:00.625 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:51:00.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "714a6b05-72a7-41cf-b268-b1e63a5c4df7", 2023-10-10T17:51:00.625 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:51:00.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:51:00.626 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:51:00.626 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:51:00.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc6bc0a12c0 msgr2=0x7fc6bc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc6bc0a12c0 0x7fc6bc0a3770 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009fd0 tx=0x7fc6c801a040 comp rx=0 tx=0).stop 2023-10-10T17:51:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d41005d0 msgr2=0x7fc6d41958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:00.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d41005d0 0x7fc6d41958a0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400d950 tx=0x7fc6c400dc60 comp rx=0 tx=0).stop 2023-10-10T17:51:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 shutdown_connections 2023-10-10T17:51:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc6d4104590 0x7fc6d4199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc6bc0a12c0 0x7fc6bc0a3770 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc6d40ffc30 0x7fc6d4195360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 --2- 172.21.15.78:0/750646924 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc6d41005d0 0x7fc6d41958a0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 >> 172.21.15.78:0/750646924 conn(0x7fc6d4075470 msgr2=0x7fc6d4102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 shutdown_connections 2023-10-10T17:51:00.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:00.568+0000 7fc6d8811700 1 -- 172.21.15.78:0/750646924 wait complete. 2023-10-10T17:51:00.631 INFO:tasks.cephfs.filesystem:scrub status for tag:714a6b05-72a7-41cf-b268-b1e63a5c4df7 - {'path': '/', 'tag': '714a6b05-72a7-41cf-b268-b1e63a5c4df7', 'options': 'recursive,force'} 2023-10-10T17:51:00.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:00 smithi078 ceph-mon[99494]: pgmap v261: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:00 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:51:00.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:00 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:51:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:00 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:51:00.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:00 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:51:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:00 smithi190 ceph-mon[101220]: pgmap v261: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:00 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:51:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:00 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:51:01.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:00 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:51:01.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:00 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:51:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:00 smithi157 ceph-mon[100896]: pgmap v261: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:00 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:51:01.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:00 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:51:01.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:00 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:51:01.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:00 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:51:02.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:01 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:51:02.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:01 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:02.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:01 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:51:02.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:01 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:02.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:01 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:51:02.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:01 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:02.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:01 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:51:02.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:01 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:01 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:51:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:01 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:01 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:51:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:01 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:02 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:51:03.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:02 smithi078 ceph-mon[99494]: pgmap v262: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:02 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:51:03.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:02 smithi190 ceph-mon[101220]: pgmap v262: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:03.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:02 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:51:03.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:02 smithi157 ceph-mon[100896]: pgmap v262: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:05.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:04 smithi078 ceph-mon[99494]: pgmap v263: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:51:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:04 smithi190 ceph-mon[101220]: pgmap v263: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:51:05.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:04 smithi157 ceph-mon[100896]: pgmap v263: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:51:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:06 smithi078 ceph-mon[99494]: pgmap v264: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:51:07.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:06 smithi190 ceph-mon[101220]: pgmap v264: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:51:07.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:06 smithi157 ceph-mon[100896]: pgmap v264: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:51:08.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:07 smithi078 ceph-mon[99494]: osdmap e99: 12 total, 12 up, 12 in 2023-10-10T17:51:08.169 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:07 smithi190 ceph-mon[101220]: osdmap e99: 12 total, 12 up, 12 in 2023-10-10T17:51:08.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:07 smithi157 ceph-mon[100896]: osdmap e99: 12 total, 12 up, 12 in 2023-10-10T17:51:09.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:08 smithi078 ceph-mon[99494]: pgmap v266: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:51:09.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:08 smithi190 ceph-mon[101220]: pgmap v266: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:51:09.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:08 smithi157 ceph-mon[100896]: pgmap v266: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:51:10.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:09 smithi078 ceph-mon[99494]: pgmap v267: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:51:10.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:09 smithi190 ceph-mon[101220]: pgmap v267: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:51:10.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:09 smithi157 ceph-mon[100896]: pgmap v267: 129 pgs: 129 active+clean; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:51:12.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:12 smithi157 ceph-mon[100896]: pgmap v268: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:51:12.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:12 smithi078 ceph-mon[99494]: pgmap v268: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:51:12.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:12 smithi190 ceph-mon[101220]: pgmap v268: 129 pgs: 3 active+clean+snaptrim, 123 active+clean, 3 active+clean+snaptrim_wait; 799 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-10-10T17:51:15.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:14 smithi078 ceph-mon[99494]: pgmap v269: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 799 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:51:15.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:14 smithi190 ceph-mon[101220]: pgmap v269: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 799 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:51:15.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:14 smithi157 ceph-mon[100896]: pgmap v269: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 799 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:51:17.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:16 smithi078 ceph-mon[99494]: pgmap v270: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 799 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:51:17.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:16 smithi190 ceph-mon[101220]: pgmap v270: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 799 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:51:17.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:16 smithi157 ceph-mon[100896]: pgmap v270: 129 pgs: 23 active+clean+snaptrim, 36 active+clean, 70 active+clean+snaptrim_wait; 799 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:51:19.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:18 smithi078 ceph-mon[99494]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 47 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 96 B/s wr, 5 op/s 2023-10-10T17:51:19.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:18 smithi190 ceph-mon[101220]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 47 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 96 B/s wr, 5 op/s 2023-10-10T17:51:19.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:18 smithi157 ceph-mon[100896]: pgmap v271: 129 pgs: 20 active+clean+snaptrim, 47 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 96 B/s wr, 5 op/s 2023-10-10T17:51:21.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:20 smithi078 ceph-mon[99494]: pgmap v272: 129 pgs: 20 active+clean+snaptrim, 47 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-10T17:51:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:20 smithi190 ceph-mon[101220]: pgmap v272: 129 pgs: 20 active+clean+snaptrim, 47 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-10T17:51:21.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:20 smithi157 ceph-mon[100896]: pgmap v272: 129 pgs: 20 active+clean+snaptrim, 47 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-10T17:51:23.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:22 smithi078 ceph-mon[99494]: pgmap v273: 129 pgs: 19 active+clean+snaptrim, 48 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:23.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:22 smithi190 ceph-mon[101220]: pgmap v273: 129 pgs: 19 active+clean+snaptrim, 48 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:23.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:22 smithi157 ceph-mon[100896]: pgmap v273: 129 pgs: 19 active+clean+snaptrim, 48 active+clean, 62 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:25.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:24 smithi078 ceph-mon[99494]: pgmap v274: 129 pgs: 14 active+clean+snaptrim, 73 active+clean, 42 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:25.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:24 smithi190 ceph-mon[101220]: pgmap v274: 129 pgs: 14 active+clean+snaptrim, 73 active+clean, 42 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:25.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:24 smithi157 ceph-mon[100896]: pgmap v274: 129 pgs: 14 active+clean+snaptrim, 73 active+clean, 42 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:51:27.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:26 smithi078 ceph-mon[99494]: pgmap v275: 129 pgs: 14 active+clean+snaptrim, 73 active+clean, 42 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-10T17:51:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:26 smithi190 ceph-mon[101220]: pgmap v275: 129 pgs: 14 active+clean+snaptrim, 73 active+clean, 42 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-10T17:51:27.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:26 smithi157 ceph-mon[100896]: pgmap v275: 129 pgs: 14 active+clean+snaptrim, 73 active+clean, 42 active+clean+snaptrim_wait; 799 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-10T17:51:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:28 smithi078 ceph-mon[99494]: pgmap v276: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2023-10-10T17:51:29.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:28 smithi078 ceph-mon[99494]: osdmap e100: 12 total, 12 up, 12 in 2023-10-10T17:51:29.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:28 smithi190 ceph-mon[101220]: pgmap v276: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2023-10-10T17:51:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:28 smithi190 ceph-mon[101220]: osdmap e100: 12 total, 12 up, 12 in 2023-10-10T17:51:29.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:28 smithi157 ceph-mon[100896]: pgmap v276: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2023-10-10T17:51:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:28 smithi157 ceph-mon[100896]: osdmap e100: 12 total, 12 up, 12 in 2023-10-10T17:51:30.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:51:30.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 -- 172.21.15.78:0/2430639482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 msgr2=0x7f70d8103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 --2- 172.21.15.78:0/2430639482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8103db0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f70c80099c0 tx=0x7f70c8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 -- 172.21.15.78:0/2430639482 shutdown_connections 2023-10-10T17:51:30.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 --2- 172.21.15.78:0/2430639482 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8104490 0x7f70d810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 --2- 172.21.15.78:0/2430639482 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70d80ff450 0x7f70d80ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 --2- 172.21.15.78:0/2430639482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8103db0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 -- 172.21.15.78:0/2430639482 >> 172.21.15.78:0/2430639482 conn(0x7f70d8077080 msgr2=0x7f70d8077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 -- 172.21.15.78:0/2430639482 shutdown_connections 2023-10-10T17:51:30.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.777+0000 7f70dfa20700 1 -- 172.21.15.78:0/2430639482 wait complete. 2023-10-10T17:51:30.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 Processor -- start 2023-10-10T17:51:30.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 -- start start 2023-10-10T17:51:30.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70d80ff450 0x7f70d8195250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8195790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8104490 0x7f70d8199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d819a230 con 0x7f70d80ffdf0 2023-10-10T17:51:30.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d819a3a0 con 0x7f70d80ff450 2023-10-10T17:51:30.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dfa20700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d819a560 con 0x7f70d8104490 2023-10-10T17:51:30.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8195790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8195790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48456/0 (socket says 172.21.15.78:48456) 2023-10-10T17:51:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 -- 172.21.15.78:0/3585788104 learned_addr learned my addr 172.21.15.78:0/3585788104 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:30.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70d80ff450 0x7f70d8195250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8104490 0x7f70d8199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 -- 172.21.15.78:0/3585788104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8104490 msgr2=0x7f70d8199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8104490 0x7f70d8199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 -- 172.21.15.78:0/3585788104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70d80ff450 msgr2=0x7f70d8195250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70d80ff450 0x7f70d8195250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70dcfbb700 1 -- 172.21.15.78:0/3585788104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c80095f0 con 0x7f70d80ffdf0 2023-10-10T17:51:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.778+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8104490 0x7f70d8199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:30.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.779+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8195790 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009990 tx=0x7f70c8004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:30.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.779+0000 7f70ce7fc700 1 -- 172.21.15.78:0/3585788104 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c801d070 con 0x7f70d80ffdf0 2023-10-10T17:51:30.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.779+0000 7f70ce7fc700 1 -- 172.21.15.78:0/3585788104 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70c8004110 con 0x7f70d80ffdf0 2023-10-10T17:51:30.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.779+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d810f210 con 0x7f70d80ffdf0 2023-10-10T17:51:30.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.779+0000 7f70ce7fc700 1 -- 172.21.15.78:0/3585788104 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8022dd0 con 0x7f70d80ffdf0 2023-10-10T17:51:30.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.779+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d810f080 con 0x7f70d80ffdf0 2023-10-10T17:51:30.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.781+0000 7f70ce7fc700 1 -- 172.21.15.78:0/3585788104 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f70c8022500 con 0x7f70d80ffdf0 2023-10-10T17:51:30.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.781+0000 7f70ce7fc700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f70c40a1340 0x7f70c40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.782+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f70c40a1340 0x7f70c40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.782+0000 7f70ce7fc700 1 -- 172.21.15.78:0/3585788104 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f70c80ce310 con 0x7f70d80ffdf0 2023-10-10T17:51:30.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.782+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f70c40a1340 0x7f70c40a37f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f70d400ac50 tx=0x7f70d400a380 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:30.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 Processor -- start 2023-10-10T17:51:30.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 -- start start 2023-10-10T17:51:30.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaacd70 0x56334aaad140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaad680 0x56334aab1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334a9b0060 con 0x56334aab1ac0 2023-10-10T17:51:30.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334aa028a0 con 0x56334aaad680 2023-10-10T17:51:30.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334a9fbd30 con 0x56334aaacd70 2023-10-10T17:51:30.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab1f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48466/0 (socket says 172.21.15.78:48466) 2023-10-10T17:51:30.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 -- 172.21.15.78:0/4073275412 learned_addr learned my addr 172.21.15.78:0/4073275412 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaacd70 0x56334aaad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 -- 172.21.15.78:0/4073275412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaacd70 msgr2=0x56334aaad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaacd70 0x56334aaad140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 -- 172.21.15.78:0/4073275412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaad680 msgr2=0x56334aab1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaad680 0x56334aab1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaad680 0x56334aab1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70ddfbd700 1 -- 172.21.15.78:0/4073275412 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56334a87d630 con 0x56334aab1ac0 2023-10-10T17:51:30.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.783+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaad680 0x56334aab1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:30.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab1f10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e160 tx=0x7f70d000e470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c40e5c5d056da37a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:30.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70c37fe700 1 -- 172.21.15.78:0/4073275412 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0018070 con 0x56334aab1ac0 2023-10-10T17:51:30.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70c37fe700 1 -- 172.21.15.78:0/4073275412 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70d0007c70 con 0x56334aab1ac0 2023-10-10T17:51:30.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70c37fe700 1 -- 172.21.15.78:0/4073275412 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d00041a0 con 0x56334aab1ac0 2023-10-10T17:51:30.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ebbacb80 1 -- 172.21.15.78:0/4073275412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 msgr2=0x56334aab1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab1f10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f70d000e160 tx=0x7f70d000e470 comp rx=0 tx=0).stop 2023-10-10T17:51:30.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ebbacb80 1 -- 172.21.15.78:0/4073275412 shutdown_connections 2023-10-10T17:51:30.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaacd70 0x56334aaad140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab1f10 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/4073275412 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaad680 0x56334aab1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.784+0000 7f70ebbacb80 1 -- 172.21.15.78:0/4073275412 >> 172.21.15.78:0/4073275412 conn(0x56334aaa0360 msgr2=0x56334aaa5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:30.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 -- 172.21.15.78:0/4073275412 shutdown_connections 2023-10-10T17:51:30.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 -- 172.21.15.78:0/4073275412 wait complete. 2023-10-10T17:51:30.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 -- start start 2023-10-10T17:51:30.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 Processor -- start 2023-10-10T17:51:30.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aab9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aaad680 0x56334aab64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aab1ac0 0x56334aab6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334a99d8e0 con 0x56334aaad680 2023-10-10T17:51:30.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334a99d580 con 0x56334aaacd70 2023-10-10T17:51:30.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.785+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334a87f920 con 0x56334aab1ac0 2023-10-10T17:51:30.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70ebbacb80 10 client.0 _ll_register_callbacks cb 0x56334a9bb780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:30.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aaad680 0x56334aab64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aaad680 0x56334aab64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48470/0 (socket says 172.21.15.78:48470) 2023-10-10T17:51:30.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 -- 172.21.15.78:0/2208738768 learned_addr learned my addr 172.21.15.78:0/2208738768 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:30.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aab1ac0 0x56334aab6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aab9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 -- 172.21.15.78:0/2208738768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aab1ac0 msgr2=0x56334aab6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aab1ac0 0x56334aab6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 -- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 msgr2=0x56334aab9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aab9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 -- 172.21.15.78:0/2208738768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d000f040 con 0x56334aaad680 2023-10-10T17:51:30.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aaad680 0x56334aab64f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f70c80cb7c0 tx=0x7f70c8009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a06b1e65af876b94 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:30.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:30.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70c0ff9700 1 -- 172.21.15.78:0/2208738768 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c809f1a0 con 0x56334aaad680 2023-10-10T17:51:30.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70c0ff9700 1 -- 172.21.15.78:0/2208738768 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f70c80a3530 con 0x56334aaad680 2023-10-10T17:51:30.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56334a9be0a0 con 0x56334aaad680 2023-10-10T17:51:30.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70c0ff9700 1 -- 172.21.15.78:0/2208738768 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70c80ce020 con 0x56334aaad680 2023-10-10T17:51:30.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70ebbacb80 10 client.17274 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.786+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56334a9c2190 con 0x56334aaad680 2023-10-10T17:51:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70c0ff9700 1 -- 172.21.15.78:0/2208738768 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56334a9be0a0 con 0x56334aaad680 2023-10-10T17:51:30.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70c0ff9700 1 -- 172.21.15.78:0/2208738768 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f70c8096f60 con 0x56334aaad680 2023-10-10T17:51:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70ebbacb80 10 client.17274 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70ebbacb80 10 client.17274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:30.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70ebbacb80 20 client.17274 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334aac65e0 0x56334aac89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70ebbacb80 4 client.17274 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:30.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56334a9ca660 con 0x56334aac65e0 2023-10-10T17:51:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334aac65e0 0x56334aac89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70c0ff9700 10 client.17274 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.787+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334aac65e0 0x56334aac89f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f713faa5250cf3eb server_cookie=6746a60a816510b7 in_seq=0 out_seq=0 2023-10-10T17:51:30.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70c0ff9700 1 -- 172.21.15.78:0/2208738768 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x56334a9ca660 con 0x56334aac65e0 2023-10-10T17:51:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70c0ff9700 10 client.17274 handle_command_reply: tid=0 2023-10-10T17:51:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 client.17274 shutdown 2023-10-10T17:51:30.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aaad680 msgr2=0x56334aab64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aaad680 0x56334aab64f0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f70c80cb7c0 tx=0x7f70c8009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 shutdown_connections 2023-10-10T17:51:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aab1ac0 0x56334aab6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334aac65e0 0x56334aac89f0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aab9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/2208738768 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aaad680 0x56334aab64f0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70c0ff9700 0 client.17274 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 >> 172.21.15.78:0/2208738768 conn(0x56334aaa0360 msgr2=0x56334aab2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 shutdown_connections 2023-10-10T17:51:30.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.788+0000 7f70ebbacb80 1 -- 172.21.15.78:0/2208738768 wait complete. 2023-10-10T17:51:30.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.789+0000 7f70ebbacb80 20 client.17274 trim_cache size 0 max 16384 2023-10-10T17:51:30.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.805+0000 7f70ebbacb80 1 Processor -- start 2023-10-10T17:51:30.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.805+0000 7f70ebbacb80 1 -- start start 2023-10-10T17:51:30.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.805+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aaa5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334aaaddd0 con 0x56334aab1ac0 2023-10-10T17:51:30.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334aab7ad0 con 0x56334aaacd70 2023-10-10T17:51:30.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334aab4a50 con 0x56334aaad680 2023-10-10T17:51:30.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48480/0 (socket says 172.21.15.78:48480) 2023-10-10T17:51:30.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 -- 172.21.15.78:0/322258898 learned_addr learned my addr 172.21.15.78:0/322258898 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:30.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aaa5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 -- 172.21.15.78:0/322258898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 msgr2=0x56334aabdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 -- 172.21.15.78:0/322258898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 msgr2=0x56334aaa5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aaa5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 -- 172.21.15.78:0/322258898 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56334a9c1ee0 con 0x56334aab1ac0 2023-10-10T17:51:30.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:30.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab5cb0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f70d000ed00 tx=0x7f70d000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ec4d5b77e69340 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:30.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70c3fff700 1 -- 172.21.15.78:0/322258898 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0018040 con 0x56334aab1ac0 2023-10-10T17:51:30.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70c3fff700 1 -- 172.21.15.78:0/322258898 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f70d000eed0 con 0x56334aab1ac0 2023-10-10T17:51:30.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.806+0000 7f70c3fff700 1 -- 172.21.15.78:0/322258898 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d001f450 con 0x56334aab1ac0 2023-10-10T17:51:30.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 -- 172.21.15.78:0/322258898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 msgr2=0x56334aab5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab5cb0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f70d000ed00 tx=0x7f70d000ed30 comp rx=0 tx=0).stop 2023-10-10T17:51:30.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 -- 172.21.15.78:0/322258898 shutdown_connections 2023-10-10T17:51:30.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aab5cb0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aaa5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/322258898 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabdbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 -- 172.21.15.78:0/322258898 >> 172.21.15.78:0/322258898 conn(0x56334aaa0360 msgr2=0x56334aac2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:30.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 -- 172.21.15.78:0/322258898 shutdown_connections 2023-10-10T17:51:30.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 -- 172.21.15.78:0/322258898 wait complete. 2023-10-10T17:51:30.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 -- start start 2023-10-10T17:51:30.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.807+0000 7f70ebbacb80 1 Processor -- start 2023-10-10T17:51:30.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aabdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ebbacb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aabc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334aabcd80 con 0x56334aab1ac0 2023-10-10T17:51:30.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334ab1a4b0 con 0x56334aaacd70 2023-10-10T17:51:30.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ebbacb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x56334ab1a5f0 con 0x56334aaad680 2023-10-10T17:51:30.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ebbacb80 10 client.0 _ll_register_callbacks cb 0x7f70c809acb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:30.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aabc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aabc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48486/0 (socket says 172.21.15.78:48486) 2023-10-10T17:51:30.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 -- 172.21.15.78:0/1975415271 learned_addr learned my addr 172.21.15.78:0/1975415271 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:30.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70dcfbb700 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aabdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 -- 172.21.15.78:0/1975415271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 msgr2=0x56334aabc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 -- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 msgr2=0x56334aabdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:30.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aabdaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.808+0000 7f70ddfbd700 1 -- 172.21.15.78:0/1975415271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56334a87d630 con 0x56334aab1ac0 2023-10-10T17:51:30.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ddfbd700 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aabc640 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000f80 tx=0x7f70d0014740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75502bedefcff65 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:30.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70c0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:30.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70c0ff9700 1 -- 172.21.15.78:0/1975415271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d0018040 con 0x56334aab1ac0 2023-10-10T17:51:30.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56334aa9db20 con 0x56334aab1ac0 2023-10-10T17:51:30.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70c0ff9700 1 -- 172.21.15.78:0/1975415271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f70d0003a00 con 0x56334aab1ac0 2023-10-10T17:51:30.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70c0ff9700 1 -- 172.21.15.78:0/1975415271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70d0003bc0 con 0x56334aab1ac0 2023-10-10T17:51:30.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 10 client.17280 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:30.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x56334a812c60 con 0x56334aab1ac0 2023-10-10T17:51:30.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70c0ff9700 1 -- 172.21.15.78:0/1975415271 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56334aa9db20 con 0x56334aab1ac0 2023-10-10T17:51:30.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70c0ff9700 1 -- 172.21.15.78:0/1975415271 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f70d0031470 con 0x56334aab1ac0 2023-10-10T17:51:30.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 10 client.17280 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:30.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:30.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 20 client.17280 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:30.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334ab1b860 0x56334ab1dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:30.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 4 client.17280 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:51:30.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.809+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56334a84af70 con 0x56334ab1b860 2023-10-10T17:51:30.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334ab1b860 0x56334ab1dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:30.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70dd7bc700 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334ab1b860 0x56334ab1dc70 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e55421d057aa635 server_cookie=e3c7abff84bf3223 in_seq=0 out_seq=0 2023-10-10T17:51:30.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70c0ff9700 10 client.17280 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:30.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70c0ff9700 1 -- 172.21.15.78:0/1975415271 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56334a84af70 con 0x56334ab1b860 2023-10-10T17:51:30.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70c0ff9700 10 client.17280 handle_command_reply: tid=0 2023-10-10T17:51:30.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 client.17280 shutdown 2023-10-10T17:51:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 msgr2=0x56334aabc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aabc640 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f70d0000f80 tx=0x7f70d0014740 comp rx=0 tx=0).stop 2023-10-10T17:51:30.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 shutdown_connections 2023-10-10T17:51:30.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x56334ab1b860 0x56334ab1dc70 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x56334aab1ac0 0x56334aabc640 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x56334aaacd70 0x56334aabdaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70c0ff9700 0 client.17280 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 --2- 172.21.15.78:0/1975415271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x56334aaad680 0x56334aabc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 >> 172.21.15.78:0/1975415271 conn(0x56334aaa0360 msgr2=0x56334aaafac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:30.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 shutdown_connections 2023-10-10T17:51:30.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.810+0000 7f70ebbacb80 1 -- 172.21.15.78:0/1975415271 wait complete. 2023-10-10T17:51:30.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70ebbacb80 20 client.17280 trim_cache size 0 max 16384 2023-10-10T17:51:30.870 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:51:30.870 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:51:30.871 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:51:30.871 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:51:30.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f70c40a1340 msgr2=0x7f70c40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f70c40a1340 0x7f70c40a37f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f70d400ac50 tx=0x7f70d400a380 comp rx=0 tx=0).stop 2023-10-10T17:51:30.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 msgr2=0x7f70d8195790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:30.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8195790 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f70c8009990 tx=0x7f70c8004710 comp rx=0 tx=0).stop 2023-10-10T17:51:30.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 shutdown_connections 2023-10-10T17:51:30.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f70d8104490 0x7f70d8199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f70c40a1340 0x7f70c40a37f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f70d80ff450 0x7f70d8195250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 --2- 172.21.15.78:0/3585788104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f70d80ffdf0 0x7f70d8195790 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 >> 172.21.15.78:0/3585788104 conn(0x7f70d8077080 msgr2=0x7f70d80fcba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:30.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 shutdown_connections 2023-10-10T17:51:30.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:30.811+0000 7f70dfa20700 1 -- 172.21.15.78:0/3585788104 wait complete. 2023-10-10T17:51:30.877 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:51:30.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:51:31.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 -- 172.21.15.78:0/2585966206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab340ffdd0 msgr2=0x7fab34103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 --2- 172.21.15.78:0/2585966206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab340ffdd0 0x7fab34103d90 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fab240099c0 tx=0x7fab24009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 -- 172.21.15.78:0/2585966206 shutdown_connections 2023-10-10T17:51:31.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 --2- 172.21.15.78:0/2585966206 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab340ff430 0x7fab340ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 --2- 172.21.15.78:0/2585966206 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab34104470 0x7fab3410c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 --2- 172.21.15.78:0/2585966206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab340ffdd0 0x7fab34103d90 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 -- 172.21.15.78:0/2585966206 >> 172.21.15.78:0/2585966206 conn(0x7fab34076340 msgr2=0x7fab34076740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 -- 172.21.15.78:0/2585966206 shutdown_connections 2023-10-10T17:51:31.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.000+0000 7fab38c82700 1 -- 172.21.15.78:0/2585966206 wait complete. 2023-10-10T17:51:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 Processor -- start 2023-10-10T17:51:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 -- start start 2023-10-10T17:51:31.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab340ff430 0x7fab34195230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab340ffdd0 0x7fab34195770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab34104470 0x7fab34199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3419a210 con 0x7fab34104470 2023-10-10T17:51:31.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3419a380 con 0x7fab340ff430 2023-10-10T17:51:31.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab38c82700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3419a540 con 0x7fab340ffdd0 2023-10-10T17:51:31.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab32d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab34104470 0x7fab34199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab32d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab34104470 0x7fab34199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48500/0 (socket says 172.21.15.78:48500) 2023-10-10T17:51:31.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab32d9d700 1 -- 172.21.15.78:0/3680882427 learned_addr learned my addr 172.21.15.78:0/3680882427 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab3259c700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab340ff430 0x7fab34195230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.001+0000 7fab31d9b700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab340ffdd0 0x7fab34195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab32d9d700 1 -- 172.21.15.78:0/3680882427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab340ffdd0 msgr2=0x7fab34195770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab32d9d700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab340ffdd0 0x7fab34195770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab32d9d700 1 -- 172.21.15.78:0/3680882427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab340ff430 msgr2=0x7fab34195230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab32d9d700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab340ff430 0x7fab34195230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab32d9d700 1 -- 172.21.15.78:0/3680882427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab240095f0 con 0x7fab34104470 2023-10-10T17:51:31.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab3259c700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab340ff430 0x7fab34195230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:31.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab32d9d700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab34104470 0x7fab34199bd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00ada0 tx=0x7fab2c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab237fe700 1 -- 172.21.15.78:0/3680882427 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c00f070 con 0x7fab34104470 2023-10-10T17:51:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3410f240 con 0x7fab34104470 2023-10-10T17:51:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab237fe700 1 -- 172.21.15.78:0/3680882427 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fab2c007b00 con 0x7fab34104470 2023-10-10T17:51:31.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab237fe700 1 -- 172.21.15.78:0/3680882427 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c005460 con 0x7fab34104470 2023-10-10T17:51:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.002+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3410f0b0 con 0x7fab34104470 2023-10-10T17:51:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.003+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3410f3d0 con 0x7fab34104470 2023-10-10T17:51:31.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.004+0000 7fab237fe700 1 -- 172.21.15.78:0/3680882427 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fab2c0055c0 con 0x7fab34104470 2023-10-10T17:51:31.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.004+0000 7fab237fe700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fab1c0a12c0 0x7fab1c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.004+0000 7fab3259c700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fab1c0a12c0 0x7fab1c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.004+0000 7fab237fe700 1 -- 172.21.15.78:0/3680882427 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fab2c0cc620 con 0x7fab34104470 2023-10-10T17:51:31.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.005+0000 7fab3259c700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fab1c0a12c0 0x7fab1c0a3770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fab28005fd0 tx=0x7fab28005e30 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.009+0000 7fab237fe700 1 -- 172.21.15.78:0/3680882427 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fab2c095730 con 0x7fab34104470 2023-10-10T17:51:31.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:30 smithi078 ceph-mon[99494]: pgmap v278: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:31.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:30 smithi190 ceph-mon[101220]: pgmap v278: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:31.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:30 smithi157 ceph-mon[100896]: pgmap v278: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.240+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab340671d0 con 0x7fab34104470 2023-10-10T17:51:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.241+0000 7fab237fe700 1 -- 172.21.15.78:0/3680882427 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+10817 (secure 0 0 0) 0x7fab2c098d50 con 0x7fab34104470 2023-10-10T17:51:31.242 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:51:31.242 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":31,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:46:23.648656+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:51:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fab1c0a12c0 msgr2=0x7fab1c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fab1c0a12c0 0x7fab1c0a3770 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fab28005fd0 tx=0x7fab28005e30 comp rx=0 tx=0).stop 2023-10-10T17:51:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab34104470 msgr2=0x7fab34199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab34104470 0x7fab34199bd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00ada0 tx=0x7fab2c00c330 comp rx=0 tx=0).stop 2023-10-10T17:51:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 shutdown_connections 2023-10-10T17:51:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fab1c0a12c0 0x7fab1c0a3770 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fab34104470 0x7fab34199bd0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fab340ff430 0x7fab34195230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 --2- 172.21.15.78:0/3680882427 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fab340ffdd0 0x7fab34195770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 >> 172.21.15.78:0/3680882427 conn(0x7fab34076340 msgr2=0x7fab340fca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 shutdown_connections 2023-10-10T17:51:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.243+0000 7fab38c82700 1 -- 172.21.15.78:0/3680882427 wait complete. 2023-10-10T17:51:31.248 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 32 2023-10-10T17:51:31.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:51:31.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 -- 172.21.15.78:0/1695563294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 msgr2=0x7ffb8810c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/1695563294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8810c060 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ffb8000b0d0 tx=0x7ffb8000b3e0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 -- 172.21.15.78:0/1695563294 shutdown_connections 2023-10-10T17:51:31.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/1695563294 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8810c060 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/1695563294 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb880ff450 0x7ffb880ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/1695563294 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb880ffdf0 0x7ffb88103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 -- 172.21.15.78:0/1695563294 >> 172.21.15.78:0/1695563294 conn(0x7ffb880770b0 msgr2=0x7ffb880774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 -- 172.21.15.78:0/1695563294 shutdown_connections 2023-10-10T17:51:31.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 -- 172.21.15.78:0/1695563294 wait complete. 2023-10-10T17:51:31.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.379+0000 7ffb8ed25700 1 Processor -- start 2023-10-10T17:51:31.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8ed25700 1 -- start start 2023-10-10T17:51:31.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8ed25700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb880ff450 0x7ffb88196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8ed25700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb880ffdf0 0x7ffb88196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8ed25700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8819af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8ed25700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8819b560 con 0x7ffb88104490 2023-10-10T17:51:31.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8ed25700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8819b6d0 con 0x7ffb880ffdf0 2023-10-10T17:51:31.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8ed25700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8819b890 con 0x7ffb880ff450 2023-10-10T17:51:31.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8819af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8819af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48514/0 (socket says 172.21.15.78:48514) 2023-10-10T17:51:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/3154847715 learned_addr learned my addr 172.21.15.78:0/3154847715 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb880ff450 0x7ffb88196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/3154847715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb880ff450 msgr2=0x7ffb88196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb880ff450 0x7ffb88196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/3154847715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb880ffdf0 msgr2=0x7ffb88196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb87fff700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb880ffdf0 0x7ffb88196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb880ffdf0 0x7ffb88196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/3154847715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb80009d00 con 0x7ffb88104490 2023-10-10T17:51:31.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.380+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8819af20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ffb80007940 tx=0x7ffb80007970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.381+0000 7ffb85ffb700 1 -- 172.21.15.78:0/3154847715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8000e040 con 0x7ffb88104490 2023-10-10T17:51:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.381+0000 7ffb85ffb700 1 -- 172.21.15.78:0/3154847715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb80003e10 con 0x7ffb88104490 2023-10-10T17:51:31.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.381+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8810f2d0 con 0x7ffb88104490 2023-10-10T17:51:31.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.381+0000 7ffb85ffb700 1 -- 172.21.15.78:0/3154847715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb80012a60 con 0x7ffb88104490 2023-10-10T17:51:31.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.381+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8810f140 con 0x7ffb88104490 2023-10-10T17:51:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.384+0000 7ffb85ffb700 1 -- 172.21.15.78:0/3154847715 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ffb80005730 con 0x7ffb88104490 2023-10-10T17:51:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.384+0000 7ffb85ffb700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ffb780a12c0 0x7ffb780a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.384+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ffb780a12c0 0x7ffb780a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.384+0000 7ffb85ffb700 1 -- 172.21.15.78:0/3154847715 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7ffb800ce850 con 0x7ffb88104490 2023-10-10T17:51:31.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ffb780a12c0 0x7ffb780a3770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009a10 tx=0x7ffb7c009190 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 Processor -- start 2023-10-10T17:51:31.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 -- start start 2023-10-10T17:51:31.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c36f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c37470 0x55ec37c3b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37b3c060 con 0x55ec37c3b8b0 2023-10-10T17:51:31.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37b8e8a0 con 0x55ec37c36b60 2023-10-10T17:51:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37b87d30 con 0x55ec37c37470 2023-10-10T17:51:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48522/0 (socket says 172.21.15.78:48522) 2023-10-10T17:51:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/922968705 learned_addr learned my addr 172.21.15.78:0/922968705 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c36f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb87fff700 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c37470 0x55ec37c3b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/922968705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c37470 msgr2=0x55ec37c3b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c37470 0x55ec37c3b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/922968705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 msgr2=0x55ec37c36f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c36f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8d2c2700 1 -- 172.21.15.78:0/922968705 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec37a09630 con 0x55ec37c3b8b0 2023-10-10T17:51:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.385+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c36f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3bd00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ffb800a34b0 tx=0x7ffb800124f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d99d8dae883e92 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb72ffd700 1 -- 172.21.15.78:0/922968705 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8009b620 con 0x55ec37c3b8b0 2023-10-10T17:51:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb72ffd700 1 -- 172.21.15.78:0/922968705 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb8009f650 con 0x55ec37c3b8b0 2023-10-10T17:51:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb72ffd700 1 -- 172.21.15.78:0/922968705 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb800862e0 con 0x55ec37c3b8b0 2023-10-10T17:51:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/922968705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c3b8b0 msgr2=0x55ec37c3bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3bd00 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ffb800a34b0 tx=0x7ffb800124f0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/922968705 shutdown_connections 2023-10-10T17:51:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3bd00 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c36f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/922968705 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c37470 0x55ec37c3b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/922968705 >> 172.21.15.78:0/922968705 conn(0x55ec37c2c360 msgr2=0x55ec37c31b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.386+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/922968705 shutdown_connections 2023-10-10T17:51:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/922968705 wait complete. 2023-10-10T17:51:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 -- start start 2023-10-10T17:51:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 Processor -- start 2023-10-10T17:51:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c36b60 0x55ec37c43060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c3b8b0 0x55ec37c40980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37b298e0 con 0x55ec37c37470 2023-10-10T17:51:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37b29580 con 0x55ec37c3b8b0 2023-10-10T17:51:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37a0b920 con 0x55ec37c36b60 2023-10-10T17:51:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb9aeb1b80 10 client.0 _ll_register_callbacks cb 0x55ec37b47780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48526/0 (socket says 172.21.15.78:48526) 2023-10-10T17:51:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.387+0000 7ffb87fff700 1 -- 172.21.15.78:0/247396362 learned_addr learned my addr 172.21.15.78:0/247396362 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c36b60 0x55ec37c43060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c3b8b0 0x55ec37c40980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb87fff700 1 -- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c36b60 msgr2=0x55ec37c43060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb87fff700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c36b60 0x55ec37c43060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb87fff700 1 -- 172.21.15.78:0/247396362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c3b8b0 msgr2=0x55ec37c40980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb87fff700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c3b8b0 0x55ec37c40980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb87fff700 1 -- 172.21.15.78:0/247396362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8810f140 con 0x55ec37c37470 2023-10-10T17:51:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c3b8b0 0x55ec37c40980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb87fff700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c402e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400e6b0 tx=0x7ffb7400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b845b5d170338d44 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb67fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb67fff700 1 -- 172.21.15.78:0/247396362 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74004a30 con 0x55ec37c37470 2023-10-10T17:51:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb67fff700 1 -- 172.21.15.78:0/247396362 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7ffb74004b90 con 0x55ec37c37470 2023-10-10T17:51:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec37c29b20 con 0x55ec37c37470 2023-10-10T17:51:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb67fff700 1 -- 172.21.15.78:0/247396362 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb7400c570 con 0x55ec37c37470 2023-10-10T17:51:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb9aeb1b80 10 client.17298 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ec37b4e190 con 0x55ec37c37470 2023-10-10T17:51:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.388+0000 7ffb67fff700 1 -- 172.21.15.78:0/247396362 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ec37c29b20 con 0x55ec37c37470 2023-10-10T17:51:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb67fff700 1 -- 172.21.15.78:0/247396362 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ffb740052e0 con 0x55ec37c37470 2023-10-10T17:51:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb9aeb1b80 10 client.17298 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb9aeb1b80 10 client.17298 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:51:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb9aeb1b80 20 client.17298 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37c50790 0x55ec37c52ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb9aeb1b80 4 client.17298 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec37b4a0a0 con 0x55ec37c50790 2023-10-10T17:51:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37c50790 0x55ec37c52ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb67fff700 10 client.17298 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:51:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.389+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37c50790 0x55ec37c52ba0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=382ff2a947099e6e server_cookie=9a786efac79cef8c in_seq=0 out_seq=0 2023-10-10T17:51:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb67fff700 1 -- 172.21.15.78:0/247396362 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ec37b4a0a0 con 0x55ec37c50790 2023-10-10T17:51:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb67fff700 10 client.17298 handle_command_reply: tid=0 2023-10-10T17:51:31.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 client.17298 shutdown 2023-10-10T17:51:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 msgr2=0x55ec37c402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c402e0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ffb7400e6b0 tx=0x7ffb7400ea70 comp rx=0 tx=0).stop 2023-10-10T17:51:31.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 shutdown_connections 2023-10-10T17:51:31.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37c50790 0x55ec37c52ba0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c36b60 0x55ec37c43060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb67fff700 0 client.17298 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:51:31.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c3b8b0 0x55ec37c40980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/247396362 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c402e0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 >> 172.21.15.78:0/247396362 conn(0x55ec37c2c360 msgr2=0x55ec37c3cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 shutdown_connections 2023-10-10T17:51:31.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/247396362 wait complete. 2023-10-10T17:51:31.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.390+0000 7ffb9aeb1b80 20 client.17298 trim_cache size 0 max 16384 2023-10-10T17:51:31.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 Processor -- start 2023-10-10T17:51:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 -- start start 2023-10-10T17:51:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c31be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c477b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37c331c0 con 0x55ec37c37470 2023-10-10T17:51:31.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37c433a0 con 0x55ec37c36b60 2023-10-10T17:51:31.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37c53200 con 0x55ec37c3b8b0 2023-10-10T17:51:31.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c477b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c477b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48534/0 (socket says 172.21.15.78:48534) 2023-10-10T17:51:31.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 -- 172.21.15.78:0/516286347 learned_addr learned my addr 172.21.15.78:0/516286347 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c31be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 -- 172.21.15.78:0/516286347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 msgr2=0x55ec37c3cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 -- 172.21.15.78:0/516286347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 msgr2=0x55ec37c31be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c31be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 -- 172.21.15.78:0/516286347 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec37c2db00 con 0x55ec37c37470 2023-10-10T17:51:31.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.403+0000 7ffb87fff700 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c477b0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ffb74001eb0 tx=0x7ffb7400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b51f5c3f6760cfc server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb737fe700 1 -- 172.21.15.78:0/516286347 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb740177d0 con 0x55ec37c37470 2023-10-10T17:51:31.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb737fe700 1 -- 172.21.15.78:0/516286347 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb74017df0 con 0x55ec37c37470 2023-10-10T17:51:31.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb737fe700 1 -- 172.21.15.78:0/516286347 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74026690 con 0x55ec37c37470 2023-10-10T17:51:31.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/516286347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 msgr2=0x55ec37c477b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c477b0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ffb74001eb0 tx=0x7ffb7400ef30 comp rx=0 tx=0).stop 2023-10-10T17:51:31.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/516286347 shutdown_connections 2023-10-10T17:51:31.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c31be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/516286347 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c477b0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/516286347 >> 172.21.15.78:0/516286347 conn(0x55ec37c2c360 msgr2=0x55ec37c42c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/516286347 shutdown_connections 2023-10-10T17:51:31.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/516286347 wait complete. 2023-10-10T17:51:31.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.404+0000 7ffb9aeb1b80 1 -- start start 2023-10-10T17:51:31.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 1 Processor -- start 2023-10-10T17:51:31.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c3fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c40070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37c40780 con 0x55ec37c37470 2023-10-10T17:51:31.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37c3eff0 con 0x55ec37c36b60 2023-10-10T17:51:31.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec37c3f1b0 con 0x55ec37c3b8b0 2023-10-10T17:51:31.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb9aeb1b80 10 client.0 _ll_register_callbacks cb 0x7ffb7400ca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:31.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c40070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c40070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48540/0 (socket says 172.21.15.78:48540) 2023-10-10T17:51:31.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 -- 172.21.15.78:0/2273371476 learned_addr learned my addr 172.21.15.78:0/2273371476 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c3fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb8d2c2700 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 -- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 msgr2=0x55ec37c3eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 -- 172.21.15.78:0/2273371476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 msgr2=0x55ec37c3fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c3fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.405+0000 7ffb87fff700 1 -- 172.21.15.78:0/2273371476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec37b4e190 con 0x55ec37c37470 2023-10-10T17:51:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb87fff700 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c40070 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ffb74005950 tx=0x7ffb740176f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7379b91b2aea117f server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb67fff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb67fff700 1 -- 172.21.15.78:0/2273371476 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7400f8f0 con 0x55ec37c37470 2023-10-10T17:51:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb67fff700 1 -- 172.21.15.78:0/2273371476 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7ffb74000e40 con 0x55ec37c37470 2023-10-10T17:51:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb8009f650 con 0x55ec37c37470 2023-10-10T17:51:31.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb67fff700 1 -- 172.21.15.78:0/2273371476 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb740177d0 con 0x55ec37c37470 2023-10-10T17:51:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 10 client.17304 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ec379b5de0 con 0x55ec37c37470 2023-10-10T17:51:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb67fff700 1 -- 172.21.15.78:0/2273371476 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb8009f650 con 0x55ec37c37470 2023-10-10T17:51:31.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb67fff700 1 -- 172.21.15.78:0/2273371476 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7ffb74026b80 con 0x55ec37c37470 2023-10-10T17:51:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 10 client.17304 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 10 client.17304 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:51:31.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 20 client.17304 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37ca4f10 0x55ec37ca7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 4 client.17304 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:51:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.406+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec37b56660 con 0x55ec37ca4f10 2023-10-10T17:51:31.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37ca4f10 0x55ec37ca7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb67fff700 10 client.17304 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:51:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb8cac1700 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37ca4f10 0x55ec37ca7320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc67458b0e73ef65 server_cookie=f382c53ccebf068d in_seq=0 out_seq=0 2023-10-10T17:51:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb67fff700 1 -- 172.21.15.78:0/2273371476 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec37b56660 con 0x55ec37ca4f10 2023-10-10T17:51:31.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb67fff700 10 client.17304 handle_command_reply: tid=0 2023-10-10T17:51:31.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 client.17304 shutdown 2023-10-10T17:51:31.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 msgr2=0x55ec37c40070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c40070 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7ffb74005950 tx=0x7ffb740176f0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 shutdown_connections 2023-10-10T17:51:31.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55ec37ca4f10 0x55ec37ca7320 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ec37c3b8b0 0x55ec37c3eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb67fff700 0 client.17304 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:51:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ec37c36b60 0x55ec37c3fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 --2- 172.21.15.78:0/2273371476 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ec37c37470 0x55ec37c40070 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.407+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 >> 172.21.15.78:0/2273371476 conn(0x55ec37c2c360 msgr2=0x55ec37c35960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 shutdown_connections 2023-10-10T17:51:31.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb9aeb1b80 1 -- 172.21.15.78:0/2273371476 wait complete. 2023-10-10T17:51:31.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb9aeb1b80 20 client.17304 trim_cache size 0 max 16384 2023-10-10T17:51:31.469 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:51:31.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ffb780a12c0 msgr2=0x7ffb780a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ffb780a12c0 0x7ffb780a3770 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c009a10 tx=0x7ffb7c009190 comp rx=0 tx=0).stop 2023-10-10T17:51:31.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 msgr2=0x7ffb8819af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8819af20 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ffb80007940 tx=0x7ffb80007970 comp rx=0 tx=0).stop 2023-10-10T17:51:31.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 shutdown_connections 2023-10-10T17:51:31.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ffb880ff450 0x7ffb88196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ffb780a12c0 0x7ffb780a3770 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ffb88104490 0x7ffb8819af20 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 --2- 172.21.15.78:0/3154847715 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ffb880ffdf0 0x7ffb88196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 >> 172.21.15.78:0/3154847715 conn(0x7ffb880770b0 msgr2=0x7ffb88067e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 shutdown_connections 2023-10-10T17:51:31.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.408+0000 7ffb8ed25700 1 -- 172.21.15.78:0/3154847715 wait complete. 2023-10-10T17:51:31.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:51:31.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 -- 172.21.15.78:0/3464883378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 msgr2=0x7f96c8103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 --2- 172.21.15.78:0/3464883378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8103db0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0099c0 tx=0x7f96bc009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 -- 172.21.15.78:0/3464883378 shutdown_connections 2023-10-10T17:51:31.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 --2- 172.21.15.78:0/3464883378 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96c80ff450 0x7f96c80ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 --2- 172.21.15.78:0/3464883378 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96c8104490 0x7f96c810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 --2- 172.21.15.78:0/3464883378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8103db0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 -- 172.21.15.78:0/3464883378 >> 172.21.15.78:0/3464883378 conn(0x7f96c80770b0 msgr2=0x7f96c80774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 -- 172.21.15.78:0/3464883378 shutdown_connections 2023-10-10T17:51:31.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.597+0000 7f96ce89a700 1 -- 172.21.15.78:0/3464883378 wait complete. 2023-10-10T17:51:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 Processor -- start 2023-10-10T17:51:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 -- start start 2023-10-10T17:51:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96c80ff450 0x7f96c8100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96c8104490 0x7f96c81a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c8101bc0 con 0x7f96c80ffdf0 2023-10-10T17:51:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c81a4b00 con 0x7f96c8104490 2023-10-10T17:51:31.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96ce89a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c81a4c40 con 0x7f96c80ff450 2023-10-10T17:51:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48554/0 (socket says 172.21.15.78:48554) 2023-10-10T17:51:31.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 -- 172.21.15.78:0/4155572287 learned_addr learned my addr 172.21.15.78:0/4155572287 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96cce37700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96c8104490 0x7f96c81a45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c7fff700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96c80ff450 0x7f96c8100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 -- 172.21.15.78:0/4155572287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96c80ff450 msgr2=0x7f96c8100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96c80ff450 0x7f96c8100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 -- 172.21.15.78:0/4155572287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96c8104490 msgr2=0x7f96c81a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96c8104490 0x7f96c81a45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.598+0000 7f96c77fe700 1 -- 172.21.15.78:0/4155572287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0095f0 con 0x7f96c80ffdf0 2023-10-10T17:51:31.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.599+0000 7f96c77fe700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8101390 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009990 tx=0x7f96bc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.599+0000 7f96c57fa700 1 -- 172.21.15.78:0/4155572287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc01d070 con 0x7f96c80ffdf0 2023-10-10T17:51:31.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.599+0000 7f96c57fa700 1 -- 172.21.15.78:0/4155572287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96bc0054e0 con 0x7f96c80ffdf0 2023-10-10T17:51:31.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.599+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c810f260 con 0x7f96c80ffdf0 2023-10-10T17:51:31.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.599+0000 7f96c57fa700 1 -- 172.21.15.78:0/4155572287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc004a70 con 0x7f96c80ffdf0 2023-10-10T17:51:31.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.599+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c810f0d0 con 0x7f96c80ffdf0 2023-10-10T17:51:31.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.602+0000 7f96c57fa700 1 -- 172.21.15.78:0/4155572287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f96bc00fb80 con 0x7f96c80ffdf0 2023-10-10T17:51:31.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.602+0000 7f96c57fa700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96b00a12c0 0x7f96b00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.602+0000 7f96c7fff700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96b00a12c0 0x7f96b00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.602+0000 7f96c57fa700 1 -- 172.21.15.78:0/4155572287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f96bc0cdb90 con 0x7f96c80ffdf0 2023-10-10T17:51:31.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c7fff700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96b00a12c0 0x7f96b00a3770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009bf0 tx=0x7f96b8009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 Processor -- start 2023-10-10T17:51:31.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 -- start start 2023-10-10T17:51:31.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e16f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e1b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e1b8b0 0x5577e5e1bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5d1c060 con 0x5577e5e17470 2023-10-10T17:51:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5d6e8a0 con 0x5577e5e16b60 2023-10-10T17:51:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96daa26b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5d67d30 con 0x5577e5e1b8b0 2023-10-10T17:51:31.620 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e1b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e1b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48562/0 (socket says 172.21.15.78:48562) 2023-10-10T17:51:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 -- 172.21.15.78:0/4095570735 learned_addr learned my addr 172.21.15.78:0/4095570735 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96cce37700 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e1b8b0 0x5577e5e1bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c7fff700 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e16f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 -- 172.21.15.78:0/4095570735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e1b8b0 msgr2=0x5577e5e1bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e1b8b0 0x5577e5e1bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.622 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 -- 172.21.15.78:0/4095570735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 msgr2=0x5577e5e16f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e16f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c77fe700 1 -- 172.21.15.78:0/4095570735 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577e5be9630 con 0x5577e5e17470 2023-10-10T17:51:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.603+0000 7f96c7fff700 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e16f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96c77fe700 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e1b370 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f96bc096e60 tx=0x7f96bc009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a8dffc2f720b01 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96ae7fc700 1 -- 172.21.15.78:0/4095570735 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc09a960 con 0x5577e5e17470 2023-10-10T17:51:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96ae7fc700 1 -- 172.21.15.78:0/4095570735 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96bc09e990 con 0x5577e5e17470 2023-10-10T17:51:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96ae7fc700 1 -- 172.21.15.78:0/4095570735 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc09a050 con 0x5577e5e17470 2023-10-10T17:51:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 -- 172.21.15.78:0/4095570735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 msgr2=0x5577e5e1b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e1b370 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f96bc096e60 tx=0x7f96bc009310 comp rx=0 tx=0).stop 2023-10-10T17:51:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 -- 172.21.15.78:0/4095570735 shutdown_connections 2023-10-10T17:51:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e1b8b0 0x5577e5e1bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e16f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 --2- 172.21.15.78:0/4095570735 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e1b370 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 -- 172.21.15.78:0/4095570735 >> 172.21.15.78:0/4095570735 conn(0x5577e5e0c360 msgr2=0x5577e5e11b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 -- 172.21.15.78:0/4095570735 shutdown_connections 2023-10-10T17:51:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.604+0000 7f96daa26b80 1 -- 172.21.15.78:0/4095570735 wait complete. 2023-10-10T17:51:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 -- start start 2023-10-10T17:51:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 Processor -- start 2023-10-10T17:51:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e22fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e17470 0x5577e5e201b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e20730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5d098e0 con 0x5577e5e1b8b0 2023-10-10T17:51:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5d09580 con 0x5577e5e16b60 2023-10-10T17:51:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5beb920 con 0x5577e5e17470 2023-10-10T17:51:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96daa26b80 10 client.0 _ll_register_callbacks cb 0x5577e5d27780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96cce37700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e20730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96cce37700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e20730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48570/0 (socket says 172.21.15.78:48570) 2023-10-10T17:51:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96cce37700 1 -- 172.21.15.78:0/3275788912 learned_addr learned my addr 172.21.15.78:0/3275788912 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96c7fff700 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e22fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.605+0000 7f96c77fe700 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e17470 0x5577e5e201b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96cce37700 1 -- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e17470 msgr2=0x5577e5e201b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96cce37700 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e17470 0x5577e5e201b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96cce37700 1 -- 172.21.15.78:0/3275788912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 msgr2=0x5577e5e22fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96cce37700 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e22fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96cce37700 1 -- 172.21.15.78:0/3275788912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc085040 con 0x5577e5e1b8b0 2023-10-10T17:51:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96cce37700 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e20730 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f96c0012d70 tx=0x7f96c0010520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdfbcf74c631a8a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96a77fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96a77fe700 1 -- 172.21.15.78:0/3275788912 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0004bb0 con 0x5577e5e1b8b0 2023-10-10T17:51:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96a77fe700 1 -- 172.21.15.78:0/3275788912 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f96c0004d10 con 0x5577e5e1b8b0 2023-10-10T17:51:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577e5e09b20 con 0x5577e5e1b8b0 2023-10-10T17:51:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96a77fe700 1 -- 172.21.15.78:0/3275788912 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96c0010b30 con 0x5577e5e1b8b0 2023-10-10T17:51:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96daa26b80 10 client.17316 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5577e5d2e190 con 0x5577e5e1b8b0 2023-10-10T17:51:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96a77fe700 1 -- 172.21.15.78:0/3275788912 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5577e5e09b20 con 0x5577e5e1b8b0 2023-10-10T17:51:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96a77fe700 1 -- 172.21.15.78:0/3275788912 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f96c00143a0 con 0x5577e5e1b8b0 2023-10-10T17:51:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96daa26b80 10 client.17316 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96daa26b80 10 client.17316 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:51:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.606+0000 7f96daa26b80 20 client.17316 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.607+0000 7f96daa26b80 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e30570 0x5577e5e32980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.607+0000 7f96daa26b80 4 client.17316 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.607+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577e5d2a0a0 con 0x5577e5e30570 2023-10-10T17:51:31.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.607+0000 7f96c7fff700 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e30570 0x5577e5e32980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.607+0000 7f96a77fe700 10 client.17316 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:51:31.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.607+0000 7f96c7fff700 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e30570 0x5577e5e32980 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a7ca9b552e101edf server_cookie=dc95cd956b846078 in_seq=0 out_seq=0 2023-10-10T17:51:31.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96a77fe700 1 -- 172.21.15.78:0/3275788912 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5577e5d2a0a0 con 0x5577e5e30570 2023-10-10T17:51:31.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96a77fe700 10 client.17316 handle_command_reply: tid=0 2023-10-10T17:51:31.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 client.17316 shutdown 2023-10-10T17:51:31.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 msgr2=0x5577e5e20730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e20730 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f96c0012d70 tx=0x7f96c0010520 comp rx=0 tx=0).stop 2023-10-10T17:51:31.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 shutdown_connections 2023-10-10T17:51:31.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e20730 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e16b60 0x5577e5e22fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e30570 0x5577e5e32980 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 --2- 172.21.15.78:0/3275788912 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e17470 0x5577e5e201b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96a77fe700 0 client.17316 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:51:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 >> 172.21.15.78:0/3275788912 conn(0x5577e5e0c360 msgr2=0x5577e5e10de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.608+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 shutdown_connections 2023-10-10T17:51:31.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.609+0000 7f96daa26b80 1 -- 172.21.15.78:0/3275788912 wait complete. 2023-10-10T17:51:31.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.609+0000 7f96daa26b80 20 client.17316 trim_cache size 0 max 16384 2023-10-10T17:51:31.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.621+0000 7f96daa26b80 1 Processor -- start 2023-10-10T17:51:31.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.621+0000 7f96daa26b80 1 -- start start 2023-10-10T17:51:31.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.621+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e277b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.621+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e17470 0x5577e5e22ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e11560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96daa26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5e09660 con 0x5577e5e1b8b0 2023-10-10T17:51:31.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96daa26b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5e2dbf0 con 0x5577e5e17470 2023-10-10T17:51:31.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96daa26b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5e32fe0 con 0x5577e5e16b60 2023-10-10T17:51:31.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e11560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e11560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48578/0 (socket says 172.21.15.78:48578) 2023-10-10T17:51:31.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 -- 172.21.15.78:0/59831861 learned_addr learned my addr 172.21.15.78:0/59831861 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96c7fff700 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e277b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 -- 172.21.15.78:0/59831861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 msgr2=0x5577e5e277b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e277b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 -- 172.21.15.78:0/59831861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e17470 msgr2=0x5577e5e22ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e17470 0x5577e5e22ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 -- 172.21.15.78:0/59831861 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577e5e0db00 con 0x5577e5e1b8b0 2023-10-10T17:51:31.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96cce37700 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e11560 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f96c0012750 tx=0x7f96c0010920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c08cedf9c6e2dd server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96aeffd700 1 -- 172.21.15.78:0/59831861 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0014aa0 con 0x5577e5e1b8b0 2023-10-10T17:51:31.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96aeffd700 1 -- 172.21.15.78:0/59831861 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96c00079c0 con 0x5577e5e1b8b0 2023-10-10T17:51:31.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.622+0000 7f96aeffd700 1 -- 172.21.15.78:0/59831861 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0003d70 con 0x5577e5e1b8b0 2023-10-10T17:51:31.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 -- 172.21.15.78:0/59831861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 msgr2=0x5577e5e11560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e11560 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f96c0012750 tx=0x7f96c0010920 comp rx=0 tx=0).stop 2023-10-10T17:51:31.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 -- 172.21.15.78:0/59831861 shutdown_connections 2023-10-10T17:51:31.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e277b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e1b8b0 0x5577e5e11560 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 --2- 172.21.15.78:0/59831861 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e17470 0x5577e5e22ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 -- 172.21.15.78:0/59831861 >> 172.21.15.78:0/59831861 conn(0x5577e5e0c360 msgr2=0x5577e5e198b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 -- 172.21.15.78:0/59831861 shutdown_connections 2023-10-10T17:51:31.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 -- 172.21.15.78:0/59831861 wait complete. 2023-10-10T17:51:31.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 -- start start 2023-10-10T17:51:31.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.623+0000 7f96daa26b80 1 Processor -- start 2023-10-10T17:51:31.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e24cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e25210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e1b8b0 0x5577e5e25750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5e27da0 con 0x5577e5e17470 2023-10-10T17:51:31.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5e833f0 con 0x5577e5e1b8b0 2023-10-10T17:51:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e5e83530 con 0x5577e5e16b60 2023-10-10T17:51:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 10 client.0 _ll_register_callbacks cb 0x7f96c0012660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:31.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e25210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e25210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48582/0 (socket says 172.21.15.78:48582) 2023-10-10T17:51:31.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 -- 172.21.15.78:0/491053138 learned_addr learned my addr 172.21.15.78:0/491053138 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c7fff700 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e24cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96cce37700 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e1b8b0 0x5577e5e25750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 -- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 msgr2=0x5577e5e24cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e24cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 -- 172.21.15.78:0/491053138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e1b8b0 msgr2=0x5577e5e25750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e1b8b0 0x5577e5e25750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 -- 172.21.15.78:0/491053138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577e5d2e190 con 0x5577e5e17470 2023-10-10T17:51:31.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96c77fe700 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e25210 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0cb550 tx=0x7f96bc005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e811a3a81bf3e01 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96a77fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:31.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96a77fe700 1 -- 172.21.15.78:0/491053138 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc0a2d20 con 0x5577e5e17470 2023-10-10T17:51:31.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96bc09e990 con 0x5577e5e17470 2023-10-10T17:51:31.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96a77fe700 1 -- 172.21.15.78:0/491053138 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f96bc09a960 con 0x5577e5e17470 2023-10-10T17:51:31.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96a77fe700 1 -- 172.21.15.78:0/491053138 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f96bc0a2aa0 con 0x5577e5e17470 2023-10-10T17:51:31.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 10 client.17322 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:31.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.624+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5577e5b95de0 con 0x5577e5e17470 2023-10-10T17:51:31.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96a77fe700 1 -- 172.21.15.78:0/491053138 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f96bc09e990 con 0x5577e5e17470 2023-10-10T17:51:31.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96a77fe700 1 -- 172.21.15.78:0/491053138 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f96bc0b4030 con 0x5577e5e17470 2023-10-10T17:51:31.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96daa26b80 10 client.17322 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:31.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96daa26b80 10 client.17322 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:51:31.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96daa26b80 20 client.17322 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:31.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96daa26b80 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e84950 0x5577e5e86d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96daa26b80 4 client.17322 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:51:31.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5577e5d36660 con 0x5577e5e84950 2023-10-10T17:51:31.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96c7fff700 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e84950 0x5577e5e86d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96a77fe700 10 client.17322 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:51:31.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.625+0000 7f96c7fff700 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e84950 0x5577e5e86d60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=236675cd29e69bde server_cookie=63d647008247764c in_seq=0 out_seq=0 2023-10-10T17:51:31.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96a77fe700 1 -- 172.21.15.78:0/491053138 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5577e5d36660 con 0x5577e5e84950 2023-10-10T17:51:31.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96a77fe700 10 client.17322 handle_command_reply: tid=0 2023-10-10T17:51:31.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 client.17322 shutdown 2023-10-10T17:51:31.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 msgr2=0x5577e5e25210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e25210 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f96bc0cb550 tx=0x7f96bc005f10 comp rx=0 tx=0).stop 2023-10-10T17:51:31.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 shutdown_connections 2023-10-10T17:51:31.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x5577e5e84950 0x5577e5e86d60 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5577e5e16b60 0x5577e5e24cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5577e5e1b8b0 0x5577e5e25750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 --2- 172.21.15.78:0/491053138 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5577e5e17470 0x5577e5e25210 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96a77fe700 0 client.17322 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:51:31.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 >> 172.21.15.78:0/491053138 conn(0x5577e5e0c360 msgr2=0x5577e5e15960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 shutdown_connections 2023-10-10T17:51:31.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 1 -- 172.21.15.78:0/491053138 wait complete. 2023-10-10T17:51:31.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96daa26b80 20 client.17322 trim_cache size 0 max 16384 2023-10-10T17:51:31.686 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:51:31.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96b00a12c0 msgr2=0x7f96b00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96b00a12c0 0x7f96b00a3770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f96b8009bf0 tx=0x7f96b8009260 comp rx=0 tx=0).stop 2023-10-10T17:51:31.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 msgr2=0x7f96c8101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8101390 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f96bc009990 tx=0x7f96bc00f690 comp rx=0 tx=0).stop 2023-10-10T17:51:31.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 shutdown_connections 2023-10-10T17:51:31.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f96c80ff450 0x7f96c8100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f96b00a12c0 0x7f96b00a3770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f96c8104490 0x7f96c81a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.626+0000 7f96ce89a700 1 --2- 172.21.15.78:0/4155572287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f96c80ffdf0 0x7f96c8101390 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.627+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 >> 172.21.15.78:0/4155572287 conn(0x7f96c80770b0 msgr2=0x7f96c8067e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.627+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 shutdown_connections 2023-10-10T17:51:31.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.627+0000 7f96ce89a700 1 -- 172.21.15.78:0/4155572287 wait complete. 2023-10-10T17:51:31.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:51:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 -- 172.21.15.78:0/2217195970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100f30 msgr2=0x7f3cc0109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/2217195970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100f30 0x7f3cc0109690 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f3cb00099c0 tx=0x7f3cb0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 -- 172.21.15.78:0/2217195970 shutdown_connections 2023-10-10T17:51:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/2217195970 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3cc0109bd0 0x7f3cc010bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/2217195970 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc0100620 0x7f3cc01009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/2217195970 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100f30 0x7f3cc0109690 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 -- 172.21.15.78:0/2217195970 >> 172.21.15.78:0/2217195970 conn(0x7f3cc0074b00 msgr2=0x7f3cc0074f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 -- 172.21.15.78:0/2217195970 shutdown_connections 2023-10-10T17:51:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.816+0000 7f3cc5abf700 1 -- 172.21.15.78:0/2217195970 wait complete. 2023-10-10T17:51:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 Processor -- start 2023-10-10T17:51:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 -- start start 2023-10-10T17:51:31.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100620 0x7f3cc0195350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3cc0100f30 0x7f3cc0195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc0109bd0 0x7f3cc0199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc019a330 con 0x7f3cc0100620 2023-10-10T17:51:31.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc019a4a0 con 0x7f3cc0109bd0 2023-10-10T17:51:31.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cc5abf700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cc019a660 con 0x7f3cc0100f30 2023-10-10T17:51:31.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cbf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100620 0x7f3cc0195350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cbf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100620 0x7f3cc0195350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48594/0 (socket says 172.21.15.78:48594) 2023-10-10T17:51:31.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cbf7fe700 1 -- 172.21.15.78:0/19164495 learned_addr learned my addr 172.21.15.78:0/19164495 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cbffff700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc0109bd0 0x7f3cc0199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.817+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3cc0100f30 0x7f3cc0195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbf7fe700 1 -- 172.21.15.78:0/19164495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3cc0100f30 msgr2=0x7f3cc0195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3cc0100f30 0x7f3cc0195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbf7fe700 1 -- 172.21.15.78:0/19164495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc0109bd0 msgr2=0x7f3cc0199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc0109bd0 0x7f3cc0199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbf7fe700 1 -- 172.21.15.78:0/19164495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb00095f0 con 0x7f3cc0100620 2023-10-10T17:51:31.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100620 0x7f3cc0195350 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400d950 tx=0x7f3cb400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbcff9700 1 -- 172.21.15.78:0/19164495 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb400f890 con 0x7f3cc0100620 2023-10-10T17:51:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbcff9700 1 -- 172.21.15.78:0/19164495 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3cb4010460 con 0x7f3cc0100620 2023-10-10T17:51:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cc010f2f0 con 0x7f3cc0100620 2023-10-10T17:51:31.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cbcff9700 1 -- 172.21.15.78:0/19164495 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb400e610 con 0x7f3cc0100620 2023-10-10T17:51:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.818+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc010f160 con 0x7f3cc0100620 2023-10-10T17:51:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.820+0000 7f3cbcff9700 1 -- 172.21.15.78:0/19164495 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3cb40108d0 con 0x7f3cc0100620 2023-10-10T17:51:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.821+0000 7f3cbcff9700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3ca80a12c0 0x7f3ca80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.821+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3ca80a12c0 0x7f3ca80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.821+0000 7f3cbcff9700 1 -- 172.21.15.78:0/19164495 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f3cb40cce90 con 0x7f3cc0100620 2023-10-10T17:51:31.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.821+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3ca80a12c0 0x7f3ca80a3770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0006010 tx=0x7f3cb0009f10 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.821+0000 7f3cd1c4bb80 1 Processor -- start 2023-10-10T17:51:31.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.821+0000 7f3cd1c4bb80 1 -- start start 2023-10-10T17:51:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7932d70 0x5558d7933140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7933680 0x5558d7937580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7937ac0 0x5558d7937f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7836060 con 0x5558d7933680 2023-10-10T17:51:31.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d78888a0 con 0x5558d7937ac0 2023-10-10T17:51:31.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7881d30 con 0x5558d7932d70 2023-10-10T17:51:31.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7933680 0x5558d7937580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7933680 0x5558d7937580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48604/0 (socket says 172.21.15.78:48604) 2023-10-10T17:51:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 -- 172.21.15.78:0/3015106893 learned_addr learned my addr 172.21.15.78:0/3015106893 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7932d70 0x5558d7933140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbffff700 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7937ac0 0x5558d7937f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 -- 172.21.15.78:0/3015106893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7932d70 msgr2=0x5558d7933140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7932d70 0x5558d7933140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 -- 172.21.15.78:0/3015106893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7937ac0 msgr2=0x5558d7937f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7937ac0 0x5558d7937f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 -- 172.21.15.78:0/3015106893 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558d7703630 con 0x5558d7933680 2023-10-10T17:51:31.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbffff700 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7937ac0 0x5558d7937f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7933680 0x5558d7937580 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0005f50 tx=0x7f3cb0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ff4266d2c43ae3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3ca5ffb700 1 -- 172.21.15.78:0/3015106893 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb000bdf0 con 0x5558d7933680 2023-10-10T17:51:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3ca5ffb700 1 -- 172.21.15.78:0/3015106893 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3cb001e040 con 0x5558d7933680 2023-10-10T17:51:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.822+0000 7f3ca5ffb700 1 -- 172.21.15.78:0/3015106893 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0004760 con 0x5558d7933680 2023-10-10T17:51:31.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/3015106893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7933680 msgr2=0x5558d7937580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7933680 0x5558d7937580 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0005f50 tx=0x7f3cb0020000 comp rx=0 tx=0).stop 2023-10-10T17:51:31.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/3015106893 shutdown_connections 2023-10-10T17:51:31.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7932d70 0x5558d7933140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7937ac0 0x5558d7937f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/3015106893 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7933680 0x5558d7937580 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/3015106893 >> 172.21.15.78:0/3015106893 conn(0x5558d7926360 msgr2=0x5558d792bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/3015106893 shutdown_connections 2023-10-10T17:51:31.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/3015106893 wait complete. 2023-10-10T17:51:31.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.823+0000 7f3cd1c4bb80 1 -- start start 2023-10-10T17:51:31.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 1 Processor -- start 2023-10-10T17:51:31.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d793f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d793c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d793cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d78238e0 con 0x5558d7937ac0 2023-10-10T17:51:31.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7823580 con 0x5558d7932d70 2023-10-10T17:51:31.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7705920 con 0x5558d7933680 2023-10-10T17:51:31.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cd1c4bb80 10 client.0 _ll_register_callbacks cb 0x5558d7841780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:31.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d793cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d793cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48612/0 (socket says 172.21.15.78:48612) 2023-10-10T17:51:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 -- 172.21.15.78:0/2203472919 learned_addr learned my addr 172.21.15.78:0/2203472919 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d793f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d793c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 -- 172.21.15.78:0/2203472919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 msgr2=0x5558d793c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d793c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 -- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 msgr2=0x5558d793f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d793f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.824+0000 7f3cbffff700 1 -- 172.21.15.78:0/2203472919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cc010c900 con 0x5558d7937ac0 2023-10-10T17:51:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3cbffff700 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d793cc50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009fd0 tx=0x7f3cb8004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21724469243f7de server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3c9effd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:31.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3c9effd700 1 -- 172.21.15.78:0/2203472919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8007710 con 0x5558d7937ac0 2023-10-10T17:51:31.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3c9effd700 1 -- 172.21.15.78:0/2203472919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f3cb8007870 con 0x5558d7937ac0 2023-10-10T17:51:31.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558d7923b20 con 0x5558d7937ac0 2023-10-10T17:51:31.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3c9effd700 1 -- 172.21.15.78:0/2203472919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cb8007a30 con 0x5558d7937ac0 2023-10-10T17:51:31.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3cd1c4bb80 10 client.17334 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:31.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5558d7848190 con 0x5558d7937ac0 2023-10-10T17:51:31.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.825+0000 7f3c9effd700 1 -- 172.21.15.78:0/2203472919 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5558d7923b20 con 0x5558d7937ac0 2023-10-10T17:51:31.860 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3c9effd700 1 -- 172.21.15.78:0/2203472919 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3cb8014070 con 0x5558d7937ac0 2023-10-10T17:51:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cd1c4bb80 10 client.17334 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cd1c4bb80 10 client.17334 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:31.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cd1c4bb80 20 client.17334 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d794cb20 0x5558d794ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cd1c4bb80 4 client.17334 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:31.862 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558d78440a0 con 0x5558d794cb20 2023-10-10T17:51:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d794cb20 0x5558d794ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3c9effd700 10 client.17334 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:31.863 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.826+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d794cb20 0x5558d794ef30 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e213ad416b5cd6d server_cookie=bf83c84acb991851 in_seq=0 out_seq=0 2023-10-10T17:51:31.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3c9effd700 1 -- 172.21.15.78:0/2203472919 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5558d78440a0 con 0x5558d794cb20 2023-10-10T17:51:31.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3c9effd700 10 client.17334 handle_command_reply: tid=0 2023-10-10T17:51:31.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 client.17334 shutdown 2023-10-10T17:51:31.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 msgr2=0x5558d793cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d793cc50 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8009fd0 tx=0x7f3cb8004320 comp rx=0 tx=0).stop 2023-10-10T17:51:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 shutdown_connections 2023-10-10T17:51:31.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d794cb20 0x5558d794ef30 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d793cc50 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d793f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.866 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2203472919 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d793c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3c9effd700 0 client.17334 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 >> 172.21.15.78:0/2203472919 conn(0x5558d7926360 msgr2=0x5558d7938f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 shutdown_connections 2023-10-10T17:51:31.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2203472919 wait complete. 2023-10-10T17:51:31.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.827+0000 7f3cd1c4bb80 20 client.17334 trim_cache size 0 max 16384 2023-10-10T17:51:31.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.840+0000 7f3cd1c4bb80 1 Processor -- start 2023-10-10T17:51:31.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cd1c4bb80 1 -- start start 2023-10-10T17:51:31.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d792a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7948300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d793d490 con 0x5558d7937ac0 2023-10-10T17:51:31.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d794a0d0 con 0x5558d7932d70 2023-10-10T17:51:31.876 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7944090 con 0x5558d7933680 2023-10-10T17:51:31.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7948300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7948300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48620/0 (socket says 172.21.15.78:48620) 2023-10-10T17:51:31.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 -- 172.21.15.78:0/1234686107 learned_addr learned my addr 172.21.15.78:0/1234686107 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d792a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 -- 172.21.15.78:0/1234686107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 msgr2=0x5558d792a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d792a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 -- 172.21.15.78:0/1234686107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 msgr2=0x5558d7943b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 -- 172.21.15.78:0/1234686107 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558d7847ee0 con 0x5558d7937ac0 2023-10-10T17:51:31.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3cbffff700 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7948300 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8007fd0 tx=0x7f3cb8016ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c1f6133900932c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.841+0000 7f3ca67fc700 1 -- 172.21.15.78:0/1234686107 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb801d3a0 con 0x5558d7937ac0 2023-10-10T17:51:31.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3ca67fc700 1 -- 172.21.15.78:0/1234686107 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3cb800f070 con 0x5558d7937ac0 2023-10-10T17:51:31.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3ca67fc700 1 -- 172.21.15.78:0/1234686107 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8007710 con 0x5558d7937ac0 2023-10-10T17:51:31.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/1234686107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 msgr2=0x5558d7948300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7948300 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8007fd0 tx=0x7f3cb8016ec0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/1234686107 shutdown_connections 2023-10-10T17:51:31.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7948300 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/1234686107 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d792a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/1234686107 >> 172.21.15.78:0/1234686107 conn(0x5558d7926360 msgr2=0x5558d7939ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/1234686107 shutdown_connections 2023-10-10T17:51:31.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.842+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/1234686107 wait complete. 2023-10-10T17:51:31.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 -- start start 2023-10-10T17:51:31.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 Processor -- start 2023-10-10T17:51:31.884 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d7943840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7941b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7942100 con 0x5558d7937ac0 2023-10-10T17:51:31.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7942270 con 0x5558d7932d70 2023-10-10T17:51:31.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d7942430 con 0x5558d7933680 2023-10-10T17:51:31.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cd1c4bb80 10 client.0 _ll_register_callbacks cb 0x5558d7841780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:31.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cbffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7941b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cbffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7941b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48626/0 (socket says 172.21.15.78:48626) 2023-10-10T17:51:31.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cbffff700 1 -- 172.21.15.78:0/2482135343 learned_addr learned my addr 172.21.15.78:0/2482135343 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:31.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cbeffd700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d7943840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.843+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cbffff700 1 -- 172.21.15.78:0/2482135343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 msgr2=0x5558d7943840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cbffff700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d7943840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cbffff700 1 -- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 msgr2=0x5558d7943300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:31.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cbffff700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cbffff700 1 -- 172.21.15.78:0/2482135343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558d7848190 con 0x5558d7937ac0 2023-10-10T17:51:31.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:31.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cbffff700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7941b40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8015d10 tx=0x7f3cb8009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e29735b0e2ce56 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:31.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3c9effd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:31.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3c9effd700 1 -- 172.21.15.78:0/2482135343 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cb8015930 con 0x5558d7937ac0 2023-10-10T17:51:31.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3c9effd700 1 -- 172.21.15.78:0/2482135343 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f3cb8004940 con 0x5558d7937ac0 2023-10-10T17:51:31.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cb001e040 con 0x5558d7937ac0 2023-10-10T17:51:31.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3c9effd700 1 -- 172.21.15.78:0/2482135343 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cb8015a90 con 0x5558d7937ac0 2023-10-10T17:51:31.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 10 client.17340 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:31.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x5558d7698c60 con 0x5558d7937ac0 2023-10-10T17:51:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3c9effd700 1 -- 172.21.15.78:0/2482135343 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3cb001e040 con 0x5558d7937ac0 2023-10-10T17:51:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3c9effd700 1 -- 172.21.15.78:0/2482135343 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f3cb8014050 con 0x5558d7937ac0 2023-10-10T17:51:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 10 client.17340 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 10 client.17340 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 20 client.17340 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d79a1860 0x5558d79a3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 4 client.17340 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:51:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.844+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5558d7850660 con 0x5558d79a1860 2023-10-10T17:51:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d79a1860 0x5558d79a3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3c9effd700 10 client.17340 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cbf7fe700 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d79a1860 0x5558d79a3c70 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=164656d5eec4f45c server_cookie=8278863563863bb1 in_seq=0 out_seq=0 2023-10-10T17:51:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3c9effd700 1 -- 172.21.15.78:0/2482135343 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5558d7850660 con 0x5558d79a1860 2023-10-10T17:51:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3c9effd700 10 client.17340 handle_command_reply: tid=0 2023-10-10T17:51:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 client.17340 shutdown 2023-10-10T17:51:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 msgr2=0x5558d7941b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7941b40 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3cb8015d10 tx=0x7f3cb8009580 comp rx=0 tx=0).stop 2023-10-10T17:51:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 shutdown_connections 2023-10-10T17:51:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5558d79a1860 0x5558d79a3c70 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5558d7937ac0 0x5558d7941b40 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5558d7932d70 0x5558d7943300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 --2- 172.21.15.78:0/2482135343 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5558d7933680 0x5558d7943840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3c9effd700 0 client.17340 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.845+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 >> 172.21.15.78:0/2482135343 conn(0x5558d7926360 msgr2=0x5558d792a550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 shutdown_connections 2023-10-10T17:51:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cd1c4bb80 1 -- 172.21.15.78:0/2482135343 wait complete. 2023-10-10T17:51:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cd1c4bb80 20 client.17340 trim_cache size 0 max 16384 2023-10-10T17:51:31.903 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:51:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3ca80a12c0 msgr2=0x7f3ca80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3ca80a12c0 0x7f3ca80a3770 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0006010 tx=0x7f3cb0009f10 comp rx=0 tx=0).stop 2023-10-10T17:51:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100620 msgr2=0x7f3cc0195350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100620 0x7f3cc0195350 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f3cb400d950 tx=0x7f3cb400dc60 comp rx=0 tx=0).stop 2023-10-10T17:51:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 shutdown_connections 2023-10-10T17:51:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3ca80a12c0 0x7f3ca80a3770 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3cc0109bd0 0x7f3cc0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3cc0100620 0x7f3cc0195350 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 --2- 172.21.15.78:0/19164495 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3cc0100f30 0x7f3cc0195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 >> 172.21.15.78:0/19164495 conn(0x7f3cc0074b00 msgr2=0x7f3cc00fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 shutdown_connections 2023-10-10T17:51:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:31.846+0000 7f3cc5abf700 1 -- 172.21.15.78:0/19164495 wait complete. 2023-10-10T17:51:32.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:31 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3680882427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:51:32.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:31 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3680882427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:51:32.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:31 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3680882427' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:51:32.908 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:51:32.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:51:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 -- 172.21.15.78:0/137126914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17840ff990 msgr2=0x7f17840ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 --2- 172.21.15.78:0/137126914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17840ff990 0x7f17840ffe00 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f17740099c0 tx=0x7f1774009cd0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 -- 172.21.15.78:0/137126914 shutdown_connections 2023-10-10T17:51:33.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 --2- 172.21.15.78:0/137126914 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1784104180 0x7f1784104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 --2- 172.21.15.78:0/137126914 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1784100340 0x7f178410c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 --2- 172.21.15.78:0/137126914 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f17840ff990 0x7f17840ffe00 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 -- 172.21.15.78:0/137126914 >> 172.21.15.78:0/137126914 conn(0x7f1784075920 msgr2=0x7f1784075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 -- 172.21.15.78:0/137126914 shutdown_connections 2023-10-10T17:51:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.051+0000 7f178c464700 1 -- 172.21.15.78:0/137126914 wait complete. 2023-10-10T17:51:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 Processor -- start 2023-10-10T17:51:33.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 -- start start 2023-10-10T17:51:33.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17840ff990 0x7f1784100e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1784100340 0x7f17841013b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1784104180 0x7f17841018f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17841a4760 con 0x7f1784100340 2023-10-10T17:51:33.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17841a48d0 con 0x7f17840ff990 2023-10-10T17:51:33.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178c464700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17841a4a90 con 0x7f1784104180 2023-10-10T17:51:33.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1784100340 0x7f17841013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1784100340 0x7f17841013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48638/0 (socket says 172.21.15.78:48638) 2023-10-10T17:51:33.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 -- 172.21.15.78:0/2746578523 learned_addr learned my addr 172.21.15.78:0/2746578523 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178aa01700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1784104180 0x7f17841018f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f178a200700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17840ff990 0x7f1784100e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 -- 172.21.15.78:0/2746578523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1784104180 msgr2=0x7f17841018f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1784104180 0x7f17841018f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 -- 172.21.15.78:0/2746578523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17840ff990 msgr2=0x7f1784100e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17840ff990 0x7f1784100e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 -- 172.21.15.78:0/2746578523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17740095f0 con 0x7f1784100340 2023-10-10T17:51:33.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.052+0000 7f17899ff700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1784100340 0x7f17841013b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f1774009990 tx=0x7f1774009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.053+0000 7f177b7fe700 1 -- 172.21.15.78:0/2746578523 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177401d070 con 0x7f1784100340 2023-10-10T17:51:33.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.053+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178410f210 con 0x7f1784100340 2023-10-10T17:51:33.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.053+0000 7f177b7fe700 1 -- 172.21.15.78:0/2746578523 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1774003f90 con 0x7f1784100340 2023-10-10T17:51:33.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.053+0000 7f177b7fe700 1 -- 172.21.15.78:0/2746578523 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1774022ce0 con 0x7f1784100340 2023-10-10T17:51:33.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.053+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178410f080 con 0x7f1784100340 2023-10-10T17:51:33.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.055+0000 7f177b7fe700 1 -- 172.21.15.78:0/2746578523 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f1774022470 con 0x7f1784100340 2023-10-10T17:51:33.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.055+0000 7f177b7fe700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17700a12c0 0x7f17700a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.055+0000 7f178a200700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17700a12c0 0x7f17700a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.055+0000 7f177b7fe700 1 -- 172.21.15.78:0/2746578523 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f17740ce2b0 con 0x7f1784100340 2023-10-10T17:51:33.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f178a200700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17700a12c0 0x7f17700a3770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1780005950 tx=0x7f17800058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 Processor -- start 2023-10-10T17:51:33.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 -- start start 2023-10-10T17:51:33.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01abe0 0x55bcba01afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba01f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01f930 0x55bcba01fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb9f200e0 con 0x55bcba01f930 2023-10-10T17:51:33.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb9f72920 con 0x55bcba01abe0 2023-10-10T17:51:33.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.056+0000 7f17985f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb9f6bdb0 con 0x55bcba01b4f0 2023-10-10T17:51:33.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01f930 0x55bcba01fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01f930 0x55bcba01fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48648/0 (socket says 172.21.15.78:48648) 2023-10-10T17:51:33.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 -- 172.21.15.78:0/2513667060 learned_addr learned my addr 172.21.15.78:0/2513667060 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178a200700 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01abe0 0x55bcba01afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f17899ff700 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba01f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 -- 172.21.15.78:0/2513667060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 msgr2=0x55bcba01f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba01f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 -- 172.21.15.78:0/2513667060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01abe0 msgr2=0x55bcba01afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01abe0 0x55bcba01afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 -- 172.21.15.78:0/2513667060 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcb9ded6b0 con 0x55bcba01f930 2023-10-10T17:51:33.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f178aa01700 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01f930 0x55bcba01fd80 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f177c00f5a0 tx=0x7f177c00f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729d5edda90a8256 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f1778dac700 1 -- 172.21.15.78:0/2513667060 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c007e00 con 0x55bcba01f930 2023-10-10T17:51:33.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f1778dac700 1 -- 172.21.15.78:0/2513667060 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f177c004bb0 con 0x55bcba01f930 2023-10-10T17:51:33.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.057+0000 7f1778dac700 1 -- 172.21.15.78:0/2513667060 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c0051a0 con 0x55bcba01f930 2023-10-10T17:51:33.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 -- 172.21.15.78:0/2513667060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01f930 msgr2=0x55bcba01fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01f930 0x55bcba01fd80 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f177c00f5a0 tx=0x7f177c00f8b0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 -- 172.21.15.78:0/2513667060 shutdown_connections 2023-10-10T17:51:33.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01f930 0x55bcba01fd80 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01abe0 0x55bcba01afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2513667060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba01f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 -- 172.21.15.78:0/2513667060 >> 172.21.15.78:0/2513667060 conn(0x55bcba0103e0 msgr2=0x55bcba015bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 -- 172.21.15.78:0/2513667060 shutdown_connections 2023-10-10T17:51:33.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 -- 172.21.15.78:0/2513667060 wait complete. 2023-10-10T17:51:33.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 -- start start 2023-10-10T17:51:33.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.058+0000 7f17985f0b80 1 Processor -- start 2023-10-10T17:51:33.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba0270c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba0248e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb9f0d960 con 0x55bcba01abe0 2023-10-10T17:51:33.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb9f0d600 con 0x55bcba01b4f0 2023-10-10T17:51:33.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcb9def9a0 con 0x55bcba01f930 2023-10-10T17:51:33.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 10 client.0 _ll_register_callbacks cb 0x55bcb9f2b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:33.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba0270c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba0270c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48650/0 (socket says 172.21.15.78:48650) 2023-10-10T17:51:33.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 -- 172.21.15.78:0/443001613 learned_addr learned my addr 172.21.15.78:0/443001613 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178aa01700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba0248e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17899ff700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 -- 172.21.15.78:0/443001613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 msgr2=0x55bcba0248e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba0248e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 -- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 msgr2=0x55bcba0242d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 -- 172.21.15.78:0/443001613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c00f1d0 con 0x55bcba01abe0 2023-10-10T17:51:33.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178aa01700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba0248e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:33.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17899ff700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:33.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f178a200700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba0270c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1780011720 tx=0x7f1780004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d45d05dd7ce774d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f176dffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:33.086 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f176dffb700 1 -- 172.21.15.78:0/443001613 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f178001f830 con 0x55bcba01abe0 2023-10-10T17:51:33.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f176dffb700 1 -- 172.21.15.78:0/443001613 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f178001f990 con 0x55bcba01abe0 2023-10-10T17:51:33.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.059+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bcba00dba0 con 0x55bcba01abe0 2023-10-10T17:51:33.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f176dffb700 1 -- 172.21.15.78:0/443001613 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f178001fb50 con 0x55bcba01abe0 2023-10-10T17:51:33.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 10 client.17352 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:33.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55bcb9f32210 con 0x55bcba01abe0 2023-10-10T17:51:33.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f176dffb700 1 -- 172.21.15.78:0/443001613 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bcba00dba0 con 0x55bcba01abe0 2023-10-10T17:51:33.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f176dffb700 1 -- 172.21.15.78:0/443001613 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f17800273b0 con 0x55bcba01abe0 2023-10-10T17:51:33.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 10 client.17352 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:33.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 10 client.17352 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:33.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 20 client.17352 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:33.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba034940 0x55bcba036d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 4 client.17352 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:33.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcb9f2e120 con 0x55bcba034940 2023-10-10T17:51:33.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.060+0000 7f17899ff700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba034940 0x55bcba036d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.061+0000 7f17899ff700 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba034940 0x55bcba036d50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6d1c6bf715adeb2 server_cookie=d750d9081c7141a9 in_seq=0 out_seq=0 2023-10-10T17:51:33.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.061+0000 7f176dffb700 10 client.17352 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f176dffb700 1 -- 172.21.15.78:0/443001613 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bcb9f2e120 con 0x55bcba034940 2023-10-10T17:51:33.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f176dffb700 10 client.17352 handle_command_reply: tid=0 2023-10-10T17:51:33.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 client.17352 shutdown 2023-10-10T17:51:33.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 msgr2=0x55bcba0270c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba0270c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f1780011720 tx=0x7f1780004750 comp rx=0 tx=0).stop 2023-10-10T17:51:33.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 shutdown_connections 2023-10-10T17:51:33.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba0248e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba034940 0x55bcba036d50 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba0270c0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f176dffb700 0 client.17352 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 --2- 172.21.15.78:0/443001613 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 >> 172.21.15.78:0/443001613 conn(0x55bcba0103e0 msgr2=0x55bcba020de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 shutdown_connections 2023-10-10T17:51:33.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 1 -- 172.21.15.78:0/443001613 wait complete. 2023-10-10T17:51:33.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.062+0000 7f17985f0b80 20 client.17352 trim_cache size 0 max 16384 2023-10-10T17:51:33.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 Processor -- start 2023-10-10T17:51:33.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 -- start start 2023-10-10T17:51:33.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba015d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba02ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcba01e270 con 0x55bcba01abe0 2023-10-10T17:51:33.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcba00da20 con 0x55bcba01b4f0 2023-10-10T17:51:33.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f17985f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcba022880 con 0x55bcba01f930 2023-10-10T17:51:33.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba015d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba015d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48658/0 (socket says 172.21.15.78:48658) 2023-10-10T17:51:33.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 -- 172.21.15.78:0/4225670578 learned_addr learned my addr 172.21.15.78:0/4225670578 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 -- 172.21.15.78:0/4225670578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 msgr2=0x55bcba02ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 --2- 172.21.15.78:0/4225670578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba02ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 -- 172.21.15.78:0/4225670578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 msgr2=0x55bcba0242d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 --2- 172.21.15.78:0/4225670578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.079+0000 7f178a200700 1 -- 172.21.15.78:0/4225670578 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcb9f31f60 con 0x55bcba01abe0 2023-10-10T17:51:33.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f178a200700 1 --2- 172.21.15.78:0/4225670578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba015d60 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1780027740 tx=0x7f1780027940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=777e32911e5c93ba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17795ad700 1 -- 172.21.15.78:0/4225670578 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780009320 con 0x55bcba01abe0 2023-10-10T17:51:33.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17795ad700 1 -- 172.21.15.78:0/4225670578 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1780009770 con 0x55bcba01abe0 2023-10-10T17:51:33.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17795ad700 1 -- 172.21.15.78:0/4225670578 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17800052a0 con 0x55bcba01abe0 2023-10-10T17:51:33.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 -- 172.21.15.78:0/4225670578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 msgr2=0x55bcba015d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 --2- 172.21.15.78:0/4225670578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba015d60 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f1780027740 tx=0x7f1780027940 comp rx=0 tx=0).stop 2023-10-10T17:51:33.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 -- 172.21.15.78:0/4225670578 shutdown_connections 2023-10-10T17:51:33.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 --2- 172.21.15.78:0/4225670578 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01f930 0x55bcba02ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 --2- 172.21.15.78:0/4225670578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba015d60 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 --2- 172.21.15.78:0/4225670578 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01b4f0 0x55bcba0242d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 -- 172.21.15.78:0/4225670578 >> 172.21.15.78:0/4225670578 conn(0x55bcba0103e0 msgr2=0x55bcba023bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 -- 172.21.15.78:0/4225670578 shutdown_connections 2023-10-10T17:51:33.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.080+0000 7f17985f0b80 1 -- 172.21.15.78:0/4225670578 wait complete. 2023-10-10T17:51:33.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 -- start start 2023-10-10T17:51:33.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 Processor -- start 2023-10-10T17:51:33.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba02bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba02a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01f930 0x55bcba02a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcba02ae90 con 0x55bcba01abe0 2023-10-10T17:51:33.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcba088810 con 0x55bcba01f930 2023-10-10T17:51:33.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcba088950 con 0x55bcba01b4f0 2023-10-10T17:51:33.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17985f0b80 10 client.0 _ll_register_callbacks cb 0x7f178000edc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:33.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba02bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba02bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48664/0 (socket says 172.21.15.78:48664) 2023-10-10T17:51:33.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 -- 172.21.15.78:0/2608116383 learned_addr learned my addr 172.21.15.78:0/2608116383 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17899ff700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba02a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178aa01700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01f930 0x55bcba02a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 -- 172.21.15.78:0/2608116383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 msgr2=0x55bcba02a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba02a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 -- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01f930 msgr2=0x55bcba02a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01f930 0x55bcba02a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178a200700 1 -- 172.21.15.78:0/2608116383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcb9f32210 con 0x55bcba01abe0 2023-10-10T17:51:33.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f17899ff700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba02a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:51:33.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.081+0000 7f178aa01700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01f930 0x55bcba02a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:33.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f178a200700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba02bc20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1780027800 tx=0x7f1780004f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d481f729ef1b267 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f176dffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:33.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f176dffb700 1 -- 172.21.15.78:0/2608116383 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780004930 con 0x55bcba01abe0 2023-10-10T17:51:33.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f176dffb700 1 -- 172.21.15.78:0/2608116383 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f1780004a90 con 0x55bcba01abe0 2023-10-10T17:51:33.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f177c004bb0 con 0x55bcba01abe0 2023-10-10T17:51:33.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f176dffb700 1 -- 172.21.15.78:0/2608116383 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1780004c50 con 0x55bcba01abe0 2023-10-10T17:51:33.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 10 client.17358 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:33.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55bcb9d82ce0 con 0x55bcba01abe0 2023-10-10T17:51:33.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f176dffb700 1 -- 172.21.15.78:0/2608116383 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f177c004bb0 con 0x55bcba01abe0 2023-10-10T17:51:33.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f176dffb700 1 -- 172.21.15.78:0/2608116383 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f178001fa50 con 0x55bcba01abe0 2023-10-10T17:51:33.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 10 client.17358 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:33.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 10 client.17358 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:33.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 20 client.17358 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:33.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba089bd0 0x55bcba08bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 4 client.17358 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:51:33.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55bcb9f3a6e0 con 0x55bcba089bd0 2023-10-10T17:51:33.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.082+0000 7f17899ff700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba089bd0 0x55bcba08bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.083+0000 7f17899ff700 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba089bd0 0x55bcba08bfb0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=722699a1bfc80c2a server_cookie=80595323cf5f81f5 in_seq=0 out_seq=0 2023-10-10T17:51:33.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.083+0000 7f176dffb700 10 client.17358 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f176dffb700 1 -- 172.21.15.78:0/2608116383 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bcb9f3a6e0 con 0x55bcba089bd0 2023-10-10T17:51:33.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f176dffb700 10 client.17358 handle_command_reply: tid=0 2023-10-10T17:51:33.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 client.17358 shutdown 2023-10-10T17:51:33.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 msgr2=0x55bcba02bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba02bc20 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f1780027800 tx=0x7f1780004f10 comp rx=0 tx=0).stop 2023-10-10T17:51:33.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 shutdown_connections 2023-10-10T17:51:33.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55bcba089bd0 0x55bcba08bfb0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bcba01f930 0x55bcba02a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bcba01abe0 0x55bcba02bc20 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 --2- 172.21.15.78:0/2608116383 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bcba01b4f0 0x55bcba02a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 >> 172.21.15.78:0/2608116383 conn(0x55bcba0103e0 msgr2=0x55bcba01f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f176dffb700 0 client.17358 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 shutdown_connections 2023-10-10T17:51:33.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 1 -- 172.21.15.78:0/2608116383 wait complete. 2023-10-10T17:51:33.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.084+0000 7f17985f0b80 20 client.17358 trim_cache size 0 max 16384 2023-10-10T17:51:33.127 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:51:33.127 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:51:33.127 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "8e09704c-9e26-44e1-bd30-70ebabda9868", 2023-10-10T17:51:33.128 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:51:33.128 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:51:33.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17700a12c0 msgr2=0x7f17700a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17700a12c0 0x7f17700a3770 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1780005950 tx=0x7f17800058e0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1784100340 msgr2=0x7f17841013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1784100340 0x7f17841013b0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f1774009990 tx=0x7f1774009580 comp rx=0 tx=0).stop 2023-10-10T17:51:33.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 shutdown_connections 2023-10-10T17:51:33.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1784104180 0x7f17841018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f17700a12c0 0x7f17700a3770 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f17840ff990 0x7f1784100e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 --2- 172.21.15.78:0/2746578523 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1784100340 0x7f17841013b0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 >> 172.21.15.78:0/2746578523 conn(0x7f1784075920 msgr2=0x7f17840fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 shutdown_connections 2023-10-10T17:51:33.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.085+0000 7f178c464700 1 -- 172.21.15.78:0/2746578523 wait complete. 2023-10-10T17:51:33.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:32 smithi078 ceph-mon[99494]: pgmap v279: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:33.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:51:33.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:32 smithi190 ceph-mon[101220]: pgmap v279: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:33.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:32 smithi157 ceph-mon[100896]: pgmap v279: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:33.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 -- 172.21.15.78:0/1661982429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 msgr2=0x7f4488104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 --2- 172.21.15.78:0/1661982429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488104d30 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f447000aaa0 tx=0x7f447000adb0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 -- 172.21.15.78:0/1661982429 shutdown_connections 2023-10-10T17:51:33.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 --2- 172.21.15.78:0/1661982429 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44880ff300 0x7f448810b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 --2- 172.21.15.78:0/1661982429 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488104d30 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 --2- 172.21.15.78:0/1661982429 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44880fe950 0x7f44880fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 -- 172.21.15.78:0/1661982429 >> 172.21.15.78:0/1661982429 conn(0x7f4488075920 msgr2=0x7f4488075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 -- 172.21.15.78:0/1661982429 shutdown_connections 2023-10-10T17:51:33.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.257+0000 7f448db60700 1 -- 172.21.15.78:0/1661982429 wait complete. 2023-10-10T17:51:33.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 Processor -- start 2023-10-10T17:51:33.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 -- start start 2023-10-10T17:51:33.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44880fe950 0x7f44881941f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44880ff300 0x7f4488194730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44881991d0 con 0x7f4488104960 2023-10-10T17:51:33.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488199340 con 0x7f44880fe950 2023-10-10T17:51:33.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f448db60700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488199500 con 0x7f44880ff300 2023-10-10T17:51:33.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f4487fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f4487fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488198b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48682/0 (socket says 172.21.15.78:48682) 2023-10-10T17:51:33.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f4487fff700 1 -- 172.21.15.78:0/1031249501 learned_addr learned my addr 172.21.15.78:0/1031249501 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f4486ffd700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44880ff300 0x7f4488194730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.258+0000 7f44877fe700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44880fe950 0x7f44881941f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4487fff700 1 -- 172.21.15.78:0/1031249501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44880ff300 msgr2=0x7f4488194730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4487fff700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44880ff300 0x7f4488194730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4487fff700 1 -- 172.21.15.78:0/1031249501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44880fe950 msgr2=0x7f44881941f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4487fff700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44880fe950 0x7f44881941f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4487fff700 1 -- 172.21.15.78:0/1031249501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447000a700 con 0x7f4488104960 2023-10-10T17:51:33.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4487fff700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488198b90 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f448000ab50 tx=0x7f448000ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4484ff9700 1 -- 172.21.15.78:0/1031249501 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448000f070 con 0x7f4488104960 2023-10-10T17:51:33.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448810e120 con 0x7f4488104960 2023-10-10T17:51:33.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4484ff9700 1 -- 172.21.15.78:0/1031249501 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4480007a80 con 0x7f4488104960 2023-10-10T17:51:33.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f4484ff9700 1 -- 172.21.15.78:0/1031249501 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480005280 con 0x7f4488104960 2023-10-10T17:51:33.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.259+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448810df90 con 0x7f4488104960 2023-10-10T17:51:33.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.261+0000 7f4484ff9700 1 -- 172.21.15.78:0/1031249501 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f448000c620 con 0x7f4488104960 2023-10-10T17:51:33.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.262+0000 7f4484ff9700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44740a12c0 0x7f44740a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.262+0000 7f44877fe700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44740a12c0 0x7f44740a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.262+0000 7f4484ff9700 1 -- 172.21.15.78:0/1031249501 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f44800cc460 con 0x7f4488104960 2023-10-10T17:51:33.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.262+0000 7f44877fe700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44740a12c0 0x7f44740a3770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f447000c6d0 tx=0x7f4470005fd0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.262+0000 7f4499cecb80 1 Processor -- start 2023-10-10T17:51:33.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4499cecb80 1 -- start start 2023-10-10T17:51:33.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a26b60 0x55ccb6a26f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a2b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a2bd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4499cecb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb692c060 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4499cecb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb697e8a0 con 0x55ccb6a27470 2023-10-10T17:51:33.281 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4499cecb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6977d30 con 0x55ccb6a26b60 2023-10-10T17:51:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a2bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a2bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48690/0 (socket says 172.21.15.78:48690) 2023-10-10T17:51:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 -- 172.21.15.78:0/3819026451 learned_addr learned my addr 172.21.15.78:0/3819026451 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f44877fe700 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a26b60 0x55ccb6a26f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4486ffd700 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a2b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 -- 172.21.15.78:0/3819026451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a26b60 msgr2=0x55ccb6a26f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a26b60 0x55ccb6a26f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 -- 172.21.15.78:0/3819026451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 msgr2=0x55ccb6a2b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a2b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 -- 172.21.15.78:0/3819026451 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccb67f9630 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4486ffd700 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a2b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.263+0000 7f4487fff700 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a2bd00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4480095d40 tx=0x7f4480098f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73bf5d9c623a671 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f447dffb700 1 -- 172.21.15.78:0/3819026451 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800a1690 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f447dffb700 1 -- 172.21.15.78:0/3819026451 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4480004dd0 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f447dffb700 1 -- 172.21.15.78:0/3819026451 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448000c8d0 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 -- 172.21.15.78:0/3819026451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a2b8b0 msgr2=0x55ccb6a2bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a2bd00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f4480095d40 tx=0x7f4480098f20 comp rx=0 tx=0).stop 2023-10-10T17:51:33.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 -- 172.21.15.78:0/3819026451 shutdown_connections 2023-10-10T17:51:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a26b60 0x55ccb6a26f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a2bd00 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 --2- 172.21.15.78:0/3819026451 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a2b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 -- 172.21.15.78:0/3819026451 >> 172.21.15.78:0/3819026451 conn(0x55ccb6a1c360 msgr2=0x55ccb6a21b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 -- 172.21.15.78:0/3819026451 shutdown_connections 2023-10-10T17:51:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.264+0000 7f4499cecb80 1 -- 172.21.15.78:0/3819026451 wait complete. 2023-10-10T17:51:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 -- start start 2023-10-10T17:51:33.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 Processor -- start 2023-10-10T17:51:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a330c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb69198e0 con 0x55ccb6a26b60 2023-10-10T17:51:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6919580 con 0x55ccb6a27470 2023-10-10T17:51:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb67fb920 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4499cecb80 10 client.0 _ll_register_callbacks cb 0x55ccb6937780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a330c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a330c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48692/0 (socket says 172.21.15.78:48692) 2023-10-10T17:51:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 -- 172.21.15.78:0/4105871515 learned_addr learned my addr 172.21.15.78:0/4105871515 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4487fff700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f4486ffd700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 -- 172.21.15.78:0/4105871515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 msgr2=0x55ccb6a30aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 -- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 msgr2=0x55ccb6a303b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a303b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.265+0000 7f44877fe700 1 -- 172.21.15.78:0/4105871515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448810df90 con 0x55ccb6a26b60 2023-10-10T17:51:33.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4486ffd700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:33.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f44877fe700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a330c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4470016000 tx=0x7f447000a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46216568dc8d183d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4466ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:33.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4466ffd700 1 -- 172.21.15.78:0/4105871515 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4470021910 con 0x55ccb6a26b60 2023-10-10T17:51:33.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4466ffd700 1 -- 172.21.15.78:0/4105871515 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f4470029040 con 0x55ccb6a26b60 2023-10-10T17:51:33.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ccb6a19b20 con 0x55ccb6a26b60 2023-10-10T17:51:33.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4466ffd700 1 -- 172.21.15.78:0/4105871515 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4470021a70 con 0x55ccb6a26b60 2023-10-10T17:51:33.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 10 client.17370 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:33.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ccb693e190 con 0x55ccb6a26b60 2023-10-10T17:51:33.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4466ffd700 1 -- 172.21.15.78:0/4105871515 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ccb6a19b20 con 0x55ccb6a26b60 2023-10-10T17:51:33.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4466ffd700 1 -- 172.21.15.78:0/4105871515 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f447000eca0 con 0x55ccb6a26b60 2023-10-10T17:51:33.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 10 client.17370 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:33.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:33.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 20 client.17370 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:33.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a40960 0x55ccb6a42d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 4 client.17370 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:51:33.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.266+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ccb693a0a0 con 0x55ccb6a40960 2023-10-10T17:51:33.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.267+0000 7f4486ffd700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a40960 0x55ccb6a42d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.267+0000 7f4486ffd700 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a40960 0x55ccb6a42d70 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de3feafa73bb905f server_cookie=4171db712e358b64 in_seq=0 out_seq=0 2023-10-10T17:51:33.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.267+0000 7f4466ffd700 10 client.17370 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4466ffd700 1 -- 172.21.15.78:0/4105871515 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55ccb693a0a0 con 0x55ccb6a40960 2023-10-10T17:51:33.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4466ffd700 10 client.17370 handle_command_reply: tid=0 2023-10-10T17:51:33.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 client.17370 shutdown 2023-10-10T17:51:33.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 msgr2=0x55ccb6a330c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a330c0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f4470016000 tx=0x7f447000a420 comp rx=0 tx=0).stop 2023-10-10T17:51:33.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 shutdown_connections 2023-10-10T17:51:33.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a40960 0x55ccb6a42d70 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a330c0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 --2- 172.21.15.78:0/4105871515 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a303b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4466ffd700 0 client.17370 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 >> 172.21.15.78:0/4105871515 conn(0x55ccb6a1c360 msgr2=0x55ccb6a2cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 shutdown_connections 2023-10-10T17:51:33.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 1 -- 172.21.15.78:0/4105871515 wait complete. 2023-10-10T17:51:33.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.268+0000 7f4499cecb80 20 client.17370 trim_cache size 0 max 16384 2023-10-10T17:51:33.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 Processor -- start 2023-10-10T17:51:33.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 -- start start 2023-10-10T17:51:33.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a32c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a27470 0x55ccb6a2fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6a31920 con 0x55ccb6a26b60 2023-10-10T17:51:33.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6a2a540 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4499cecb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6a27900 con 0x55ccb6a27470 2023-10-10T17:51:33.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f44877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a32c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f44877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a32c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48700/0 (socket says 172.21.15.78:48700) 2023-10-10T17:51:33.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f44877fe700 1 -- 172.21.15.78:0/481808049 learned_addr learned my addr 172.21.15.78:0/481808049 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4486ffd700 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a27470 0x55ccb6a2fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f44877fe700 1 -- 172.21.15.78:0/481808049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a27470 msgr2=0x55ccb6a2fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f44877fe700 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a27470 0x55ccb6a2fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f44877fe700 1 -- 172.21.15.78:0/481808049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a2b8b0 msgr2=0x55ccb6a30140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f4487fff700 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.285+0000 7f44877fe700 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f44877fe700 1 -- 172.21.15.78:0/481808049 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ccb693dee0 con 0x55ccb6a26b60 2023-10-10T17:51:33.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4486ffd700 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a27470 0x55ccb6a2fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:33.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4487fff700 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:51:33.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f44877fe700 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a32c60 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4470005c90 tx=0x7f4470005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504dfbd01e679318 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f447e7fc700 1 -- 172.21.15.78:0/481808049 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447002e980 con 0x55ccb6a26b60 2023-10-10T17:51:33.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f447e7fc700 1 -- 172.21.15.78:0/481808049 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4470021710 con 0x55ccb6a26b60 2023-10-10T17:51:33.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f447e7fc700 1 -- 172.21.15.78:0/481808049 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447000e690 con 0x55ccb6a26b60 2023-10-10T17:51:33.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 -- 172.21.15.78:0/481808049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 msgr2=0x55ccb6a32c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a32c60 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4470005c90 tx=0x7f4470005900 comp rx=0 tx=0).stop 2023-10-10T17:51:33.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 -- 172.21.15.78:0/481808049 shutdown_connections 2023-10-10T17:51:33.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a30140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a32c60 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 --2- 172.21.15.78:0/481808049 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a27470 0x55ccb6a2fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 -- 172.21.15.78:0/481808049 >> 172.21.15.78:0/481808049 conn(0x55ccb6a1c360 msgr2=0x55ccb6a299d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 -- 172.21.15.78:0/481808049 shutdown_connections 2023-10-10T17:51:33.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.286+0000 7f4499cecb80 1 -- 172.21.15.78:0/481808049 wait complete. 2023-10-10T17:51:33.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 -- start start 2023-10-10T17:51:33.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 Processor -- start 2023-10-10T17:51:33.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a35b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a8d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6a8ddc0 con 0x55ccb6a26b60 2023-10-10T17:51:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6a8df30 con 0x55ccb6a27470 2023-10-10T17:51:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ccb6a8e0f0 con 0x55ccb6a2b8b0 2023-10-10T17:51:33.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4499cecb80 10 client.0 _ll_register_callbacks cb 0x7f4470003a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:51:33.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f44877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a35b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f44877fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a35b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48706/0 (socket says 172.21.15.78:48706) 2023-10-10T17:51:33.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f44877fe700 1 -- 172.21.15.78:0/2888795083 learned_addr learned my addr 172.21.15.78:0/2888795083 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:51:33.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4486ffd700 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.287+0000 7f4487fff700 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f44877fe700 1 -- 172.21.15.78:0/2888795083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 msgr2=0x55ccb6a8d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f44877fe700 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a8d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f44877fe700 1 -- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 msgr2=0x55ccb6a360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:51:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f44877fe700 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f44877fe700 1 -- 172.21.15.78:0/2888795083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ccb693e190 con 0x55ccb6a26b60 2023-10-10T17:51:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f44877fe700 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a35b80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4470005990 tx=0x7f447000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5deec09653c831d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:51:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4466ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:51:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4466ffd700 1 -- 172.21.15.78:0/2888795083 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447002e7f0 con 0x55ccb6a26b60 2023-10-10T17:51:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4466ffd700 1 -- 172.21.15.78:0/2888795083 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f4470012070 con 0x55ccb6a26b60 2023-10-10T17:51:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4480004dd0 con 0x55ccb6a26b60 2023-10-10T17:51:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4466ffd700 1 -- 172.21.15.78:0/2888795083 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4470021c10 con 0x55ccb6a26b60 2023-10-10T17:51:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 10 client.17376 fetch_fsmap learned FSMap version 32 2023-10-10T17:51:33.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x55ccb678ec60 con 0x55ccb6a26b60 2023-10-10T17:51:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4466ffd700 1 -- 172.21.15.78:0/2888795083 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4480004dd0 con 0x55ccb6a26b60 2023-10-10T17:51:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4466ffd700 1 -- 172.21.15.78:0/2888795083 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 32) v1 ==== 7494+0+0 (secure 0 0 0) 0x7f447000ec20 con 0x55ccb6a26b60 2023-10-10T17:51:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 10 client.17376 fetch_fsmap finished waiting for FSMap version 32 2023-10-10T17:51:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 10 client.17376 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:51:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 20 client.17376 populate_metadata read hostname 'smithi078' 2023-10-10T17:51:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a95300 0x55ccb6a97710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:51:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 4 client.17376 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:51:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.288+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ccb6946660 con 0x55ccb6a95300 2023-10-10T17:51:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4486ffd700 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a95300 0x55ccb6a97710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:51:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4466ffd700 10 client.17376 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4486ffd700 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a95300 0x55ccb6a97710 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cc1c22bf661110e server_cookie=699c96362822c22f in_seq=0 out_seq=0 2023-10-10T17:51:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4466ffd700 1 -- 172.21.15.78:0/2888795083 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ccb6946660 con 0x55ccb6a95300 2023-10-10T17:51:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4466ffd700 10 client.17376 handle_command_reply: tid=0 2023-10-10T17:51:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 client.17376 shutdown 2023-10-10T17:51:33.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 msgr2=0x55ccb6a35b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a35b80 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4470005990 tx=0x7f447000cc20 comp rx=0 tx=0).stop 2023-10-10T17:51:33.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 shutdown_connections 2023-10-10T17:51:33.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55ccb6a2b8b0 0x55ccb6a8d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55ccb6a95300 0x55ccb6a97710 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55ccb6a26b60 0x55ccb6a35b80 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 --2- 172.21.15.78:0/2888795083 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55ccb6a27470 0x55ccb6a360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4466ffd700 0 client.17376 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:51:33.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.289+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 >> 172.21.15.78:0/2888795083 conn(0x55ccb6a1c360 msgr2=0x55ccb6a2a950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 shutdown_connections 2023-10-10T17:51:33.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f4499cecb80 1 -- 172.21.15.78:0/2888795083 wait complete. 2023-10-10T17:51:33.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f4499cecb80 20 client.17376 trim_cache size 0 max 16384 2023-10-10T17:51:33.333 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:51:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:51:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:51:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "8e09704c-9e26-44e1-bd30-70ebabda9868": { 2023-10-10T17:51:33.334 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:51:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "8e09704c-9e26-44e1-bd30-70ebabda9868", 2023-10-10T17:51:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:51:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:51:33.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:51:33.336 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:51:33.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44740a12c0 msgr2=0x7f44740a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44740a12c0 0x7f44740a3770 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f447000c6d0 tx=0x7f4470005fd0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 msgr2=0x7f4488198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:51:33.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488198b90 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f448000ab50 tx=0x7f448000ae60 comp rx=0 tx=0).stop 2023-10-10T17:51:33.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 shutdown_connections 2023-10-10T17:51:33.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f44740a12c0 0x7f44740a3770 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4488104960 0x7f4488198b90 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f44880fe950 0x7f44881941f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 --2- 172.21.15.78:0/1031249501 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f44880ff300 0x7f4488194730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:51:33.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 >> 172.21.15.78:0/1031249501 conn(0x7f4488075920 msgr2=0x7f44880fcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:51:33.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 shutdown_connections 2023-10-10T17:51:33.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:51:33.290+0000 7f448db60700 1 -- 172.21.15.78:0/1031249501 wait complete. 2023-10-10T17:51:33.341 INFO:tasks.cephfs.filesystem:scrub status for tag:8e09704c-9e26-44e1-bd30-70ebabda9868 - {'path': '/', 'tag': '8e09704c-9e26-44e1-bd30-70ebabda9868', 'options': 'recursive,force'} 2023-10-10T17:51:34.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:33 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:51:34.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:33 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:34.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:33 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:51:34.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:33 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:34.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:33 smithi078 ceph-mon[99494]: pgmap v280: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:34.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:33 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:51:34.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:33 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:34.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:33 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:51:34.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:33 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:34.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:33 smithi190 ceph-mon[101220]: pgmap v280: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:34.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:33 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:51:34.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:33 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:34.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:33 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:51:34.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:33 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:51:34.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:33 smithi157 ceph-mon[100896]: pgmap v280: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:36.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:36 smithi157 ceph-mon[100896]: pgmap v281: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:36.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:36 smithi078 ceph-mon[99494]: pgmap v281: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:36.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:36 smithi190 ceph-mon[101220]: pgmap v281: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:37.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:37 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:51:37.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:37 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:51:37.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:37 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:51:38.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:38 smithi157 ceph-mon[100896]: pgmap v282: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-10T17:51:38.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:38 smithi078 ceph-mon[99494]: pgmap v282: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-10T17:51:38.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:38 smithi190 ceph-mon[101220]: pgmap v282: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2023-10-10T17:51:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:40 smithi078 ceph-mon[99494]: pgmap v283: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-10-10T17:51:41.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:40 smithi190 ceph-mon[101220]: pgmap v283: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-10-10T17:51:41.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:40 smithi157 ceph-mon[100896]: pgmap v283: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2023-10-10T17:51:43.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:42 smithi078 ceph-mon[99494]: pgmap v284: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-10T17:51:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:42 smithi190 ceph-mon[101220]: pgmap v284: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-10T17:51:43.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:42 smithi157 ceph-mon[100896]: pgmap v284: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 6 op/s 2023-10-10T17:51:45.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:44 smithi078 ceph-mon[99494]: pgmap v285: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-10T17:51:45.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:44 smithi190 ceph-mon[101220]: pgmap v285: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-10T17:51:45.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:44 smithi157 ceph-mon[100896]: pgmap v285: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2023-10-10T17:51:47.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:46 smithi078 ceph-mon[99494]: pgmap v286: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:47.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:46 smithi190 ceph-mon[101220]: pgmap v286: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:47.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:46 smithi157 ceph-mon[100896]: pgmap v286: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:51:48.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:47 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:51:48.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:47 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:51:48.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:47 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:51:49.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:48 smithi078 ceph-mon[99494]: pgmap v287: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:51:49.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:48 smithi190 ceph-mon[101220]: pgmap v287: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:51:49.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:48 smithi157 ceph-mon[100896]: pgmap v287: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 7 op/s 2023-10-10T17:51:51.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:50 smithi078 ceph-mon[99494]: pgmap v288: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:50 smithi190 ceph-mon[101220]: pgmap v288: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:51.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:50 smithi157 ceph-mon[100896]: pgmap v288: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:53.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:52 smithi078 ceph-mon[99494]: pgmap v289: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:51:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:52 smithi190 ceph-mon[101220]: pgmap v289: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:51:53.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:52 smithi157 ceph-mon[100896]: pgmap v289: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:51:55.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:54 smithi078 ceph-mon[99494]: pgmap v290: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:51:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:54 smithi190 ceph-mon[101220]: pgmap v290: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:51:55.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:54 smithi157 ceph-mon[100896]: pgmap v290: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-10-10T17:51:57.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:56 smithi078 ceph-mon[99494]: pgmap v291: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:57.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:56 smithi190 ceph-mon[101220]: pgmap v291: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:57.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:56 smithi157 ceph-mon[100896]: pgmap v291: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:51:59.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:58 smithi078 ceph-mon[99494]: pgmap v292: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:51:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:58 smithi190 ceph-mon[101220]: pgmap v292: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:51:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:58 smithi157 ceph-mon[100896]: pgmap v292: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:00.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:51:59 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:52:00.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:51:59 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:52:00.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:51:59 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:52:01.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:00 smithi078 ceph-mon[99494]: pgmap v293: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:01.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:00 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:01.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:00 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:52:01.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:00 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:52:01.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:00 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:52:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:00 smithi190 ceph-mon[101220]: pgmap v293: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:00 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:00 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:52:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:00 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:52:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:00 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:52:01.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:00 smithi157 ceph-mon[100896]: pgmap v293: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:00 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:00 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:52:01.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:00 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:52:01.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:00 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:52:02.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:01 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:52:02.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:01 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:52:02.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:01 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:52:02.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:01 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:52:02.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:01 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:01 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:01 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:52:02.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:01 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:52:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:01 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:52:02.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:01 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:52:02.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:01 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:52:02.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:01 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:52:03.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:02 smithi078 ceph-mon[99494]: pgmap v294: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:03.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:02 smithi190 ceph-mon[101220]: pgmap v294: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:03.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:02 smithi157 ceph-mon[100896]: pgmap v294: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:03.342 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:52:03.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 -- 172.21.15.78:0/4078223498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 msgr2=0x7fcad00ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 --2- 172.21.15.78:0/4078223498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad00ff820 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fcacc0099c0 tx=0x7fcacc009cd0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 -- 172.21.15.78:0/4078223498 shutdown_connections 2023-10-10T17:52:03.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 --2- 172.21.15.78:0/4078223498 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcad0104490 0x7fcad010c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 --2- 172.21.15.78:0/4078223498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad00ff820 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 --2- 172.21.15.78:0/4078223498 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcad00ffdf0 0x7fcad0103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 -- 172.21.15.78:0/4078223498 >> 172.21.15.78:0/4078223498 conn(0x7fcad00770b0 msgr2=0x7fcad00774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 -- 172.21.15.78:0/4078223498 shutdown_connections 2023-10-10T17:52:03.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.481+0000 7fcad8679700 1 -- 172.21.15.78:0/4078223498 wait complete. 2023-10-10T17:52:03.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 Processor -- start 2023-10-10T17:52:03.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 -- start start 2023-10-10T17:52:03.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad01952d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcad00ffdf0 0x7fcad0195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcad0104490 0x7fcad0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad019a2b0 con 0x7fcad00ff450 2023-10-10T17:52:03.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad019a420 con 0x7fcad00ffdf0 2023-10-10T17:52:03.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad8679700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad019a5e0 con 0x7fcad0104490 2023-10-10T17:52:03.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad6415700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad01952d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad6415700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad01952d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48720/0 (socket says 172.21.15.78:48720) 2023-10-10T17:52:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad6415700 1 -- 172.21.15.78:0/2410296099 learned_addr learned my addr 172.21.15.78:0/2410296099 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad5c14700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcad00ffdf0 0x7fcad0195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad6c16700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcad0104490 0x7fcad0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad6415700 1 -- 172.21.15.78:0/2410296099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcad0104490 msgr2=0x7fcad0199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.482+0000 7fcad6415700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcad0104490 0x7fcad0199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcad6415700 1 -- 172.21.15.78:0/2410296099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcad00ffdf0 msgr2=0x7fcad0195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcad6415700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcad00ffdf0 0x7fcad0195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcad6415700 1 -- 172.21.15.78:0/2410296099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc0095f0 con 0x7fcad00ff450 2023-10-10T17:52:03.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcad6c16700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcad0104490 0x7fcad0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:03.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcad6415700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad01952d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00b580 tx=0x7fcacc00bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcac77fe700 1 -- 172.21.15.78:0/2410296099 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc01d070 con 0x7fcad00ff450 2023-10-10T17:52:03.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad010f250 con 0x7fcad00ff450 2023-10-10T17:52:03.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcac77fe700 1 -- 172.21.15.78:0/2410296099 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcacc022470 con 0x7fcad00ff450 2023-10-10T17:52:03.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcac77fe700 1 -- 172.21.15.78:0/2410296099 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc017650 con 0x7fcad00ff450 2023-10-10T17:52:03.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.483+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad010f0c0 con 0x7fcad00ff450 2023-10-10T17:52:03.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.485+0000 7fcac77fe700 1 -- 172.21.15.78:0/2410296099 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fcacc0225e0 con 0x7fcad00ff450 2023-10-10T17:52:03.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.485+0000 7fcac77fe700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcabc0a12c0 0x7fcabc0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.485+0000 7fcad5c14700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcabc0a12c0 0x7fcabc0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.486+0000 7fcac77fe700 1 -- 172.21.15.78:0/2410296099 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fcacc0ce040 con 0x7fcad00ff450 2023-10-10T17:52:03.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.486+0000 7fcad5c14700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcabc0a12c0 0x7fcabc0a3770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcad01968e0 tx=0x7fcac0009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.486+0000 7fcae4805b80 1 Processor -- start 2023-10-10T17:52:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcae4805b80 1 -- start start 2023-10-10T17:52:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 0x55a124a8ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a90470 0x55a124a94370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a94d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcae4805b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124995060 con 0x55a124a90470 2023-10-10T17:52:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcae4805b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1249e78a0 con 0x55a124a8fb60 2023-10-10T17:52:03.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcae4805b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1249e0d30 con 0x55a124a948b0 2023-10-10T17:52:03.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a90470 0x55a124a94370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a90470 0x55a124a94370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48728/0 (socket says 172.21.15.78:48728) 2023-10-10T17:52:03.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 -- 172.21.15.78:0/1687411267 learned_addr learned my addr 172.21.15.78:0/1687411267 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:03.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad6c16700 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a94d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 -- 172.21.15.78:0/1687411267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 msgr2=0x55a124a94d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a94d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 -- 172.21.15.78:0/1687411267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 msgr2=0x55a124a8ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad6415700 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 0x55a124a8ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 0x55a124a8ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 -- 172.21.15.78:0/1687411267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a124862630 con 0x55a124a90470 2023-10-10T17:52:03.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcad5c14700 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a90470 0x55a124a94370 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fcac0011cf0 tx=0x7fcac0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e598cee5917998 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.487+0000 7fcac4ff9700 1 -- 172.21.15.78:0/1687411267 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac000eaf0 con 0x55a124a90470 2023-10-10T17:52:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcac4ff9700 1 -- 172.21.15.78:0/1687411267 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcac0021070 con 0x55a124a90470 2023-10-10T17:52:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcac4ff9700 1 -- 172.21.15.78:0/1687411267 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac001d670 con 0x55a124a90470 2023-10-10T17:52:03.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 -- 172.21.15.78:0/1687411267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a90470 msgr2=0x55a124a94370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a90470 0x55a124a94370 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fcac0011cf0 tx=0x7fcac0014000 comp rx=0 tx=0).stop 2023-10-10T17:52:03.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 -- 172.21.15.78:0/1687411267 shutdown_connections 2023-10-10T17:52:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a94d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 0x55a124a8ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1687411267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a90470 0x55a124a94370 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 -- 172.21.15.78:0/1687411267 >> 172.21.15.78:0/1687411267 conn(0x55a124a85360 msgr2=0x55a124a8ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 -- 172.21.15.78:0/1687411267 shutdown_connections 2023-10-10T17:52:03.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.488+0000 7fcae4805b80 1 -- 172.21.15.78:0/1687411267 wait complete. 2023-10-10T17:52:03.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 -- start start 2023-10-10T17:52:03.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 Processor -- start 2023-10-10T17:52:03.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a9bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 0x55a124a99250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a948b0 0x55a124a99880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1249828e0 con 0x55a124a8fb60 2023-10-10T17:52:03.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124982580 con 0x55a124a948b0 2023-10-10T17:52:03.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124864920 con 0x55a124a90470 2023-10-10T17:52:03.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcae4805b80 10 client.0 _ll_register_callbacks cb 0x55a1249a0780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:03.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a9bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a9bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48734/0 (socket says 172.21.15.78:48734) 2023-10-10T17:52:03.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 -- 172.21.15.78:0/3266346292 learned_addr learned my addr 172.21.15.78:0/3266346292 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:03.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad5c14700 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 0x55a124a99250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 -- 172.21.15.78:0/3266346292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 msgr2=0x55a124a99250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 0x55a124a99250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6c16700 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a948b0 0x55a124a99880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 -- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a948b0 msgr2=0x55a124a99880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a948b0 0x55a124a99880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.489+0000 7fcad6415700 1 -- 172.21.15.78:0/3266346292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac0017070 con 0x55a124a8fb60 2023-10-10T17:52:03.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcad6415700 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a9bfd0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009990 tx=0x7fcacc09a9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a01c9d32ecc5070 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcab9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:03.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcab9ffb700 1 -- 172.21.15.78:0/3266346292 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0a32f0 con 0x55a124a8fb60 2023-10-10T17:52:03.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcab9ffb700 1 -- 172.21.15.78:0/3266346292 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fcacc09a420 con 0x55a124a8fb60 2023-10-10T17:52:03.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a124a82b20 con 0x55a124a8fb60 2023-10-10T17:52:03.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcab9ffb700 1 -- 172.21.15.78:0/3266346292 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcacc09a5e0 con 0x55a124a8fb60 2023-10-10T17:52:03.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 10 client.17388 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:03.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a1249a7190 con 0x55a124a8fb60 2023-10-10T17:52:03.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcab9ffb700 1 -- 172.21.15.78:0/3266346292 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcacc0b9030 con 0x55a124a8fb60 2023-10-10T17:52:03.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcab9ffb700 1 -- 172.21.15.78:0/3266346292 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fcacc0aeba0 con 0x55a124a8fb60 2023-10-10T17:52:03.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 10 client.17388 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:03.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 10 client.17388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:03.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 20 client.17388 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:03.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124aa9520 0x55a124aab930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 4 client.17388 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:03.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a1249a30a0 con 0x55a124aa9520 2023-10-10T17:52:03.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.490+0000 7fcad5c14700 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124aa9520 0x55a124aab930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.491+0000 7fcab9ffb700 10 client.17388 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:03.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.491+0000 7fcad5c14700 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124aa9520 0x55a124aab930 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41c48b369635244 server_cookie=5cd9c908c8d51d71 in_seq=0 out_seq=0 2023-10-10T17:52:03.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcab9ffb700 1 -- 172.21.15.78:0/3266346292 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a1249a30a0 con 0x55a124aa9520 2023-10-10T17:52:03.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcab9ffb700 10 client.17388 handle_command_reply: tid=0 2023-10-10T17:52:03.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 client.17388 shutdown 2023-10-10T17:52:03.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 msgr2=0x55a124a9bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a9bfd0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009990 tx=0x7fcacc09a9b0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 shutdown_connections 2023-10-10T17:52:03.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124aa9520 0x55a124aab930 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a948b0 0x55a124a99880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcab9ffb700 0 client.17388 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:03.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a9bfd0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 --2- 172.21.15.78:0/3266346292 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 0x55a124a99250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 >> 172.21.15.78:0/3266346292 conn(0x55a124a85360 msgr2=0x55a124a89de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 shutdown_connections 2023-10-10T17:52:03.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 1 -- 172.21.15.78:0/3266346292 wait complete. 2023-10-10T17:52:03.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.492+0000 7fcae4805b80 20 client.17388 trim_cache size 0 max 16384 2023-10-10T17:52:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 Processor -- start 2023-10-10T17:52:03.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 -- start start 2023-10-10T17:52:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a96cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a90470 0x55a124aa09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a95390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124a9a860 con 0x55a124a8fb60 2023-10-10T17:52:03.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124a84ee0 con 0x55a124a90470 2023-10-10T17:52:03.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcae4805b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124a93180 con 0x55a124a948b0 2023-10-10T17:52:03.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a96cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a96cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48742/0 (socket says 172.21.15.78:48742) 2023-10-10T17:52:03.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 -- 172.21.15.78:0/1790099742 learned_addr learned my addr 172.21.15.78:0/1790099742 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:03.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad5c14700 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a90470 0x55a124aa09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6c16700 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a95390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 -- 172.21.15.78:0/1790099742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 msgr2=0x55a124a95390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a95390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 -- 172.21.15.78:0/1790099742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a90470 msgr2=0x55a124aa09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a90470 0x55a124aa09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.509+0000 7fcad6415700 1 -- 172.21.15.78:0/1790099742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a124a86b00 con 0x55a124a8fb60 2023-10-10T17:52:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcad5c14700 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a90470 0x55a124aa09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:03.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcad6c16700 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a95390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcad6415700 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a96cc0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fcacc09ac30 tx=0x7fcacc009290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcf3225ec8e1ff97 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcac57fa700 1 -- 172.21.15.78:0/1790099742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0cbba0 con 0x55a124a8fb60 2023-10-10T17:52:03.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcac57fa700 1 -- 172.21.15.78:0/1790099742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcacc0b9070 con 0x55a124a8fb60 2023-10-10T17:52:03.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcac57fa700 1 -- 172.21.15.78:0/1790099742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc09a420 con 0x55a124a8fb60 2023-10-10T17:52:03.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 -- 172.21.15.78:0/1790099742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 msgr2=0x55a124a96cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a96cc0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fcacc09ac30 tx=0x7fcacc009290 comp rx=0 tx=0).stop 2023-10-10T17:52:03.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 -- 172.21.15.78:0/1790099742 shutdown_connections 2023-10-10T17:52:03.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a948b0 0x55a124a95390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a8fb60 0x55a124a96cc0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 --2- 172.21.15.78:0/1790099742 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a90470 0x55a124aa09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 -- 172.21.15.78:0/1790099742 >> 172.21.15.78:0/1790099742 conn(0x55a124a85360 msgr2=0x55a124a94c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 -- 172.21.15.78:0/1790099742 shutdown_connections 2023-10-10T17:52:03.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.510+0000 7fcae4805b80 1 -- 172.21.15.78:0/1790099742 wait complete. 2023-10-10T17:52:03.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 -- start start 2023-10-10T17:52:03.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 Processor -- start 2023-10-10T17:52:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 0x55a124a97a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 0x55a124a97f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a948b0 0x55a124a984d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124aa0fb0 con 0x55a124a948b0 2023-10-10T17:52:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124a98a10 con 0x55a124a8fb60 2023-10-10T17:52:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a124a98bd0 con 0x55a124a90470 2023-10-10T17:52:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcae4805b80 10 client.0 _ll_register_callbacks cb 0x7fcacc09adf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a948b0 0x55a124a984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a948b0 0x55a124a984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48750/0 (socket says 172.21.15.78:48750) 2023-10-10T17:52:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 -- 172.21.15.78:0/103809118 learned_addr learned my addr 172.21.15.78:0/103809118 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 -- 172.21.15.78:0/103809118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 msgr2=0x55a124a97f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 0x55a124a97f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 -- 172.21.15.78:0/103809118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 msgr2=0x55a124a97a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:03.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 0x55a124a97a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 -- 172.21.15.78:0/103809118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1249a7190 con 0x55a124a948b0 2023-10-10T17:52:03.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcad6c16700 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a948b0 0x55a124a984d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcac8007a80 tx=0x7fcac8007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a756f161dccb7ed server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcab9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.511+0000 7fcab9ffb700 1 -- 172.21.15.78:0/103809118 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac800e040 con 0x55a124a948b0 2023-10-10T17:52:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcab9ffb700 1 -- 172.21.15.78:0/103809118 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fcac8004bb0 con 0x55a124a948b0 2023-10-10T17:52:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcac0021070 con 0x55a124a948b0 2023-10-10T17:52:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcab9ffb700 1 -- 172.21.15.78:0/103809118 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcac8004d70 con 0x55a124a948b0 2023-10-10T17:52:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 10 client.17394 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:03.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a12480ede0 con 0x55a124a948b0 2023-10-10T17:52:03.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcab9ffb700 1 -- 172.21.15.78:0/103809118 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcac0021070 con 0x55a124a948b0 2023-10-10T17:52:03.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcab9ffb700 1 -- 172.21.15.78:0/103809118 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fcac801d020 con 0x55a124a948b0 2023-10-10T17:52:03.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 10 client.17394 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:03.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 10 client.17394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:03.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 20 client.17394 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:03.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124afdf80 0x55a124afe380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 4 client.17394 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:52:03.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a1249af660 con 0x55a124afdf80 2023-10-10T17:52:03.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.512+0000 7fcad6415700 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124afdf80 0x55a124afe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcab9ffb700 10 client.17394 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:03.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcad6415700 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124afdf80 0x55a124afe380 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1be7eee4404a59d4 server_cookie=24b461da6a377ff6 in_seq=0 out_seq=0 2023-10-10T17:52:03.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcab9ffb700 1 -- 172.21.15.78:0/103809118 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a1249af660 con 0x55a124afdf80 2023-10-10T17:52:03.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcab9ffb700 10 client.17394 handle_command_reply: tid=0 2023-10-10T17:52:03.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 client.17394 shutdown 2023-10-10T17:52:03.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a948b0 msgr2=0x55a124a984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a948b0 0x55a124a984d0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fcac8007a80 tx=0x7fcac8007e40 comp rx=0 tx=0).stop 2023-10-10T17:52:03.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 shutdown_connections 2023-10-10T17:52:03.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55a124afdf80 0x55a124afe380 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a124a948b0 0x55a124a984d0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcab9ffb700 0 client.17394 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:03.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a124a8fb60 0x55a124a97a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 --2- 172.21.15.78:0/103809118 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a124a90470 0x55a124a97f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 >> 172.21.15.78:0/103809118 conn(0x55a124a85360 msgr2=0x55a124a95510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 shutdown_connections 2023-10-10T17:52:03.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 1 -- 172.21.15.78:0/103809118 wait complete. 2023-10-10T17:52:03.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.513+0000 7fcae4805b80 20 client.17394 trim_cache size 0 max 16384 2023-10-10T17:52:03.558 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:03.558 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:52:03.559 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:52:03.559 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:03.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcabc0a12c0 msgr2=0x7fcabc0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcabc0a12c0 0x7fcabc0a3770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcad01968e0 tx=0x7fcac0009260 comp rx=0 tx=0).stop 2023-10-10T17:52:03.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 msgr2=0x7fcad01952d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad01952d0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fcacc00b580 tx=0x7fcacc00bfd0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 shutdown_connections 2023-10-10T17:52:03.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcad0104490 0x7fcad0199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcabc0a12c0 0x7fcabc0a3770 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcad00ff450 0x7fcad01952d0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 --2- 172.21.15.78:0/2410296099 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcad00ffdf0 0x7fcad0195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 >> 172.21.15.78:0/2410296099 conn(0x7fcad00770b0 msgr2=0x7fcad00fcc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 shutdown_connections 2023-10-10T17:52:03.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.514+0000 7fcad8679700 1 -- 172.21.15.78:0/2410296099 wait complete. 2023-10-10T17:52:03.564 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:52:03.564 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:52:03.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2271730269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac0104190 msgr2=0x7f0ac0104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2271730269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac0104190 0x7f0ac0104560 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f0ab00099c0 tx=0x7f0ab0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2271730269 shutdown_connections 2023-10-10T17:52:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2271730269 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ac0100560 0x7f0ac010c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2271730269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac0104190 0x7f0ac0104560 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2271730269 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ac00ffbb0 0x7f0ac0100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2271730269 >> 172.21.15.78:0/2271730269 conn(0x7f0ac0074840 msgr2=0x7f0ac0074c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2271730269 shutdown_connections 2023-10-10T17:52:03.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.684+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2271730269 wait complete. 2023-10-10T17:52:03.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 Processor -- start 2023-10-10T17:52:03.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 -- start start 2023-10-10T17:52:03.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac00ffbb0 0x7f0ac0197150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ac0100560 0x7f0ac0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ac0104190 0x7f0ac01a21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01a2710 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01a2850 con 0x7f0ac0104190 2023-10-10T17:52:03.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.685+0000 7f0ac58cb700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01a2990 con 0x7f0ac0100560 2023-10-10T17:52:03.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac00ffbb0 0x7f0ac0197150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac00ffbb0 0x7f0ac0197150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48762/0 (socket says 172.21.15.78:48762) 2023-10-10T17:52:03.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 -- 172.21.15.78:0/2726187540 learned_addr learned my addr 172.21.15.78:0/2726187540 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:03.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abe7fc700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ac0100560 0x7f0ac0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 -- 172.21.15.78:0/2726187540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ac0100560 msgr2=0x7f0ac0197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ac0100560 0x7f0ac0197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 -- 172.21.15.78:0/2726187540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ac0104190 msgr2=0x7f0ac01a21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abf7fe700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ac0104190 0x7f0ac01a21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ac0104190 0x7f0ac01a21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 -- 172.21.15.78:0/2726187540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00095f0 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abf7fe700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ac0104190 0x7f0ac01a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:03.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0abeffd700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac00ffbb0 0x7f0ac0197150 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009990 tx=0x7f0ab0009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0ac48c9700 1 -- 172.21.15.78:0/2726187540 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab001d070 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac010f200 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0ac48c9700 1 -- 172.21.15.78:0/2726187540 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ab0004010 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0ac48c9700 1 -- 172.21.15.78:0/2726187540 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0022d40 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.686+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac010f070 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.687+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac010f390 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.689+0000 7f0ac48c9700 1 -- 172.21.15.78:0/2726187540 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f0ab0022470 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.689+0000 7f0ac48c9700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0aa80a12c0 0x7f0aa80a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:03.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.689+0000 7f0ac48c9700 1 -- 172.21.15.78:0/2726187540 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f0ab00ce8a0 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.689+0000 7f0abe7fc700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0aa80a12c0 0x7f0aa80a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:03.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.690+0000 7f0abe7fc700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0aa80a12c0 0x7f0aa80a3770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4005950 tx=0x7f0ab40058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:03.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.692+0000 7f0ac48c9700 1 -- 172.21.15.78:0/2726187540 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0ab00979b0 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.927+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ac00ff1b0 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.928+0000 7f0ac48c9700 1 -- 172.21.15.78:0/2726187540 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+10813 (secure 0 0 0) 0x7f0ac00ff1b0 con 0x7f0ac00ffbb0 2023-10-10T17:52:03.930 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:52:03.930 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:51:59.786011+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:52:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0aa80a12c0 msgr2=0x7f0aa80a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0aa80a12c0 0x7f0aa80a3770 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4005950 tx=0x7f0ab40058e0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac00ffbb0 msgr2=0x7f0ac0197150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac00ffbb0 0x7f0ac0197150 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009990 tx=0x7f0ab0009580 comp rx=0 tx=0).stop 2023-10-10T17:52:03.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 shutdown_connections 2023-10-10T17:52:03.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f0aa80a12c0 0x7f0aa80a3770 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0ac0104190 0x7f0ac01a21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0ac00ffbb0 0x7f0ac0197150 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 --2- 172.21.15.78:0/2726187540 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0ac0100560 0x7f0ac0197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:03.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 >> 172.21.15.78:0/2726187540 conn(0x7f0ac0074840 msgr2=0x7f0ac00fc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:03.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 shutdown_connections 2023-10-10T17:52:03.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:03.930+0000 7f0ac58cb700 1 -- 172.21.15.78:0/2726187540 wait complete. 2023-10-10T17:52:03.937 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 34 2023-10-10T17:52:03.945 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:52:04.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 -- 172.21.15.78:0/2409237628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc0ff450 msgr2=0x7fb7cc0ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2409237628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc0ff450 0x7fb7cc0ff820 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00099c0 tx=0x7fb7c0009cd0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 -- 172.21.15.78:0/2409237628 shutdown_connections 2023-10-10T17:52:04.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2409237628 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7cc104490 0x7fb7cc10c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2409237628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc0ff450 0x7fb7cc0ff820 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2409237628 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7cc0ffdf0 0x7fb7cc103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 -- 172.21.15.78:0/2409237628 >> 172.21.15.78:0/2409237628 conn(0x7fb7cc0770b0 msgr2=0x7fb7cc0774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 -- 172.21.15.78:0/2409237628 shutdown_connections 2023-10-10T17:52:04.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 -- 172.21.15.78:0/2409237628 wait complete. 2023-10-10T17:52:04.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.067+0000 7fb7d1933700 1 Processor -- start 2023-10-10T17:52:04.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7d1933700 1 -- start start 2023-10-10T17:52:04.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7d1933700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7cc0ff450 0x7fb7cc100e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7d1933700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7cc0ffdf0 0x7fb7cc101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7d1933700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc104490 0x7fb7cc1a45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7d1933700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc101bc0 con 0x7fb7cc104490 2023-10-10T17:52:04.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7d1933700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc1a4b20 con 0x7fb7cc0ff450 2023-10-10T17:52:04.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7d1933700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7cc1a4c60 con 0x7fb7cc0ffdf0 2023-10-10T17:52:04.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc104490 0x7fb7cc1a45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc104490 0x7fb7cc1a45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48780/0 (socket says 172.21.15.78:48780) 2023-10-10T17:52:04.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/2171825500 learned_addr learned my addr 172.21.15.78:0/2171825500 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7ca7fc700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7cc0ffdf0 0x7fb7cc101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7cc0ff450 0x7fb7cc100e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/2171825500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7cc0ffdf0 msgr2=0x7fb7cc101390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7cc0ffdf0 0x7fb7cc101390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/2171825500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7cc0ff450 msgr2=0x7fb7cc100e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7cc0ff450 0x7fb7cc100e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/2171825500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7c00095f0 con 0x7fb7cc104490 2023-10-10T17:52:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.068+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc104490 0x7fb7cc1a45e0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400ef30 tx=0x7fb7c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.069+0000 7fb7d0931700 1 -- 172.21.15.78:0/2171825500 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4018070 con 0x7fb7cc104490 2023-10-10T17:52:04.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.069+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc10f250 con 0x7fb7cc104490 2023-10-10T17:52:04.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.069+0000 7fb7d0931700 1 -- 172.21.15.78:0/2171825500 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7c400f040 con 0x7fb7cc104490 2023-10-10T17:52:04.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.069+0000 7fb7d0931700 1 -- 172.21.15.78:0/2171825500 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c400cd00 con 0x7fb7cc104490 2023-10-10T17:52:04.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.069+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc10f0c0 con 0x7fb7cc104490 2023-10-10T17:52:04.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.071+0000 7fb7d0931700 1 -- 172.21.15.78:0/2171825500 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fb7c4003a00 con 0x7fb7cc104490 2023-10-10T17:52:04.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.071+0000 7fb7d0931700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb7b40a1340 0x7fb7b40a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.072+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb7b40a1340 0x7fb7b40a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.072+0000 7fb7d0931700 1 -- 172.21.15.78:0/2171825500 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb7c40ccbc0 con 0x7fb7cc104490 2023-10-10T17:52:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.072+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb7b40a1340 0x7fb7b40a37f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009990 tx=0x7fb7c0005fb0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.072+0000 7fb7ddabfb80 1 Processor -- start 2023-10-10T17:52:04.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.072+0000 7fb7ddabfb80 1 -- start start 2023-10-10T17:52:04.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d17f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d1c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d1cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466c1d060 con 0x55a466d17b60 2023-10-10T17:52:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466c6f8a0 con 0x55a466d18470 2023-10-10T17:52:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466c68d30 con 0x55a466d1c8b0 2023-10-10T17:52:04.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d17f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d17f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48784/0 (socket says 172.21.15.78:48784) 2023-10-10T17:52:04.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 -- 172.21.15.78:0/2900194848 learned_addr learned my addr 172.21.15.78:0/2900194848 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ca7fc700 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d1c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d1cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 -- 172.21.15.78:0/2900194848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 msgr2=0x55a466d1cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d1cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 -- 172.21.15.78:0/2900194848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 msgr2=0x55a466d1c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d1c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 -- 172.21.15.78:0/2900194848 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a466aea630 con 0x55a466d17b60 2023-10-10T17:52:04.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7ca7fc700 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d1c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:04.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d1cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:04.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d17f30 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00099c0 tx=0x7fb7c001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9638c5207bf93e server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7b1ffb700 1 -- 172.21.15.78:0/2900194848 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000b990 con 0x55a466d17b60 2023-10-10T17:52:04.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7b1ffb700 1 -- 172.21.15.78:0/2900194848 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7c001e040 con 0x55a466d17b60 2023-10-10T17:52:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.073+0000 7fb7b1ffb700 1 -- 172.21.15.78:0/2900194848 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000f6c0 con 0x55a466d17b60 2023-10-10T17:52:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2900194848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 msgr2=0x55a466d17f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d17f30 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00099c0 tx=0x7fb7c001f000 comp rx=0 tx=0).stop 2023-10-10T17:52:04.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2900194848 shutdown_connections 2023-10-10T17:52:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d1cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d17f30 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2900194848 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d1c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2900194848 >> 172.21.15.78:0/2900194848 conn(0x55a466d0d360 msgr2=0x55a466d12b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2900194848 shutdown_connections 2023-10-10T17:52:04.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2900194848 wait complete. 2023-10-10T17:52:04.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.074+0000 7fb7ddabfb80 1 -- start start 2023-10-10T17:52:04.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 1 Processor -- start 2023-10-10T17:52:04.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d23fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d18470 0x55a466d21240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d1c8b0 0x55a466d217c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466c0a8e0 con 0x55a466d18470 2023-10-10T17:52:04.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466c0a580 con 0x55a466d1c8b0 2023-10-10T17:52:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466aec920 con 0x55a466d17b60 2023-10-10T17:52:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ddabfb80 10 client.0 _ll_register_callbacks cb 0x55a466c28780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:04.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d18470 0x55a466d21240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d18470 0x55a466d21240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48792/0 (socket says 172.21.15.78:48792) 2023-10-10T17:52:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 -- 172.21.15.78:0/2710506964 learned_addr learned my addr 172.21.15.78:0/2710506964 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d1c8b0 0x55a466d217c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d23fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 -- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 msgr2=0x55a466d23fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d23fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 -- 172.21.15.78:0/2710506964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d1c8b0 msgr2=0x55a466d217c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d1c8b0 0x55a466d217c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 -- 172.21.15.78:0/2710506964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc10c8a0 con 0x55a466d18470 2023-10-10T17:52:04.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7ca7fc700 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d18470 0x55a466d21240 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00b9c0 tx=0x7fb7bc00bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381a9fde5781ceaa server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7aaffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.075+0000 7fb7aaffd700 1 -- 172.21.15.78:0/2710506964 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc003d70 con 0x55a466d18470 2023-10-10T17:52:04.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7aaffd700 1 -- 172.21.15.78:0/2710506964 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb7bc004830 con 0x55a466d18470 2023-10-10T17:52:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a466d0ab20 con 0x55a466d18470 2023-10-10T17:52:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7aaffd700 1 -- 172.21.15.78:0/2710506964 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7bc0049f0 con 0x55a466d18470 2023-10-10T17:52:04.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 10 client.17412 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a466c2f190 con 0x55a466d18470 2023-10-10T17:52:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7aaffd700 1 -- 172.21.15.78:0/2710506964 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55a466d0ab20 con 0x55a466d18470 2023-10-10T17:52:04.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7aaffd700 1 -- 172.21.15.78:0/2710506964 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb7bc01a590 con 0x55a466d18470 2023-10-10T17:52:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 10 client.17412 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 10 client.17412 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:52:04.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 20 client.17412 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d31860 0x55a466d33c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 4 client.17412 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a466c2b0a0 con 0x55a466d31860 2023-10-10T17:52:04.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.076+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d31860 0x55a466d33c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.077+0000 7fb7aaffd700 10 client.17412 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.077+0000 7fb7caffd700 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d31860 0x55a466d33c70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ace58674ee297b5c server_cookie=30eae51ad723c048 in_seq=0 out_seq=0 2023-10-10T17:52:04.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.077+0000 7fb7aaffd700 1 -- 172.21.15.78:0/2710506964 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55a466c2b0a0 con 0x55a466d31860 2023-10-10T17:52:04.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.077+0000 7fb7aaffd700 10 client.17412 handle_command_reply: tid=0 2023-10-10T17:52:04.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.077+0000 7fb7ddabfb80 1 client.17412 shutdown 2023-10-10T17:52:04.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d18470 msgr2=0x55a466d21240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d18470 0x55a466d21240 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00b9c0 tx=0x7fb7bc00bcd0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 shutdown_connections 2023-10-10T17:52:04.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d31860 0x55a466d33c70 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d23fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d1c8b0 0x55a466d217c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7aaffd700 0 client.17412 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/2710506964 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d18470 0x55a466d21240 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 >> 172.21.15.78:0/2710506964 conn(0x55a466d0d360 msgr2=0x55a466d11de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 shutdown_connections 2023-10-10T17:52:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/2710506964 wait complete. 2023-10-10T17:52:04.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.078+0000 7fb7ddabfb80 20 client.17412 trim_cache size 0 max 16384 2023-10-10T17:52:04.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.090+0000 7fb7ddabfb80 1 Processor -- start 2023-10-10T17:52:04.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ddabfb80 1 -- start start 2023-10-10T17:52:04.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d28980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d17070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d23af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466d22350 con 0x55a466d17b60 2023-10-10T17:52:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466d2e130 con 0x55a466d18470 2023-10-10T17:52:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466d1c470 con 0x55a466d1c8b0 2023-10-10T17:52:04.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d28980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d28980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48798/0 (socket says 172.21.15.78:48798) 2023-10-10T17:52:04.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 -- 172.21.15.78:0/1241858368 learned_addr learned my addr 172.21.15.78:0/1241858368 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d23af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7ca7fc700 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d17070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 -- 172.21.15.78:0/1241858368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 msgr2=0x55a466d23af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d23af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 -- 172.21.15.78:0/1241858368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 msgr2=0x55a466d17070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d17070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 -- 172.21.15.78:0/1241858368 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a466d20db0 con 0x55a466d17b60 2023-10-10T17:52:04.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d23af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7caffd700 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d28980 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00058e0 tx=0x7fb7c00045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=908590359f696fc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.091+0000 7fb7b27fc700 1 -- 172.21.15.78:0/1241858368 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c0036040 con 0x55a466d17b60 2023-10-10T17:52:04.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7b27fc700 1 -- 172.21.15.78:0/1241858368 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7c0009290 con 0x55a466d17b60 2023-10-10T17:52:04.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7b27fc700 1 -- 172.21.15.78:0/1241858368 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c000bb00 con 0x55a466d17b60 2023-10-10T17:52:04.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/1241858368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 msgr2=0x55a466d28980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d28980 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00058e0 tx=0x7fb7c00045d0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/1241858368 shutdown_connections 2023-10-10T17:52:04.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d1c8b0 0x55a466d23af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d17b60 0x55a466d28980 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/1241858368 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d17070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/1241858368 >> 172.21.15.78:0/1241858368 conn(0x55a466d0d360 msgr2=0x55a466d16960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/1241858368 shutdown_connections 2023-10-10T17:52:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/1241858368 wait complete. 2023-10-10T17:52:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.092+0000 7fb7ddabfb80 1 -- start start 2023-10-10T17:52:04.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 1 Processor -- start 2023-10-10T17:52:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d26290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d267d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d1c8b0 0x55a466d26d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466d84890 con 0x55a466d1c8b0 2023-10-10T17:52:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466d84a00 con 0x55a466d18470 2023-10-10T17:52:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a466d84bc0 con 0x55a466d17b60 2023-10-10T17:52:04.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7ddabfb80 10 client.0 _ll_register_callbacks cb 0x7fb7bc004350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d1c8b0 0x55a466d26d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d1c8b0 0x55a466d26d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48808/0 (socket says 172.21.15.78:48808) 2023-10-10T17:52:04.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/3687626539 learned_addr learned my addr 172.21.15.78:0/3687626539 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7caffd700 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d26290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 msgr2=0x55a466d26290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d26290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/3687626539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 msgr2=0x55a466d267d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d267d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 -- 172.21.15.78:0/3687626539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a466aea630 con 0x55a466d1c8b0 2023-10-10T17:52:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7cb7fe700 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d1c8b0 0x55a466d26d30 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40090a0 tx=0x7fb7c4004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c9100d13556327c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.093+0000 7fb7aaffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:04.142 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7aaffd700 1 -- 172.21.15.78:0/3687626539 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c4084040 con 0x55a466d1c8b0 2023-10-10T17:52:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7aaffd700 1 -- 172.21.15.78:0/3687626539 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fb7c4099930 con 0x55a466d1c8b0 2023-10-10T17:52:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7c001e040 con 0x55a466d1c8b0 2023-10-10T17:52:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7aaffd700 1 -- 172.21.15.78:0/3687626539 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7c40c6070 con 0x55a466d1c8b0 2023-10-10T17:52:04.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 10 client.17418 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:04.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a466a96de0 con 0x55a466d1c8b0 2023-10-10T17:52:04.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7aaffd700 1 -- 172.21.15.78:0/3687626539 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb7c001e040 con 0x55a466d1c8b0 2023-10-10T17:52:04.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7aaffd700 1 -- 172.21.15.78:0/3687626539 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb7c40a1860 con 0x55a466d1c8b0 2023-10-10T17:52:04.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 10 client.17418 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:04.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 10 client.17418 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:52:04.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 20 client.17418 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:04.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d86200 0x55a466d88610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 4 client.17418 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:52:04.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a466c37660 con 0x55a466d86200 2023-10-10T17:52:04.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.094+0000 7fb7caffd700 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d86200 0x55a466d88610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7aaffd700 10 client.17418 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:04.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7caffd700 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d86200 0x55a466d88610 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e2242b55e0a50150 server_cookie=7505627014ef84a7 in_seq=0 out_seq=0 2023-10-10T17:52:04.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7aaffd700 1 -- 172.21.15.78:0/3687626539 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a466c37660 con 0x55a466d86200 2023-10-10T17:52:04.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7aaffd700 10 client.17418 handle_command_reply: tid=0 2023-10-10T17:52:04.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 client.17418 shutdown 2023-10-10T17:52:04.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d1c8b0 msgr2=0x55a466d26d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d1c8b0 0x55a466d26d30 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb7c40090a0 tx=0x7fb7c4004ad0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 shutdown_connections 2023-10-10T17:52:04.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x55a466d86200 0x55a466d88610 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55a466d17b60 0x55a466d26290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55a466d1c8b0 0x55a466d26d30 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7aaffd700 0 client.17418 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:04.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 --2- 172.21.15.78:0/3687626539 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55a466d18470 0x55a466d267d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 >> 172.21.15.78:0/3687626539 conn(0x55a466d0d360 msgr2=0x55a466d171f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 shutdown_connections 2023-10-10T17:52:04.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 1 -- 172.21.15.78:0/3687626539 wait complete. 2023-10-10T17:52:04.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.095+0000 7fb7ddabfb80 20 client.17418 trim_cache size 0 max 16384 2023-10-10T17:52:04.151 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:52:04.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb7b40a1340 msgr2=0x7fb7b40a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb7b40a1340 0x7fb7b40a37f0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0009990 tx=0x7fb7c0005fb0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc104490 msgr2=0x7fb7cc1a45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc104490 0x7fb7cc1a45e0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb7c400ef30 tx=0x7fb7c400c3a0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 shutdown_connections 2023-10-10T17:52:04.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fb7b40a1340 0x7fb7b40a37f0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb7cc104490 0x7fb7cc1a45e0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fb7cc0ff450 0x7fb7cc100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 --2- 172.21.15.78:0/2171825500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fb7cc0ffdf0 0x7fb7cc101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 >> 172.21.15.78:0/2171825500 conn(0x7fb7cc0770b0 msgr2=0x7fb7cc103030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 shutdown_connections 2023-10-10T17:52:04.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.096+0000 7fb7d1933700 1 -- 172.21.15.78:0/2171825500 wait complete. 2023-10-10T17:52:04.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:52:04.282 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 -- 172.21.15.78:0/1190793915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 msgr2=0x7fe6d810d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 --2- 172.21.15.78:0/1190793915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d810d190 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000b0a0 tx=0x7fe6d000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 -- 172.21.15.78:0/1190793915 shutdown_connections 2023-10-10T17:52:04.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 --2- 172.21.15.78:0/1190793915 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d8106360 0x7fe6d8106730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 --2- 172.21.15.78:0/1190793915 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d810d190 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 --2- 172.21.15.78:0/1190793915 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d8068e70 0x7fe6d80692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 -- 172.21.15.78:0/1190793915 >> 172.21.15.78:0/1190793915 conn(0x7fe6d8075470 msgr2=0x7fe6d8075870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 -- 172.21.15.78:0/1190793915 shutdown_connections 2023-10-10T17:52:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 -- 172.21.15.78:0/1190793915 wait complete. 2023-10-10T17:52:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 Processor -- start 2023-10-10T17:52:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.279+0000 7fe6e036e700 1 -- start start 2023-10-10T17:52:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d8068e70 0x7fe6d81964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d8106360 0x7fe6d819ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d819b4a0 con 0x7fe6d8069820 2023-10-10T17:52:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d819b610 con 0x7fe6d8106360 2023-10-10T17:52:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d819b7d0 con 0x7fe6d8068e70 2023-10-10T17:52:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d8196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48818/0 (socket says 172.21.15.78:48818) 2023-10-10T17:52:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 -- 172.21.15.78:0/500332561 learned_addr learned my addr 172.21.15.78:0/500332561 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6de10a700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d8068e70 0x7fe6d81964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6de90b700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d8106360 0x7fe6d819ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 -- 172.21.15.78:0/500332561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d8068e70 msgr2=0x7fe6d81964c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d8068e70 0x7fe6d81964c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 -- 172.21.15.78:0/500332561 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d8106360 msgr2=0x7fe6d819ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d8106360 0x7fe6d819ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 -- 172.21.15.78:0/500332561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d0009d00 con 0x7fe6d8069820 2023-10-10T17:52:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6dd909700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d8196a00 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800e9a0 tx=0x7fe6c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6cf7fe700 1 -- 172.21.15.78:0/500332561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8009790 con 0x7fe6d8069820 2023-10-10T17:52:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6cf7fe700 1 -- 172.21.15.78:0/500332561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c800cbd0 con 0x7fe6d8069820 2023-10-10T17:52:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d8110360 con 0x7fe6d8069820 2023-10-10T17:52:04.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6cf7fe700 1 -- 172.21.15.78:0/500332561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8018680 con 0x7fe6d8069820 2023-10-10T17:52:04.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.280+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d81101d0 con 0x7fe6d8069820 2023-10-10T17:52:04.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.282+0000 7fe6cf7fe700 1 -- 172.21.15.78:0/500332561 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fe6c800cd40 con 0x7fe6d8069820 2023-10-10T17:52:04.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.283+0000 7fe6cf7fe700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe6c40a12c0 0x7fe6c40a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.283+0000 7fe6de10a700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe6c40a12c0 0x7fe6c40a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.283+0000 7fe6cf7fe700 1 -- 172.21.15.78:0/500332561 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fe6c8014070 con 0x7fe6d8069820 2023-10-10T17:52:04.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.283+0000 7fe6de10a700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe6c40a12c0 0x7fe6c40a3770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009be0 tx=0x7fe6d4009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.284+0000 7fe6ec4fab80 1 Processor -- start 2023-10-10T17:52:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.284+0000 7fe6ec4fab80 1 -- start start 2023-10-10T17:52:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.284+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56321b60 0x555d56321f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56326370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d56326d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d56227060 con 0x555d56322470 2023-10-10T17:52:04.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d562798a0 con 0x555d56321b60 2023-10-10T17:52:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d56272d30 con 0x555d563268b0 2023-10-10T17:52:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56326370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56326370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48826/0 (socket says 172.21.15.78:48826) 2023-10-10T17:52:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 -- 172.21.15.78:0/2873754446 learned_addr learned my addr 172.21.15.78:0/2873754446 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6de90b700 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d56326d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6de10a700 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56321b60 0x555d56321f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 -- 172.21.15.78:0/2873754446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 msgr2=0x555d56326d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d56326d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 -- 172.21.15.78:0/2873754446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56321b60 msgr2=0x555d56321f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56321b60 0x555d56321f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 -- 172.21.15.78:0/2873754446 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d560f4630 con 0x555d56322470 2023-10-10T17:52:04.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6de90b700 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d56326d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:04.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6de10a700 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56321b60 0x555d56321f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:04.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6dd909700 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56326370 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8096590 tx=0x7fe6c8099a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f42ed894900902 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ccdac700 1 -- 172.21.15.78:0/2873754446 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80cd1b0 con 0x555d56322470 2023-10-10T17:52:04.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ccdac700 1 -- 172.21.15.78:0/2873754446 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c80a24e0 con 0x555d56322470 2023-10-10T17:52:04.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.285+0000 7fe6ccdac700 1 -- 172.21.15.78:0/2873754446 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80a22b0 con 0x555d56322470 2023-10-10T17:52:04.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2873754446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 msgr2=0x555d56326370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56326370 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8096590 tx=0x7fe6c8099a70 comp rx=0 tx=0).stop 2023-10-10T17:52:04.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2873754446 shutdown_connections 2023-10-10T17:52:04.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d56326d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56321b60 0x555d56321f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2873754446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56326370 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2873754446 >> 172.21.15.78:0/2873754446 conn(0x555d56317360 msgr2=0x555d5631cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2873754446 shutdown_connections 2023-10-10T17:52:04.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2873754446 wait complete. 2023-10-10T17:52:04.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 -- start start 2023-10-10T17:52:04.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.286+0000 7fe6ec4fab80 1 Processor -- start 2023-10-10T17:52:04.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d5632b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56322470 0x555d5632b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d5632be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d562148e0 con 0x555d56321b60 2023-10-10T17:52:04.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d56214580 con 0x555d56322470 2023-10-10T17:52:04.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d560f6920 con 0x555d563268b0 2023-10-10T17:52:04.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6ec4fab80 10 client.0 _ll_register_callbacks cb 0x555d56232780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:04.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d5632b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d5632b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48830/0 (socket says 172.21.15.78:48830) 2023-10-10T17:52:04.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 -- 172.21.15.78:0/25716873 learned_addr learned my addr 172.21.15.78:0/25716873 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de90b700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d5632be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6dd909700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56322470 0x555d5632b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 -- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 msgr2=0x555d5632be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d5632be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 -- 172.21.15.78:0/25716873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56322470 msgr2=0x555d5632b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56322470 0x555d5632b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 -- 172.21.15.78:0/25716873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d81101d0 con 0x555d56321b60 2023-10-10T17:52:04.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6dd909700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56322470 0x555d5632b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:04.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6de10a700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d5632b3a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fe6d401d7e0 tx=0x7fe6d40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ece297f06dff79 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6c1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:04.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.287+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/25716873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d400e640 con 0x555d56321b60 2023-10-10T17:52:04.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/25716873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fe6d400e7a0 con 0x555d56321b60 2023-10-10T17:52:04.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555d56314b20 con 0x555d56321b60 2023-10-10T17:52:04.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/25716873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6d400e960 con 0x555d56321b60 2023-10-10T17:52:04.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 10 client.17430 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:04.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x555d56239190 con 0x555d56321b60 2023-10-10T17:52:04.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/25716873 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555d56314b20 con 0x555d56321b60 2023-10-10T17:52:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/25716873 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fe6d401ec20 con 0x555d56321b60 2023-10-10T17:52:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 10 client.17430 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 10 client.17430 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:52:04.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 20 client.17430 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:04.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5633b9e0 0x555d5633ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 4 client.17430 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:04.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d562350a0 con 0x555d5633b9e0 2023-10-10T17:52:04.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.288+0000 7fe6dd909700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5633b9e0 0x555d5633ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.289+0000 7fe6dd909700 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5633b9e0 0x555d5633ddf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=37dc3879d68f7ddd server_cookie=86ed2df45f157a79 in_seq=0 out_seq=0 2023-10-10T17:52:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.289+0000 7fe6c1ffb700 10 client.17430 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:04.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.289+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/25716873 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x555d562350a0 con 0x555d5633b9e0 2023-10-10T17:52:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.289+0000 7fe6c1ffb700 10 client.17430 handle_command_reply: tid=0 2023-10-10T17:52:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.289+0000 7fe6ec4fab80 1 client.17430 shutdown 2023-10-10T17:52:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 msgr2=0x555d5632b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d5632b3a0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fe6d401d7e0 tx=0x7fe6d40047c0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 shutdown_connections 2023-10-10T17:52:04.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5633b9e0 0x555d5633ddf0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d563268b0 0x555d5632be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6c1ffb700 0 client.17430 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:04.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d5632b3a0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/25716873 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d56322470 0x555d5632b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 >> 172.21.15.78:0/25716873 conn(0x555d56317360 msgr2=0x555d5631bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 shutdown_connections 2023-10-10T17:52:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/25716873 wait complete. 2023-10-10T17:52:04.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.290+0000 7fe6ec4fab80 20 client.17430 trim_cache size 0 max 16384 2023-10-10T17:52:04.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.302+0000 7fe6ec4fab80 1 Processor -- start 2023-10-10T17:52:04.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.302+0000 7fe6ec4fab80 1 -- start start 2023-10-10T17:52:04.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.302+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d56337cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.302+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56322470 0x555d56324930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.302+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d56324e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5631e230 con 0x555d56321b60 2023-10-10T17:52:04.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d56322bc0 con 0x555d563268b0 2023-10-10T17:52:04.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d56316ee0 con 0x555d56322470 2023-10-10T17:52:04.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d56337cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d56337cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48838/0 (socket says 172.21.15.78:48838) 2023-10-10T17:52:04.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 -- 172.21.15.78:0/1235623055 learned_addr learned my addr 172.21.15.78:0/1235623055 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6dd909700 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56322470 0x555d56324930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 -- 172.21.15.78:0/1235623055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56322470 msgr2=0x555d56324930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de90b700 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d56324e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56322470 0x555d56324930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 -- 172.21.15.78:0/1235623055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 msgr2=0x555d56324e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d56324e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 -- 172.21.15.78:0/1235623055 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d56318b00 con 0x555d56321b60 2023-10-10T17:52:04.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6dd909700 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56322470 0x555d56324930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:04.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6de10a700 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d56337cb0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4016040 tx=0x7fe6d401b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c3dd3c62d54aa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6cd5ad700 1 -- 172.21.15.78:0/1235623055 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d401b680 con 0x555d56321b60 2023-10-10T17:52:04.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6cd5ad700 1 -- 172.21.15.78:0/1235623055 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6d401bd80 con 0x555d56321b60 2023-10-10T17:52:04.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.303+0000 7fe6cd5ad700 1 -- 172.21.15.78:0/1235623055 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d4003680 con 0x555d56321b60 2023-10-10T17:52:04.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/1235623055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 msgr2=0x555d56337cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d56337cb0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4016040 tx=0x7fe6d401b5b0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/1235623055 shutdown_connections 2023-10-10T17:52:04.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d56324e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56321b60 0x555d56337cb0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/1235623055 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56322470 0x555d56324930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/1235623055 >> 172.21.15.78:0/1235623055 conn(0x555d56317360 msgr2=0x555d5632bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/1235623055 shutdown_connections 2023-10-10T17:52:04.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/1235623055 wait complete. 2023-10-10T17:52:04.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 -- start start 2023-10-10T17:52:04.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 Processor -- start 2023-10-10T17:52:04.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56321b60 0x555d5632ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.304+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56330460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d563309a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d56332f80 con 0x555d56322470 2023-10-10T17:52:04.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5638e8a0 con 0x555d563268b0 2023-10-10T17:52:04.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5638e9e0 con 0x555d56321b60 2023-10-10T17:52:04.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 10 client.0 _ll_register_callbacks cb 0x7fe6d401d3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:04.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56330460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56330460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48846/0 (socket says 172.21.15.78:48846) 2023-10-10T17:52:04.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 -- 172.21.15.78:0/2734235718 learned_addr learned my addr 172.21.15.78:0/2734235718 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6de90b700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d563309a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 -- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56321b60 msgr2=0x555d5632ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6de10a700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56321b60 0x555d5632ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56321b60 0x555d5632ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 -- 172.21.15.78:0/2734235718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 msgr2=0x555d563309a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d563309a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 -- 172.21.15.78:0/2734235718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d56239190 con 0x555d56322470 2023-10-10T17:52:04.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6de90b700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d563309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:04.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6de10a700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56321b60 0x555d5632ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:04.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6dd909700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56330460 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80ca7c0 tx=0x7fe6c801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf42083a134f21c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6c1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:04.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/2734235718 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c809a5c0 con 0x555d56322470 2023-10-10T17:52:04.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/2734235718 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7fe6c8099780 con 0x555d56322470 2023-10-10T17:52:04.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6c80a24e0 con 0x555d56322470 2023-10-10T17:52:04.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/2734235718 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6c80b8250 con 0x555d56322470 2023-10-10T17:52:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 10 client.17436 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x555d560a0de0 con 0x555d56322470 2023-10-10T17:52:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.305+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/2734235718 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe6c80ca020 con 0x555d56322470 2023-10-10T17:52:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/2734235718 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fe6c8099180 con 0x555d56322470 2023-10-10T17:52:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6ec4fab80 10 client.17436 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6ec4fab80 10 client.17436 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:52:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6ec4fab80 20 client.17436 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5638fcb0 0x555d56392090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6ec4fab80 4 client.17436 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:52:04.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555d56241660 con 0x555d5638fcb0 2023-10-10T17:52:04.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6de10a700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5638fcb0 0x555d56392090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6de10a700 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5638fcb0 0x555d56392090 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=716c4dcab51ed89f server_cookie=628f2cd3c154dcfa in_seq=0 out_seq=0 2023-10-10T17:52:04.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6c1ffb700 10 client.17436 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:04.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6c1ffb700 1 -- 172.21.15.78:0/2734235718 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555d56241660 con 0x555d5638fcb0 2023-10-10T17:52:04.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6c1ffb700 10 client.17436 handle_command_reply: tid=0 2023-10-10T17:52:04.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.306+0000 7fe6ec4fab80 1 client.17436 shutdown 2023-10-10T17:52:04.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 msgr2=0x555d56330460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56330460 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe6c80ca7c0 tx=0x7fe6c801f000 comp rx=0 tx=0).stop 2023-10-10T17:52:04.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 shutdown_connections 2023-10-10T17:52:04.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x555d5638fcb0 0x555d56392090 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x555d56321b60 0x555d5632ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6c1ffb700 0 client.17436 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:04.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x555d563268b0 0x555d563309a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 --2- 172.21.15.78:0/2734235718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x555d56322470 0x555d56330460 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 >> 172.21.15.78:0/2734235718 conn(0x555d56317360 msgr2=0x555d56324930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 shutdown_connections 2023-10-10T17:52:04.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 1 -- 172.21.15.78:0/2734235718 wait complete. 2023-10-10T17:52:04.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6ec4fab80 20 client.17436 trim_cache size 0 max 16384 2023-10-10T17:52:04.352 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:52:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe6c40a12c0 msgr2=0x7fe6c40a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe6c40a12c0 0x7fe6c40a3770 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe6d4009be0 tx=0x7fe6d4009260 comp rx=0 tx=0).stop 2023-10-10T17:52:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 msgr2=0x7fe6d8196a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d8196a00 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fe6c800e9a0 tx=0x7fe6c800ecb0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 shutdown_connections 2023-10-10T17:52:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fe6d8068e70 0x7fe6d81964c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fe6c40a12c0 0x7fe6c40a3770 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fe6d8106360 0x7fe6d819ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 --2- 172.21.15.78:0/500332561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6d8069820 0x7fe6d8196a00 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 >> 172.21.15.78:0/500332561 conn(0x7fe6d8075470 msgr2=0x7fe6d80fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 shutdown_connections 2023-10-10T17:52:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.307+0000 7fe6e036e700 1 -- 172.21.15.78:0/500332561 wait complete. 2023-10-10T17:52:04.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:52:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 -- 172.21.15.78:0/1594970771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24640ff300 msgr2=0x7f246410b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 --2- 172.21.15.78:0/1594970771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24640ff300 0x7f246410b790 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f245c00c210 tx=0x7f245c00c520 comp rx=0 tx=0).stop 2023-10-10T17:52:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 -- 172.21.15.78:0/1594970771 shutdown_connections 2023-10-10T17:52:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 --2- 172.21.15.78:0/1594970771 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2464104960 0x7f2464104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 --2- 172.21.15.78:0/1594970771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24640ff300 0x7f246410b790 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 --2- 172.21.15.78:0/1594970771 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24640fe950 0x7f24640fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 -- 172.21.15.78:0/1594970771 >> 172.21.15.78:0/1594970771 conn(0x7f2464075920 msgr2=0x7f2464075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 -- 172.21.15.78:0/1594970771 shutdown_connections 2023-10-10T17:52:04.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 -- 172.21.15.78:0/1594970771 wait complete. 2023-10-10T17:52:04.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.480+0000 7f2469f0c700 1 Processor -- start 2023-10-10T17:52:04.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2469f0c700 1 -- start start 2023-10-10T17:52:04.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2469f0c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24640fe950 0x7f2464196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2469f0c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24640ff300 0x7f2464196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2469f0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2464104960 0x7f246419ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2469f0c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246419b4b0 con 0x7f2464104960 2023-10-10T17:52:04.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2469f0c700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246419b620 con 0x7f24640fe950 2023-10-10T17:52:04.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2469f0c700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246419b7e0 con 0x7f24640ff300 2023-10-10T17:52:04.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2464104960 0x7f246419ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2464104960 0x7f246419ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48862/0 (socket says 172.21.15.78:48862) 2023-10-10T17:52:04.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 -- 172.21.15.78:0/4050283765 learned_addr learned my addr 172.21.15.78:0/4050283765 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2462ffd700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24640ff300 0x7f2464196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f24637fe700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24640fe950 0x7f2464196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 -- 172.21.15.78:0/4050283765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24640ff300 msgr2=0x7f2464196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24640ff300 0x7f2464196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 -- 172.21.15.78:0/4050283765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24640fe950 msgr2=0x7f2464196520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24640fe950 0x7f2464196520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 -- 172.21.15.78:0/4050283765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c00ae10 con 0x7f2464104960 2023-10-10T17:52:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.481+0000 7f2463fff700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2464104960 0x7f246419ae70 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f245c0062a0 tx=0x7f245c0082f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.482+0000 7f2460ff9700 1 -- 172.21.15.78:0/4050283765 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c00f040 con 0x7f2464104960 2023-10-10T17:52:04.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.482+0000 7f2460ff9700 1 -- 172.21.15.78:0/4050283765 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f245c00a4e0 con 0x7f2464104960 2023-10-10T17:52:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.482+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246410e100 con 0x7f2464104960 2023-10-10T17:52:04.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.482+0000 7f2460ff9700 1 -- 172.21.15.78:0/4050283765 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c01e7f0 con 0x7f2464104960 2023-10-10T17:52:04.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.482+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246410df70 con 0x7f2464104960 2023-10-10T17:52:04.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.484+0000 7f2460ff9700 1 -- 172.21.15.78:0/4050283765 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f245c0074a0 con 0x7f2464104960 2023-10-10T17:52:04.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.484+0000 7f2460ff9700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f24500a12c0 0x7f24500a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.484+0000 7f2460ff9700 1 -- 172.21.15.78:0/4050283765 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f245c0cec00 con 0x7f2464104960 2023-10-10T17:52:04.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.484+0000 7f24637fe700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f24500a12c0 0x7f24500a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f24637fe700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f24500a12c0 0x7f24500a3770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f244c005950 tx=0x7f244c0058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 Processor -- start 2023-10-10T17:52:04.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 -- start start 2023-10-10T17:52:04.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebb9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebba470 0x55cc9ebbe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebbed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9eabf060 con 0x55cc9ebb9b60 2023-10-10T17:52:04.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9eb118a0 con 0x55cc9ebba470 2023-10-10T17:52:04.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2476098b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9eb0ad30 con 0x55cc9ebbe8b0 2023-10-10T17:52:04.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f24637fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebb9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f24637fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebb9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48866/0 (socket says 172.21.15.78:48866) 2023-10-10T17:52:04.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f24637fe700 1 -- 172.21.15.78:0/1293475769 learned_addr learned my addr 172.21.15.78:0/1293475769 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2462ffd700 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebba470 0x55cc9ebbe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.485+0000 7f2463fff700 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebbed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f24637fe700 1 -- 172.21.15.78:0/1293475769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 msgr2=0x55cc9ebbed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f24637fe700 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebbed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f24637fe700 1 -- 172.21.15.78:0/1293475769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebba470 msgr2=0x55cc9ebbe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f24637fe700 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebba470 0x55cc9ebbe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f24637fe700 1 -- 172.21.15.78:0/1293475769 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc9e98c630 con 0x55cc9ebb9b60 2023-10-10T17:52:04.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f24637fe700 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebb9f30 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f244c00fcc0 tx=0x7f244c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b000a5beb503264a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f2459ffb700 1 -- 172.21.15.78:0/1293475769 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c011c40 con 0x55cc9ebb9b60 2023-10-10T17:52:04.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f2459ffb700 1 -- 172.21.15.78:0/1293475769 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f244c011da0 con 0x55cc9ebb9b60 2023-10-10T17:52:04.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f2459ffb700 1 -- 172.21.15.78:0/1293475769 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c01d610 con 0x55cc9ebb9b60 2023-10-10T17:52:04.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f2476098b80 1 -- 172.21.15.78:0/1293475769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 msgr2=0x55cc9ebb9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.486+0000 7f2476098b80 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebb9f30 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f244c00fcc0 tx=0x7f244c012000 comp rx=0 tx=0).stop 2023-10-10T17:52:04.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- 172.21.15.78:0/1293475769 shutdown_connections 2023-10-10T17:52:04.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebbed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebb9f30 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 --2- 172.21.15.78:0/1293475769 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebba470 0x55cc9ebbe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- 172.21.15.78:0/1293475769 >> 172.21.15.78:0/1293475769 conn(0x55cc9ebaf360 msgr2=0x55cc9ebb4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- 172.21.15.78:0/1293475769 shutdown_connections 2023-10-10T17:52:04.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- 172.21.15.78:0/1293475769 wait complete. 2023-10-10T17:52:04.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- start start 2023-10-10T17:52:04.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 Processor -- start 2023-10-10T17:52:04.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebc5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ebc3210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebc37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9eaac8e0 con 0x55cc9ebba470 2023-10-10T17:52:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9eaac580 con 0x55cc9ebb9b60 2023-10-10T17:52:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.487+0000 7f2476098b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9e98e920 con 0x55cc9ebbe8b0 2023-10-10T17:52:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2476098b80 10 client.0 _ll_register_callbacks cb 0x55cc9eaca780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ebc3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f24637fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebc5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ebc3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48874/0 (socket says 172.21.15.78:48874) 2023-10-10T17:52:04.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f24637fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebc5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50594/0 (socket says 172.21.15.78:50594) 2023-10-10T17:52:04.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 -- 172.21.15.78:0/1927315297 learned_addr learned my addr 172.21.15.78:0/1927315297 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2463fff700 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebc37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 -- 172.21.15.78:0/1927315297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 msgr2=0x55cc9ebc37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebc37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 -- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 msgr2=0x55cc9ebc5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebc5f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 -- 172.21.15.78:0/1927315297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c00f970 con 0x55cc9ebba470 2023-10-10T17:52:04.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f24637fe700 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebc5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:52:04.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2462ffd700 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ebc3210 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f24540126b0 tx=0x7f24540129c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c069eb04d1e492a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2442ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:04.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2442ffd700 1 -- 172.21.15.78:0/1927315297 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454017070 con 0x55cc9ebba470 2023-10-10T17:52:04.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2442ffd700 1 -- 172.21.15.78:0/1927315297 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f2454004050 con 0x55cc9ebba470 2023-10-10T17:52:04.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc9ebacb20 con 0x55cc9ebba470 2023-10-10T17:52:04.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2442ffd700 1 -- 172.21.15.78:0/1927315297 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2454004210 con 0x55cc9ebba470 2023-10-10T17:52:04.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2476098b80 10 client.17448 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:04.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.488+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55cc9ead1190 con 0x55cc9ebba470 2023-10-10T17:52:04.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2442ffd700 1 -- 172.21.15.78:0/1927315297 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55cc9ebacb20 con 0x55cc9ebba470 2023-10-10T17:52:04.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2442ffd700 1 -- 172.21.15.78:0/1927315297 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f2454010450 con 0x55cc9ebba470 2023-10-10T17:52:04.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2476098b80 10 client.17448 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:04.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2476098b80 10 client.17448 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:04.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2476098b80 20 client.17448 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:04.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2476098b80 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ebd3810 0x55cc9ebd5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2476098b80 4 client.17448 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:04.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc9eacd0a0 con 0x55cc9ebd3810 2023-10-10T17:52:04.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f24637fe700 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ebd3810 0x55cc9ebd5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f2442ffd700 10 client.17448 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:04.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.489+0000 7f24637fe700 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ebd3810 0x55cc9ebd5c20 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cc1b20fee1d0e5e server_cookie=fdef0b7439f9c3c9 in_seq=0 out_seq=0 2023-10-10T17:52:04.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2442ffd700 1 -- 172.21.15.78:0/1927315297 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55cc9eacd0a0 con 0x55cc9ebd3810 2023-10-10T17:52:04.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2442ffd700 10 client.17448 handle_command_reply: tid=0 2023-10-10T17:52:04.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 client.17448 shutdown 2023-10-10T17:52:04.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 msgr2=0x55cc9ebc3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ebc3210 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f24540126b0 tx=0x7f24540129c0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 shutdown_connections 2023-10-10T17:52:04.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebc37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ebd3810 0x55cc9ebd5c20 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebc5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 --2- 172.21.15.78:0/1927315297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ebc3210 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2442ffd700 0 client.17448 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 >> 172.21.15.78:0/1927315297 conn(0x55cc9ebaf360 msgr2=0x55cc9ebb3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 shutdown_connections 2023-10-10T17:52:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.490+0000 7f2476098b80 1 -- 172.21.15.78:0/1927315297 wait complete. 2023-10-10T17:52:04.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.491+0000 7f2476098b80 20 client.17448 trim_cache size 0 max 16384 2023-10-10T17:52:04.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 Processor -- start 2023-10-10T17:52:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 -- start start 2023-10-10T17:52:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebbd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebba470 0x55cc9ebcfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebb4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9ebc6790 con 0x55cc9ebb9b60 2023-10-10T17:52:04.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9eacd0a0 con 0x55cc9ebbe8b0 2023-10-10T17:52:04.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f2476098b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9ebac660 con 0x55cc9ebba470 2023-10-10T17:52:04.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.503+0000 7f24637fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebbd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebbd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48880/0 (socket says 172.21.15.78:48880) 2023-10-10T17:52:04.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 -- 172.21.15.78:0/821684466 learned_addr learned my addr 172.21.15.78:0/821684466 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 -- 172.21.15.78:0/821684466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebba470 msgr2=0x55cc9ebcfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 --2- 172.21.15.78:0/821684466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebba470 0x55cc9ebcfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 -- 172.21.15.78:0/821684466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebbe8b0 msgr2=0x55cc9ebb4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 --2- 172.21.15.78:0/821684466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebb4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 -- 172.21.15.78:0/821684466 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc9ebb0b00 con 0x55cc9ebb9b60 2023-10-10T17:52:04.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f24637fe700 1 --2- 172.21.15.78:0/821684466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebbd1a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f244c0117d0 tx=0x7f244c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac836d419ee8d794 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f245a7fc700 1 -- 172.21.15.78:0/821684466 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c02d030 con 0x55cc9ebb9b60 2023-10-10T17:52:04.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f245a7fc700 1 -- 172.21.15.78:0/821684466 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f244c02e040 con 0x55cc9ebb9b60 2023-10-10T17:52:04.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.504+0000 7f245a7fc700 1 -- 172.21.15.78:0/821684466 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c005330 con 0x55cc9ebb9b60 2023-10-10T17:52:04.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- 172.21.15.78:0/821684466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 msgr2=0x55cc9ebbd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 --2- 172.21.15.78:0/821684466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebbd1a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f244c0117d0 tx=0x7f244c015000 comp rx=0 tx=0).stop 2023-10-10T17:52:04.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- 172.21.15.78:0/821684466 shutdown_connections 2023-10-10T17:52:04.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 --2- 172.21.15.78:0/821684466 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ebb4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 --2- 172.21.15.78:0/821684466 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebb9b60 0x55cc9ebbd1a0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 --2- 172.21.15.78:0/821684466 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebba470 0x55cc9ebcfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- 172.21.15.78:0/821684466 >> 172.21.15.78:0/821684466 conn(0x55cc9ebaf360 msgr2=0x55cc9ebbda10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- 172.21.15.78:0/821684466 shutdown_connections 2023-10-10T17:52:04.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- 172.21.15.78:0/821684466 wait complete. 2023-10-10T17:52:04.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- start start 2023-10-10T17:52:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 Processor -- start 2023-10-10T17:52:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ec245b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ec24af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ec1e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9ec1ef30 con 0x55cc9ebba470 2023-10-10T17:52:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9ec1f0a0 con 0x55cc9ebb9b60 2023-10-10T17:52:04.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc9ec1f260 con 0x55cc9ebbe8b0 2023-10-10T17:52:04.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2476098b80 10 client.0 _ll_register_callbacks cb 0x7f2454009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:04.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.505+0000 7f2462ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ec24af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ec24af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48888/0 (socket says 172.21.15.78:48888) 2023-10-10T17:52:04.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 -- 172.21.15.78:0/2498324888 learned_addr learned my addr 172.21.15.78:0/2498324888 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:04.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f24637fe700 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ec245b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2463fff700 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ec1e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 -- 172.21.15.78:0/2498324888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 msgr2=0x55cc9ec1e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ec1e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 -- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 msgr2=0x55cc9ec245b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:04.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ec245b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 -- 172.21.15.78:0/2498324888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc9e98c630 con 0x55cc9ebba470 2023-10-10T17:52:04.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2462ffd700 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ec24af0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f2454009e70 tx=0x7f2454011ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b402faaf8f40f19 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:04.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2442ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:04.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2442ffd700 1 -- 172.21.15.78:0/2498324888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454017070 con 0x55cc9ebba470 2023-10-10T17:52:04.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f244c011da0 con 0x55cc9ebba470 2023-10-10T17:52:04.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2442ffd700 1 -- 172.21.15.78:0/2498324888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f2454000e40 con 0x55cc9ebba470 2023-10-10T17:52:04.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2442ffd700 1 -- 172.21.15.78:0/2498324888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24540092d0 con 0x55cc9ebba470 2023-10-10T17:52:04.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2476098b80 10 client.17454 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:04.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55cc9e938de0 con 0x55cc9ebba470 2023-10-10T17:52:04.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.506+0000 7f2442ffd700 1 -- 172.21.15.78:0/2498324888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f244c011da0 con 0x55cc9ebba470 2023-10-10T17:52:04.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2442ffd700 1 -- 172.21.15.78:0/2498324888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f24540050c0 con 0x55cc9ebba470 2023-10-10T17:52:04.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2476098b80 10 client.17454 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:04.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2476098b80 10 client.17454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:04.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2476098b80 20 client.17454 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:04.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2476098b80 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ec2c430 0x55cc9ec2e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:04.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2476098b80 4 client.17454 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:52:04.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cc9ead9660 con 0x55cc9ec2c430 2023-10-10T17:52:04.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f24637fe700 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ec2c430 0x55cc9ec2e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:04.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f2442ffd700 10 client.17454 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:04.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.507+0000 7f24637fe700 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ec2c430 0x55cc9ec2e840 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16865a78c3226abc server_cookie=4f6c5dae6944b802 in_seq=0 out_seq=0 2023-10-10T17:52:04.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2442ffd700 1 -- 172.21.15.78:0/2498324888 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cc9ead9660 con 0x55cc9ec2c430 2023-10-10T17:52:04.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2442ffd700 10 client.17454 handle_command_reply: tid=0 2023-10-10T17:52:04.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 client.17454 shutdown 2023-10-10T17:52:04.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 msgr2=0x55cc9ec24af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ec24af0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f2454009e70 tx=0x7f2454011ff0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 shutdown_connections 2023-10-10T17:52:04.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55cc9ebbe8b0 0x55cc9ec1e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55cc9ec2c430 0x55cc9ec2e840 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55cc9ebb9b60 0x55cc9ec245b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2442ffd700 0 client.17454 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:04.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 --2- 172.21.15.78:0/2498324888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55cc9ebba470 0x55cc9ec24af0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 >> 172.21.15.78:0/2498324888 conn(0x55cc9ebaf360 msgr2=0x55cc9ebcfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 shutdown_connections 2023-10-10T17:52:04.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 1 -- 172.21.15.78:0/2498324888 wait complete. 2023-10-10T17:52:04.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.508+0000 7f2476098b80 20 client.17454 trim_cache size 0 max 16384 2023-10-10T17:52:04.560 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:52:04.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f24500a12c0 msgr2=0x7f24500a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f24500a12c0 0x7f24500a3770 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f244c005950 tx=0x7f244c0058e0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2464104960 msgr2=0x7f246419ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:04.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2464104960 0x7f246419ae70 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f245c0062a0 tx=0x7f245c0082f0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 shutdown_connections 2023-10-10T17:52:04.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f24500a12c0 0x7f24500a3770 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2464104960 0x7f246419ae70 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f24640fe950 0x7f2464196520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 --2- 172.21.15.78:0/4050283765 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f24640ff300 0x7f2464196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 >> 172.21.15.78:0/4050283765 conn(0x7f2464075920 msgr2=0x7f24640fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 shutdown_connections 2023-10-10T17:52:04.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:04.509+0000 7f2469f0c700 1 -- 172.21.15.78:0/4050283765 wait complete. 2023-10-10T17:52:05.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:04 smithi078 ceph-mon[99494]: pgmap v295: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:52:05.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:04 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2726187540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:52:05.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:04 smithi190 ceph-mon[101220]: pgmap v295: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:52:05.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:04 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2726187540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:52:05.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:04 smithi157 ceph-mon[100896]: pgmap v295: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-10-10T17:52:05.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:04 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2726187540' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:52:05.567 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-10-10T17:52:05.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-10-10T17:52:05.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.704+0000 7f46ecf07700 1 -- 172.21.15.78:0/3215399297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 msgr2=0x7f46e810a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.704+0000 7f46ecf07700 1 --2- 172.21.15.78:0/3215399297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e810a370 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f46dc009990 tx=0x7f46dc009ca0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 -- 172.21.15.78:0/3215399297 shutdown_connections 2023-10-10T17:52:05.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 --2- 172.21.15.78:0/3215399297 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46e8101390 0x7f46e8101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 --2- 172.21.15.78:0/3215399297 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46e810a8b0 0x7f46e810ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 --2- 172.21.15.78:0/3215399297 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e810a370 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 -- 172.21.15.78:0/3215399297 >> 172.21.15.78:0/3215399297 conn(0x7f46e80770b0 msgr2=0x7f46e80774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 -- 172.21.15.78:0/3215399297 shutdown_connections 2023-10-10T17:52:05.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 -- 172.21.15.78:0/3215399297 wait complete. 2023-10-10T17:52:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 Processor -- start 2023-10-10T17:52:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.705+0000 7f46ecf07700 1 -- start start 2023-10-10T17:52:05.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46e8101390 0x7f46e8196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e8196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46e810a8b0 0x7f46e819add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e819b410 con 0x7f46e8101ca0 2023-10-10T17:52:05.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e819b580 con 0x7f46e810a8b0 2023-10-10T17:52:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e819b740 con 0x7f46e8101390 2023-10-10T17:52:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e8196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e659c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46e8101390 0x7f46e8196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e8196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48902/0 (socket says 172.21.15.78:48902) 2023-10-10T17:52:05.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 -- 172.21.15.78:0/562074647 learned_addr learned my addr 172.21.15.78:0/562074647 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46e810a8b0 0x7f46e819add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 -- 172.21.15.78:0/562074647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46e8101390 msgr2=0x7f46e8196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46e8101390 0x7f46e8196430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 -- 172.21.15.78:0/562074647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46e810a8b0 msgr2=0x7f46e819add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46e810a8b0 0x7f46e819add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 -- 172.21.15.78:0/562074647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc0095f0 con 0x7f46e8101ca0 2023-10-10T17:52:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e8196970 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0095c0 tx=0x7f46dc00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46d77fe700 1 -- 172.21.15.78:0/562074647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc01d070 con 0x7f46e8101ca0 2023-10-10T17:52:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e8110390 con 0x7f46e8101ca0 2023-10-10T17:52:05.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46d77fe700 1 -- 172.21.15.78:0/562074647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46dc005030 con 0x7f46e8101ca0 2023-10-10T17:52:05.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46d77fe700 1 -- 172.21.15.78:0/562074647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc004a20 con 0x7f46e8101ca0 2023-10-10T17:52:05.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.706+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e8110200 con 0x7f46e8101ca0 2023-10-10T17:52:05.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.708+0000 7f46d77fe700 1 -- 172.21.15.78:0/562074647 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f46dc005410 con 0x7f46e8101ca0 2023-10-10T17:52:05.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.709+0000 7f46d77fe700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46d00a12c0 0x7f46d00a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.709+0000 7f46d77fe700 1 -- 172.21.15.78:0/562074647 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f46dc0cdef0 con 0x7f46e8101ca0 2023-10-10T17:52:05.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.709+0000 7f46e659c700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46d00a12c0 0x7f46d00a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.709+0000 7f46e659c700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46d00a12c0 0x7f46d00a3770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f46d8005950 tx=0x7f46d800b410 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 Processor -- start 2023-10-10T17:52:05.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 -- start start 2023-10-10T17:52:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9f93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615ec9fd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9fdd40 0x5615ec9fe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec8fa0e0 con 0x5615ec9f9900 2023-10-10T17:52:05.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec94c920 con 0x5615ec9fdd40 2023-10-10T17:52:05.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46f9093b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec945db0 con 0x5615ec9f8ff0 2023-10-10T17:52:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.710+0000 7f46e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615ec9fd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615ec9fd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48910/0 (socket says 172.21.15.78:48910) 2023-10-10T17:52:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 -- 172.21.15.78:0/4282167754 learned_addr learned my addr 172.21.15.78:0/4282167754 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9fdd40 0x5615ec9fe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e659c700 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9f93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 -- 172.21.15.78:0/4282167754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 msgr2=0x5615ec9f93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9f93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 -- 172.21.15.78:0/4282167754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9fdd40 msgr2=0x5615ec9fe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9fdd40 0x5615ec9fe190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 -- 172.21.15.78:0/4282167754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615ec7c76b0 con 0x5615ec9f9900 2023-10-10T17:52:05.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615ec9fd800 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0a2b50 tx=0x7f46dc09a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12eeda3c32b81e6a server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46d4ff9700 1 -- 172.21.15.78:0/4282167754 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc099060 con 0x5615ec9f9900 2023-10-10T17:52:05.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46d4ff9700 1 -- 172.21.15.78:0/4282167754 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46dc09af20 con 0x5615ec9f9900 2023-10-10T17:52:05.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.711+0000 7f46d4ff9700 1 -- 172.21.15.78:0/4282167754 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0a2e50 con 0x5615ec9f9900 2023-10-10T17:52:05.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 -- 172.21.15.78:0/4282167754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 msgr2=0x5615ec9fd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615ec9fd800 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0a2b50 tx=0x7f46dc09a6f0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 -- 172.21.15.78:0/4282167754 shutdown_connections 2023-10-10T17:52:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9f93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9fdd40 0x5615ec9fe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 --2- 172.21.15.78:0/4282167754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615ec9fd800 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 -- 172.21.15.78:0/4282167754 >> 172.21.15.78:0/4282167754 conn(0x5615ec9ea3e0 msgr2=0x5615ec9efbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 -- 172.21.15.78:0/4282167754 shutdown_connections 2023-10-10T17:52:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 -- 172.21.15.78:0/4282167754 wait complete. 2023-10-10T17:52:05.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 -- start start 2023-10-10T17:52:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.712+0000 7f46f9093b80 1 Processor -- start 2023-10-10T17:52:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca05490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615eca026a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9fdd40 0x5615eca02c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46f9093b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec8e7960 con 0x5615ec9f9900 2023-10-10T17:52:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46f9093b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec8e7600 con 0x5615ec9f8ff0 2023-10-10T17:52:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46f9093b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec7c99a0 con 0x5615ec9fdd40 2023-10-10T17:52:05.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46f9093b80 10 client.0 _ll_register_callbacks cb 0x5615ec905800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615eca026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615eca026a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48916/0 (socket says 172.21.15.78:48916) 2023-10-10T17:52:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 -- 172.21.15.78:0/402730112 learned_addr learned my addr 172.21.15.78:0/402730112 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9fdd40 0x5615eca02c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e659c700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca05490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 -- 172.21.15.78:0/402730112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9fdd40 msgr2=0x5615eca02c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9fdd40 0x5615eca02c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 -- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 msgr2=0x5615eca05490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca05490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 -- 172.21.15.78:0/402730112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e8110200 con 0x5615ec9f9900 2023-10-10T17:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e659c700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca05490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:05.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615eca026a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f46dc09ac00 tx=0x7f46dc0a31f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a183f011364904 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46cdffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:05.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.713+0000 7f46cdffb700 1 -- 172.21.15.78:0/402730112 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0a2bf0 con 0x5615ec9f9900 2023-10-10T17:52:05.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46cdffb700 1 -- 172.21.15.78:0/402730112 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f46dc0a2d50 con 0x5615ec9f9900 2023-10-10T17:52:05.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615ec9e7ba0 con 0x5615ec9f9900 2023-10-10T17:52:05.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46cdffb700 1 -- 172.21.15.78:0/402730112 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46dc004b80 con 0x5615ec9f9900 2023-10-10T17:52:05.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 10 client.17466 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:05.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5615ec90c210 con 0x5615ec9f9900 2023-10-10T17:52:05.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46cdffb700 1 -- 172.21.15.78:0/402730112 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46dc064080 con 0x5615ec9f9900 2023-10-10T17:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46cdffb700 1 -- 172.21.15.78:0/402730112 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f46dc096c50 con 0x5615ec9f9900 2023-10-10T17:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 10 client.17466 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:05.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 20 client.17466 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca12510 0x5615eca148f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 4 client.17466 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:05.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5615ec908120 con 0x5615eca12510 2023-10-10T17:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.714+0000 7f46e659c700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca12510 0x5615eca148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.715+0000 7f46e659c700 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca12510 0x5615eca148f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10f550166ee4fd0d server_cookie=9a95fb296ee3409b in_seq=0 out_seq=0 2023-10-10T17:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.715+0000 7f46cdffb700 10 client.17466 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46cdffb700 1 -- 172.21.15.78:0/402730112 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5615ec908120 con 0x5615eca12510 2023-10-10T17:52:05.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46cdffb700 10 client.17466 handle_command_reply: tid=0 2023-10-10T17:52:05.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 client.17466 shutdown 2023-10-10T17:52:05.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 msgr2=0x5615eca026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615eca026a0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f46dc09ac00 tx=0x7f46dc0a31f0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 shutdown_connections 2023-10-10T17:52:05.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9fdd40 0x5615eca02c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca12510 0x5615eca148f0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca05490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 --2- 172.21.15.78:0/402730112 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9f9900 0x5615eca026a0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46cdffb700 0 client.17466 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 >> 172.21.15.78:0/402730112 conn(0x5615ec9ea3e0 msgr2=0x5615ec9eee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 shutdown_connections 2023-10-10T17:52:05.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 1 -- 172.21.15.78:0/402730112 wait complete. 2023-10-10T17:52:05.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.717+0000 7f46f9093b80 20 client.17466 trim_cache size 0 max 16384 2023-10-10T17:52:05.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 Processor -- start 2023-10-10T17:52:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 -- start start 2023-10-10T17:52:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca09c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f9900 0x5615eca015b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615ec9fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec9fa050 con 0x5615ec9fdd40 2023-10-10T17:52:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec9fc5f0 con 0x5615ec9f8ff0 2023-10-10T17:52:05.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46f9093b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec9e76e0 con 0x5615ec9f9900 2023-10-10T17:52:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615ec9fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615ec9fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48926/0 (socket says 172.21.15.78:48926) 2023-10-10T17:52:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 -- 172.21.15.78:0/1029393271 learned_addr learned my addr 172.21.15.78:0/1029393271 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f9900 0x5615eca015b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e659c700 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca09c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 -- 172.21.15.78:0/1029393271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f9900 msgr2=0x5615eca015b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f9900 0x5615eca015b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 -- 172.21.15.78:0/1029393271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 msgr2=0x5615eca09c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca09c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 -- 172.21.15.78:0/1029393271 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615ec9fe3e0 con 0x5615ec9fdd40 2023-10-10T17:52:05.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f9900 0x5615eca015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:05.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615ec9fffb0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f46e0009e50 tx=0x7f46e0009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fb7498df5c6083 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46d57fa700 1 -- 172.21.15.78:0/1029393271 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0017440 con 0x5615ec9fdd40 2023-10-10T17:52:05.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46d57fa700 1 -- 172.21.15.78:0/1029393271 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46e000f040 con 0x5615ec9fdd40 2023-10-10T17:52:05.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.733+0000 7f46d57fa700 1 -- 172.21.15.78:0/1029393271 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e001f8d0 con 0x5615ec9fdd40 2023-10-10T17:52:05.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 -- 172.21.15.78:0/1029393271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 msgr2=0x5615ec9fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615ec9fffb0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f46e0009e50 tx=0x7f46e0009100 comp rx=0 tx=0).stop 2023-10-10T17:52:05.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 -- 172.21.15.78:0/1029393271 shutdown_connections 2023-10-10T17:52:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615ec9fffb0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f8ff0 0x5615eca09c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 --2- 172.21.15.78:0/1029393271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f9900 0x5615eca015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 -- 172.21.15.78:0/1029393271 >> 172.21.15.78:0/1029393271 conn(0x5615ec9ea3e0 msgr2=0x5615eca00ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 -- 172.21.15.78:0/1029393271 shutdown_connections 2023-10-10T17:52:05.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 -- 172.21.15.78:0/1029393271 wait complete. 2023-10-10T17:52:05.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 -- start start 2023-10-10T17:52:05.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.734+0000 7f46f9093b80 1 Processor -- start 2023-10-10T17:52:05.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9ef030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f9900 0x5615ec9ef570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46f9093b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615eca66320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46f9093b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615ec9efd10 con 0x5615ec9fdd40 2023-10-10T17:52:05.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46f9093b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615eca66890 con 0x5615ec9f9900 2023-10-10T17:52:05.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46f9093b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615eca66a50 con 0x5615ec9f8ff0 2023-10-10T17:52:05.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46f9093b80 10 client.0 _ll_register_callbacks cb 0x5615ec905800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:05.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615eca66320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615eca66320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48932/0 (socket says 172.21.15.78:48932) 2023-10-10T17:52:05.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 -- 172.21.15.78:0/3100427021 learned_addr learned my addr 172.21.15.78:0/3100427021 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e659c700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9ef030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f9900 0x5615ec9ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 -- 172.21.15.78:0/3100427021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 msgr2=0x5615ec9ef030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9ef030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 -- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f9900 msgr2=0x5615ec9ef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f9900 0x5615ec9ef570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 -- 172.21.15.78:0/3100427021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e0009b30 con 0x5615ec9fdd40 2023-10-10T17:52:05.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e5d9b700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f9900 0x5615ec9ef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:05.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46e6d9d700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615eca66320 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f46e000b010 tx=0x7f46e00053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4afdd5f3920a2a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46cdffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:05.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.735+0000 7f46cdffb700 1 -- 172.21.15.78:0/3100427021 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e0005850 con 0x5615ec9fdd40 2023-10-10T17:52:05.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46dc09af20 con 0x5615ec9fdd40 2023-10-10T17:52:05.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46cdffb700 1 -- 172.21.15.78:0/3100427021 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f46e00059b0 con 0x5615ec9fdd40 2023-10-10T17:52:05.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46cdffb700 1 -- 172.21.15.78:0/3100427021 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46e0003a40 con 0x5615ec9fdd40 2023-10-10T17:52:05.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 10 client.17472 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:05.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5615ec773e60 con 0x5615ec9fdd40 2023-10-10T17:52:05.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46cdffb700 1 -- 172.21.15.78:0/3100427021 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f46dc09af20 con 0x5615ec9fdd40 2023-10-10T17:52:05.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46cdffb700 1 -- 172.21.15.78:0/3100427021 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f46e0014050 con 0x5615ec9fdd40 2023-10-10T17:52:05.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 10 client.17472 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 20 client.17472 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca681f0 0x5615eca6a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 4 client.17472 mds_command: new command op to 24500 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-10-10T17:52:05.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5615ec9146e0 con 0x5615eca681f0 2023-10-10T17:52:05.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.736+0000 7f46e659c700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca681f0 0x5615eca6a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.737+0000 7f46e659c700 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca681f0 0x5615eca6a600 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce1ab9f2c0e43b7f server_cookie=afdd74af28bc092b in_seq=0 out_seq=0 2023-10-10T17:52:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.737+0000 7f46cdffb700 10 client.17472 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46cdffb700 1 -- 172.21.15.78:0/3100427021 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5615ec9146e0 con 0x5615eca681f0 2023-10-10T17:52:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46cdffb700 10 client.17472 handle_command_reply: tid=0 2023-10-10T17:52:05.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 client.17472 shutdown 2023-10-10T17:52:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 msgr2=0x5615eca66320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615eca66320 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f46e000b010 tx=0x7f46e00053a0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 shutdown_connections 2023-10-10T17:52:05.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5615ec9f8ff0 0x5615ec9ef030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5615eca681f0 0x5615eca6a600 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5615ec9fdd40 0x5615eca66320 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 --2- 172.21.15.78:0/3100427021 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5615ec9f9900 0x5615ec9ef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46cdffb700 0 client.17472 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 >> 172.21.15.78:0/3100427021 conn(0x5615ec9ea3e0 msgr2=0x5615ec9f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 shutdown_connections 2023-10-10T17:52:05.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 1 -- 172.21.15.78:0/3100427021 wait complete. 2023-10-10T17:52:05.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.738+0000 7f46f9093b80 20 client.17472 trim_cache size 0 max 16384 2023-10-10T17:52:05.779 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "return_code": 0, 2023-10-10T17:52:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "scrub_tag": "3a991db7-a377-4138-93a1-0797c702d854", 2023-10-10T17:52:05.780 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": "asynchronous" 2023-10-10T17:52:05.780 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46d00a12c0 msgr2=0x7f46d00a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46d00a12c0 0x7f46d00a3770 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f46d8005950 tx=0x7f46d800b410 comp rx=0 tx=0).stop 2023-10-10T17:52:05.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 msgr2=0x7f46e8196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e8196970 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0095c0 tx=0x7f46dc00f690 comp rx=0 tx=0).stop 2023-10-10T17:52:05.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 shutdown_connections 2023-10-10T17:52:05.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f46e8101390 0x7f46e8196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f46d00a12c0 0x7f46d00a3770 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f46e810a8b0 0x7f46e819add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 --2- 172.21.15.78:0/562074647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46e8101ca0 0x7f46e8196970 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 >> 172.21.15.78:0/562074647 conn(0x7f46e80770b0 msgr2=0x7f46e80fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 shutdown_connections 2023-10-10T17:52:05.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.739+0000 7f46ecf07700 1 -- 172.21.15.78:0/562074647 wait complete. 2023-10-10T17:52:05.785 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:52:05.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 -- 172.21.15.78:0/777581016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 msgr2=0x7f10400ffe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 --2- 172.21.15.78:0/777581016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f10400ffe00 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f103c0099c0 tx=0x7f103c009cd0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 -- 172.21.15.78:0/777581016 shutdown_connections 2023-10-10T17:52:05.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 --2- 172.21.15.78:0/777581016 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1040104180 0x7f1040104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 --2- 172.21.15.78:0/777581016 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1040100340 0x7f104010c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 --2- 172.21.15.78:0/777581016 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f10400ffe00 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 -- 172.21.15.78:0/777581016 >> 172.21.15.78:0/777581016 conn(0x7f1040075920 msgr2=0x7f1040075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 -- 172.21.15.78:0/777581016 shutdown_connections 2023-10-10T17:52:05.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 -- 172.21.15.78:0/777581016 wait complete. 2023-10-10T17:52:05.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 Processor -- start 2023-10-10T17:52:05.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.909+0000 7f1047ba4700 1 -- start start 2023-10-10T17:52:05.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f1040196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1040100340 0x7f10401968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1040104180 0x7f104019ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104019b340 con 0x7f10400ff990 2023-10-10T17:52:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104019b4b0 con 0x7f1040100340 2023-10-10T17:52:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104019b670 con 0x7f1040104180 2023-10-10T17:52:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f1040196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f1040196360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48942/0 (socket says 172.21.15.78:48942) 2023-10-10T17:52:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 -- 172.21.15.78:0/2849653663 learned_addr learned my addr 172.21.15.78:0/2849653663 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1046141700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1040104180 0x7f104019ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 -- 172.21.15.78:0/2849653663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1040104180 msgr2=0x7f104019ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1040104180 0x7f104019ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 -- 172.21.15.78:0/2849653663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1040100340 msgr2=0x7f10401968a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f104513f700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1040100340 0x7f10401968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1040100340 0x7f10401968a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 -- 172.21.15.78:0/2849653663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c0095f0 con 0x7f10400ff990 2023-10-10T17:52:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f104513f700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1040100340 0x7f10401968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:05.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1045940700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f1040196360 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f103000c920 tx=0x7f103000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1036ffd700 1 -- 172.21.15.78:0/2849653663 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030007b00 con 0x7f10400ff990 2023-10-10T17:52:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104010f210 con 0x7f10400ff990 2023-10-10T17:52:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1036ffd700 1 -- 172.21.15.78:0/2849653663 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1030004cf0 con 0x7f10400ff990 2023-10-10T17:52:05.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1036ffd700 1 -- 172.21.15.78:0/2849653663 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1030005610 con 0x7f10400ff990 2023-10-10T17:52:05.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.910+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104010f080 con 0x7f10400ff990 2023-10-10T17:52:05.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.912+0000 7f1036ffd700 1 -- 172.21.15.78:0/2849653663 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f1030004750 con 0x7f10400ff990 2023-10-10T17:52:05.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.913+0000 7f1036ffd700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f102c0a12c0 0x7f102c0a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.913+0000 7f1036ffd700 1 -- 172.21.15.78:0/2849653663 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f10300ccfd0 con 0x7f10400ff990 2023-10-10T17:52:05.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.913+0000 7f104513f700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f102c0a12c0 0x7f102c0a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.913+0000 7f104513f700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f102c0a12c0 0x7f102c0a3770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f103c006010 tx=0x7f103c000bc0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.914+0000 7f1053d30b80 1 Processor -- start 2023-10-10T17:52:05.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1053d30b80 1 -- start start 2023-10-10T17:52:05.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e5f70 0x5570d48e6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e6880 0x5570d48ea780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48eacc0 0x5570d48eb110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1053d30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d47e7060 con 0x5570d48eacc0 2023-10-10T17:52:05.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1053d30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d48398a0 con 0x5570d48e5f70 2023-10-10T17:52:05.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1053d30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d4832d30 con 0x5570d48e6880 2023-10-10T17:52:05.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48eacc0 0x5570d48eb110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48eacc0 0x5570d48eb110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48954/0 (socket says 172.21.15.78:48954) 2023-10-10T17:52:05.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 -- 172.21.15.78:0/3952714125 learned_addr learned my addr 172.21.15.78:0/3952714125 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f104513f700 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e6880 0x5570d48ea780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1045940700 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e5f70 0x5570d48e6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 -- 172.21.15.78:0/3952714125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e6880 msgr2=0x5570d48ea780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e6880 0x5570d48ea780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 -- 172.21.15.78:0/3952714125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e5f70 msgr2=0x5570d48e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e5f70 0x5570d48e6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 -- 172.21.15.78:0/3952714125 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d46b4630 con 0x5570d48eacc0 2023-10-10T17:52:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.915+0000 7f1046141700 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48eacc0 0x5570d48eb110 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1038009d90 tx=0x7f10380077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f5799ecc31ec86 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f102bfff700 1 -- 172.21.15.78:0/3952714125 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038019070 con 0x5570d48eacc0 2023-10-10T17:52:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f102bfff700 1 -- 172.21.15.78:0/3952714125 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f103800f050 con 0x5570d48eacc0 2023-10-10T17:52:05.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f102bfff700 1 -- 172.21.15.78:0/3952714125 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1038015df0 con 0x5570d48eacc0 2023-10-10T17:52:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 -- 172.21.15.78:0/3952714125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48eacc0 msgr2=0x5570d48eb110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48eacc0 0x5570d48eb110 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f1038009d90 tx=0x7f10380077e0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 -- 172.21.15.78:0/3952714125 shutdown_connections 2023-10-10T17:52:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48eacc0 0x5570d48eb110 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e5f70 0x5570d48e6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 --2- 172.21.15.78:0/3952714125 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e6880 0x5570d48ea780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 -- 172.21.15.78:0/3952714125 >> 172.21.15.78:0/3952714125 conn(0x5570d48d7360 msgr2=0x5570d48dcb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 -- 172.21.15.78:0/3952714125 shutdown_connections 2023-10-10T17:52:05.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.916+0000 7f1053d30b80 1 -- 172.21.15.78:0/3952714125 wait complete. 2023-10-10T17:52:05.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 -- start start 2023-10-10T17:52:05.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 Processor -- start 2023-10-10T17:52:05.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e5f70 0x5570d48f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e6880 0x5570d48ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48eacc0 0x5570d48efc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d47d48e0 con 0x5570d48e5f70 2023-10-10T17:52:05.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d47d4580 con 0x5570d48e6880 2023-10-10T17:52:05.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d46b6920 con 0x5570d48eacc0 2023-10-10T17:52:05.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1053d30b80 10 client.0 _ll_register_callbacks cb 0x5570d47f2780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1045940700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e5f70 0x5570d48f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1045940700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e5f70 0x5570d48f23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48956/0 (socket says 172.21.15.78:48956) 2023-10-10T17:52:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1045940700 1 -- 172.21.15.78:0/1355929791 learned_addr learned my addr 172.21.15.78:0/1355929791 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.917+0000 7f1046141700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48eacc0 0x5570d48efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f104513f700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e6880 0x5570d48ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1045940700 1 -- 172.21.15.78:0/1355929791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48eacc0 msgr2=0x5570d48efc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1045940700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48eacc0 0x5570d48efc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1045940700 1 -- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e6880 msgr2=0x5570d48ef6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1045940700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e6880 0x5570d48ef6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1045940700 1 -- 172.21.15.78:0/1355929791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10380099c0 con 0x5570d48e5f70 2023-10-10T17:52:05.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1046141700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48eacc0 0x5570d48efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f104513f700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e6880 0x5570d48ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:52:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1045940700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e5f70 0x5570d48f23b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1030099a10 tx=0x7f1030099d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206665714382ca32 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f10297fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f10297fa700 1 -- 172.21.15.78:0/1355929791 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10300b1070 con 0x5570d48e5f70 2023-10-10T17:52:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f10297fa700 1 -- 172.21.15.78:0/1355929791 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f1030007c60 con 0x5570d48e5f70 2023-10-10T17:52:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5570d48d4b20 con 0x5570d48e5f70 2023-10-10T17:52:05.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f10297fa700 1 -- 172.21.15.78:0/1355929791 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1030004a00 con 0x5570d48e5f70 2023-10-10T17:52:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 10 client.17484 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5570d47f9190 con 0x5570d48e5f70 2023-10-10T17:52:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f10297fa700 1 -- 172.21.15.78:0/1355929791 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10300c8080 con 0x5570d48e5f70 2023-10-10T17:52:05.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f10297fa700 1 -- 172.21.15.78:0/1355929791 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f103009a000 con 0x5570d48e5f70 2023-10-10T17:52:05.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 10 client.17484 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:05.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 10 client.17484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:05.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 20 client.17484 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d48ffbc0 0x5570d4901fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 4 client.17484 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.918+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570d47f50a0 con 0x5570d48ffbc0 2023-10-10T17:52:05.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.919+0000 7f104513f700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d48ffbc0 0x5570d4901fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.919+0000 7f10297fa700 10 client.17484 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.919+0000 7f104513f700 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d48ffbc0 0x5570d4901fd0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84c5e113a3625ddf server_cookie=6df7a357822cb6d6 in_seq=0 out_seq=0 2023-10-10T17:52:05.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f10297fa700 1 -- 172.21.15.78:0/1355929791 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5570d47f50a0 con 0x5570d48ffbc0 2023-10-10T17:52:05.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f10297fa700 10 client.17484 handle_command_reply: tid=0 2023-10-10T17:52:05.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 client.17484 shutdown 2023-10-10T17:52:05.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e5f70 msgr2=0x5570d48f23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e5f70 0x5570d48f23b0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f1030099a10 tx=0x7f1030099d10 comp rx=0 tx=0).stop 2023-10-10T17:52:05.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 shutdown_connections 2023-10-10T17:52:05.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48eacc0 0x5570d48efc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d48ffbc0 0x5570d4901fd0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e5f70 0x5570d48f23b0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 --2- 172.21.15.78:0/1355929791 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48e6880 0x5570d48ef6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f10297fa700 0 client.17484 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 >> 172.21.15.78:0/1355929791 conn(0x5570d48d7360 msgr2=0x5570d48ec170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 shutdown_connections 2023-10-10T17:52:05.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 1 -- 172.21.15.78:0/1355929791 wait complete. 2023-10-10T17:52:05.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.920+0000 7f1053d30b80 20 client.17484 trim_cache size 0 max 16384 2023-10-10T17:52:05.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 Processor -- start 2023-10-10T17:52:05.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 -- start start 2023-10-10T17:52:05.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d48f6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d48f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 0x5570d48dc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d48d4740 con 0x5570d48e6880 2023-10-10T17:52:05.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d48d6ee0 con 0x5570d48eacc0 2023-10-10T17:52:05.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1053d30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d48de1c0 con 0x5570d48e5f70 2023-10-10T17:52:05.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d48f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d48f1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48966/0 (socket says 172.21.15.78:48966) 2023-10-10T17:52:05.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 -- 172.21.15.78:0/672509870 learned_addr learned my addr 172.21.15.78:0/672509870 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1045940700 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d48f6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 -- 172.21.15.78:0/672509870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 msgr2=0x5570d48f6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d48f6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 -- 172.21.15.78:0/672509870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 msgr2=0x5570d48dc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 0x5570d48dc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 -- 172.21.15.78:0/672509870 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d48d8b00 con 0x5570d48e6880 2023-10-10T17:52:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f104513f700 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d48f1f50 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f103c015000 tx=0x7f103c01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66df292a6e94578d server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.937+0000 7f1034dac700 1 -- 172.21.15.78:0/672509870 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c034ea0 con 0x5570d48e6880 2023-10-10T17:52:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1034dac700 1 -- 172.21.15.78:0/672509870 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f103c02b030 con 0x5570d48e6880 2023-10-10T17:52:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1034dac700 1 -- 172.21.15.78:0/672509870 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c004620 con 0x5570d48e6880 2023-10-10T17:52:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 -- 172.21.15.78:0/672509870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 msgr2=0x5570d48f1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.965 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d48f1f50 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f103c015000 tx=0x7f103c01e000 comp rx=0 tx=0).stop 2023-10-10T17:52:05.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 -- 172.21.15.78:0/672509870 shutdown_connections 2023-10-10T17:52:05.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d48f6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 0x5570d48dc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 --2- 172.21.15.78:0/672509870 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d48f1f50 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 -- 172.21.15.78:0/672509870 >> 172.21.15.78:0/672509870 conn(0x5570d48d7360 msgr2=0x5570d48fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 -- 172.21.15.78:0/672509870 shutdown_connections 2023-10-10T17:52:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 -- 172.21.15.78:0/672509870 wait complete. 2023-10-10T17:52:05.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 -- start start 2023-10-10T17:52:05.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.938+0000 7f1053d30b80 1 Processor -- start 2023-10-10T17:52:05.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d494cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d494d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1053d30b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 0x5570d48f4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1053d30b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d48f5370 con 0x5570d48e6880 2023-10-10T17:52:05.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1053d30b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d48f54e0 con 0x5570d48eacc0 2023-10-10T17:52:05.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1053d30b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d48f56a0 con 0x5570d48e5f70 2023-10-10T17:52:05.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1053d30b80 10 client.0 _ll_register_callbacks cb 0x7f1030099bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:05.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d494d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d494d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48972/0 (socket says 172.21.15.78:48972) 2023-10-10T17:52:05.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 -- 172.21.15.78:0/478165801 learned_addr learned my addr 172.21.15.78:0/478165801 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:05.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f1045940700 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d494cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 -- 172.21.15.78:0/478165801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 msgr2=0x5570d494cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d494cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 -- 172.21.15.78:0/478165801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 msgr2=0x5570d48f4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:05.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 0x5570d48f4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 -- 172.21.15.78:0/478165801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104010c8a0 con 0x5570d48e6880 2023-10-10T17:52:05.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f104513f700 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d494d2e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f103c0098e0 tx=0x7f103c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fcd071e7c00109 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:05.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.939+0000 7f10297fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:05.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f10297fa700 1 -- 172.21.15.78:0/478165801 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c00bcc0 con 0x5570d48e6880 2023-10-10T17:52:05.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f10297fa700 1 -- 172.21.15.78:0/478165801 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9432+0+0 (secure 0 0 0) 0x7f103c00be20 con 0x5570d48e6880 2023-10-10T17:52:05.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f103800f050 con 0x5570d48e6880 2023-10-10T17:52:05.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f10297fa700 1 -- 172.21.15.78:0/478165801 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f103c034400 con 0x5570d48e6880 2023-10-10T17:52:05.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 10 client.17490 fetch_fsmap learned FSMap version 34 2023-10-10T17:52:05.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5570d4660de0 con 0x5570d48e6880 2023-10-10T17:52:05.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f10297fa700 1 -- 172.21.15.78:0/478165801 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f103800f050 con 0x5570d48e6880 2023-10-10T17:52:05.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f10297fa700 1 -- 172.21.15.78:0/478165801 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 34) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f103c0038c0 con 0x5570d48e6880 2023-10-10T17:52:05.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 10 client.17490 fetch_fsmap finished waiting for FSMap version 34 2023-10-10T17:52:05.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 10 client.17490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:05.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 20 client.17490 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:05.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d4954790 0x5570d4956ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:05.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 4 client.17490 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:52:05.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5570d4801660 con 0x5570d4954790 2023-10-10T17:52:05.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.940+0000 7f1045940700 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d4954790 0x5570d4956ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:05.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f10297fa700 10 client.17490 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1045940700 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d4954790 0x5570d4956ba0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4a4d55490d2ae69 server_cookie=b1177653b257b17a in_seq=0 out_seq=0 2023-10-10T17:52:05.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f10297fa700 1 -- 172.21.15.78:0/478165801 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5570d4801660 con 0x5570d4954790 2023-10-10T17:52:05.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f10297fa700 10 client.17490 handle_command_reply: tid=0 2023-10-10T17:52:05.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 client.17490 shutdown 2023-10-10T17:52:05.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 msgr2=0x5570d494d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d494d2e0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f103c0098e0 tx=0x7f103c0057c0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 shutdown_connections 2023-10-10T17:52:05.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5570d48e5f70 0x5570d494cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5570d4954790 0x5570d4956ba0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5570d48eacc0 0x5570d48f4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 --2- 172.21.15.78:0/478165801 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5570d48e6880 0x5570d494d2e0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f10297fa700 0 client.17490 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:05.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 >> 172.21.15.78:0/478165801 conn(0x5570d48d7360 msgr2=0x5570d48e0960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 shutdown_connections 2023-10-10T17:52:05.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 1 -- 172.21.15.78:0/478165801 wait complete. 2023-10-10T17:52:05.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.941+0000 7f1053d30b80 20 client.17490 trim_cache size 0 max 16384 2023-10-10T17:52:05.982 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:05.982 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "IDLE (waiting for more scrubs)", 2023-10-10T17:52:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": { 2023-10-10T17:52:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "3a991db7-a377-4138-93a1-0797c702d854": { 2023-10-10T17:52:05.983 INFO:teuthology.orchestra.run.smithi078.stdout: "path": "/", 2023-10-10T17:52:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tag": "3a991db7-a377-4138-93a1-0797c702d854", 2023-10-10T17:52:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: "options": "recursive,force" 2023-10-10T17:52:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:05.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:05.985 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:05.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f102c0a12c0 msgr2=0x7f102c0a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f102c0a12c0 0x7f102c0a3770 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f103c006010 tx=0x7f103c000bc0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 msgr2=0x7f1040196360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f1040196360 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f103000c920 tx=0x7f103000cce0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 shutdown_connections 2023-10-10T17:52:05.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1040104180 0x7f104019ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f102c0a12c0 0x7f102c0a3770 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f10400ff990 0x7f1040196360 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 --2- 172.21.15.78:0/2849653663 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1040100340 0x7f10401968a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 >> 172.21.15.78:0/2849653663 conn(0x7f1040075920 msgr2=0x7f10400fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 shutdown_connections 2023-10-10T17:52:05.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:05.942+0000 7f1047ba4700 1 -- 172.21.15.78:0/2849653663 wait complete. 2023-10-10T17:52:05.990 INFO:tasks.cephfs.filesystem:scrub status for tag:3a991db7-a377-4138-93a1-0797c702d854 - {'path': '/', 'tag': '3a991db7-a377-4138-93a1-0797c702d854', 'options': 'recursive,force'} 2023-10-10T17:52:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:06 smithi078 ceph-mon[99494]: pgmap v296: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:52:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:06 smithi078 ceph-mon[99494]: scrub queued for path: / 2023-10-10T17:52:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:06 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:52:07.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:06 smithi078 ceph-mon[99494]: scrub summary: active paths [/] 2023-10-10T17:52:07.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:06 smithi078 ceph-mon[99494]: scrub summary: idle+waiting paths [/] 2023-10-10T17:52:07.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:06 smithi190 ceph-mon[101220]: pgmap v296: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:52:07.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:06 smithi190 ceph-mon[101220]: scrub queued for path: / 2023-10-10T17:52:07.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:06 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:52:07.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:06 smithi190 ceph-mon[101220]: scrub summary: active paths [/] 2023-10-10T17:52:07.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:06 smithi190 ceph-mon[101220]: scrub summary: idle+waiting paths [/] 2023-10-10T17:52:07.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:06 smithi157 ceph-mon[100896]: pgmap v296: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-10-10T17:52:07.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:06 smithi157 ceph-mon[100896]: scrub queued for path: / 2023-10-10T17:52:07.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:06 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:52:07.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:06 smithi157 ceph-mon[100896]: scrub summary: active paths [/] 2023-10-10T17:52:07.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:06 smithi157 ceph-mon[100896]: scrub summary: idle+waiting paths [/] 2023-10-10T17:52:08.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:07 smithi078 ceph-mon[99494]: scrub summary: idle 2023-10-10T17:52:08.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:07 smithi078 ceph-mon[99494]: osdmap e101: 12 total, 12 up, 12 in 2023-10-10T17:52:08.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:07 smithi190 ceph-mon[101220]: scrub summary: idle 2023-10-10T17:52:08.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:07 smithi190 ceph-mon[101220]: osdmap e101: 12 total, 12 up, 12 in 2023-10-10T17:52:08.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:07 smithi157 ceph-mon[100896]: scrub summary: idle 2023-10-10T17:52:08.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:07 smithi157 ceph-mon[100896]: osdmap e101: 12 total, 12 up, 12 in 2023-10-10T17:52:09.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:08 smithi078 ceph-mon[99494]: pgmap v298: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:09.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:08 smithi190 ceph-mon[101220]: pgmap v298: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:09.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:08 smithi157 ceph-mon[100896]: pgmap v298: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:10.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:09 smithi078 ceph-mon[99494]: pgmap v299: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:10.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:09 smithi190 ceph-mon[101220]: pgmap v299: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:10.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:09 smithi157 ceph-mon[100896]: pgmap v299: 129 pgs: 129 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:12.733 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:12 smithi157 ceph-mon[100896]: pgmap v300: 129 pgs: 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 120 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:12.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:12 smithi078 ceph-mon[99494]: pgmap v300: 129 pgs: 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 120 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:12.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:12 smithi190 ceph-mon[101220]: pgmap v300: 129 pgs: 4 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 120 active+clean; 799 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-10-10T17:52:15.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:14 smithi078 ceph-mon[99494]: pgmap v301: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:52:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:14 smithi190 ceph-mon[101220]: pgmap v301: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:52:15.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:14 smithi157 ceph-mon[100896]: pgmap v301: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:52:17.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:16 smithi078 ceph-mon[99494]: pgmap v302: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:52:17.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:16 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:17.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:16 smithi190 ceph-mon[101220]: pgmap v302: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:52:17.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:16 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:16 smithi157 ceph-mon[100896]: pgmap v302: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:52:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:16 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:18.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:17 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:18.168 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:17 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:18.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:17 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:19.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:18 smithi078 ceph-mon[99494]: pgmap v303: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 96 B/s wr, 5 op/s 2023-10-10T17:52:19.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:18 smithi190 ceph-mon[101220]: pgmap v303: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 96 B/s wr, 5 op/s 2023-10-10T17:52:19.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:18 smithi157 ceph-mon[100896]: pgmap v303: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 96 B/s wr, 5 op/s 2023-10-10T17:52:20.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:19 smithi078 ceph-mon[99494]: pgmap v304: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-10T17:52:20.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:19 smithi190 ceph-mon[101220]: pgmap v304: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-10T17:52:20.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:19 smithi157 ceph-mon[100896]: pgmap v304: 129 pgs: 23 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 36 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2023-10-10T17:52:21.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:20 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:21.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:20 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:21.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:20 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:22.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:21 smithi190 ceph-mon[101220]: pgmap v305: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:22.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:21 smithi157 ceph-mon[100896]: pgmap v305: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:22.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:21 smithi078 ceph-mon[99494]: pgmap v305: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:25.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:24 smithi078 ceph-mon[99494]: pgmap v306: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:25.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:24 smithi190 ceph-mon[101220]: pgmap v306: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:25.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:24 smithi157 ceph-mon[100896]: pgmap v306: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:52:27.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:26 smithi078 ceph-mon[99494]: pgmap v307: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-10T17:52:27.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:26 smithi078 ceph-mon[99494]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:27.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:26 smithi190 ceph-mon[101220]: pgmap v307: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-10T17:52:27.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:26 smithi190 ceph-mon[101220]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:27.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:26 smithi157 ceph-mon[100896]: pgmap v307: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-10-10T17:52:27.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:26 smithi157 ceph-mon[100896]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 3 up:standby-replay 6 up:standby 2023-10-10T17:52:29.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:28 smithi078 ceph-mon[99494]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:29.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:28 smithi190 ceph-mon[101220]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:29.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:28 smithi157 ceph-mon[100896]: pgmap v308: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:31.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:30 smithi078 ceph-mon[99494]: pgmap v309: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-10T17:52:31.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:30 smithi190 ceph-mon[101220]: pgmap v309: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-10T17:52:31.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:30 smithi157 ceph-mon[100896]: pgmap v309: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-10T17:52:33.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:32 smithi078 ceph-mon[99494]: pgmap v310: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-10-10T17:52:33.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:32 smithi190 ceph-mon[101220]: pgmap v310: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-10-10T17:52:33.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:32 smithi157 ceph-mon[100896]: pgmap v310: 129 pgs: 20 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 42 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-10-10T17:52:35.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:34 smithi078 ceph-mon[99494]: pgmap v311: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-10-10T17:52:35.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:34 smithi190 ceph-mon[101220]: pgmap v311: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-10-10T17:52:35.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:34 smithi157 ceph-mon[100896]: pgmap v311: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-10-10T17:52:35.991 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-10-10T17:52:36.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.250+0000 7f54b58d0700 1 -- 172.21.15.78:0/213290661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 msgr2=0x7f54b00ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.250+0000 7f54b58d0700 1 --2- 172.21.15.78:0/213290661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 0x7f54b00ff820 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f54a40098f0 tx=0x7f54a4009c00 comp rx=0 tx=0).stop 2023-10-10T17:52:36.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.250+0000 7f54b58d0700 1 -- 172.21.15.78:0/213290661 shutdown_connections 2023-10-10T17:52:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.250+0000 7f54b58d0700 1 --2- 172.21.15.78:0/213290661 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54b0104490 0x7f54b010c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.250+0000 7f54b58d0700 1 --2- 172.21.15.78:0/213290661 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 0x7f54b00ff820 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.250+0000 7f54b58d0700 1 --2- 172.21.15.78:0/213290661 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 0x7f54b0103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.250+0000 7f54b58d0700 1 -- 172.21.15.78:0/213290661 >> 172.21.15.78:0/213290661 conn(0x7f54b0077080 msgr2=0x7f54b0077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.252+0000 7f54b58d0700 1 -- 172.21.15.78:0/213290661 shutdown_connections 2023-10-10T17:52:36.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.252+0000 7f54b58d0700 1 -- 172.21.15.78:0/213290661 wait complete. 2023-10-10T17:52:36.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.253+0000 7f54b58d0700 1 Processor -- start 2023-10-10T17:52:36.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54b58d0700 1 -- start start 2023-10-10T17:52:36.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54b58d0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 0x7f54b0072690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54b58d0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 0x7f54b006a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54b58d0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54b0104490 0x7f54b006ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54b58d0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b006b350 con 0x7f54b0104490 2023-10-10T17:52:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54b58d0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b01aa0a0 con 0x7f54b00ff450 2023-10-10T17:52:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54b58d0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b01aa1e0 con 0x7f54b00ffdf0 2023-10-10T17:52:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54ae7fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 0x7f54b006a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54aeffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 0x7f54b0072690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54ae7fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 0x7f54b006a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40516/0 (socket says 172.21.15.78:40516) 2023-10-10T17:52:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54aeffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 0x7f54b0072690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50706/0 (socket says 172.21.15.78:50706) 2023-10-10T17:52:36.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.256+0000 7f54ae7fc700 1 -- 172.21.15.78:0/2297900683 learned_addr learned my addr 172.21.15.78:0/2297900683 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:36.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54ae7fc700 1 -- 172.21.15.78:0/2297900683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 msgr2=0x7f54b0072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54ae7fc700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 0x7f54b0072690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54ae7fc700 1 -- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54b0104490 msgr2=0x7f54b006ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54ae7fc700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54b0104490 0x7f54b006ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54ae7fc700 1 -- 172.21.15.78:0/2297900683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a4009520 con 0x7f54b00ffdf0 2023-10-10T17:52:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54ae7fc700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 0x7f54b006a5e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f54a000e9c0 tx=0x7f54a000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54b48ce700 1 -- 172.21.15.78:0/2297900683 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0009690 con 0x7f54b00ffdf0 2023-10-10T17:52:36.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b010f2f0 con 0x7f54b00ffdf0 2023-10-10T17:52:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b010f160 con 0x7f54b00ffdf0 2023-10-10T17:52:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54b48ce700 1 -- 172.21.15.78:0/2297900683 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54a000ce80 con 0x7f54b00ffdf0 2023-10-10T17:52:36.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.257+0000 7f54b48ce700 1 -- 172.21.15.78:0/2297900683 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a0018420 con 0x7f54b00ffdf0 2023-10-10T17:52:36.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.262+0000 7f54b48ce700 1 -- 172.21.15.78:0/2297900683 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f54a0018660 con 0x7f54b00ffdf0 2023-10-10T17:52:36.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.263+0000 7f54b48ce700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54980a1340 0x7f54980a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.263+0000 7f54b48ce700 1 -- 172.21.15.78:0/2297900683 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f54a0014070 con 0x7f54b00ffdf0 2023-10-10T17:52:36.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 Processor -- start 2023-10-10T17:52:36.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 -- start start 2023-10-10T17:52:36.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54aeffd700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54980a1340 0x7f54980a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2260450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 0x5610b2252de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2267ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2158060 con 0x5610b2253320 2023-10-10T17:52:36.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b21aa8a0 con 0x5610b2252970 2023-10-10T17:52:36.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b21a3d30 con 0x5610b2260080 2023-10-10T17:52:36.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.264+0000 7f54af7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2267ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2267ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:48996/0 (socket says 172.21.15.78:48996) 2023-10-10T17:52:36.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 -- 172.21.15.78:0/1736833901 learned_addr learned my addr 172.21.15.78:0/1736833901 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:36.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54aeffd700 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2260450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 -- 172.21.15.78:0/1736833901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 msgr2=0x5610b2260450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2260450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 -- 172.21.15.78:0/1736833901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 msgr2=0x5610b2252de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 0x5610b2252de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 -- 172.21.15.78:0/1736833901 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610b2025630 con 0x5610b2253320 2023-10-10T17:52:36.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54aeffd700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54980a1340 0x7f54980a37f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f54a40098c0 tx=0x7f54a400f800 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.265+0000 7f54af7fe700 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2267ce0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009be0 tx=0x7f54a8009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5b15e8ac8e05f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.267+0000 7f5495ffb700 1 -- 172.21.15.78:0/1736833901 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8004bb0 con 0x5610b2253320 2023-10-10T17:52:36.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.267+0000 7f5495ffb700 1 -- 172.21.15.78:0/1736833901 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54a8004d10 con 0x5610b2253320 2023-10-10T17:52:36.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.267+0000 7f5495ffb700 1 -- 172.21.15.78:0/1736833901 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8035660 con 0x5610b2253320 2023-10-10T17:52:36.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.269+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1736833901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 msgr2=0x5610b2267ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.269+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2267ce0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f54a8009be0 tx=0x7f54a8009ef0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.275+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1736833901 shutdown_connections 2023-10-10T17:52:36.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.275+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2260450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.275+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2267ce0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.275+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1736833901 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 0x5610b2252de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.275+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1736833901 >> 172.21.15.78:0/1736833901 conn(0x5610b2248360 msgr2=0x5610b224d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:36.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.277+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1736833901 shutdown_connections 2023-10-10T17:52:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.281+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1736833901 wait complete. 2023-10-10T17:52:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.281+0000 7f54c1a5cb80 1 -- start start 2023-10-10T17:52:36.283 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.281+0000 7f54c1a5cb80 1 Processor -- start 2023-10-10T17:52:36.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b2267c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2264ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 0x5610b2265520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b21458e0 con 0x5610b2253320 2023-10-10T17:52:36.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2145580 con 0x5610b2260080 2023-10-10T17:52:36.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2027920 con 0x5610b2252970 2023-10-10T17:52:36.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54c1a5cb80 10 client.0 _ll_register_callbacks cb 0x5610b2163780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:36.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54aeffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b2267c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54aeffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b2267c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40528/0 (socket says 172.21.15.78:40528) 2023-10-10T17:52:36.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.290+0000 7f54aeffd700 1 -- 172.21.15.78:0/65029272 learned_addr learned my addr 172.21.15.78:0/65029272 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:36.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.291+0000 7f54aeffd700 1 -- 172.21.15.78:0/65029272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 msgr2=0x5610b2265520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.291+0000 7f54aeffd700 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 0x5610b2265520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.291+0000 7f54aeffd700 1 -- 172.21.15.78:0/65029272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 msgr2=0x5610b2264ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.291+0000 7f54aeffd700 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2264ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.291+0000 7f54aeffd700 1 -- 172.21.15.78:0/65029272 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a80098c0 con 0x5610b2252970 2023-10-10T17:52:36.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.292+0000 7f54aeffd700 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b2267c20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f54a4016000 tx=0x7f54a4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ec8203cae599e65 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.295+0000 7f548effd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:52:36.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.295+0000 7f548effd700 1 -- 172.21.15.78:0/65029272 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a400bc20 con 0x5610b2252970 2023-10-10T17:52:36.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.295+0000 7f548effd700 1 -- 172.21.15.78:0/65029272 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f54a4005440 con 0x5610b2252970 2023-10-10T17:52:36.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.302+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610b2245b20 con 0x5610b2252970 2023-10-10T17:52:36.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.309+0000 7f548effd700 1 -- 172.21.15.78:0/65029272 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54a4005600 con 0x5610b2252970 2023-10-10T17:52:36.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.314+0000 7f54c1a5cb80 10 client.26800 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:36.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.314+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5610b216a190 con 0x5610b2252970 2023-10-10T17:52:36.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f548effd700 1 -- 172.21.15.78:0/65029272 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5610b2245b20 con 0x5610b2252970 2023-10-10T17:52:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f548effd700 1 -- 172.21.15.78:0/65029272 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54a40275a0 con 0x5610b2252970 2023-10-10T17:52:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f54c1a5cb80 10 client.26800 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:36.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f54c1a5cb80 10 client.26800 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f54c1a5cb80 20 client.26800 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b2257ba0 0x5610b2259fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f54c1a5cb80 4 client.26800 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:36.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.319+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610b21660a0 con 0x5610b2257ba0 2023-10-10T17:52:36.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.323+0000 7f54ae7fc700 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b2257ba0 0x5610b2259fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.324+0000 7f54ae7fc700 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b2257ba0 0x5610b2259fd0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=992e53ff7c626f06 server_cookie=ec2519cb3eefd87 in_seq=0 out_seq=0 2023-10-10T17:52:36.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.327+0000 7f548effd700 10 client.26800 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:36.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.327+0000 7f548effd700 1 -- 172.21.15.78:0/65029272 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x5610b21660a0 con 0x5610b2257ba0 2023-10-10T17:52:36.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.327+0000 7f548effd700 10 client.26800 handle_command_reply: tid=0 2023-10-10T17:52:36.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.327+0000 7f54c1a5cb80 1 client.26800 shutdown 2023-10-10T17:52:36.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.337+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 msgr2=0x5610b2267c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.337+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b2267c20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f54a4016000 tx=0x7f54a4009c00 comp rx=0 tx=0).stop 2023-10-10T17:52:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 shutdown_connections 2023-10-10T17:52:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b2267c20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b2257ba0 0x5610b2259fd0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 0x5610b2265520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/65029272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2264ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 >> 172.21.15.78:0/65029272 conn(0x5610b2248360 msgr2=0x5610b224cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:36.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f548effd700 0 client.26800 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:36.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.346+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 shutdown_connections 2023-10-10T17:52:36.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.354+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/65029272 wait complete. 2023-10-10T17:52:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.354+0000 7f54c1a5cb80 20 client.26800 trim_cache size 0 max 16384 2023-10-10T17:52:36.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.383+0000 7f54c1a5cb80 1 Processor -- start 2023-10-10T17:52:36.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54c1a5cb80 1 -- start start 2023-10-10T17:52:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b226c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b22677c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 0x5610b225f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2245660 con 0x5610b2253320 2023-10-10T17:52:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2247ee0 con 0x5610b2260080 2023-10-10T17:52:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2253b80 con 0x5610b2252970 2023-10-10T17:52:36.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54aeffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b226c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54aeffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b226c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40536/0 (socket says 172.21.15.78:40536) 2023-10-10T17:52:36.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54aeffd700 1 -- 172.21.15.78:0/3147384751 learned_addr learned my addr 172.21.15.78:0/3147384751 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54ae7fc700 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b22677c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.388+0000 7f54af7fe700 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 0x5610b225f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.389+0000 7f54aeffd700 1 -- 172.21.15.78:0/3147384751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 msgr2=0x5610b225f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.389+0000 7f54aeffd700 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 0x5610b225f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.389+0000 7f54aeffd700 1 -- 172.21.15.78:0/3147384751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 msgr2=0x5610b22677c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.389+0000 7f54aeffd700 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b22677c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.389+0000 7f54aeffd700 1 -- 172.21.15.78:0/3147384751 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610b2169ee0 con 0x5610b2252970 2023-10-10T17:52:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.389+0000 7f54aeffd700 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b226c2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009820 tx=0x7f54a4009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=208568ea2b40b64c server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.389+0000 7f54967fc700 1 -- 172.21.15.78:0/3147384751 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a4061070 con 0x5610b2252970 2023-10-10T17:52:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.390+0000 7f54967fc700 1 -- 172.21.15.78:0/3147384751 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54a4005700 con 0x5610b2252970 2023-10-10T17:52:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.390+0000 7f54967fc700 1 -- 172.21.15.78:0/3147384751 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a40276e0 con 0x5610b2252970 2023-10-10T17:52:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.390+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/3147384751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 msgr2=0x5610b226c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.390+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b226c2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009820 tx=0x7f54a4009850 comp rx=0 tx=0).stop 2023-10-10T17:52:36.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/3147384751 shutdown_connections 2023-10-10T17:52:36.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2252970 0x5610b226c2d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2260080 0x5610b225f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/3147384751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b22677c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/3147384751 >> 172.21.15.78:0/3147384751 conn(0x5610b2248360 msgr2=0x5610b2253f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:36.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/3147384751 shutdown_connections 2023-10-10T17:52:36.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/3147384751 wait complete. 2023-10-10T17:52:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.391+0000 7f54c1a5cb80 1 -- start start 2023-10-10T17:52:36.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 1 Processor -- start 2023-10-10T17:52:36.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 0x5610b226c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2261760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2261ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2262570 con 0x5610b2253320 2023-10-10T17:52:36.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2278ad0 con 0x5610b2252970 2023-10-10T17:52:36.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610b2278c90 con 0x5610b2260080 2023-10-10T17:52:36.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.392+0000 7f54c1a5cb80 10 client.0 _ll_register_callbacks cb 0x7f54a400f530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:36.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2261ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2261ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40544/0 (socket says 172.21.15.78:40544) 2023-10-10T17:52:36.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 -- 172.21.15.78:0/1150591114 learned_addr learned my addr 172.21.15.78:0/1150591114 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:36.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 -- 172.21.15.78:0/1150591114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 msgr2=0x5610b226c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 0x5610b226c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 -- 172.21.15.78:0/1150591114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 msgr2=0x5610b2261760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:36.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2261760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.393+0000 7f54af7fe700 1 -- 172.21.15.78:0/1150591114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610b216a190 con 0x5610b2260080 2023-10-10T17:52:36.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f54af7fe700 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2261ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f54a8036000 tx=0x7f54a8035bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19fa332f8a847687 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f548effd700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:52:36.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f548effd700 1 -- 172.21.15.78:0/1150591114 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54a8006000 con 0x5610b2260080 2023-10-10T17:52:36.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f548effd700 1 -- 172.21.15.78:0/1150591114 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f54a8003a00 con 0x5610b2260080 2023-10-10T17:52:36.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5610b2172660 con 0x5610b2260080 2023-10-10T17:52:36.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f548effd700 1 -- 172.21.15.78:0/1150591114 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54a8003bc0 con 0x5610b2260080 2023-10-10T17:52:36.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f54c1a5cb80 10 client.26806 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:36.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.394+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5610b1fbac60 con 0x5610b2260080 2023-10-10T17:52:36.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f548effd700 1 -- 172.21.15.78:0/1150591114 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5610b2172660 con 0x5610b2260080 2023-10-10T17:52:36.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f548effd700 1 -- 172.21.15.78:0/1150591114 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54a803d050 con 0x5610b2260080 2023-10-10T17:52:36.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f54c1a5cb80 10 client.26806 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:36.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f54c1a5cb80 10 client.26806 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:36.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f54c1a5cb80 20 client.26806 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:36.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b227a140 0x5610b227c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f54c1a5cb80 4 client.26806 mds_command: new command op to 24500 tid=0[{"prefix": "scrub status"}] 2023-10-10T17:52:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.396+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5610b1ff2f70 con 0x5610b227a140 2023-10-10T17:52:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.397+0000 7f54aeffd700 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b227a140 0x5610b227c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.397+0000 7f548effd700 10 client.26806 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:36.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.397+0000 7f54aeffd700 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b227a140 0x5610b227c540 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c238c8873312988 server_cookie=41eea39f2cb6f330 in_seq=0 out_seq=0 2023-10-10T17:52:36.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.398+0000 7f548effd700 1 -- 172.21.15.78:0/1150591114 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5610b1ff2f70 con 0x5610b227a140 2023-10-10T17:52:36.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.398+0000 7f548effd700 10 client.26806 handle_command_reply: tid=0 2023-10-10T17:52:36.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.398+0000 7f54c1a5cb80 1 client.26806 shutdown 2023-10-10T17:52:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 msgr2=0x5610b2261ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2261ea0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f54a8036000 tx=0x7f54a8035bc0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 shutdown_connections 2023-10-10T17:52:36.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x5610b2260080 0x5610b2261ea0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x5610b227a140 0x5610b227c540 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x5610b2252970 0x5610b226c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 --2- 172.21.15.78:0/1150591114 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5610b2253320 0x5610b2261760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.399+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 >> 172.21.15.78:0/1150591114 conn(0x5610b2248360 msgr2=0x5610b2251960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:36.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.403+0000 7f548effd700 0 client.26806 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:36.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.409+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 shutdown_connections 2023-10-10T17:52:36.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.418+0000 7f54c1a5cb80 1 -- 172.21.15.78:0/1150591114 wait complete. 2023-10-10T17:52:36.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.418+0000 7f54c1a5cb80 20 client.26806 trim_cache size 0 max 16384 2023-10-10T17:52:36.419 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "status": "no active scrubs running", 2023-10-10T17:52:36.420 INFO:teuthology.orchestra.run.smithi078.stdout: "scrubs": {} 2023-10-10T17:52:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:36.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.419+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54980a1340 msgr2=0x7f54980a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.419+0000 7f54b58d0700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54980a1340 0x7f54980a37f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f54a40098c0 tx=0x7f54a400f800 comp rx=0 tx=0).stop 2023-10-10T17:52:36.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.419+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 msgr2=0x7f54b006a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.419+0000 7f54b58d0700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 0x7f54b006a5e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f54a000e9c0 tx=0x7f54a000ecd0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.429+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 shutdown_connections 2023-10-10T17:52:36.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.429+0000 7f54b58d0700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54980a1340 0x7f54980a37f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.429+0000 7f54b58d0700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54b0104490 0x7f54b006ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.429+0000 7f54b58d0700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54b00ff450 0x7f54b0072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.429+0000 7f54b58d0700 1 --2- 172.21.15.78:0/2297900683 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54b00ffdf0 0x7f54b006a5e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.429+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 >> 172.21.15.78:0/2297900683 conn(0x7f54b0077080 msgr2=0x7f54b00fcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:36.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.436+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 shutdown_connections 2023-10-10T17:52:36.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.438+0000 7f54b58d0700 1 -- 172.21.15.78:0/2297900683 wait complete. 2023-10-10T17:52:36.464 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-10-10T17:52:36.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:52:36.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.689+0000 7fcec6a2d700 1 -- 172.21.15.78:0/3761943279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 msgr2=0x7fcec0108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.689+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/3761943279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec0108130 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fceb00099b0 tx=0x7fceb0009cc0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.700+0000 7fcec6a2d700 1 -- 172.21.15.78:0/3761943279 shutdown_connections 2023-10-10T17:52:36.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.700+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/3761943279 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec0108130 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.700+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/3761943279 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcec0072ff0 0x7fcec00799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.700+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/3761943279 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcec0108700 0x7fcec0072910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.700+0000 7fcec6a2d700 1 -- 172.21.15.78:0/3761943279 >> 172.21.15.78:0/3761943279 conn(0x7fcec006c550 msgr2=0x7fcec00716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:36.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.703+0000 7fcec6a2d700 1 -- 172.21.15.78:0/3761943279 shutdown_connections 2023-10-10T17:52:36.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.703+0000 7fcec6a2d700 1 -- 172.21.15.78:0/3761943279 wait complete. 2023-10-10T17:52:36.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.704+0000 7fcec6a2d700 1 Processor -- start 2023-10-10T17:52:36.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.704+0000 7fcec6a2d700 1 -- start start 2023-10-10T17:52:36.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.704+0000 7fcec6a2d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcec0072ff0 0x7fcec0080090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.704+0000 7fcec6a2d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec00805d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcec6a2d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcec0108700 0x7fcec01bb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcec6a2d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec01bbe80 con 0x7fcec0072ff0 2023-10-10T17:52:36.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcec6a2d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec01bbfc0 con 0x7fcec0108700 2023-10-10T17:52:36.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcec6a2d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec01bc180 con 0x7fcec0107d60 2023-10-10T17:52:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec00805d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec00805d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40556/0 (socket says 172.21.15.78:40556) 2023-10-10T17:52:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 -- 172.21.15.78:0/2976011810 learned_addr learned my addr 172.21.15.78:0/2976011810 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:36.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcec4fca700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcec0108700 0x7fcec01bb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 -- 172.21.15.78:0/2976011810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcec0108700 msgr2=0x7fcec01bb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcec0108700 0x7fcec01bb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 -- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcec0072ff0 msgr2=0x7fcec0080090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:36.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcec0072ff0 0x7fcec0080090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:36.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcebf7fe700 1 -- 172.21.15.78:0/2976011810 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb00095e0 con 0x7fcec0107d60 2023-10-10T17:52:36.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.705+0000 7fcec4fca700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcec0108700 0x7fcec01bb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:52:36.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.706+0000 7fcebf7fe700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec00805d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009750 tx=0x7fceb4009a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.707+0000 7fcebd7fa700 1 -- 172.21.15.78:0/2976011810 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4004830 con 0x7fcec0107d60 2023-10-10T17:52:36.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.707+0000 7fcebd7fa700 1 -- 172.21.15.78:0/2976011810 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fceb4010040 con 0x7fcec0107d60 2023-10-10T17:52:36.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.707+0000 7fcebd7fa700 1 -- 172.21.15.78:0/2976011810 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb4014880 con 0x7fcec0107d60 2023-10-10T17:52:36.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.709+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec007cc30 con 0x7fcec0107d60 2023-10-10T17:52:36.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.709+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec007caa0 con 0x7fcec0107d60 2023-10-10T17:52:36.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.710+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec007cdc0 con 0x7fcec0107d60 2023-10-10T17:52:36.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.712+0000 7fcebd7fa700 1 -- 172.21.15.78:0/2976011810 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fceb4004990 con 0x7fcec0107d60 2023-10-10T17:52:36.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.713+0000 7fcebd7fa700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcea80a5750 0x7fcea80a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:36.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.713+0000 7fcebd7fa700 1 -- 172.21.15.78:0/2976011810 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fceb40ccfa0 con 0x7fcec0107d60 2023-10-10T17:52:36.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.713+0000 7fcebffff700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcea80a5750 0x7fcea80a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:36.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.716+0000 7fcebffff700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcea80a5750 0x7fcea80a7c00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009fd0 tx=0x7fceb0012510 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:36.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:36.719+0000 7fcebd7fa700 1 -- 172.21.15.78:0/2976011810 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fceb4095fd0 con 0x7fcec0107d60 2023-10-10T17:52:36.819 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:36 smithi078 ceph-mon[99494]: pgmap v312: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-10T17:52:37.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.140+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcec0067a60 con 0x7fcec0107d60 2023-10-10T17:52:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.144+0000 7fcebd7fa700 1 -- 172.21.15.78:0/2976011810 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+10808 (secure 0 0 0) 0x7fceb4018090 con 0x7fcec0107d60 2023-10-10T17:52:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:52:37.145 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:52:25.779986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:52:37.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.152+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcea80a5750 msgr2=0x7fcea80a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.152+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcea80a5750 0x7fcea80a7c00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009fd0 tx=0x7fceb0012510 comp rx=0 tx=0).stop 2023-10-10T17:52:37.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.152+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 msgr2=0x7fcec00805d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.152+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec00805d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009750 tx=0x7fceb4009a60 comp rx=0 tx=0).stop 2023-10-10T17:52:37.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.153+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 shutdown_connections 2023-10-10T17:52:37.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.153+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fcea80a5750 0x7fcea80a7c00 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fceb0009fd0 tx=0x7fceb0012510 comp rx=0 tx=0).stop 2023-10-10T17:52:37.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.153+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fcec0108700 0x7fcec01bb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.153+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcec0072ff0 0x7fcec0080090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.153+0000 7fcec6a2d700 1 --2- 172.21.15.78:0/2976011810 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fcec0107d60 0x7fcec00805d0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009750 tx=0x7fceb4009a60 comp rx=0 tx=0).stop 2023-10-10T17:52:37.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.153+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 >> 172.21.15.78:0/2976011810 conn(0x7fcec006c550 msgr2=0x7fcec010abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.155+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 shutdown_connections 2023-10-10T17:52:37.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.161+0000 7fcec6a2d700 1 -- 172.21.15.78:0/2976011810 wait complete. 2023-10-10T17:52:37.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:36 smithi190 ceph-mon[101220]: pgmap v312: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-10T17:52:37.173 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 38 2023-10-10T17:52:37.193 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-10-10T17:52:37.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:36 smithi157 ceph-mon[100896]: pgmap v312: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-10-10T17:52:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.428+0000 7f5445216700 1 -- 172.21.15.78:0/3411309574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5440108030 msgr2=0x7f54401084a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.428+0000 7f5445216700 1 --2- 172.21.15.78:0/3411309574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5440108030 0x7f54401084a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5434009990 tx=0x7f5434009ca0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.428+0000 7f5445216700 1 -- 172.21.15.78:0/3411309574 shutdown_connections 2023-10-10T17:52:37.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.428+0000 7f5445216700 1 --2- 172.21.15.78:0/3411309574 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f54400777b0 0x7f5440079c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.428+0000 7f5445216700 1 --2- 172.21.15.78:0/3411309574 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f544010c360 0x7f5440107af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.428+0000 7f5445216700 1 --2- 172.21.15.78:0/3411309574 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5440108030 0x7f54401084a0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5434009990 tx=0x7f5434009ca0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.428+0000 7f5445216700 1 -- 172.21.15.78:0/3411309574 >> 172.21.15.78:0/3411309574 conn(0x7f544006c7b0 msgr2=0x7f5440071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.434+0000 7f5445216700 1 -- 172.21.15.78:0/3411309574 shutdown_connections 2023-10-10T17:52:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.434+0000 7f5445216700 1 -- 172.21.15.78:0/3411309574 wait complete. 2023-10-10T17:52:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 Processor -- start 2023-10-10T17:52:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 -- start start 2023-10-10T17:52:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54400777b0 0x7f5440076c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f544010c360 0x7f5440077190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5440073cb0 0x7f5440074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54400748e0 con 0x7f544010c360 2023-10-10T17:52:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5440074a50 con 0x7f54400777b0 2023-10-10T17:52:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.435+0000 7f5445216700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54401bc8c0 con 0x7f5440073cb0 2023-10-10T17:52:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.436+0000 7f543f59e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5440073cb0 0x7f5440074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.436+0000 7f543f59e700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5440073cb0 0x7f5440074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40570/0 (socket says 172.21.15.78:40570) 2023-10-10T17:52:37.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.436+0000 7f543f59e700 1 -- 172.21.15.78:0/3606884317 learned_addr learned my addr 172.21.15.78:0/3606884317 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.436+0000 7f543ed9d700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54400777b0 0x7f5440076c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.437+0000 7f543f59e700 1 -- 172.21.15.78:0/3606884317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54400777b0 msgr2=0x7f5440076c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.437+0000 7f543f59e700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54400777b0 0x7f5440076c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.437+0000 7f543f59e700 1 -- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f544010c360 msgr2=0x7f5440077190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.437+0000 7f543f59e700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f544010c360 0x7f5440077190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.437+0000 7f543f59e700 1 -- 172.21.15.78:0/3606884317 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54340095f0 con 0x7f5440073cb0 2023-10-10T17:52:37.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.437+0000 7f543f59e700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5440073cb0 0x7f5440074110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f543800ef30 tx=0x7f543800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.438+0000 7f5427fff700 1 -- 172.21.15.78:0/3606884317 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5438018070 con 0x7f5440073cb0 2023-10-10T17:52:37.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.438+0000 7f5427fff700 1 -- 172.21.15.78:0/3606884317 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f543800f040 con 0x7f5440073cb0 2023-10-10T17:52:37.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.438+0000 7f5427fff700 1 -- 172.21.15.78:0/3606884317 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543800cd50 con 0x7f5440073cb0 2023-10-10T17:52:37.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.439+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544007edd0 con 0x7f5440073cb0 2023-10-10T17:52:37.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.439+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544007ec40 con 0x7f5440073cb0 2023-10-10T17:52:37.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.443+0000 7f54513a2b80 1 Processor -- start 2023-10-10T17:52:37.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f54513a2b80 1 -- start start 2023-10-10T17:52:37.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc4205740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc4209b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc420a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f54513a2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc4102060 con 0x563dc4205c80 2023-10-10T17:52:37.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f54513a2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc41548a0 con 0x563dc420a0c0 2023-10-10T17:52:37.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f54513a2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc414dd30 con 0x563dc4205370 2023-10-10T17:52:37.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f543e59c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc420a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f543e59c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc420a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50768/0 (socket says 172.21.15.78:50768) 2023-10-10T17:52:37.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.444+0000 7f543e59c700 1 -- 172.21.15.78:0/3515197815 learned_addr learned my addr 172.21.15.78:0/3515197815 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.445+0000 7f5427fff700 1 -- 172.21.15.78:0/3606884317 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5438004ad0 con 0x7f5440073cb0 2023-10-10T17:52:37.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.446+0000 7f543e59c700 1 -- 172.21.15.78:0/3515197815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 msgr2=0x563dc4205740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.446+0000 7f543e59c700 1 --2- 172.21.15.78:0/3515197815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc4205740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.446+0000 7f543e59c700 1 -- 172.21.15.78:0/3515197815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 msgr2=0x563dc4209b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.446+0000 7f543e59c700 1 --2- 172.21.15.78:0/3515197815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc4209b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.446+0000 7f543e59c700 1 -- 172.21.15.78:0/3515197815 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563dc3fcf630 con 0x563dc420a0c0 2023-10-10T17:52:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.446+0000 7f5427fff700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54280a1330 0x7f54280a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.446+0000 7f5427fff700 1 -- 172.21.15.78:0/3606884317 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f54380cc4a0 con 0x7f5440073cb0 2023-10-10T17:52:37.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.457+0000 7f543e59c700 1 --2- 172.21.15.78:0/3515197815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc420a510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5434006010 tx=0x7f5434005ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79599fbe70a86d49 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.459+0000 7f543ed9d700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54280a1330 0x7f54280a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.459+0000 7f54257fa700 1 -- 172.21.15.78:0/3515197815 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543400b950 con 0x563dc420a0c0 2023-10-10T17:52:37.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.459+0000 7f54257fa700 1 -- 172.21.15.78:0/3515197815 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54340047e0 con 0x563dc420a0c0 2023-10-10T17:52:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.459+0000 7f54257fa700 1 -- 172.21.15.78:0/3515197815 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434005320 con 0x563dc420a0c0 2023-10-10T17:52:37.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.460+0000 7f54513a2b80 1 -- 172.21.15.78:0/3515197815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 msgr2=0x563dc420a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.460+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3515197815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc420a510 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5434006010 tx=0x7f5434005ea0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.460+0000 7f543ed9d700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54280a1330 0x7f54280a37e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5430006b30 tx=0x7f5430006ac0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.462+0000 7f54513a2b80 1 -- 172.21.15.78:0/3515197815 shutdown_connections 2023-10-10T17:52:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.462+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3515197815 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc4205740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.462+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3515197815 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc420a510 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.462+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3515197815 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc4209b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.462+0000 7f54513a2b80 1 -- 172.21.15.78:0/3515197815 >> 172.21.15.78:0/3515197815 conn(0x563dc41f2360 msgr2=0x563dc41f7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.462+0000 7f54513a2b80 1 -- 172.21.15.78:0/3515197815 shutdown_connections 2023-10-10T17:52:37.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.462+0000 7f54513a2b80 1 -- 172.21.15.78:0/3515197815 wait complete. 2023-10-10T17:52:37.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 -- start start 2023-10-10T17:52:37.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 Processor -- start 2023-10-10T17:52:37.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc420ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc4205c80 0x563dc420f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc420a0c0 0x563dc420f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc40ef8e0 con 0x563dc420a0c0 2023-10-10T17:52:37.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc40ef580 con 0x563dc4205c80 2023-10-10T17:52:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc3fd1920 con 0x563dc4205370 2023-10-10T17:52:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f54513a2b80 10 client.0 _ll_register_callbacks cb 0x563dc410d780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:37.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f543f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc420a0c0 0x563dc420f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f543f59e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc420a0c0 0x563dc420f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49056/0 (socket says 172.21.15.78:49056) 2023-10-10T17:52:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f543f59e700 1 -- 172.21.15.78:0/677608062 learned_addr learned my addr 172.21.15.78:0/677608062 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f543ed9d700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc420ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.463+0000 7f543e59c700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc4205c80 0x563dc420f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f543f59e700 1 -- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 msgr2=0x563dc420ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f543f59e700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc420ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f543f59e700 1 -- 172.21.15.78:0/677608062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc4205c80 msgr2=0x563dc420f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f543f59e700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc4205c80 0x563dc420f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f543f59e700 1 -- 172.21.15.78:0/677608062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544007a330 con 0x563dc420a0c0 2023-10-10T17:52:37.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f543ed9d700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc420ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:52:37.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f543f59e700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc420a0c0 0x563dc420f670 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f54380a10b0 tx=0x7f5438099190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a02f24bded7048c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f5422ff5700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:37.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f5422ff5700 1 -- 172.21.15.78:0/677608062 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54380cc2b0 con 0x563dc420a0c0 2023-10-10T17:52:37.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f5422ff5700 1 -- 172.21.15.78:0/677608062 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f54380a15e0 con 0x563dc420a0c0 2023-10-10T17:52:37.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563dc41efb20 con 0x563dc420a0c0 2023-10-10T17:52:37.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f5422ff5700 1 -- 172.21.15.78:0/677608062 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5438098680 con 0x563dc420a0c0 2023-10-10T17:52:37.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f54513a2b80 10 client.17502 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:37.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.464+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x563dc4114190 con 0x563dc420a0c0 2023-10-10T17:52:37.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f5422ff5700 1 -- 172.21.15.78:0/677608062 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563dc41efb20 con 0x563dc420a0c0 2023-10-10T17:52:37.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f5422ff5700 1 -- 172.21.15.78:0/677608062 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f54380b7200 con 0x563dc420a0c0 2023-10-10T17:52:37.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f54513a2b80 10 client.17502 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:37.479 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f54513a2b80 10 client.17502 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:52:37.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f54513a2b80 20 client.17502 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:37.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f54513a2b80 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc42279f0 0x563dc4229e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f54513a2b80 4 client.17502 mds_command: new command op to 24472 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:37.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563dc41100a0 con 0x563dc42279f0 2023-10-10T17:52:37.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.465+0000 7f543ed9d700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc42279f0 0x563dc4229e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.466+0000 7f5422ff5700 10 client.17502 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:37.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.466+0000 7f543ed9d700 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc42279f0 0x563dc4229e00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15177cb99d9d5168 server_cookie=b7fad6980a5499d8 in_seq=0 out_seq=0 2023-10-10T17:52:37.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f5422ff5700 1 -- 172.21.15.78:0/677608062 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x563dc41100a0 con 0x563dc42279f0 2023-10-10T17:52:37.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f5422ff5700 10 client.17502 handle_command_reply: tid=0 2023-10-10T17:52:37.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 client.17502 shutdown 2023-10-10T17:52:37.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc420a0c0 msgr2=0x563dc420f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc420a0c0 0x563dc420f670 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f54380a10b0 tx=0x7f5438099190 comp rx=0 tx=0).stop 2023-10-10T17:52:37.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 shutdown_connections 2023-10-10T17:52:37.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc42279f0 0x563dc4229e00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc420ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc420a0c0 0x563dc420f670 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 --2- 172.21.15.78:0/677608062 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc4205c80 0x563dc420f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f5422ff5700 0 client.17502 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:37.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.467+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 >> 172.21.15.78:0/677608062 conn(0x563dc41f2360 msgr2=0x563dc420b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.470+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 shutdown_connections 2023-10-10T17:52:37.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.477+0000 7f54513a2b80 1 -- 172.21.15.78:0/677608062 wait complete. 2023-10-10T17:52:37.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.477+0000 7f54513a2b80 20 client.17502 trim_cache size 0 max 16384 2023-10-10T17:52:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 Processor -- start 2023-10-10T17:52:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 -- start start 2023-10-10T17:52:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc4200cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc41f7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc41f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc41f87d0 con 0x563dc4205c80 2023-10-10T17:52:37.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc4208c90 con 0x563dc420a0c0 2023-10-10T17:52:37.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f54513a2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc4201200 con 0x563dc4205370 2023-10-10T17:52:37.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f543e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc41f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f543e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc41f7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49060/0 (socket says 172.21.15.78:49060) 2023-10-10T17:52:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.502+0000 7f543e59c700 1 -- 172.21.15.78:0/1498521630 learned_addr learned my addr 172.21.15.78:0/1498521630 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.503+0000 7f543e59c700 1 -- 172.21.15.78:0/1498521630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 msgr2=0x563dc4200cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:52:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.503+0000 7f543e59c700 1 --2- 172.21.15.78:0/1498521630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc4200cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.503+0000 7f543e59c700 1 -- 172.21.15.78:0/1498521630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 msgr2=0x563dc41f7aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:52:37.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.503+0000 7f543e59c700 1 --2- 172.21.15.78:0/1498521630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc41f7aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.503+0000 7f543e59c700 1 -- 172.21.15.78:0/1498521630 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563dc420a9d0 con 0x563dc4205c80 2023-10-10T17:52:37.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.503+0000 7f543e59c700 1 --2- 172.21.15.78:0/1498521630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc41f7560 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5434000c00 tx=0x7f5434005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de312dcb64a9d3ba server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.504+0000 7f5425ffb700 1 -- 172.21.15.78:0/1498521630 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434029070 con 0x563dc4205c80 2023-10-10T17:52:37.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.504+0000 7f5425ffb700 1 -- 172.21.15.78:0/1498521630 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f543402e460 con 0x563dc4205c80 2023-10-10T17:52:37.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.504+0000 7f5425ffb700 1 -- 172.21.15.78:0/1498521630 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543400f6c0 con 0x563dc4205c80 2023-10-10T17:52:37.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.505+0000 7f54513a2b80 1 -- 172.21.15.78:0/1498521630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 msgr2=0x563dc41f7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.505+0000 7f54513a2b80 1 --2- 172.21.15.78:0/1498521630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc41f7560 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f5434000c00 tx=0x7f5434005f10 comp rx=0 tx=0).stop 2023-10-10T17:52:37.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.506+0000 7f54513a2b80 1 -- 172.21.15.78:0/1498521630 shutdown_connections 2023-10-10T17:52:37.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.506+0000 7f54513a2b80 1 --2- 172.21.15.78:0/1498521630 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205370 0x563dc4200cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.506+0000 7f54513a2b80 1 --2- 172.21.15.78:0/1498521630 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc41f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.506+0000 7f54513a2b80 1 --2- 172.21.15.78:0/1498521630 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205c80 0x563dc41f7560 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.506+0000 7f54513a2b80 1 -- 172.21.15.78:0/1498521630 >> 172.21.15.78:0/1498521630 conn(0x563dc41f2360 msgr2=0x563dc4208180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.506+0000 7f54513a2b80 1 -- 172.21.15.78:0/1498521630 shutdown_connections 2023-10-10T17:52:37.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.506+0000 7f54513a2b80 1 -- 172.21.15.78:0/1498521630 wait complete. 2023-10-10T17:52:37.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 -- start start 2023-10-10T17:52:37.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 Processor -- start 2023-10-10T17:52:37.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205370 0x563dc41fecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205c80 0x563dc41ff1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc41ff730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc4201020 con 0x563dc4205370 2023-10-10T17:52:37.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc427a8b0 con 0x563dc420a0c0 2023-10-10T17:52:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.507+0000 7f54513a2b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x563dc427aa20 con 0x563dc4205c80 2023-10-10T17:52:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f54513a2b80 10 client.0 _ll_register_callbacks cb 0x7f5438098af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:37.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205c80 0x563dc41ff1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205c80 0x563dc41ff1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40592/0 (socket says 172.21.15.78:40592) 2023-10-10T17:52:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 -- 172.21.15.78:0/3213329500 learned_addr learned my addr 172.21.15.78:0/3213329500 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543f59e700 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc41ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543ed9d700 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205370 0x563dc41fecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 -- 172.21.15.78:0/3213329500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 msgr2=0x563dc41ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc41ff730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 -- 172.21.15.78:0/3213329500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205370 msgr2=0x563dc41fecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205370 0x563dc41fecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 -- 172.21.15.78:0/3213329500 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563dc3fcf630 con 0x563dc4205c80 2023-10-10T17:52:37.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.508+0000 7f543e59c700 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205c80 0x563dc41ff1f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5434009550 tx=0x7f543402ef80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88f8fffa82f985f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.512+0000 7f5422ff5700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:52:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.512+0000 7f5422ff5700 1 -- 172.21.15.78:0/3213329500 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5434029070 con 0x563dc4205c80 2023-10-10T17:52:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.512+0000 7f5422ff5700 1 -- 172.21.15.78:0/3213329500 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f543400bd40 con 0x563dc4205c80 2023-10-10T17:52:37.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.512+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54340047e0 con 0x563dc4205c80 2023-10-10T17:52:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.513+0000 7f5422ff5700 1 -- 172.21.15.78:0/3213329500 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5434032040 con 0x563dc4205c80 2023-10-10T17:52:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.513+0000 7f54513a2b80 10 client.26824 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:37.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.513+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x563dc3f7bde0 con 0x563dc4205c80 2023-10-10T17:52:37.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.513+0000 7f5422ff5700 1 -- 172.21.15.78:0/3213329500 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f54340047e0 con 0x563dc4205c80 2023-10-10T17:52:37.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.513+0000 7f5422ff5700 1 -- 172.21.15.78:0/3213329500 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5434022030 con 0x563dc4205c80 2023-10-10T17:52:37.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f54513a2b80 10 client.26824 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:37.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f54513a2b80 10 client.26824 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2023-10-10T17:52:37.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f54513a2b80 20 client.26824 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:37.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc427bc50 0x563dc427e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f54513a2b80 4 client.26824 mds_command: new command op to 24472 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:52:37.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 --> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563dc411c660 con 0x563dc427bc50 2023-10-10T17:52:37.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f543ed9d700 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc427bc50 0x563dc427e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f5422ff5700 10 client.26824 ms_handle_connect on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:37.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f543ed9d700 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc427bc50 0x563dc427e060 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=438a19bd31187382 server_cookie=3058dc4072656109 in_seq=0 out_seq=0 2023-10-10T17:52:37.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f5422ff5700 1 -- 172.21.15.78:0/3213329500 <== mds.2 v2:172.21.15.190:6836/661255149 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563dc411c660 con 0x563dc427bc50 2023-10-10T17:52:37.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f5422ff5700 10 client.26824 handle_command_reply: tid=0 2023-10-10T17:52:37.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.514+0000 7f54513a2b80 1 client.26824 shutdown 2023-10-10T17:52:37.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.515+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205c80 msgr2=0x563dc41ff1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.515+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205c80 0x563dc41ff1f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5434009550 tx=0x7f543402ef80 comp rx=0 tx=0).stop 2023-10-10T17:52:37.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.516+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 shutdown_connections 2023-10-10T17:52:37.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.516+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x563dc420a0c0 0x563dc41ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.516+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x563dc4205370 0x563dc41fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.516+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:6836/661255149,v1:172.21.15.190:6837/661255149] conn(0x563dc427bc50 0x563dc427e060 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.516+0000 7f54513a2b80 1 --2- 172.21.15.78:0/3213329500 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x563dc4205c80 0x563dc41ff1f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.516+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 >> 172.21.15.78:0/3213329500 conn(0x563dc41f2360 msgr2=0x563dc4209140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.519+0000 7f5422ff5700 0 client.26824 ms_handle_reset on v2:172.21.15.190:6836/661255149 2023-10-10T17:52:37.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.522+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 shutdown_connections 2023-10-10T17:52:37.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.522+0000 7f54513a2b80 1 -- 172.21.15.78:0/3213329500 wait complete. 2023-10-10T17:52:37.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.522+0000 7f54513a2b80 20 client.26824 trim_cache size 0 max 16384 2023-10-10T17:52:37.532 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:52:37.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54280a1330 msgr2=0x7f54280a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54280a1330 0x7f54280a37e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5430006b30 tx=0x7f5430006ac0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5440073cb0 msgr2=0x7f5440074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5440073cb0 0x7f5440074110 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f543800ef30 tx=0x7f543800c3a0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 shutdown_connections 2023-10-10T17:52:37.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5440073cb0 0x7f5440074110 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f54280a1330 0x7f54280a37e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f54400777b0 0x7f5440076c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 --2- 172.21.15.78:0/3606884317 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f544010c360 0x7f5440077190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 >> 172.21.15.78:0/3606884317 conn(0x7f544006c7b0 msgr2=0x7f5440108be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 shutdown_connections 2023-10-10T17:52:37.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.523+0000 7f5445216700 1 -- 172.21.15.78:0/3606884317 wait complete. 2023-10-10T17:52:37.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-10-10T17:52:37.575 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-10-10T17:52:37.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.792+0000 7fa5f9767700 1 -- 172.21.15.78:0/1766745844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f4104960 msgr2=0x7fa5f4104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.792+0000 7fa5f9767700 1 --2- 172.21.15.78:0/1766745844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f4104960 0x7fa5f4104d30 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009990 tx=0x7fa5e8009ca0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.803+0000 7fa5f9767700 1 -- 172.21.15.78:0/1766745844 shutdown_connections 2023-10-10T17:52:37.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.803+0000 7fa5f9767700 1 --2- 172.21.15.78:0/1766745844 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f40ff300 0x7fa5f410b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.803+0000 7fa5f9767700 1 --2- 172.21.15.78:0/1766745844 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f4104960 0x7fa5f4104d30 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.803+0000 7fa5f9767700 1 --2- 172.21.15.78:0/1766745844 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f40fe950 0x7fa5f40fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.803+0000 7fa5f9767700 1 -- 172.21.15.78:0/1766745844 >> 172.21.15.78:0/1766745844 conn(0x7fa5f4075920 msgr2=0x7fa5f4075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.812+0000 7fa5f9767700 1 -- 172.21.15.78:0/1766745844 shutdown_connections 2023-10-10T17:52:37.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.814+0000 7fa5f9767700 1 -- 172.21.15.78:0/1766745844 wait complete. 2023-10-10T17:52:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.814+0000 7fa5f9767700 1 Processor -- start 2023-10-10T17:52:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f9767700 1 -- start start 2023-10-10T17:52:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f9767700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f40fe950 0x7fa5f406fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f9767700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f40ff300 0x7fa5f4070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f9767700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f4104960 0x7fa5f410a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f9767700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f410a9e0 con 0x7fa5f40ff300 2023-10-10T17:52:37.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f9767700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f410ab50 con 0x7fa5f4104960 2023-10-10T17:52:37.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f9767700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f410ad10 con 0x7fa5f40fe950 2023-10-10T17:52:37.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f2ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f40fe950 0x7fa5f406fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f2ffd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f40fe950 0x7fa5f406fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40606/0 (socket says 172.21.15.78:40606) 2023-10-10T17:52:37.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f2ffd700 1 -- 172.21.15.78:0/389006508 learned_addr learned my addr 172.21.15.78:0/389006508 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.815+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f4104960 0x7fa5f410a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.816+0000 7fa5f37fe700 1 -- 172.21.15.78:0/389006508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f40fe950 msgr2=0x7fa5f406fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.816+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f40fe950 0x7fa5f406fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.816+0000 7fa5f37fe700 1 -- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f40ff300 msgr2=0x7fa5f4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.816+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f40ff300 0x7fa5f4070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.816+0000 7fa5f37fe700 1 -- 172.21.15.78:0/389006508 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e80095f0 con 0x7fa5f4104960 2023-10-10T17:52:37.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.817+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f4104960 0x7fa5f410a3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00bab0 tx=0x7fa5ec00bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.819+0000 7fa5dbfff700 1 -- 172.21.15.78:0/389006508 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec00c9b0 con 0x7fa5f4104960 2023-10-10T17:52:37.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.819+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f406ae20 con 0x7fa5f4104960 2023-10-10T17:52:37.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.819+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f406ac90 con 0x7fa5f4104960 2023-10-10T17:52:37.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.819+0000 7fa5dbfff700 1 -- 172.21.15.78:0/389006508 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5ec014440 con 0x7fa5f4104960 2023-10-10T17:52:37.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.819+0000 7fa5dbfff700 1 -- 172.21.15.78:0/389006508 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0125a0 con 0x7fa5f4104960 2023-10-10T17:52:37.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 Processor -- start 2023-10-10T17:52:37.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 -- start start 2023-10-10T17:52:37.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322b9b60 0x55bc322b9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 0x55bc322be370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 0x55bc322bed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc321bf060 con 0x55bc322be8b0 2023-10-10T17:52:37.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc322118a0 con 0x55bc322ba470 2023-10-10T17:52:37.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc3220ad30 con 0x55bc322b9b60 2023-10-10T17:52:37.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa5f27fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322b9b60 0x55bc322b9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa5f27fc700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322b9b60 0x55bc322b9f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40612/0 (socket says 172.21.15.78:40612) 2023-10-10T17:52:37.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa5f27fc700 1 -- 172.21.15.78:0/307503906 learned_addr learned my addr 172.21.15.78:0/307503906 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.825+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 0x55bc322bed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.826+0000 7fa5f27fc700 1 -- 172.21.15.78:0/307503906 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 msgr2=0x55bc322be370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:52:37.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.826+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 0x55bc322be370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.826+0000 7fa5f27fc700 1 -- 172.21.15.78:0/307503906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 msgr2=0x55bc322bed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.826+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 0x55bc322bed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.826+0000 7fa5f27fc700 1 -- 172.21.15.78:0/307503906 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc3208c630 con 0x55bc322b9b60 2023-10-10T17:52:37.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.826+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322b9b60 0x55bc322b9f30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40127f0 tx=0x7fa5e400af90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2484a198ed086856 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.828+0000 7fa5d97fa700 1 -- 172.21.15.78:0/307503906 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4012230 con 0x55bc322b9b60 2023-10-10T17:52:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.828+0000 7fa5d97fa700 1 -- 172.21.15.78:0/307503906 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5e4018070 con 0x55bc322b9b60 2023-10-10T17:52:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.828+0000 7fa5d97fa700 1 -- 172.21.15.78:0/307503906 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e400d640 con 0x55bc322b9b60 2023-10-10T17:52:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.829+0000 7fa6058f3b80 1 -- 172.21.15.78:0/307503906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322b9b60 msgr2=0x55bc322b9f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.829+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322b9b60 0x55bc322b9f30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa5e40127f0 tx=0x7fa5e400af90 comp rx=0 tx=0).stop 2023-10-10T17:52:37.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.830+0000 7fa5dbfff700 1 -- 172.21.15.78:0/389006508 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa5ec00cb10 con 0x7fa5f4104960 2023-10-10T17:52:37.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa5dbfff700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5dc0a14d0 0x7fa5dc0a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa5dbfff700 1 -- 172.21.15.78:0/389006508 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fa5ec0cc540 con 0x7fa5f4104960 2023-10-10T17:52:37.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5dc0a14d0 0x7fa5dc0a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa6058f3b80 1 -- 172.21.15.78:0/307503906 shutdown_connections 2023-10-10T17:52:37.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322b9b60 0x55bc322b9f30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 0x55bc322bed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/307503906 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 0x55bc322be370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.831+0000 7fa6058f3b80 1 -- 172.21.15.78:0/307503906 >> 172.21.15.78:0/307503906 conn(0x55bc322af360 msgr2=0x55bc322b4b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.834+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5dc0a14d0 0x7fa5dc0a3980 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4005d40 tx=0x7fa5e400a3f0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.835+0000 7fa6058f3b80 1 -- 172.21.15.78:0/307503906 shutdown_connections 2023-10-10T17:52:37.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.835+0000 7fa6058f3b80 1 -- 172.21.15.78:0/307503906 wait complete. 2023-10-10T17:52:37.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.835+0000 7fa6058f3b80 1 -- start start 2023-10-10T17:52:37.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.835+0000 7fa6058f3b80 1 Processor -- start 2023-10-10T17:52:37.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:37 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2976011810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:52:37.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322ba470 0x55bc322c32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 0x55bc322c3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc321ac8e0 con 0x55bc322be8b0 2023-10-10T17:52:37.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc321ac580 con 0x55bc322b9b60 2023-10-10T17:52:37.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc3208e920 con 0x55bc322ba470 2023-10-10T17:52:37.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa6058f3b80 10 client.0 _ll_register_callbacks cb 0x55bc321ca780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:37.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50812/0 (socket says 172.21.15.78:50812) 2023-10-10T17:52:37.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 -- 172.21.15.78:0/2986906456 learned_addr learned my addr 172.21.15.78:0/2986906456 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322ba470 0x55bc322c32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 -- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322ba470 msgr2=0x55bc322c32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322ba470 0x55bc322c32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 -- 172.21.15.78:0/2986906456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 msgr2=0x55bc322c3a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:52:37.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 0x55bc322c3a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.844 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.836+0000 7fa5f2ffd700 1 -- 172.21.15.78:0/2986906456 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e40124d0 con 0x55bc322b9b60 2023-10-10T17:52:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.837+0000 7fa5f2ffd700 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c60e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8005b40 tx=0x7fa5e8003730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e078dda33c5e3281 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.837+0000 7fa5d6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.157:3300/0 2023-10-10T17:52:37.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.843+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/2986906456 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e801d030 con 0x55bc322b9b60 2023-10-10T17:52:37.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.843+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/2986906456 <== mon.2 v2:172.21.15.157:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fa5e8003e10 con 0x55bc322b9b60 2023-10-10T17:52:37.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.843+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bc322acb20 con 0x55bc322b9b60 2023-10-10T17:52:37.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.847+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/2986906456 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5e80052c0 con 0x55bc322b9b60 2023-10-10T17:52:37.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.849+0000 7fa6058f3b80 10 client.26630 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:37.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.849+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55bc321d1190 con 0x55bc322b9b60 2023-10-10T17:52:37.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/2986906456 <== mon.2 v2:172.21.15.157:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bc322acb20 con 0x55bc322b9b60 2023-10-10T17:52:37.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/2986906456 <== mon.2 v2:172.21.15.157:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa5e804c070 con 0x55bc322b9b60 2023-10-10T17:52:37.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa6058f3b80 10 client.26630 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:37.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa6058f3b80 10 client.26630 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:52:37.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa6058f3b80 20 client.26630 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:37.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc322d39c0 0x55bc322d5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa6058f3b80 4 client.26630 mds_command: new command op to 24490 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:37.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.850+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bc321cd0a0 con 0x55bc322d39c0 2023-10-10T17:52:37.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.852+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc322d39c0 0x55bc322d5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.854+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc322d39c0 0x55bc322d5dd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81527d3b97eff5e5 server_cookie=94ef98cc2874f348 in_seq=0 out_seq=0 2023-10-10T17:52:37.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.854+0000 7fa5d6ff5700 10 client.26630 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:37.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.856+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/2986906456 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55bc321cd0a0 con 0x55bc322d39c0 2023-10-10T17:52:37.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.856+0000 7fa5d6ff5700 10 client.26630 handle_command_reply: tid=0 2023-10-10T17:52:37.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.856+0000 7fa6058f3b80 1 client.26630 shutdown 2023-10-10T17:52:37.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.868+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 msgr2=0x55bc322c60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.868+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c60e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8005b40 tx=0x7fa5e8003730 comp rx=0 tx=0).stop 2023-10-10T17:52:37.869 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.868+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 shutdown_connections 2023-10-10T17:52:37.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322be8b0 0x55bc322c3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c60e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc322d39c0 0x55bc322d5dd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/2986906456 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322ba470 0x55bc322c32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 >> 172.21.15.78:0/2986906456 conn(0x55bc322af360 msgr2=0x55bc322bfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa5d6ff5700 0 client.26630 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:37.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 shutdown_connections 2023-10-10T17:52:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 1 -- 172.21.15.78:0/2986906456 wait complete. 2023-10-10T17:52:37.872 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.869+0000 7fa6058f3b80 20 client.26630 trim_cache size 0 max 16384 2023-10-10T17:52:37.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.892+0000 7fa6058f3b80 1 Processor -- start 2023-10-10T17:52:37.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.893+0000 7fa6058f3b80 1 -- start start 2023-10-10T17:52:37.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322ba470 0x55bc322c0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc322d0350 con 0x55bc322ba470 2023-10-10T17:52:37.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc322c42e0 con 0x55bc322b9b60 2023-10-10T17:52:37.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc322b61c0 con 0x55bc322be8b0 2023-10-10T17:52:37.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322ba470 0x55bc322c0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.897 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322ba470 0x55bc322c0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49094/0 (socket says 172.21.15.78:49094) 2023-10-10T17:52:37.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 -- 172.21.15.78:0/4262843027 learned_addr learned my addr 172.21.15.78:0/4262843027 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 -- 172.21.15.78:0/4262843027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 msgr2=0x55bc322c11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 -- 172.21.15.78:0/4262843027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 msgr2=0x55bc322c5c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-10T17:52:37.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c5c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.899 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.894+0000 7fa5f27fc700 1 -- 172.21.15.78:0/4262843027 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc321d0ee0 con 0x55bc322ba470 2023-10-10T17:52:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.895+0000 7fa5f27fc700 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322ba470 0x55bc322c0c90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4012de0 tx=0x7fa5e4011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e47780c7345715 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.900 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.895+0000 7fa5d9ffb700 1 -- 172.21.15.78:0/4262843027 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4011790 con 0x55bc322ba470 2023-10-10T17:52:37.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.895+0000 7fa5d9ffb700 1 -- 172.21.15.78:0/4262843027 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5e40118f0 con 0x55bc322ba470 2023-10-10T17:52:37.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.895+0000 7fa5d9ffb700 1 -- 172.21.15.78:0/4262843027 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4005020 con 0x55bc322ba470 2023-10-10T17:52:37.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.896+0000 7fa6058f3b80 1 -- 172.21.15.78:0/4262843027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322ba470 msgr2=0x55bc322c0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.896+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322ba470 0x55bc322c0c90 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4012de0 tx=0x7fa5e4011be0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.896+0000 7fa6058f3b80 1 -- 172.21.15.78:0/4262843027 shutdown_connections 2023-10-10T17:52:37.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.896+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.896+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322b9b60 0x55bc322c5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.896+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/4262843027 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322ba470 0x55bc322c0c90 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.896+0000 7fa6058f3b80 1 -- 172.21.15.78:0/4262843027 >> 172.21.15.78:0/4262843027 conn(0x55bc322af360 msgr2=0x55bc322bc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.903+0000 7fa6058f3b80 1 -- 172.21.15.78:0/4262843027 shutdown_connections 2023-10-10T17:52:37.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.907+0000 7fa6058f3b80 1 -- 172.21.15.78:0/4262843027 wait complete. 2023-10-10T17:52:37.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.907+0000 7fa6058f3b80 1 -- start start 2023-10-10T17:52:37.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.907+0000 7fa6058f3b80 1 Processor -- start 2023-10-10T17:52:37.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.907+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322b9b60 0x55bc322caa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.907+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 0x55bc322c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.907+0000 7fa6058f3b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.908+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc322c94a0 con 0x55bc322b9b60 2023-10-10T17:52:37.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.908+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc322c9610 con 0x55bc322ba470 2023-10-10T17:52:37.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.908+0000 7fa6058f3b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc322c97d0 con 0x55bc322be8b0 2023-10-10T17:52:37.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.908+0000 7fa6058f3b80 10 client.0 _ll_register_callbacks cb 0x7fa5e800bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:37.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.911+0000 7fa5f37fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.911+0000 7fa5f37fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40632/0 (socket says 172.21.15.78:40632) 2023-10-10T17:52:37.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5f37fe700 1 -- 172.21.15.78:0/3471127641 learned_addr learned my addr 172.21.15.78:0/3471127641 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5f37fe700 1 -- 172.21.15.78:0/3471127641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 msgr2=0x55bc322c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:37.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 0x55bc322c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5f37fe700 1 -- 172.21.15.78:0/3471127641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322b9b60 msgr2=0x55bc322caa40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-10T17:52:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322b9b60 0x55bc322caa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5f37fe700 1 -- 172.21.15.78:0/3471127641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc3208c630 con 0x55bc322be8b0 2023-10-10T17:52:37.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5f37fe700 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c8f60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec095760 tx=0x7fa5ec09d3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c2a0978c703af server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.912+0000 7fa5d6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:52:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.916+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/3471127641 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec098750 con 0x55bc322be8b0 2023-10-10T17:52:37.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.916+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/3471127641 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fa5ec0988b0 con 0x55bc322be8b0 2023-10-10T17:52:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.917+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5e4018070 con 0x55bc322be8b0 2023-10-10T17:52:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.918+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/3471127641 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5ec0a1220 con 0x55bc322be8b0 2023-10-10T17:52:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.918+0000 7fa6058f3b80 10 client.26839 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:37.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.918+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55bc32021c60 con 0x55bc322be8b0 2023-10-10T17:52:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.918+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/3471127641 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa5e4018070 con 0x55bc322be8b0 2023-10-10T17:52:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.918+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/3471127641 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa5ec0981c0 con 0x55bc322be8b0 2023-10-10T17:52:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.919+0000 7fa6058f3b80 10 client.26839 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.919+0000 7fa6058f3b80 10 client.26839 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2023-10-10T17:52:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.919+0000 7fa6058f3b80 20 client.26839 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.919+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc32327d90 0x55bc3232a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.919+0000 7fa6058f3b80 4 client.26839 mds_command: new command op to 24490 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:52:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.919+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 --> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bc321d9660 con 0x55bc32327d90 2023-10-10T17:52:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.921+0000 7fa5f2ffd700 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc32327d90 0x55bc3232a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.922+0000 7fa5f2ffd700 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc32327d90 0x55bc3232a1a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88715bfb72d8a0b3 server_cookie=36a1dc299e739bd5 in_seq=0 out_seq=0 2023-10-10T17:52:37.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.925+0000 7fa5d6ff5700 10 client.26839 ms_handle_connect on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.925+0000 7fa5d6ff5700 1 -- 172.21.15.78:0/3471127641 <== mds.1 v2:172.21.15.190:6838/486459111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bc321d9660 con 0x55bc32327d90 2023-10-10T17:52:37.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.925+0000 7fa5d6ff5700 10 client.26839 handle_command_reply: tid=0 2023-10-10T17:52:37.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.925+0000 7fa6058f3b80 1 client.26839 shutdown 2023-10-10T17:52:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.928+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 msgr2=0x55bc322c8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.928+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c8f60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec095760 tx=0x7fa5ec09d3c0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.929+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 shutdown_connections 2023-10-10T17:52:37.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.929+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:6838/486459111,v1:172.21.15.190:6839/486459111] conn(0x55bc32327d90 0x55bc3232a1a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.929+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55bc322be8b0 0x55bc322c8f60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.929+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55bc322b9b60 0x55bc322caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.929+0000 7fa6058f3b80 1 --2- 172.21.15.78:0/3471127641 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55bc322ba470 0x55bc322c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.929+0000 7fa5d6ff5700 0 client.26839 ms_handle_reset on v2:172.21.15.190:6838/486459111 2023-10-10T17:52:37.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.929+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 >> 172.21.15.78:0/3471127641 conn(0x55bc322af360 msgr2=0x55bc322c0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.930+0000 7ff87077a700 1 -- 172.21.15.78:0/2072000486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 msgr2=0x7ff86810cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.930+0000 7ff87077a700 1 --2- 172.21.15.78:0/2072000486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 0x7ff86810cc60 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff86000b0a0 tx=0x7ff86000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.931+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 shutdown_connections 2023-10-10T17:52:37.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.935+0000 7ff87077a700 1 -- 172.21.15.78:0/2072000486 shutdown_connections 2023-10-10T17:52:37.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.935+0000 7ff87077a700 1 --2- 172.21.15.78:0/2072000486 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 0x7ff86810cc60 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.935+0000 7ff87077a700 1 --2- 172.21.15.78:0/2072000486 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff868107050 0x7ff8681136e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.935+0000 7ff87077a700 1 --2- 172.21.15.78:0/2072000486 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff86810d230 0x7ff868106920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.935+0000 7ff87077a700 1 -- 172.21.15.78:0/2072000486 >> 172.21.15.78:0/2072000486 conn(0x7ff8680747d0 msgr2=0x7ff868076be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.935+0000 7ff87077a700 1 -- 172.21.15.78:0/2072000486 shutdown_connections 2023-10-10T17:52:37.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.935+0000 7ff87077a700 1 -- 172.21.15.78:0/2072000486 wait complete. 2023-10-10T17:52:37.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 Processor -- start 2023-10-10T17:52:37.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 -- start start 2023-10-10T17:52:37.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff868107050 0x7ff8681a2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 0x7ff8681a2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86810d230 0x7ff86819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86819cf20 con 0x7ff868107050 2023-10-10T17:52:37.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86819d090 con 0x7ff86810d230 2023-10-10T17:52:37.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff87077a700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86819d250 con 0x7ff86810c890 2023-10-10T17:52:37.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff86e516700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff868107050 0x7ff8681a2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff86e516700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff868107050 0x7ff8681a2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49112/0 (socket says 172.21.15.78:49112) 2023-10-10T17:52:37.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.936+0000 7ff86e516700 1 -- 172.21.15.78:0/3222593184 learned_addr learned my addr 172.21.15.78:0/3222593184 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:37.943 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86dd15700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 0x7ff8681a2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86ed17700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86810d230 0x7ff86819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86e516700 1 -- 172.21.15.78:0/3222593184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 msgr2=0x7ff8681a2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86e516700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 0x7ff8681a2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.944 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86e516700 1 -- 172.21.15.78:0/3222593184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86810d230 msgr2=0x7ff86819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86e516700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86810d230 0x7ff86819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86e516700 1 -- 172.21.15.78:0/3222593184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff860009d00 con 0x7ff868107050 2023-10-10T17:52:37.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86dd15700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 0x7ff8681a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:52:37.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.937+0000 7ff86e516700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff868107050 0x7ff8681a2840 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff86000b070 tx=0x7ff860004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.946 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.942+0000 7fa6058f3b80 1 -- 172.21.15.78:0/3471127641 wait complete. 2023-10-10T17:52:37.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.942+0000 7fa6058f3b80 20 client.26839 trim_cache size 0 max 16384 2023-10-10T17:52:37.947 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:52:37.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.943+0000 7ff85f7fe700 1 -- 172.21.15.78:0/3222593184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86000e040 con 0x7ff868107050 2023-10-10T17:52:37.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.943+0000 7ff85f7fe700 1 -- 172.21.15.78:0/3222593184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff860003a00 con 0x7ff868107050 2023-10-10T17:52:37.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.943+0000 7ff85f7fe700 1 -- 172.21.15.78:0/3222593184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860004580 con 0x7ff868107050 2023-10-10T17:52:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.943+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8681169c0 con 0x7ff868107050 2023-10-10T17:52:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.943+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff868116830 con 0x7ff868107050 2023-10-10T17:52:37.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.947+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c005320 con 0x7ff868107050 2023-10-10T17:52:37.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.949+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5dc0a14d0 msgr2=0x7fa5dc0a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.949+0000 7fa5f9767700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5dc0a14d0 0x7fa5dc0a3980 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4005d40 tx=0x7fa5e400a3f0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.949+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f4104960 msgr2=0x7fa5f410a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:37.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.949+0000 7fa5f9767700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f4104960 0x7fa5f410a3d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00bab0 tx=0x7fa5ec00bdc0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 shutdown_connections 2023-10-10T17:52:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa5f40fe950 0x7fa5f406fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa5dc0a14d0 0x7fa5dc0a3980 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa5f4104960 0x7fa5f410a3d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 --2- 172.21.15.78:0/389006508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa5f40ff300 0x7fa5f4070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 >> 172.21.15.78:0/389006508 conn(0x7fa5f4075920 msgr2=0x7fa5f4100490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 shutdown_connections 2023-10-10T17:52:37.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.950+0000 7fa5f9767700 1 -- 172.21.15.78:0/389006508 wait complete. 2023-10-10T17:52:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.959+0000 7ff85f7fe700 1 -- 172.21.15.78:0/3222593184 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff8600046e0 con 0x7ff868107050 2023-10-10T17:52:37.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.959+0000 7ff85f7fe700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8540a1410 0x7ff8540a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.960+0000 7ff85f7fe700 1 -- 172.21.15.78:0/3222593184 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7ff8600ce080 con 0x7ff868107050 2023-10-10T17:52:37.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.960+0000 7ff86dd15700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8540a1410 0x7ff8540a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:37.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.960+0000 7ff86dd15700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8540a1410 0x7ff8540a38c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff868106a70 tx=0x7ff864009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:37.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:37.970+0000 7ff85f7fe700 1 -- 172.21.15.78:0/3222593184 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff8600971c0 con 0x7ff868107050 2023-10-10T17:52:37.977 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-10-10T17:52:38.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:37 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2976011810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:52:38.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:37 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2976011810' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:52:38.273 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.273+0000 7f2237c8d700 1 -- 172.21.15.78:0/1524574988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2230107af0 msgr2=0x7f2230107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.273+0000 7f2237c8d700 1 --2- 172.21.15.78:0/1524574988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2230107af0 0x7f2230107f60 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f222c009990 tx=0x7f222c009ca0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.279+0000 7f2237c8d700 1 -- 172.21.15.78:0/1524574988 shutdown_connections 2023-10-10T17:52:38.286 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.279+0000 7f2237c8d700 1 --2- 172.21.15.78:0/1524574988 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f223010bf10 0x7f223010c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.279+0000 7f2237c8d700 1 --2- 172.21.15.78:0/1524574988 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f22301084a0 0x7f22300797c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.279+0000 7f2237c8d700 1 --2- 172.21.15.78:0/1524574988 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2230107af0 0x7f2230107f60 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.287 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.279+0000 7f2237c8d700 1 -- 172.21.15.78:0/1524574988 >> 172.21.15.78:0/1524574988 conn(0x7f223006c7b0 msgr2=0x7f2230071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.296+0000 7f2237c8d700 1 -- 172.21.15.78:0/1524574988 shutdown_connections 2023-10-10T17:52:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.300+0000 7f2237c8d700 1 -- 172.21.15.78:0/1524574988 wait complete. 2023-10-10T17:52:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.300+0000 7f2237c8d700 1 Processor -- start 2023-10-10T17:52:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.300+0000 7f2237c8d700 1 -- start start 2023-10-10T17:52:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2237c8d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2230107af0 0x7f22300740c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2237c8d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22301084a0 0x7f2230074600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2237c8d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f223010bf10 0x7f22301c20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2237c8d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22301c26a0 con 0x7f22301084a0 2023-10-10T17:52:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2237c8d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22301c2810 con 0x7f223010bf10 2023-10-10T17:52:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2237c8d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22301c29d0 con 0x7f2230107af0 2023-10-10T17:52:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2235a29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2230107af0 0x7f22300740c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2235a29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2230107af0 0x7f22300740c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40654/0 (socket says 172.21.15.78:40654) 2023-10-10T17:52:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2235a29700 1 -- 172.21.15.78:0/898084817 learned_addr learned my addr 172.21.15.78:0/898084817 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f223622a700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f223010bf10 0x7f22301c20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f223622a700 1 -- 172.21.15.78:0/898084817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2230107af0 msgr2=0x7f22300740c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f223622a700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2230107af0 0x7f22300740c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f223622a700 1 -- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22301084a0 msgr2=0x7f2230074600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f223622a700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22301084a0 0x7f2230074600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f223622a700 1 -- 172.21.15.78:0/898084817 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c0095f0 con 0x7f223010bf10 2023-10-10T17:52:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.301+0000 7f2235a29700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2230107af0 0x7f22300740c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:52:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.304+0000 7f223622a700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f223010bf10 0x7f22301c20a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f222000c7a0 tx=0x7f222000cab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.305+0000 7f2226ffd700 1 -- 172.21.15.78:0/898084817 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220004a30 con 0x7f223010bf10 2023-10-10T17:52:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.305+0000 7f2226ffd700 1 -- 172.21.15.78:0/898084817 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22200054e0 con 0x7f223010bf10 2023-10-10T17:52:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.305+0000 7f2226ffd700 1 -- 172.21.15.78:0/898084817 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222000eb70 con 0x7f223010bf10 2023-10-10T17:52:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.308+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223007cb80 con 0x7f223010bf10 2023-10-10T17:52:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.308+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223007c9f0 con 0x7f223010bf10 2023-10-10T17:52:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.315+0000 7f2226ffd700 1 -- 172.21.15.78:0/898084817 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f2220004b90 con 0x7f223010bf10 2023-10-10T17:52:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.315+0000 7f2226ffd700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f221c0a1490 0x7f221c0a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.316+0000 7f2226ffd700 1 -- 172.21.15.78:0/898084817 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f22200ccd90 con 0x7f223010bf10 2023-10-10T17:52:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.319+0000 7f2235a29700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f221c0a1490 0x7f221c0a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.319+0000 7f2243e19b80 1 Processor -- start 2023-10-10T17:52:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.319+0000 7f2235a29700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f221c0a1490 0x7f221c0a3940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f222800c550 tx=0x7f222800b040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2243e19b80 1 -- start start 2023-10-10T17:52:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 0x55c6057c7f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057cc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6057ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2243e19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6056cd060 con 0x55c6057c7b60 2023-10-10T17:52:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2243e19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60571f8a0 con 0x55c6057cc8b0 2023-10-10T17:52:38.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2243e19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c605718d30 con 0x55c6057c8470 2023-10-10T17:52:38.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.320+0000 7f2235228700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f2235228700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40662/0 (socket says 172.21.15.78:40662) 2023-10-10T17:52:38.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f2235228700 1 -- 172.21.15.78:0/698858203 learned_addr learned my addr 172.21.15.78:0/698858203 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:38.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f223622a700 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6057ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f223622a700 1 -- 172.21.15.78:0/698858203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 msgr2=0x55c6057cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f223622a700 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057cc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f223622a700 1 -- 172.21.15.78:0/698858203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 msgr2=0x55c6057c7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f223622a700 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 0x55c6057c7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f223622a700 1 -- 172.21.15.78:0/698858203 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c60559a630 con 0x55c6057cc8b0 2023-10-10T17:52:38.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.321+0000 7f2235228700 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:52:38.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.324+0000 7f223622a700 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6057ccd00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f22200ca0c0 tx=0x7f2220005650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad46e1f65c2133af server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.331+0000 7f221bfff700 1 -- 172.21.15.78:0/698858203 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22200b2260 con 0x55c6057cc8b0 2023-10-10T17:52:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.331+0000 7f221bfff700 1 -- 172.21.15.78:0/698858203 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22200a1f10 con 0x55c6057cc8b0 2023-10-10T17:52:38.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.331+0000 7f221bfff700 1 -- 172.21.15.78:0/698858203 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220099f80 con 0x55c6057cc8b0 2023-10-10T17:52:38.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.334+0000 7f2243e19b80 1 -- 172.21.15.78:0/698858203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 msgr2=0x55c6057ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.334+0000 7f2243e19b80 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6057ccd00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f22200ca0c0 tx=0x7f2220005650 comp rx=0 tx=0).stop 2023-10-10T17:52:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.342+0000 7f2243e19b80 1 -- 172.21.15.78:0/698858203 shutdown_connections 2023-10-10T17:52:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.342+0000 7f2243e19b80 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6057ccd00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.342+0000 7f2243e19b80 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 0x55c6057c7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.342+0000 7f2243e19b80 1 --2- 172.21.15.78:0/698858203 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.342+0000 7f2243e19b80 1 -- 172.21.15.78:0/698858203 >> 172.21.15.78:0/698858203 conn(0x55c6057bd360 msgr2=0x55c6057c2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.349+0000 7f2243e19b80 1 -- 172.21.15.78:0/698858203 shutdown_connections 2023-10-10T17:52:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.352+0000 7f2243e19b80 1 -- 172.21.15.78:0/698858203 wait complete. 2023-10-10T17:52:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.352+0000 7f2243e19b80 1 -- start start 2023-10-10T17:52:38.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.352+0000 7f2243e19b80 1 Processor -- start 2023-10-10T17:52:38.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c7b60 0x55c6057d40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057d1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2243e19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6056ba8e0 con 0x55c6057cc8b0 2023-10-10T17:52:38.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2243e19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6056ba580 con 0x55c6057c7b60 2023-10-10T17:52:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2243e19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60559c920 con 0x55c6057c8470 2023-10-10T17:52:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2243e19b80 10 client.0 _ll_register_callbacks cb 0x55c6056d8780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:38.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2235228700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2235228700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40666/0 (socket says 172.21.15.78:40666) 2023-10-10T17:52:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f2235228700 1 -- 172.21.15.78:0/718974074 learned_addr learned my addr 172.21.15.78:0/718974074 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.353+0000 7f223622a700 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057d1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f223622a700 1 -- 172.21.15.78:0/718974074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 msgr2=0x55c6057d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f223622a700 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f223622a700 1 -- 172.21.15.78:0/718974074 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c7b60 msgr2=0x55c6057d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f223622a700 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c7b60 0x55c6057d40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f223622a700 1 -- 172.21.15.78:0/718974074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223007c9f0 con 0x55c6057cc8b0 2023-10-10T17:52:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f223622a700 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057d1aa0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x55c6057d4620 tx=0x7f2220098070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539fa59e5de757b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f22197fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-10-10T17:52:38.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f22197fa700 1 -- 172.21.15.78:0/718974074 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22200991b0 con 0x55c6057cc8b0 2023-10-10T17:52:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f22197fa700 1 -- 172.21.15.78:0/718974074 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f2220099d20 con 0x55c6057cc8b0 2023-10-10T17:52:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.354+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6057bab20 con 0x55c6057cc8b0 2023-10-10T17:52:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f22197fa700 1 -- 172.21.15.78:0/718974074 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2220099ee0 con 0x55c6057cc8b0 2023-10-10T17:52:38.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 10 client.17529 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55c6056df190 con 0x55c6057cc8b0 2023-10-10T17:52:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f22197fa700 1 -- 172.21.15.78:0/718974074 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22200c2030 con 0x55c6057cc8b0 2023-10-10T17:52:38.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f22197fa700 1 -- 172.21.15.78:0/718974074 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2220098a20 con 0x55c6057cc8b0 2023-10-10T17:52:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 10 client.17529 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 10 client.17529 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 20 client.17529 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:38.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6057e1400 0x55c6057e3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 4 client.17529 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-10T17:52:38.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.355+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c6056db0a0 con 0x55c6057e1400 2023-10-10T17:52:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.358+0000 7f2235a29700 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6057e1400 0x55c6057e3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.358+0000 7f2235a29700 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6057e1400 0x55c6057e3810 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=637fb3909bce8793 server_cookie=32176680743b6ed in_seq=0 out_seq=0 2023-10-10T17:52:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.359+0000 7f22197fa700 10 client.17529 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.362+0000 7f22197fa700 1 -- 172.21.15.78:0/718974074 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18291 (crc 0 0 0) 0x55c6056db0a0 con 0x55c6057e1400 2023-10-10T17:52:38.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.362+0000 7f22197fa700 10 client.17529 handle_command_reply: tid=0 2023-10-10T17:52:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.362+0000 7f2243e19b80 1 client.17529 shutdown 2023-10-10T17:52:38.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 msgr2=0x55c6057d1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057d1aa0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x55c6057d4620 tx=0x7f2220098070 comp rx=0 tx=0).stop 2023-10-10T17:52:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 shutdown_connections 2023-10-10T17:52:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6057e1400 0x55c6057e3810 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057d1aa0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c7b60 0x55c6057d40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 --2- 172.21.15.78:0/718974074 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f22197fa700 0 client.17529 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:38.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.366+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 >> 172.21.15.78:0/718974074 conn(0x55c6057bd360 msgr2=0x55c6057cdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.368+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 shutdown_connections 2023-10-10T17:52:38.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.368+0000 7f2243e19b80 1 -- 172.21.15.78:0/718974074 wait complete. 2023-10-10T17:52:38.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.368+0000 7f2243e19b80 20 client.17529 trim_cache size 0 max 16384 2023-10-10T17:52:38.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 Processor -- start 2023-10-10T17:52:38.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 -- start start 2023-10-10T17:52:38.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c7b60 0x55c6057cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c8470 0x55c6057d8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6057d1de0 con 0x55c6057cc8b0 2023-10-10T17:52:38.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6057cb4b0 con 0x55c6057c8470 2023-10-10T17:52:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.389+0000 7f2243e19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6057c8900 con 0x55c6057c7b60 2023-10-10T17:52:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c7b60 0x55c6057cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c7b60 0x55c6057cf580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40672/0 (socket says 172.21.15.78:40672) 2023-10-10T17:52:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 -- 172.21.15.78:0/2317221499 learned_addr learned my addr 172.21.15.78:0/2317221499 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235228700 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c8470 0x55c6057d8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f223622a700 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 -- 172.21.15.78:0/2317221499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c8470 msgr2=0x55c6057d8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c8470 0x55c6057d8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 -- 172.21.15.78:0/2317221499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 msgr2=0x55c6057cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057cdc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 -- 172.21.15.78:0/2317221499 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6056deee0 con 0x55c6057c7b60 2023-10-10T17:52:38.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2235a29700 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c7b60 0x55c6057cf580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55c6057e40f0 tx=0x7f2228004b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e0f2dbb34caeda8 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.390+0000 7f2224dac700 1 -- 172.21.15.78:0/2317221499 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228015e20 con 0x55c6057c7b60 2023-10-10T17:52:38.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.391+0000 7f2224dac700 1 -- 172.21.15.78:0/2317221499 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f222801b040 con 0x55c6057c7b60 2023-10-10T17:52:38.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.391+0000 7f2224dac700 1 -- 172.21.15.78:0/2317221499 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228021650 con 0x55c6057c7b60 2023-10-10T17:52:38.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.391+0000 7f2243e19b80 1 -- 172.21.15.78:0/2317221499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c7b60 msgr2=0x55c6057cf580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.392+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c7b60 0x55c6057cf580 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x55c6057e40f0 tx=0x7f2228004b80 comp rx=0 tx=0).stop 2023-10-10T17:52:38.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.392+0000 7f2243e19b80 1 -- 172.21.15.78:0/2317221499 shutdown_connections 2023-10-10T17:52:38.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.392+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c7b60 0x55c6057cf580 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.392+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057cc8b0 0x55c6057cdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.392+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2317221499 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057c8470 0x55c6057d8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.392+0000 7f2243e19b80 1 -- 172.21.15.78:0/2317221499 >> 172.21.15.78:0/2317221499 conn(0x55c6057bd360 msgr2=0x55c6057c1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.393+0000 7f2243e19b80 1 -- 172.21.15.78:0/2317221499 shutdown_connections 2023-10-10T17:52:38.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.393+0000 7f2243e19b80 1 -- 172.21.15.78:0/2317221499 wait complete. 2023-10-10T17:52:38.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 -- start start 2023-10-10T17:52:38.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 Processor -- start 2023-10-10T17:52:38.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 0x55c6057d4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d4f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6058342b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6057d5670 con 0x55c6057c7b60 2023-10-10T17:52:38.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6058347f0 con 0x55c6057cc8b0 2023-10-10T17:52:38.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c605834930 con 0x55c6057c8470 2023-10-10T17:52:38.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.394+0000 7f2243e19b80 10 client.0 _ll_register_callbacks cb 0x7f222c009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-10T17:52:38.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40680/0 (socket says 172.21.15.78:40680) 2023-10-10T17:52:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 -- 172.21.15.78:0/2179852751 learned_addr learned my addr 172.21.15.78:0/2179852751 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:38.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 -- 172.21.15.78:0/2179852751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 msgr2=0x55c6058342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6058342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 -- 172.21.15.78:0/2179852751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 msgr2=0x55c6057d4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-10T17:52:38.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 0x55c6057d4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.397+0000 7f2235228700 1 -- 172.21.15.78:0/2179852751 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c60552fc60 con 0x55c6057c8470 2023-10-10T17:52:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.398+0000 7f2235228700 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d4f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55c6057ba6c0 tx=0x7f222c00b580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fea3f4ed01bdead7 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.399+0000 7f22197fa700 10 client.0 ms_handle_connect on v2:172.21.15.190:3300/0 2023-10-10T17:52:38.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.399+0000 7f22197fa700 1 -- 172.21.15.78:0/2179852751 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f222c01f040 con 0x55c6057c8470 2023-10-10T17:52:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.399+0000 7f22197fa700 1 -- 172.21.15.78:0/2179852751 <== mon.1 v2:172.21.15.190:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f222c0050b0 con 0x55c6057c8470 2023-10-10T17:52:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.399+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22200a1f10 con 0x55c6057c8470 2023-10-10T17:52:38.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.401+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 --> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7ff84c000bf0 con 0x7ff8540a1410 2023-10-10T17:52:38.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.404+0000 7f22197fa700 1 -- 172.21.15.78:0/2179852751 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f222c005920 con 0x55c6057c8470 2023-10-10T17:52:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.409+0000 7f2243e19b80 10 client.26860 fetch_fsmap learned FSMap version 38 2023-10-10T17:52:38.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.409+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55c605546de0 con 0x55c6057c8470 2023-10-10T17:52:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f22197fa700 1 -- 172.21.15.78:0/2179852751 <== mon.1 v2:172.21.15.190:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22200a1f10 con 0x55c6057c8470 2023-10-10T17:52:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f22197fa700 1 -- 172.21.15.78:0/2179852751 <== mon.1 v2:172.21.15.190:3300/0 5 ==== fsmap(e 38) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f222c025cd0 con 0x55c6057c8470 2023-10-10T17:52:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f2243e19b80 10 client.26860 fetch_fsmap finished waiting for FSMap version 38 2023-10-10T17:52:38.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f2243e19b80 10 client.26860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-10-10T17:52:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f2243e19b80 20 client.26860 populate_metadata read hostname 'smithi078' 2023-10-10T17:52:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6058360b0 0x55c6058384c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f2243e19b80 4 client.26860 mds_command: new command op to 24500 tid=0[{"prefix": "damage ls"}] 2023-10-10T17:52:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 --> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c6056e7660 con 0x55c6058360b0 2023-10-10T17:52:38.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.411+0000 7f2235a29700 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6058360b0 0x55c6058384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.412+0000 7f2235a29700 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6058360b0 0x55c6058384c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de3c5a901fcd93b8 server_cookie=39fcab730ec62705 in_seq=0 out_seq=0 2023-10-10T17:52:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f22197fa700 10 client.26860 ms_handle_connect on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:38.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f22197fa700 1 -- 172.21.15.78:0/2179852751 <== mds.0 v2:172.21.15.157:6838/2554315829 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c6056e7660 con 0x55c6058360b0 2023-10-10T17:52:38.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f22197fa700 10 client.26860 handle_command_reply: tid=0 2023-10-10T17:52:38.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 client.26860 shutdown 2023-10-10T17:52:38.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 msgr2=0x55c6057d4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d4f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x55c6057ba6c0 tx=0x7f222c00b580 comp rx=0 tx=0).stop 2023-10-10T17:52:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 shutdown_connections 2023-10-10T17:52:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.157:6838/2554315829,v1:172.21.15.157:6839/2554315829] conn(0x55c6058360b0 0x55c6058384c0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x55c6057cc8b0 0x55c6058342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55c6057c7b60 0x55c6057d4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 --2- 172.21.15.78:0/2179852751 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x55c6057c8470 0x55c6057d4f90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.414+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 >> 172.21.15.78:0/2179852751 conn(0x55c6057bd360 msgr2=0x55c6057ca9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.417+0000 7f22197fa700 0 client.26860 ms_handle_reset on v2:172.21.15.157:6838/2554315829 2023-10-10T17:52:38.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.417+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 shutdown_connections 2023-10-10T17:52:38.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.418+0000 7f2243e19b80 1 -- 172.21.15.78:0/2179852751 wait complete. 2023-10-10T17:52:38.424 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.424+0000 7f2243e19b80 20 client.26860 trim_cache size 0 max 16384 2023-10-10T17:52:38.425 INFO:teuthology.orchestra.run.smithi078.stdout:[] 2023-10-10T17:52:38.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f221c0a1490 msgr2=0x7f221c0a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f221c0a1490 0x7f221c0a3940 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f222800c550 tx=0x7f222800b040 comp rx=0 tx=0).stop 2023-10-10T17:52:38.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f223010bf10 msgr2=0x7f22301c20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f223010bf10 0x7f22301c20a0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f222000c7a0 tx=0x7f222000cab0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 shutdown_connections 2023-10-10T17:52:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2230107af0 0x7f22300740c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f221c0a1490 0x7f221c0a3940 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f222800c550 tx=0x7f222800b040 comp rx=0 tx=0).stop 2023-10-10T17:52:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f223010bf10 0x7f22301c20a0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 --2- 172.21.15.78:0/898084817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f22301084a0 0x7f2230074600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.428+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 >> 172.21.15.78:0/898084817 conn(0x7f223006c7b0 msgr2=0x7f2230070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.433+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 shutdown_connections 2023-10-10T17:52:38.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.433+0000 7f2237c8d700 1 -- 172.21.15.78:0/898084817 wait complete. 2023-10-10T17:52:38.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.462+0000 7ff85f7fe700 1 -- 172.21.15.78:0/3222593184 <== mgr.15588 v2:172.21.15.78:6800/2386743294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+335 (secure 0 0 0) 0x7ff84c000bf0 con 0x7ff8540a1410 2023-10-10T17:52:38.463 INFO:teuthology.orchestra.run.smithi078.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-10-10T00:00:00", "created": "2023-10-10T17:42:36", "first": "2023-10-10T17:43:00", "last": "2023-10-10T17:52:00", "last_pruned": "2023-10-10T17:52:00", "created_count": 10, "pruned_count": 4, "active": true} 2023-10-10T17:52:38.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.470+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8540a1410 msgr2=0x7ff8540a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.470+0000 7ff87077a700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8540a1410 0x7ff8540a38c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff868106a70 tx=0x7ff864009260 comp rx=0 tx=0).stop 2023-10-10T17:52:38.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.470+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff868107050 msgr2=0x7ff8681a2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.470+0000 7ff87077a700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff868107050 0x7ff8681a2840 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff86000b070 tx=0x7ff860004a50 comp rx=0 tx=0).stop 2023-10-10T17:52:38.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.471+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 shutdown_connections 2023-10-10T17:52:38.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.471+0000 7ff87077a700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8540a1410 0x7ff8540a38c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.471+0000 7ff87077a700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff86810d230 0x7ff86819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.471+0000 7ff87077a700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff868107050 0x7ff8681a2840 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.471+0000 7ff87077a700 1 --2- 172.21.15.78:0/3222593184 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff86810c890 0x7ff8681a2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.471+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 >> 172.21.15.78:0/3222593184 conn(0x7ff8680747d0 msgr2=0x7ff86810a460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.475+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 shutdown_connections 2023-10-10T17:52:38.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.475+0000 7ff87077a700 1 -- 172.21.15.78:0/3222593184 wait complete. 2023-10-10T17:52:38.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-10-10T17:52:38.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.784+0000 7fdb8dc56700 1 -- 172.21.15.78:0/352013036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 msgr2=0x7fdb880ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.784+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/352013036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb880ffd00 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fdb78009990 tx=0x7fdb78009ca0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.791+0000 7fdb8dc56700 1 -- 172.21.15.78:0/352013036 shutdown_connections 2023-10-10T17:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.791+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/352013036 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb88111410 0x7fdb881137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.791+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/352013036 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb880ff4c0 0x7fdb88106520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.791+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/352013036 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb880ffd00 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.791+0000 7fdb8dc56700 1 -- 172.21.15.78:0/352013036 >> 172.21.15.78:0/352013036 conn(0x7fdb88074a50 msgr2=0x7fdb88074e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:38.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.795+0000 7fdb8dc56700 1 -- 172.21.15.78:0/352013036 shutdown_connections 2023-10-10T17:52:38.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.796+0000 7fdb8dc56700 1 -- 172.21.15.78:0/352013036 wait complete. 2023-10-10T17:52:38.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.796+0000 7fdb8dc56700 1 Processor -- start 2023-10-10T17:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb8dc56700 1 -- start start 2023-10-10T17:52:38.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb8dc56700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb880ff4c0 0x7fdb881a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb8dc56700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb881a2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb8dc56700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb88111410 0x7fdb8819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb8dc56700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8819cfc0 con 0x7fdb880ff890 2023-10-10T17:52:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb8dc56700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8819d130 con 0x7fdb880ff4c0 2023-10-10T17:52:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb8dc56700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8819d2f0 con 0x7fdb88111410 2023-10-10T17:52:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb86ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb881a2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb877fe700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb880ff4c0 0x7fdb881a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb86ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb881a2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49164/0 (socket says 172.21.15.78:49164) 2023-10-10T17:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb86ffd700 1 -- 172.21.15.78:0/3614609974 learned_addr learned my addr 172.21.15.78:0/3614609974 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:38.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.797+0000 7fdb87fff700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb88111410 0x7fdb8819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.798+0000 7fdb86ffd700 1 -- 172.21.15.78:0/3614609974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb88111410 msgr2=0x7fdb8819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.798+0000 7fdb86ffd700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb88111410 0x7fdb8819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.798+0000 7fdb86ffd700 1 -- 172.21.15.78:0/3614609974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb880ff4c0 msgr2=0x7fdb881a28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:38.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.798+0000 7fdb86ffd700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb880ff4c0 0x7fdb881a28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:38.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.798+0000 7fdb86ffd700 1 -- 172.21.15.78:0/3614609974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780095f0 con 0x7fdb880ff890 2023-10-10T17:52:38.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.798+0000 7fdb86ffd700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb881a2e30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fdb78000c00 tx=0x7fdb78005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.802+0000 7fdb84ff9700 1 -- 172.21.15.78:0/3614609974 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7801d070 con 0x7fdb880ff890 2023-10-10T17:52:38.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.802+0000 7fdb84ff9700 1 -- 172.21.15.78:0/3614609974 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdb78022470 con 0x7fdb880ff890 2023-10-10T17:52:38.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.802+0000 7fdb84ff9700 1 -- 172.21.15.78:0/3614609974 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78017650 con 0x7fdb880ff890 2023-10-10T17:52:38.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.802+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb88116970 con 0x7fdb880ff890 2023-10-10T17:52:38.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.802+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb881167e0 con 0x7fdb880ff890 2023-10-10T17:52:38.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.803+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb88116b00 con 0x7fdb880ff890 2023-10-10T17:52:38.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.806+0000 7fdb84ff9700 1 -- 172.21.15.78:0/3614609974 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fdb780177b0 con 0x7fdb880ff890 2023-10-10T17:52:38.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.806+0000 7fdb84ff9700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb700a14d0 0x7fdb700a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.807+0000 7fdb84ff9700 1 -- 172.21.15.78:0/3614609974 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fdb780cea40 con 0x7fdb880ff890 2023-10-10T17:52:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.807+0000 7fdb877fe700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb700a14d0 0x7fdb700a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:38.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.807+0000 7fdb877fe700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb700a14d0 0x7fdb700a3980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c006fd0 tx=0x7fdb7c008040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:38.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:38.812+0000 7fdb84ff9700 1 -- 172.21.15.78:0/3614609974 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fdb78097b40 con 0x7fdb880ff890 2023-10-10T17:52:38.846 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:38 smithi078 ceph-mon[99494]: pgmap v313: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:39.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:38 smithi190 ceph-mon[101220]: pgmap v313: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:39.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.198+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 --> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fdb8819df10 con 0x7fdb700a14d0 2023-10-10T17:52:39.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:38 smithi157 ceph-mon[100896]: pgmap v313: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:39.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.249+0000 7fdb84ff9700 1 -- 172.21.15.78:0/3614609974 <== mgr.15588 v2:172.21.15.78:6800/2386743294 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fdb8819df10 con 0x7fdb700a14d0 2023-10-10T17:52:39.253 INFO:teuthology.orchestra.run.smithi078.stdout:/ 1M 6M3h 2023-10-10T17:52:39.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.259+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb700a14d0 msgr2=0x7fdb700a3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.259+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb700a14d0 0x7fdb700a3980 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c006fd0 tx=0x7fdb7c008040 comp rx=0 tx=0).stop 2023-10-10T17:52:39.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.259+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 msgr2=0x7fdb881a2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:39.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.259+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb881a2e30 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fdb78000c00 tx=0x7fdb78005740 comp rx=0 tx=0).stop 2023-10-10T17:52:39.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.263+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 shutdown_connections 2023-10-10T17:52:39.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.263+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fdb88111410 0x7fdb8819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.263+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fdb700a14d0 0x7fdb700a3980 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c006fd0 tx=0x7fdb7c008040 comp rx=0 tx=0).stop 2023-10-10T17:52:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.263+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fdb880ff4c0 0x7fdb881a28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.263+0000 7fdb8dc56700 1 --2- 172.21.15.78:0/3614609974 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdb880ff890 0x7fdb881a2e30 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.263+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 >> 172.21.15.78:0/3614609974 conn(0x7fdb88074a50 msgr2=0x7fdb881117e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:39.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.267+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 shutdown_connections 2023-10-10T17:52:39.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.268+0000 7fdb8dc56700 1 -- 172.21.15.78:0/3614609974 wait complete. 2023-10-10T17:52:39.292 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2023-10-10T17:52:39.304 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2023-10-10T17:52:39.460 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2023-10-10T17:52:39.460 INFO:tasks.fwd_scrub:done joining 2023-10-10T17:52:39.461 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2023-10-10T17:52:39.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:52:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.702+0000 7f078e676700 1 -- 172.21.15.78:0/609775281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f078810d0f0 msgr2=0x7f078810d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.702+0000 7f078e676700 1 --2- 172.21.15.78:0/609775281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f078810d0f0 0x7f078810d590 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f078000b0a0 tx=0x7f078000b3b0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.703+0000 7f078e676700 1 -- 172.21.15.78:0/609775281 shutdown_connections 2023-10-10T17:52:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.703+0000 7f078e676700 1 --2- 172.21.15.78:0/609775281 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f078810d0f0 0x7f078810d590 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.703+0000 7f078e676700 1 --2- 172.21.15.78:0/609775281 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0788107af0 0x7f0788107ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.703+0000 7f078e676700 1 --2- 172.21.15.78:0/609775281 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 0x7f078810cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.703+0000 7f078e676700 1 -- 172.21.15.78:0/609775281 >> 172.21.15.78:0/609775281 conn(0x7f078806c550 msgr2=0x7f07880716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.703+0000 7f078e676700 1 -- 172.21.15.78:0/609775281 shutdown_connections 2023-10-10T17:52:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.703+0000 7f078e676700 1 -- 172.21.15.78:0/609775281 wait complete. 2023-10-10T17:52:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.704+0000 7f078e676700 1 Processor -- start 2023-10-10T17:52:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.704+0000 7f078e676700 1 -- start start 2023-10-10T17:52:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078e676700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0788107af0 0x7f07881a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078e676700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 0x7f07881a7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078e676700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f078810d0f0 0x7f07881b9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078e676700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078811af30 con 0x7f0788107af0 2023-10-10T17:52:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078e676700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078811b0a0 con 0x7f078810d0f0 2023-10-10T17:52:39.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078e676700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07881ba120 con 0x7f0788108400 2023-10-10T17:52:39.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f078810d0f0 0x7f07881b9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:39.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f077ffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 0x7f07881a7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:39.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f078810d0f0 0x7f07881b9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50900/0 (socket says 172.21.15.78:50900) 2023-10-10T17:52:39.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 -- 172.21.15.78:0/1908136541 learned_addr learned my addr 172.21.15.78:0/1908136541 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:39.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f077ffff700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 0x7f07881a7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40706/0 (socket says 172.21.15.78:40706) 2023-10-10T17:52:39.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 -- 172.21.15.78:0/1908136541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 msgr2=0x7f07881a7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:39.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 0x7f07881a7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 -- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0788107af0 msgr2=0x7f07881a6f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:52:39.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0788107af0 0x7f07881a6f10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:39.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f078cc13700 1 -- 172.21.15.78:0/1908136541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0780009d00 con 0x7f078810d0f0 2023-10-10T17:52:39.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.705+0000 7f077ffff700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 0x7f07881a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:39.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.706+0000 7f078cc13700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f078810d0f0 0x7f07881b9be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0780009fd0 tx=0x7f07800123f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:39.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.706+0000 7f0785ffb700 1 -- 172.21.15.78:0/1908136541 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078000e040 con 0x7f078810d0f0 2023-10-10T17:52:39.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.706+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0788110ab0 con 0x7f078810d0f0 2023-10-10T17:52:39.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.706+0000 7f0785ffb700 1 -- 172.21.15.78:0/1908136541 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0780012c60 con 0x7f078810d0f0 2023-10-10T17:52:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.706+0000 7f0785ffb700 1 -- 172.21.15.78:0/1908136541 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780005680 con 0x7f078810d0f0 2023-10-10T17:52:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.706+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0788110920 con 0x7f078810d0f0 2023-10-10T17:52:39.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.707+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07680052e0 con 0x7f078810d0f0 2023-10-10T17:52:39.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.709+0000 7f0785ffb700 1 -- 172.21.15.78:0/1908136541 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f07800127a0 con 0x7f078810d0f0 2023-10-10T17:52:39.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.710+0000 7f0785ffb700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f07740a1380 0x7f07740a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:39.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.710+0000 7f0785ffb700 1 -- 172.21.15.78:0/1908136541 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f07800cea70 con 0x7f078810d0f0 2023-10-10T17:52:39.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.710+0000 7f0787fff700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f07740a1380 0x7f07740a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.710+0000 7f0787fff700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f07740a1380 0x7f07740a3830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0778009710 tx=0x7f0778006c60 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:39.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:39.713+0000 7f0785ffb700 1 -- 172.21.15.78:0/1908136541 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f0780097bc0 con 0x7f078810d0f0 2023-10-10T17:52:40.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.062+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0768005c80 con 0x7f078810d0f0 2023-10-10T17:52:40.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.067+0000 7f0785ffb700 1 -- 172.21.15.78:0/1908136541 <== mon.2 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+25283 (secure 0 0 0) 0x7f078009b1e0 con 0x7f078810d0f0 2023-10-10T17:52:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:52:40.074 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":101,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:52:06.802798+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1054117069":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/1411676003":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/540877351":"2023-10-11T17:42:35.200450+0000","172.21.15.78:6801/321866266":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/951456648":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/321866266":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]},{"pool":4,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:52:40.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.078+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f07740a1380 msgr2=0x7f07740a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:40.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.078+0000 7f078e676700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f07740a1380 0x7f07740a3830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0778009710 tx=0x7f0778006c60 comp rx=0 tx=0).stop 2023-10-10T17:52:40.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.078+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f078810d0f0 msgr2=0x7f07881b9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:40.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.078+0000 7f078e676700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f078810d0f0 0x7f07881b9be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0780009fd0 tx=0x7f07800123f0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.089+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 shutdown_connections 2023-10-10T17:52:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.089+0000 7f078e676700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f07740a1380 0x7f07740a3830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.089+0000 7f078e676700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f078810d0f0 0x7f07881b9be0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.089+0000 7f078e676700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0788107af0 0x7f07881a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.089+0000 7f078e676700 1 --2- 172.21.15.78:0/1908136541 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0788108400 0x7f07881a7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.089+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 >> 172.21.15.78:0/1908136541 conn(0x7f078806c550 msgr2=0x7f078810abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:40.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.097+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 shutdown_connections 2023-10-10T17:52:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.101+0000 7f078e676700 1 -- 172.21.15.78:0/1908136541 wait complete. 2023-10-10T17:52:40.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:39 smithi078 ceph-mon[99494]: from='client.17523 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:52:40.128 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 101, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:52:40.129 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:52:40.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:39 smithi190 ceph-mon[101220]: from='client.17523 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:52:40.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:39 smithi157 ceph-mon[100896]: from='client.17523 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:52:40.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.347+0000 7fd115dd0700 1 -- 172.21.15.78:0/2721182056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 msgr2=0x7fd1100ff550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:40.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.347+0000 7fd115dd0700 1 --2- 172.21.15.78:0/2721182056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 0x7fd1100ff550 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd10401a9a0 tx=0x7fd10401acb0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.348+0000 7fd115dd0700 1 -- 172.21.15.78:0/2721182056 shutdown_connections 2023-10-10T17:52:40.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.348+0000 7fd115dd0700 1 --2- 172.21.15.78:0/2721182056 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd110104680 0x7fd110104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.348+0000 7fd115dd0700 1 --2- 172.21.15.78:0/2721182056 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 0x7fd1100ff550 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.348+0000 7fd115dd0700 1 --2- 172.21.15.78:0/2721182056 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1100ffa90 0x7fd110104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.348+0000 7fd115dd0700 1 -- 172.21.15.78:0/2721182056 >> 172.21.15.78:0/2721182056 conn(0x7fd1100770b0 msgr2=0x7fd1100774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:40.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.348+0000 7fd115dd0700 1 -- 172.21.15.78:0/2721182056 shutdown_connections 2023-10-10T17:52:40.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.348+0000 7fd115dd0700 1 -- 172.21.15.78:0/2721182056 wait complete. 2023-10-10T17:52:40.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 Processor -- start 2023-10-10T17:52:40.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 -- start start 2023-10-10T17:52:40.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 0x7fd11010f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:40.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1100ffa90 0x7fd11010f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:40.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd110104680 0x7fd110199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019a540 con 0x7fd1100ff180 2023-10-10T17:52:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019a6b0 con 0x7fd1100ffa90 2023-10-10T17:52:40.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd115dd0700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019a870 con 0x7fd110104680 2023-10-10T17:52:40.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 0x7fd11010f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:40.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 0x7fd11010f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49186/0 (socket says 172.21.15.78:49186) 2023-10-10T17:52:40.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.349+0000 7fd10f7fe700 1 -- 172.21.15.78:0/4050123445 learned_addr learned my addr 172.21.15.78:0/4050123445 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10effd700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1100ffa90 0x7fd11010f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10effd700 1 -- 172.21.15.78:0/4050123445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd110104680 msgr2=0x7fd110199f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:52:40.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10effd700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd110104680 0x7fd110199f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10effd700 1 -- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 msgr2=0x7fd11010f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:40.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10effd700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 0x7fd11010f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10effd700 1 -- 172.21.15.78:0/4050123445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd100009520 con 0x7fd1100ffa90 2023-10-10T17:52:40.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10effd700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1100ffa90 0x7fd11010f8a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9f0 tx=0x7fd10000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:40.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.350+0000 7fd10cff9700 1 -- 172.21.15.78:0/4050123445 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10000cb70 con 0x7fd1100ffa90 2023-10-10T17:52:40.362 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.351+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10401a600 con 0x7fd1100ffa90 2023-10-10T17:52:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.351+0000 7fd10cff9700 1 -- 172.21.15.78:0/4050123445 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd1000044e0 con 0x7fd1100ffa90 2023-10-10T17:52:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.351+0000 7fd10cff9700 1 -- 172.21.15.78:0/4050123445 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100003d00 con 0x7fd1100ffa90 2023-10-10T17:52:40.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.351+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd110107ef0 con 0x7fd1100ffa90 2023-10-10T17:52:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.353+0000 7fd10cff9700 1 -- 172.21.15.78:0/4050123445 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fd100004020 con 0x7fd1100ffa90 2023-10-10T17:52:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.354+0000 7fd10cff9700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0f80a1340 0x7fd0f80a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:40.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.354+0000 7fd10cff9700 1 -- 172.21.15.78:0/4050123445 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fd100014070 con 0x7fd1100ffa90 2023-10-10T17:52:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.355+0000 7fd10f7fe700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0f80a1340 0x7fd0f80a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.355+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd110107d60 con 0x7fd1100ffa90 2023-10-10T17:52:40.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.355+0000 7fd10f7fe700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0f80a1340 0x7fd0f80a37f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd104006010 tx=0x7fd10402a040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:40.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.366+0000 7fd10cff9700 1 -- 172.21.15.78:0/4050123445 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fd100096430 con 0x7fd1100ffa90 2023-10-10T17:52:40.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.700+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd110108080 con 0x7fd1100ffa90 2023-10-10T17:52:40.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.701+0000 7fd10cff9700 1 -- 172.21.15.78:0/4050123445 <== mon.2 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd100003e60 con 0x7fd1100ffa90 2023-10-10T17:52:40.705 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:52:40.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.711+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0f80a1340 msgr2=0x7fd0f80a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:40.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.711+0000 7fd115dd0700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0f80a1340 0x7fd0f80a37f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd104006010 tx=0x7fd10402a040 comp rx=0 tx=0).stop 2023-10-10T17:52:40.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.711+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1100ffa90 msgr2=0x7fd11010f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:40.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.711+0000 7fd115dd0700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1100ffa90 0x7fd11010f8a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9f0 tx=0x7fd10000ed00 comp rx=0 tx=0).stop 2023-10-10T17:52:40.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.716+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 shutdown_connections 2023-10-10T17:52:40.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.716+0000 7fd115dd0700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fd110104680 0x7fd110199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.716+0000 7fd115dd0700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fd0f80a1340 0x7fd0f80a37f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.716+0000 7fd115dd0700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1100ff180 0x7fd11010f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.716+0000 7fd115dd0700 1 --2- 172.21.15.78:0/4050123445 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fd1100ffa90 0x7fd11010f8a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:40.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.716+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 >> 172.21.15.78:0/4050123445 conn(0x7fd1100770b0 msgr2=0x7fd1101021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:40.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.724+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 shutdown_connections 2023-10-10T17:52:40.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.727+0000 7fd115dd0700 1 -- 172.21.15.78:0/4050123445 wait complete. 2023-10-10T17:52:40.750 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:52:40.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.997+0000 7f3106a82700 1 -- 172.21.15.78:0/658842943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3100100a50 msgr2=0x7f3100100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:40.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.997+0000 7f3106a82700 1 --2- 172.21.15.78:0/658842943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3100100a50 0x7f3100100ec0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f30f4009920 tx=0x7f30f4009c30 comp rx=0 tx=0).stop 2023-10-10T17:52:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.998+0000 7f3106a82700 1 -- 172.21.15.78:0/658842943 shutdown_connections 2023-10-10T17:52:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.998+0000 7f3106a82700 1 --2- 172.21.15.78:0/658842943 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 0x7f3100105650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.998+0000 7f3106a82700 1 --2- 172.21.15.78:0/658842943 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3100101400 0x7f310010d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.998+0000 7f3106a82700 1 --2- 172.21.15.78:0/658842943 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3100100a50 0x7f3100100ec0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:40.998+0000 7f3106a82700 1 -- 172.21.15.78:0/658842943 >> 172.21.15.78:0/658842943 conn(0x7f31000775e0 msgr2=0x7f31000779e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:41.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.002+0000 7f3106a82700 1 -- 172.21.15.78:0/658842943 shutdown_connections 2023-10-10T17:52:41.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.004+0000 7f3106a82700 1 -- 172.21.15.78:0/658842943 wait complete. 2023-10-10T17:52:41.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.005+0000 7f3106a82700 1 Processor -- start 2023-10-10T17:52:41.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.005+0000 7f3106a82700 1 -- start start 2023-10-10T17:52:41.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f3106a82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3100100a50 0x7f310010d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f3106a82700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3100101400 0x7f310010df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f3106a82700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 0x7f310010e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f3106a82700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310010c8e0 con 0x7f3100100a50 2023-10-10T17:52:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f3106a82700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310010ca50 con 0x7f3100101400 2023-10-10T17:52:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f3106a82700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310010cc10 con 0x7f3100105280 2023-10-10T17:52:41.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 0x7f310010e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 0x7f310010e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40730/0 (socket says 172.21.15.78:40730) 2023-10-10T17:52:41.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 -- 172.21.15.78:0/1935403776 learned_addr learned my addr 172.21.15.78:0/1935403776 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:41.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f30fffff700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3100101400 0x7f310010df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:41.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 -- 172.21.15.78:0/1935403776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3100101400 msgr2=0x7f310010df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:41.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3100101400 0x7f310010df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 -- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3100100a50 msgr2=0x7f310010d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:41.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3100100a50 0x7f310010d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f310501f700 1 -- 172.21.15.78:0/1935403776 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f4009520 con 0x7f3100105280 2023-10-10T17:52:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.006+0000 7f30fffff700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3100101400 0x7f310010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-10T17:52:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.007+0000 7f310501f700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 0x7f310010e450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f30f800c110 tx=0x7f30f800c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:41.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.007+0000 7f30fdffb700 1 -- 172.21.15.78:0/1935403776 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8017070 con 0x7f3100105280 2023-10-10T17:52:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.007+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310006aff0 con 0x7f3100105280 2023-10-10T17:52:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.007+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310006ae60 con 0x7f3100105280 2023-10-10T17:52:41.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.007+0000 7f30fdffb700 1 -- 172.21.15.78:0/1935403776 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f30f800e050 con 0x7f3100105280 2023-10-10T17:52:41.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.007+0000 7f30fdffb700 1 -- 172.21.15.78:0/1935403776 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8007a00 con 0x7f3100105280 2023-10-10T17:52:41.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.010+0000 7f30fdffb700 1 -- 172.21.15.78:0/1935403776 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f30f8007c40 con 0x7f3100105280 2023-10-10T17:52:41.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.011+0000 7f30fdffb700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f30f00a1400 0x7f30f00a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.018 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.011+0000 7f30fdffb700 1 -- 172.21.15.78:0/1935403776 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f30f80ccbd0 con 0x7f3100105280 2023-10-10T17:52:41.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.014+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e4005310 con 0x7f3100105280 2023-10-10T17:52:41.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.014+0000 7f310481e700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f30f00a1400 0x7f30f00a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:41.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.014+0000 7f310481e700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f30f00a1400 0x7f30f00a38b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005950 tx=0x7f30ec0058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:41.020 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.019+0000 7f30fdffb700 1 -- 172.21.15.78:0/1935403776 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f30f8095bc0 con 0x7f3100105280 2023-10-10T17:52:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:40 smithi078 ceph-mon[99494]: from='client.17538 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:52:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:40 smithi078 ceph-mon[99494]: pgmap v314: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:41.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:40 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1908136541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:52:41.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:40 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4050123445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:52:41.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:40 smithi190 ceph-mon[101220]: from='client.17538 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:52:41.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:40 smithi190 ceph-mon[101220]: pgmap v314: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:41.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:40 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1908136541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:52:41.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:40 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4050123445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:52:41.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:40 smithi157 ceph-mon[100896]: from='client.17538 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-10T17:52:41.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:40 smithi157 ceph-mon[100896]: pgmap v314: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:41.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:40 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1908136541' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:52:41.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:40 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4050123445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:52:41.363 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:52:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.359+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f30e4005f60 con 0x7f3100105280 2023-10-10T17:52:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.359+0000 7f30fdffb700 1 -- 172.21.15.78:0/1935403776 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+11 (secure 0 0 0) 0x7f30f80991e0 con 0x7f3100105280 2023-10-10T17:52:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.363+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f30f00a1400 msgr2=0x7f30f00a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.363+0000 7f3106a82700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f30f00a1400 0x7f30f00a38b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005950 tx=0x7f30ec0058e0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.363+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 msgr2=0x7f310010e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.363+0000 7f3106a82700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 0x7f310010e450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f30f800c110 tx=0x7f30f800c4d0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.364+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 shutdown_connections 2023-10-10T17:52:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.364+0000 7f3106a82700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3100105280 0x7f310010e450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.364+0000 7f3106a82700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f30f00a1400 0x7f30f00a38b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.364+0000 7f3106a82700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3100100a50 0x7f310010d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.364+0000 7f3106a82700 1 --2- 172.21.15.78:0/1935403776 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3100101400 0x7f310010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.364+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 >> 172.21.15.78:0/1935403776 conn(0x7f31000775e0 msgr2=0x7f31000ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.367+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 shutdown_connections 2023-10-10T17:52:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.371+0000 7f3106a82700 1 -- 172.21.15.78:0/1935403776 wait complete. 2023-10-10T17:52:41.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:52:41.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.638+0000 7f9128d48700 1 -- 172.21.15.78:0/2046941176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 msgr2=0x7f912410c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:41.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.638+0000 7f9128d48700 1 --2- 172.21.15.78:0/2046941176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912410c060 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f911c009f90 tx=0x7f911c00b330 comp rx=0 tx=0).stop 2023-10-10T17:52:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.638+0000 7f9128d48700 1 -- 172.21.15.78:0/2046941176 shutdown_connections 2023-10-10T17:52:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.638+0000 7f9128d48700 1 --2- 172.21.15.78:0/2046941176 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912410c060 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.638+0000 7f9128d48700 1 --2- 172.21.15.78:0/2046941176 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91240ff450 0x7f91240ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.638+0000 7f9128d48700 1 --2- 172.21.15.78:0/2046941176 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91240ffdf0 0x7f9124103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.638+0000 7f9128d48700 1 -- 172.21.15.78:0/2046941176 >> 172.21.15.78:0/2046941176 conn(0x7f91240770b0 msgr2=0x7f91240774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.639+0000 7f9128d48700 1 -- 172.21.15.78:0/2046941176 shutdown_connections 2023-10-10T17:52:41.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.639+0000 7f9128d48700 1 -- 172.21.15.78:0/2046941176 wait complete. 2023-10-10T17:52:41.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.640+0000 7f9128d48700 1 Processor -- start 2023-10-10T17:52:41.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.640+0000 7f9128d48700 1 -- start start 2023-10-10T17:52:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9128d48700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91240ff450 0x7f9124198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9128d48700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91240ffdf0 0x7f9124198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9128d48700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9128d48700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912419d5e0 con 0x7f91240ff450 2023-10-10T17:52:41.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9128d48700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912419d750 con 0x7f91240ffdf0 2023-10-10T17:52:41.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9128d48700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912419d910 con 0x7f9124104490 2023-10-10T17:52:41.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9122d9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:41.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9122d9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912419cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40742/0 (socket says 172.21.15.78:40742) 2023-10-10T17:52:41.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.641+0000 7f9122d9d700 1 -- 172.21.15.78:0/584391042 learned_addr learned my addr 172.21.15.78:0/584391042 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:41.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.644+0000 7f9122d9d700 1 -- 172.21.15.78:0/584391042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91240ffdf0 msgr2=0x7f9124198b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:52:41.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.644+0000 7f9122d9d700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91240ffdf0 0x7f9124198b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.644+0000 7f9122d9d700 1 -- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91240ff450 msgr2=0x7f9124198600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:52:41.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.644+0000 7f9122d9d700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91240ff450 0x7f9124198600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:41.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.644+0000 7f9122d9d700 1 -- 172.21.15.78:0/584391042 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c009c40 con 0x7f9124104490 2023-10-10T17:52:41.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.651+0000 7f9122d9d700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912419cfa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f911c004c60 tx=0x7f911c004d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:41.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.651+0000 7f91137fe700 1 -- 172.21.15.78:0/584391042 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c00e040 con 0x7f9124104490 2023-10-10T17:52:41.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.651+0000 7f91137fe700 1 -- 172.21.15.78:0/584391042 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f911c00be00 con 0x7f9124104490 2023-10-10T17:52:41.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.651+0000 7f91137fe700 1 -- 172.21.15.78:0/584391042 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c01b680 con 0x7f9124104490 2023-10-10T17:52:41.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.653+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912410e1e0 con 0x7f9124104490 2023-10-10T17:52:41.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.653+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912410e050 con 0x7f9124104490 2023-10-10T17:52:41.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.654+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912410e370 con 0x7f9124104490 2023-10-10T17:52:41.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.657+0000 7f91137fe700 1 -- 172.21.15.78:0/584391042 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f911c01b7e0 con 0x7f9124104490 2023-10-10T17:52:41.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.658+0000 7f91137fe700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f910c0a1300 0x7f910c0a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:41.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.661+0000 7f912259c700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f910c0a1300 0x7f910c0a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:41.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.662+0000 7f91137fe700 1 -- 172.21.15.78:0/584391042 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f911c0cddb0 con 0x7f9124104490 2023-10-10T17:52:41.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.662+0000 7f912259c700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f910c0a1300 0x7f910c0a37b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9114005fd0 tx=0x7f9114009310 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:41.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:41.671+0000 7f91137fe700 1 -- 172.21.15.78:0/584391042 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f911c09a900 con 0x7f9124104490 2023-10-10T17:52:42.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.021+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9124067b00 con 0x7f9124104490 2023-10-10T17:52:42.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.021+0000 7f91137fe700 1 -- 172.21.15.78:0/584391042 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+11 (secure 0 0 0) 0x7f911c09a490 con 0x7f9124104490 2023-10-10T17:52:42.025 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:52:42.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f910c0a1300 msgr2=0x7f910c0a37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f910c0a1300 0x7f910c0a37b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f9114005fd0 tx=0x7f9114009310 comp rx=0 tx=0).stop 2023-10-10T17:52:42.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 msgr2=0x7f912419cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912419cfa0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f911c004c60 tx=0x7f911c004d40 comp rx=0 tx=0).stop 2023-10-10T17:52:42.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 shutdown_connections 2023-10-10T17:52:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9124104490 0x7f912419cfa0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f910c0a1300 0x7f910c0a37b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f91240ff450 0x7f9124198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 --2- 172.21.15.78:0/584391042 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f91240ffdf0 0x7f9124198b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.031+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 >> 172.21.15.78:0/584391042 conn(0x7f91240770b0 msgr2=0x7f91240fcbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:42.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.036+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 shutdown_connections 2023-10-10T17:52:42.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.040+0000 7f9128d48700 1 -- 172.21.15.78:0/584391042 wait complete. 2023-10-10T17:52:42.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:52:42.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:41 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1935403776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:52:42.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:41 smithi190 ceph-mon[101220]: pgmap v315: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:52:42.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:41 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1935403776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:52:42.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:41 smithi157 ceph-mon[100896]: pgmap v315: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:52:42.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.307+0000 7f98a0939700 1 -- 172.21.15.78:0/2837096592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c10c980 msgr2=0x7f989c10cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.310 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.307+0000 7f98a0939700 1 --2- 172.21.15.78:0/2837096592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c10c980 0x7f989c10cd50 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f988c01a9a0 tx=0x7f988c01acb0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.308+0000 7f98a0939700 1 -- 172.21.15.78:0/2837096592 shutdown_connections 2023-10-10T17:52:42.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.308+0000 7f98a0939700 1 --2- 172.21.15.78:0/2837096592 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c1074b0 0x7f989c113740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.308+0000 7f98a0939700 1 --2- 172.21.15.78:0/2837096592 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c10c980 0x7f989c10cd50 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.308+0000 7f98a0939700 1 --2- 172.21.15.78:0/2837096592 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c106b00 0x7f989c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.308+0000 7f98a0939700 1 -- 172.21.15.78:0/2837096592 >> 172.21.15.78:0/2837096592 conn(0x7f989c073dd0 msgr2=0x7f989c0761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:42.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.321+0000 7f98a0939700 1 -- 172.21.15.78:0/2837096592 shutdown_connections 2023-10-10T17:52:42.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.323+0000 7f98a0939700 1 -- 172.21.15.78:0/2837096592 wait complete. 2023-10-10T17:52:42.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 Processor -- start 2023-10-10T17:52:42.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 -- start start 2023-10-10T17:52:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c106b00 0x7f989c10eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c1074b0 0x7f989c10f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10c980 0x7f989c1a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c1a0550 con 0x7f989c106b00 2023-10-10T17:52:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c1a0690 con 0x7f989c1074b0 2023-10-10T17:52:42.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.324+0000 7f98a0939700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c1a07d0 con 0x7f989c10c980 2023-10-10T17:52:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c1074b0 0x7f989c10f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f989ad9d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10c980 0x7f989c1a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c1074b0 0x7f989c10f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50944/0 (socket says 172.21.15.78:50944) 2023-10-10T17:52:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 -- 172.21.15.78:0/4263571271 learned_addr learned my addr 172.21.15.78:0/4263571271 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f989a59c700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c106b00 0x7f989c10eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 -- 172.21.15.78:0/4263571271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10c980 msgr2=0x7f989c1a0010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10c980 0x7f989c1a0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 -- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c106b00 msgr2=0x7f989c10eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c106b00 0x7f989c10eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 -- 172.21.15.78:0/4263571271 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c01a600 con 0x7f989c1074b0 2023-10-10T17:52:42.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f989a59c700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c106b00 0x7f989c10eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:52:42.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.325+0000 7f9899d9b700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c1074b0 0x7f989c10f3e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f989000e9a0 tx=0x7f989000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:42.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.328+0000 7f988b7fe700 1 -- 172.21.15.78:0/4263571271 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890009790 con 0x7f989c1074b0 2023-10-10T17:52:42.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.328+0000 7f988b7fe700 1 -- 172.21.15.78:0/4263571271 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9890010430 con 0x7f989c1074b0 2023-10-10T17:52:42.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.329+0000 7f988b7fe700 1 -- 172.21.15.78:0/4263571271 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9890018680 con 0x7f989c1074b0 2023-10-10T17:52:42.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.329+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c06cd70 con 0x7f989c1074b0 2023-10-10T17:52:42.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.329+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c06cbe0 con 0x7f989c1074b0 2023-10-10T17:52:42.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.330+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989c06cf00 con 0x7f989c1074b0 2023-10-10T17:52:42.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.332+0000 7f988b7fe700 1 -- 172.21.15.78:0/4263571271 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f98900105a0 con 0x7f989c1074b0 2023-10-10T17:52:42.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.333+0000 7f988b7fe700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f98840a1450 0x7f98840a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.333+0000 7f988b7fe700 1 -- 172.21.15.78:0/4263571271 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f9890014070 con 0x7f989c1074b0 2023-10-10T17:52:42.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.334+0000 7f989a59c700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f98840a1450 0x7f98840a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.335+0000 7f989a59c700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f98840a1450 0x7f98840a3900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f988c01c2f0 tx=0x7f988c02a040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:42.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.336+0000 7f988b7fe700 1 -- 172.21.15.78:0/4263571271 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f98900962a0 con 0x7f989c1074b0 2023-10-10T17:52:42.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:41 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1935403776' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:52:42.385 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:41 smithi078 ceph-mon[99494]: pgmap v315: 129 pgs: 15 active+clean+snaptrim, 48 active+clean+snaptrim_wait, 66 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:52:42.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.686+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f989c1a1320 con 0x7f989c1074b0 2023-10-10T17:52:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.689+0000 7f988b7fe700 1 -- 172.21.15.78:0/4263571271 <== mon.2 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v101) v1 ==== 103+0+11 (secure 0 0 0) 0x7f98900998c0 con 0x7f989c1074b0 2023-10-10T17:52:42.693 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:52:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.699+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f98840a1450 msgr2=0x7f98840a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.699+0000 7f98a0939700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f98840a1450 0x7f98840a3900 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f988c01c2f0 tx=0x7f988c02a040 comp rx=0 tx=0).stop 2023-10-10T17:52:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.699+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c1074b0 msgr2=0x7f989c10f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.699+0000 7f98a0939700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c1074b0 0x7f989c10f3e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f989000e9a0 tx=0x7f989000ecb0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.702+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 shutdown_connections 2023-10-10T17:52:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.702+0000 7f98a0939700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f989c10c980 0x7f989c1a0010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.702+0000 7f98a0939700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f98840a1450 0x7f98840a3900 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.702+0000 7f98a0939700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f989c106b00 0x7f989c10eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.702+0000 7f98a0939700 1 --2- 172.21.15.78:0/4263571271 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f989c1074b0 0x7f989c10f3e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.702+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 >> 172.21.15.78:0/4263571271 conn(0x7f989c073dd0 msgr2=0x7f989c075410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.706+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 shutdown_connections 2023-10-10T17:52:42.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.708+0000 7f98a0939700 1 -- 172.21.15.78:0/4263571271 wait complete. 2023-10-10T17:52:42.739 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2023-10-10T17:52:42.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.969+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/478225060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 msgr2=0x7f9b740fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.969+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/478225060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 0x7f9b740fe940 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400c310 tx=0x7f9b6400c620 comp rx=0 tx=0).stop 2023-10-10T17:52:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.974+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/478225060 shutdown_connections 2023-10-10T17:52:42.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.974+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/478225060 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b740fee80 0x7f9b7410b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.974+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/478225060 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b741044e0 0x7f9b741048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.974+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/478225060 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 0x7f9b740fe940 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.974+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/478225060 >> 172.21.15.78:0/478225060 conn(0x7f9b740742a0 msgr2=0x7f9b740746a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:42.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.981+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/478225060 shutdown_connections 2023-10-10T17:52:42.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.981+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/478225060 wait complete. 2023-10-10T17:52:42.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 Processor -- start 2023-10-10T17:52:42.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 -- start start 2023-10-10T17:52:42.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 0x7f9b74114420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b740fee80 0x7f9b74114980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.985 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b741044e0 0x7f9b7410f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7410fcb0 con 0x7f9b740fee80 2023-10-10T17:52:42.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7410fe20 con 0x7f9b741044e0 2023-10-10T17:52:42.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b7a8ea700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7410ffe0 con 0x7f9b740fe4d0 2023-10-10T17:52:42.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b740fee80 0x7f9b74114980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b740fee80 0x7f9b74114980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49238/0 (socket says 172.21.15.78:49238) 2023-10-10T17:52:42.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 -- 172.21.15.78:0/3035753081 learned_addr learned my addr 172.21.15.78:0/3035753081 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:52:42.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b73fff700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 0x7f9b74114420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b78e87700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b741044e0 0x7f9b7410f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 -- 172.21.15.78:0/3035753081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 msgr2=0x7f9b74114420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 0x7f9b74114420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 -- 172.21.15.78:0/3035753081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b741044e0 msgr2=0x7f9b7410f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:42.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b741044e0 0x7f9b7410f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:42.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.983+0000 7f9b6bfff700 1 -- 172.21.15.78:0/3035753081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c009520 con 0x7f9b740fee80 2023-10-10T17:52:42.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.985+0000 7f9b6bfff700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b740fee80 0x7f9b74114980 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f9b64012bd0 tx=0x7f9b64012c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:42.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.988+0000 7f9b71ffb700 1 -- 172.21.15.78:0/3035753081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6400f070 con 0x7f9b740fee80 2023-10-10T17:52:42.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.988+0000 7f9b71ffb700 1 -- 172.21.15.78:0/3035753081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9b64005910 con 0x7f9b740fee80 2023-10-10T17:52:42.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.988+0000 7f9b71ffb700 1 -- 172.21.15.78:0/3035753081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6401dad0 con 0x7f9b740fee80 2023-10-10T17:52:42.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.988+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b6400c050 con 0x7f9b740fee80 2023-10-10T17:52:42.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.988+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7410dc50 con 0x7f9b740fee80 2023-10-10T17:52:42.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.991+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b54005310 con 0x7f9b740fee80 2023-10-10T17:52:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.992+0000 7f9b71ffb700 1 -- 172.21.15.78:0/3035753081 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f9b64005450 con 0x7f9b740fee80 2023-10-10T17:52:42.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.993+0000 7f9b71ffb700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b600a1410 0x7f9b600a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:52:42.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.993+0000 7f9b71ffb700 1 -- 172.21.15.78:0/3035753081 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f9b640ce330 con 0x7f9b740fee80 2023-10-10T17:52:42.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.994+0000 7f9b73fff700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b600a1410 0x7f9b600a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:52:42.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:42.995+0000 7f9b73fff700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b600a1410 0x7f9b600a38c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009a10 tx=0x7f9b5c009190 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:52:43.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.003+0000 7f9b71ffb700 1 -- 172.21.15.78:0/3035753081 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f9b640973f0 con 0x7f9b740fee80 2023-10-10T17:52:43.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/584391042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:52:43.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:42 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4263571271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:52:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:42 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/584391042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:52:43.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:42 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4263571271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:52:43.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:42 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/584391042' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:52:43.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:42 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4263571271' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:52:43.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.456+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f9b54005cb0 con 0x7f9b740fee80 2023-10-10T17:52:43.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.468+0000 7f9b71ffb700 1 -- 172.21.15.78:0/3035753081 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v22) v1 ==== 81+0+733772 (secure 0 0 0) 0x7f9b6409aa10 con 0x7f9b740fee80 2023-10-10T17:52:43.479 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:52:43.480 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 22, 2023-10-10T17:52:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "active_gid": 15588, 2023-10-10T17:52:43.481 INFO:teuthology.orchestra.run.smithi078.stdout: "active_name": "x", 2023-10-10T17:52:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addrs": { 2023-10-10T17:52:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2023-10-10T17:52:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.482 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2023-10-10T17:52:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2023-10-10T17:52:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2386743294 2023-10-10T17:52:43.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2023-10-10T17:52:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2023-10-10T17:52:43.484 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2386743294 2023-10-10T17:52:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:43.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "active_addr": "172.21.15.78:6801/2386743294", 2023-10-10T17:52:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "active_change": "2023-10-10T17:42:35.200791+0000", 2023-10-10T17:52:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "active_mgr_features": 4540138322906710015, 2023-10-10T17:52:43.486 INFO:teuthology.orchestra.run.smithi078.stdout: "available": true, 2023-10-10T17:52:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "standbys": [ 2023-10-10T17:52:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.487 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 25097, 2023-10-10T17:52:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "y", 2023-10-10T17:52:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540138322906710015, 2023-10-10T17:52:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2023-10-10T17:52:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2023-10-10T17:52:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:43.489 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:43.490 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.491 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.492 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2023-10-10T17:52:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.494 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.496 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.497 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.498 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:43.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.502 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:43.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:43.505 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.506 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.507 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.508 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.510 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.511 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.512 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.513 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2023-10-10T17:52:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2023-10-10T17:52:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.514 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.515 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2023-10-10T17:52:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.517 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2023-10-10T17:52:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2023-10-10T17:52:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2023-10-10T17:52:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.519 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2023-10-10T17:52:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2023-10-10T17:52:43.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2023-10-10T17:52:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.522 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.523 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2023-10-10T17:52:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.524 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2023-10-10T17:52:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2023-10-10T17:52:43.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.526 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.527 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2023-10-10T17:52:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2023-10-10T17:52:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2023-10-10T17:52:43.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2023-10-10T17:52:43.530 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2023-10-10T17:52:43.531 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2023-10-10T17:52:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2023-10-10T17:52:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.533 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.534 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2023-10-10T17:52:43.535 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2023-10-10T17:52:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2023-10-10T17:52:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.537 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.538 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2023-10-10T17:52:43.539 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.540 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2023-10-10T17:52:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2023-10-10T17:52:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.541 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.542 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2023-10-10T17:52:43.543 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2023-10-10T17:52:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:43.546 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2023-10-10T17:52:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2023-10-10T17:52:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.547 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.548 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2023-10-10T17:52:43.549 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.550 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2023-10-10T17:52:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2023-10-10T17:52:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.551 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.553 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2023-10-10T17:52:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.554 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2023-10-10T17:52:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-10T17:52:43.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2023-10-10T17:52:43.556 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2023-10-10T17:52:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.557 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2023-10-10T17:52:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2023-10-10T17:52:43.558 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-10-10T17:52:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-10T17:52:43.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2023-10-10T17:52:43.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2023-10-10T17:52:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.561 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2023-10-10T17:52:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2023-10-10T17:52:43.562 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-10-10T17:52:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.563 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2023-10-10T17:52:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2023-10-10T17:52:43.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2023-10-10T17:52:43.565 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.566 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-10-10T17:52:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-10-10T17:52:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2023-10-10T17:52:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2023-10-10T17:52:43.568 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2023-10-10T17:52:43.569 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2023-10-10T17:52:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2023-10-10T17:52:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2023-10-10T17:52:43.570 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-10-10T17:52:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2023-10-10T17:52:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2023-10-10T17:52:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.572 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2023-10-10T17:52:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.573 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2023-10-10T17:52:43.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-10T17:52:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.575 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2023-10-10T17:52:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2023-10-10T17:52:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:43.576 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2023-10-10T17:52:43.577 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2023-10-10T17:52:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-10-10T17:52:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-10T17:52:43.578 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.579 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.581 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.582 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.583 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.584 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:43.585 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.586 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.587 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.588 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:43.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.590 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.591 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.592 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.593 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.595 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.596 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.597 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.598 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2023-10-10T17:52:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2023-10-10T17:52:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:43.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-10-10T17:52:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.602 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2023-10-10T17:52:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2023-10-10T17:52:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.603 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2023-10-10T17:52:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.604 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2023-10-10T17:52:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2023-10-10T17:52:43.605 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap" 2023-10-10T17:52:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2023-10-10T17:52:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.607 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2023-10-10T17:52:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2023-10-10T17:52:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.609 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-10-10T17:52:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.610 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:43.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.613 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-10-10T17:52:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.614 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2023-10-10T17:52:43.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2023-10-10T17:52:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.616 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.617 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2023-10-10T17:52:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-10-10T17:52:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.618 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2023-10-10T17:52:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2023-10-10T17:52:43.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.620 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.621 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-10-10T17:52:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2023-10-10T17:52:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:43.624 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2023-10-10T17:52:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2023-10-10T17:52:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:43.625 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2023-10-10T17:52:43.626 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-10-10T17:52:43.627 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.628 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2023-10-10T17:52:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2023-10-10T17:52:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.629 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2023-10-10T17:52:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.630 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-10-10T17:52:43.631 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.632 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2023-10-10T17:52:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2023-10-10T17:52:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2023-10-10T17:52:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.634 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-10-10T17:52:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.636 INFO:teuthology.orchestra.run.smithi078.stdout: "alertmanager_web_password": { 2023-10-10T17:52:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alertmanager_web_password", 2023-10-10T17:52:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.638 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Alertmanager web password", 2023-10-10T17:52:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "alertmanager_web_user": { 2023-10-10T17:52:43.640 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alertmanager_web_user", 2023-10-10T17:52:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.642 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Alertmanager web user", 2023-10-10T17:52:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2023-10-10T17:52:43.644 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2023-10-10T17:52:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.645 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.646 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-10-10T17:52:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-10-10T17:52:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.647 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2023-10-10T17:52:43.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2023-10-10T17:52:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.649 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.650 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2023-10-10T17:52:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.651 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2023-10-10T17:52:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2023-10-10T17:52:43.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2023-10-10T17:52:43.653 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.654 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-10-10T17:52:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2023-10-10T17:52:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2023-10-10T17:52:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.656 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.657 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-10-10T17:52:43.658 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2023-10-10T17:52:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2023-10-10T17:52:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.660 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.661 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-10-10T17:52:43.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.663 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2023-10-10T17:52:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2023-10-10T17:52:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.665 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-10-10T17:52:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2023-10-10T17:52:43.667 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2023-10-10T17:52:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-10-10T17:52:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.670 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2023-10-10T17:52:43.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2023-10-10T17:52:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2023-10-10T17:52:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.673 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2023-10-10T17:52:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2023-10-10T17:52:43.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2023-10-10T17:52:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.676 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-10-10T17:52:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.677 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2023-10-10T17:52:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.678 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2023-10-10T17:52:43.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2023-10-10T17:52:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-10-10T17:52:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.681 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2023-10-10T17:52:43.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2023-10-10T17:52:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-10-10T17:52:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2023-10-10T17:52:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.686 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2023-10-10T17:52:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2023-10-10T17:52:43.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-10-10T17:52:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2023-10-10T17:52:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2023-10-10T17:52:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2023-10-10T17:52:43.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-10-10T17:52:43.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.693 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2023-10-10T17:52:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2023-10-10T17:52:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2023-10-10T17:52:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.695 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-10-10T17:52:43.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2023-10-10T17:52:43.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2023-10-10T17:52:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2023-10-10T17:52:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.699 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-10-10T17:52:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2023-10-10T17:52:43.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.702 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2023-10-10T17:52:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2023-10-10T17:52:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-10-10T17:52:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.704 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2023-10-10T17:52:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2023-10-10T17:52:43.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2023-10-10T17:52:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-10-10T17:52:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.708 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.709 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2023-10-10T17:52:43.710 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2023-10-10T17:52:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-10-10T17:52:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2023-10-10T17:52:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2023-10-10T17:52:43.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-10-10T17:52:43.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.716 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2023-10-10T17:52:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2023-10-10T17:52:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2023-10-10T17:52:43.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-10-10T17:52:43.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2023-10-10T17:52:43.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2023-10-10T17:52:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2023-10-10T17:52:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.722 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.723 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2023-10-10T17:52:43.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.725 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2023-10-10T17:52:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2023-10-10T17:52:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-10-10T17:52:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2023-10-10T17:52:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2023-10-10T17:52:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2023-10-10T17:52:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-10-10T17:52:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2023-10-10T17:52:43.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2023-10-10T17:52:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.734 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2023-10-10T17:52:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.735 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-10-10T17:52:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2023-10-10T17:52:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2023-10-10T17:52:43.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.738 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2023-10-10T17:52:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.739 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2023-10-10T17:52:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2023-10-10T17:52:43.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2023-10-10T17:52:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.742 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2023-10-10T17:52:43.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2023-10-10T17:52:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2023-10-10T17:52:43.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.747 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.747 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.747 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:43.747 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.748 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2023-10-10T17:52:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.749 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2023-10-10T17:52:43.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2023-10-10T17:52:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.751 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.752 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2023-10-10T17:52:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.753 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:43.754 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:43.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.758 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.759 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.760 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.761 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.762 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.763 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2023-10-10T17:52:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2023-10-10T17:52:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-10-10T17:52:43.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.769 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-10-10T17:52:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:43.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.773 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.774 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.775 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.776 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:43.778 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.780 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.782 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2023-10-10T17:52:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-10-10T17:52:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.783 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.784 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-10-10T17:52:43.785 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-10-10T17:52:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-10-10T17:52:43.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2023-10-10T17:52:43.788 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.789 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-10-10T17:52:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2023-10-10T17:52:43.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2023-10-10T17:52:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2023-10-10T17:52:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2023-10-10T17:52:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2023-10-10T17:52:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.796 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.797 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-10-10T17:52:43.798 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2023-10-10T17:52:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2023-10-10T17:52:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.800 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.801 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.802 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2023-10-10T17:52:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2023-10-10T17:52:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2023-10-10T17:52:43.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.805 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2023-10-10T17:52:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.806 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2023-10-10T17:52:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2023-10-10T17:52:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2023-10-10T17:52:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2023-10-10T17:52:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2023-10-10T17:52:43.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-10-10T17:52:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-10-10T17:52:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.812 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_web_password": { 2023-10-10T17:52:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_web_password", 2023-10-10T17:52:43.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.814 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus web password", 2023-10-10T17:52:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.816 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_web_user": { 2023-10-10T17:52:43.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_web_user", 2023-10-10T17:52:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus web user", 2023-10-10T17:52:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.821 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2023-10-10T17:52:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2023-10-10T17:52:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-10-10T17:52:43.824 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.825 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2023-10-10T17:52:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2023-10-10T17:52:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.826 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.827 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-10-10T17:52:43.828 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2023-10-10T17:52:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2023-10-10T17:52:43.830 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.831 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-10-10T17:52:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.833 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2023-10-10T17:52:43.834 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2023-10-10T17:52:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-10-10T17:52:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.838 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2023-10-10T17:52:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2023-10-10T17:52:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.839 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.840 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-10-10T17:52:43.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2023-10-10T17:52:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2023-10-10T17:52:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.843 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2023-10-10T17:52:43.844 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.845 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2023-10-10T17:52:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2023-10-10T17:52:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2023-10-10T17:52:43.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.848 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.849 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-10-10T17:52:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.850 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2023-10-10T17:52:43.851 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2023-10-10T17:52:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.853 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-10-10T17:52:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.855 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2023-10-10T17:52:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2023-10-10T17:52:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.857 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.858 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-10-10T17:52:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2023-10-10T17:52:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2023-10-10T17:52:43.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.861 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2023-10-10T17:52:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.863 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2023-10-10T17:52:43.864 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2023-10-10T17:52:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.865 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.866 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-10-10T17:52:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.870 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2023-10-10T17:52:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2023-10-10T17:52:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.871 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.872 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.873 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-10-10T17:52:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.874 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2023-10-10T17:52:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:43.876 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:43.877 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.879 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.880 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.881 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.882 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.885 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.886 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.888 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:43.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:43.890 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:43.891 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:43.892 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.893 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.894 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.898 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2023-10-10T17:52:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2023-10-10T17:52:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2023-10-10T17:52:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2023-10-10T17:52:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2023-10-10T17:52:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2023-10-10T17:52:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2023-10-10T17:52:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2023-10-10T17:52:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2023-10-10T17:52:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-10-10T17:52:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-10-10T17:52:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.912 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.913 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2023-10-10T17:52:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2023-10-10T17:52:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.914 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.915 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.916 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-10-10T17:52:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-10-10T17:52:43.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.919 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.920 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2023-10-10T17:52:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2023-10-10T17:52:43.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.924 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.925 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-10-10T17:52:43.926 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-10-10T17:52:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.928 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.930 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2023-10-10T17:52:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2023-10-10T17:52:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.931 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.932 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.933 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-10-10T17:52:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-10-10T17:52:43.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.936 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.937 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.938 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-10-10T17:52:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-10-10T17:52:43.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.940 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.941 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.942 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-10-10T17:52:43.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-10-10T17:52:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-10-10T17:52:43.947 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-10-10T17:52:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.949 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.950 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.951 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2023-10-10T17:52:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-10-10T17:52:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.952 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.953 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.955 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2023-10-10T17:52:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-10-10T17:52:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2023-10-10T17:52:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-10-10T17:52:43.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.962 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-10-10T17:52:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-10-10T17:52:43.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2023-10-10T17:52:43.968 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2023-10-10T17:52:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.970 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.971 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-10-10T17:52:43.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-10-10T17:52:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.973 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.974 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.976 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2023-10-10T17:52:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2023-10-10T17:52:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.979 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.980 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2023-10-10T17:52:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2023-10-10T17:52:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:43.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.983 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-10-10T17:52:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-10-10T17:52:43.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.986 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.987 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-10-10T17:52:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-10-10T17:52:43.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.990 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.991 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.992 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-10-10T17:52:43.993 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-10-10T17:52:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.994 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.995 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:43.996 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2023-10-10T17:52:43.997 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-10-10T17:52:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:43.998 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:43.999 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.001 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2023-10-10T17:52:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2023-10-10T17:52:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-10-10T17:52:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-10-10T17:52:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-10-10T17:52:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-10-10T17:52:44.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.012 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-10-10T17:52:44.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-10-10T17:52:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.015 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.016 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.017 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-10-10T17:52:44.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-10-10T17:52:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.019 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.020 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.022 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-10-10T17:52:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-10-10T17:52:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.023 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.024 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.025 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.026 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-10-10T17:52:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-10-10T17:52:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-10-10T17:52:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-10-10T17:52:44.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.033 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-10-10T17:52:44.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-10-10T17:52:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.036 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2023-10-10T17:52:44.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2023-10-10T17:52:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.043 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-10-10T17:52:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-10-10T17:52:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.044 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-10-10T17:52:44.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.046 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-10-10T17:52:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-10-10T17:52:44.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.049 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.050 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.051 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-10-10T17:52:44.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-10-10T17:52:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2023-10-10T17:52:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.056 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2023-10-10T17:52:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-10-10T17:52:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2023-10-10T17:52:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.058 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.060 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2023-10-10T17:52:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2023-10-10T17:52:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-10-10T17:52:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-10-10T17:52:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2023-10-10T17:52:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2023-10-10T17:52:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2023-10-10T17:52:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2023-10-10T17:52:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.074 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.075 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.077 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-10-10T17:52:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-10-10T17:52:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.078 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.079 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.080 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-10-10T17:52:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-10-10T17:52:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.082 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:44.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.084 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-10-10T17:52:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-10-10T17:52:44.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.087 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.088 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.089 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2023-10-10T17:52:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2023-10-10T17:52:44.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.091 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.093 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2023-10-10T17:52:44.094 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2023-10-10T17:52:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.096 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2023-10-10T17:52:44.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2023-10-10T17:52:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.100 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2023-10-10T17:52:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.102 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2023-10-10T17:52:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2023-10-10T17:52:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.103 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2023-10-10T17:52:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.104 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.105 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.106 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2023-10-10T17:52:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2023-10-10T17:52:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.108 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.109 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.111 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.112 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.113 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.114 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.115 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.116 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.118 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.119 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.120 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.123 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.124 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.125 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.127 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.130 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.131 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2023-10-10T17:52:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2023-10-10T17:52:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.132 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.133 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2023-10-10T17:52:44.134 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2023-10-10T17:52:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2023-10-10T17:52:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.136 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.138 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:44.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2023-10-10T17:52:44.141 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:44.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.145 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2023-10-10T17:52:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.146 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2023-10-10T17:52:44.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2023-10-10T17:52:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.150 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.152 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2023-10-10T17:52:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2023-10-10T17:52:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.154 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2023-10-10T17:52:44.154 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.154 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.155 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.155 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.155 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.155 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.156 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.156 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2023-10-10T17:52:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2023-10-10T17:52:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2023-10-10T17:52:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.159 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2023-10-10T17:52:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.160 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.161 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2023-10-10T17:52:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2023-10-10T17:52:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.162 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2023-10-10T17:52:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2023-10-10T17:52:44.163 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2023-10-10T17:52:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.164 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.165 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2023-10-10T17:52:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2023-10-10T17:52:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.166 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.167 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.168 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2023-10-10T17:52:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.171 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2023-10-10T17:52:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2023-10-10T17:52:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2023-10-10T17:52:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.178 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.179 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.180 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.181 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.182 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.183 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.184 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.185 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.188 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.189 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.190 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.192 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.194 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.196 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2023-10-10T17:52:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2023-10-10T17:52:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.197 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2023-10-10T17:52:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.198 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-10-10T17:52:44.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2023-10-10T17:52:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2023-10-10T17:52:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.201 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2023-10-10T17:52:44.202 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2023-10-10T17:52:44.203 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2023-10-10T17:52:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2023-10-10T17:52:44.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2023-10-10T17:52:44.206 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.207 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2023-10-10T17:52:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2023-10-10T17:52:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2023-10-10T17:52:44.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.210 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2023-10-10T17:52:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.211 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2023-10-10T17:52:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.212 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2023-10-10T17:52:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2023-10-10T17:52:44.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.214 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.215 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-10-10T17:52:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:44.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.218 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.219 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2023-10-10T17:52:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.220 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2023-10-10T17:52:44.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2023-10-10T17:52:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2023-10-10T17:52:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.223 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-10-10T17:52:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2023-10-10T17:52:44.226 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.227 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.228 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.229 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.230 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.231 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.232 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.235 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.240 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.241 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.242 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.243 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.245 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.246 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.247 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2023-10-10T17:52:44.248 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2023-10-10T17:52:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2023-10-10T17:52:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.250 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.251 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2023-10-10T17:52:44.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2023-10-10T17:52:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2023-10-10T17:52:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.256 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.257 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.258 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.259 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2023-10-10T17:52:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2023-10-10T17:52:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2023-10-10T17:52:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.262 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2023-10-10T17:52:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2023-10-10T17:52:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.263 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2023-10-10T17:52:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-10-10T17:52:44.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2023-10-10T17:52:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2023-10-10T17:52:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.267 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2023-10-10T17:52:44.268 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-10-10T17:52:44.269 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2023-10-10T17:52:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2023-10-10T17:52:44.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.272 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.273 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2023-10-10T17:52:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:44.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2023-10-10T17:52:44.276 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2023-10-10T17:52:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.277 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-10-10T17:52:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.278 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.280 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.281 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.282 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.283 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.285 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.287 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.288 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.289 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.290 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.291 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.292 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.293 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.294 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.295 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.297 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.298 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2023-10-10T17:52:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2023-10-10T17:52:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.300 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.301 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2023-10-10T17:52:44.302 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2023-10-10T17:52:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2023-10-10T17:52:44.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2023-10-10T17:52:44.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.306 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2023-10-10T17:52:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2023-10-10T17:52:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2023-10-10T17:52:44.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.309 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2023-10-10T17:52:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.310 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-10-10T17:52:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.311 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2023-10-10T17:52:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2023-10-10T17:52:44.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.313 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2023-10-10T17:52:44.314 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-10-10T17:52:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.315 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2023-10-10T17:52:44.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2023-10-10T17:52:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.318 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.318 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.318 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.319 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.319 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2023-10-10T17:52:44.319 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.320 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.320 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2023-10-10T17:52:44.321 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2023-10-10T17:52:44.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.321 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.322 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2023-10-10T17:52:44.322 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.322 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.323 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.323 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-10-10T17:52:44.323 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.324 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2023-10-10T17:52:44.326 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.326 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.326 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.327 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.327 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.327 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.328 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.328 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.328 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.329 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.329 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.329 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.329 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.330 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.330 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.330 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.331 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.331 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.331 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.332 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.332 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.333 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.333 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.334 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.336 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.336 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.337 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.338 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.338 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.339 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.339 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.339 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.340 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.340 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.340 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.340 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.341 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.341 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.341 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.342 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.342 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.342 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.343 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.343 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.343 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.344 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.344 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.345 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.346 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.346 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.347 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.348 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2023-10-10T17:52:44.348 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.349 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.349 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.349 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.350 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.350 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.351 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.351 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.351 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.351 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.352 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.352 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.352 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.353 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.353 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.353 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.353 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.354 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.354 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.354 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.355 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.356 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.357 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.357 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.358 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.358 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.358 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.358 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.359 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.360 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.360 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.361 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.361 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.361 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.362 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.362 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.362 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.363 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.363 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.363 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.363 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.364 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.364 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.364 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.365 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.365 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.366 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.367 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.367 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.367 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.369 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.370 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.370 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.370 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.371 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.372 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2023-10-10T17:52:44.373 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.373 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.373 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.374 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2023-10-10T17:52:44.374 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2023-10-10T17:52:44.374 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.374 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2023-10-10T17:52:44.375 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2023-10-10T17:52:44.376 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2023-10-10T17:52:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2023-10-10T17:52:44.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.379 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.379 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2023-10-10T17:52:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.380 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-10-10T17:52:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.381 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.384 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.385 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.385 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.385 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.385 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.386 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.386 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.386 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.387 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.387 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.387 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.388 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.388 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.388 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.389 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.389 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.390 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.390 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.390 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.391 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.391 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.391 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.391 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.392 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.392 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.393 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.393 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.393 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.394 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.394 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.395 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.395 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.395 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.395 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.396 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.396 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.396 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.397 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.397 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.397 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.398 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.398 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.399 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.399 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.399 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.400 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.400 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.401 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.402 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.402 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.402 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.403 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2023-10-10T17:52:44.404 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.405 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.405 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.405 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2023-10-10T17:52:44.406 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2023-10-10T17:52:44.406 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.406 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.407 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.407 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2023-10-10T17:52:44.407 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.407 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2023-10-10T17:52:44.408 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-10-10T17:52:44.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.409 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.409 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.410 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.410 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.410 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.411 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.411 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.411 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.412 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.412 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.412 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.412 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.413 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.413 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.413 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.414 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.414 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.414 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.414 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.415 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.416 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.416 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.417 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.417 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.418 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.418 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.418 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.419 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.419 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.420 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.420 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.421 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.421 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.421 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.422 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.422 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.423 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.423 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.423 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.424 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.424 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.424 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.424 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.425 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.425 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.426 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.426 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.427 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.428 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.428 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.428 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.429 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.429 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.429 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.429 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.430 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2023-10-10T17:52:44.430 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2023-10-10T17:52:44.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.431 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.432 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.432 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.432 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.433 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2023-10-10T17:52:44.433 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-10-10T17:52:44.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.434 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.434 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2023-10-10T17:52:44.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2023-10-10T17:52:44.435 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.435 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.436 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2023-10-10T17:52:44.436 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.436 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.437 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.437 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2023-10-10T17:52:44.437 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.437 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.438 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.438 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2023-10-10T17:52:44.439 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2023-10-10T17:52:44.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.439 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.440 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2023-10-10T17:52:44.440 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.440 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.441 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.441 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2023-10-10T17:52:44.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.442 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2023-10-10T17:52:44.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2023-10-10T17:52:44.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.443 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.444 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.444 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.444 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.445 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2023-10-10T17:52:44.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.446 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.447 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2023-10-10T17:52:44.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2023-10-10T17:52:44.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.448 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.448 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2023-10-10T17:52:44.448 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.449 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.449 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.449 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2023-10-10T17:52:44.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-10-10T17:52:44.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.450 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2023-10-10T17:52:44.452 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.452 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.452 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.453 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.453 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.454 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.455 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.456 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.456 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.456 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.457 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.457 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.457 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.457 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.458 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.459 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.459 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.459 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.460 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.460 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.460 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.461 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.461 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.461 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.461 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.462 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.462 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.463 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.463 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.464 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.464 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.465 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.467 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.467 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.467 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.468 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.468 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.469 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.469 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.470 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.470 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.470 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.471 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.471 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.471 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.471 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.472 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.472 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.473 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.474 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2023-10-10T17:52:44.474 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.475 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.475 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.475 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.476 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.477 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.477 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.477 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.478 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.478 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.478 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.478 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.479 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.479 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.479 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.480 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.480 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.480 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.481 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.481 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.482 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.482 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.482 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.483 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.483 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.483 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.483 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.484 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.484 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.484 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.485 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.485 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.486 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.486 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.486 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.487 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.487 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.487 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.487 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.488 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.488 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.488 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.489 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.489 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.489 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.490 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.490 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.490 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.490 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.491 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.492 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.492 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.492 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.493 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.493 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.494 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.494 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.494 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.495 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.495 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.495 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.497 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2023-10-10T17:52:44.497 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.497 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.497 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.498 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.498 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.499 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.499 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.500 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.501 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.502 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.502 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.502 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.502 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.503 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.504 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.505 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.506 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.507 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.507 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.507 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.508 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.509 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.510 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.510 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.510 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.511 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.511 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.511 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.511 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.512 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.512 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.512 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.513 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.513 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.514 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.515 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.515 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.516 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.516 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.517 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.518 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2023-10-10T17:52:44.520 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.520 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.520 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.520 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2023-10-10T17:52:44.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2023-10-10T17:52:44.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.521 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.522 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.522 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.523 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.523 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.523 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-10-10T17:52:44.523 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.524 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.525 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.525 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.525 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.526 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.526 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.527 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.527 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.527 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.528 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.528 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.528 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.528 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.529 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.529 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.529 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.530 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.530 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.531 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.531 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.532 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.532 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.533 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.533 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.533 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.534 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.534 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.534 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.535 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.535 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.536 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.537 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.537 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.537 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.537 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.538 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.538 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.538 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.539 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.539 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.539 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.540 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.540 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.540 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.541 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.541 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.542 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.543 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.543 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.544 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.544 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.545 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2023-10-10T17:52:44.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2023-10-10T17:52:44.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.546 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.547 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.547 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.547 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.548 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.548 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2023-10-10T17:52:44.548 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2023-10-10T17:52:44.549 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2023-10-10T17:52:44.549 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.549 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2023-10-10T17:52:44.550 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.550 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.552 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2023-10-10T17:52:44.552 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.552 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.552 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.553 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.554 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.555 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.555 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.555 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.555 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.556 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.556 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.556 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.557 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.557 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.557 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.557 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.558 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.558 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.559 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.560 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.560 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.561 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.561 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.561 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.562 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.562 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.563 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.563 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.563 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.564 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.565 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.565 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.565 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.565 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.566 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.566 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.566 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.567 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.567 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.567 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.568 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.568 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.568 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.568 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.569 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.569 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.570 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.570 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.571 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.571 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.571 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.573 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.573 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2023-10-10T17:52:44.576 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.576 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.577 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.577 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.577 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.578 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.579 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.580 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.581 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.584 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.585 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.586 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.587 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.587 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.588 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.588 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.588 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.589 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.589 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.589 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.589 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.590 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.590 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.590 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.591 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.591 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.591 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.592 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.592 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.592 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.593 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.594 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.594 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.594 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.594 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.595 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.595 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.595 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.595 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.596 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.597 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2023-10-10T17:52:44.597 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.597 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.598 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.598 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.599 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "noautoscale": { 2023-10-10T17:52:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "noautoscale", 2023-10-10T17:52:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "global autoscale flag", 2023-10-10T17:52:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-10-10T17:52:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2023-10-10T17:52:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2023-10-10T17:52:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2023-10-10T17:52:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2023-10-10T17:52:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2023-10-10T17:52:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-10-10T17:52:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2023-10-10T17:52:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2023-10-10T17:52:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2023-10-10T17:52:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2023-10-10T17:52:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2023-10-10T17:52:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2023-10-10T17:52:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.635 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.635 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.635 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.636 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.636 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.636 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.637 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.637 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.638 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.638 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.638 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.639 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.639 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.639 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.640 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.640 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.640 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.641 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.641 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.641 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.641 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.642 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.642 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.642 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.643 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.643 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.643 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.644 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.645 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.645 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.645 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.645 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.646 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.646 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.646 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.647 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.648 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.648 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.648 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.649 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.649 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.649 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.649 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.650 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.650 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.650 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.651 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.653 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.654 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.655 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2023-10-10T17:52:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2023-10-10T17:52:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.657 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2023-10-10T17:52:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.658 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2023-10-10T17:52:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.660 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.662 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2023-10-10T17:52:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2023-10-10T17:52:44.665 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2023-10-10T17:52:44.666 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2023-10-10T17:52:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.667 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.668 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.669 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2023-10-10T17:52:44.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2023-10-10T17:52:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.672 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-10-10T17:52:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-10-10T17:52:44.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.675 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.676 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.677 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.686 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.687 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.689 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.691 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2023-10-10T17:52:44.693 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2023-10-10T17:52:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.694 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.695 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.696 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2023-10-10T17:52:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-10-10T17:52:44.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2023-10-10T17:52:44.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.699 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2023-10-10T17:52:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2023-10-10T17:52:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:44.701 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2023-10-10T17:52:44.702 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.703 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.704 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2023-10-10T17:52:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.706 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-10-10T17:52:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:44.708 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2023-10-10T17:52:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.710 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-10-10T17:52:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.711 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2023-10-10T17:52:44.712 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2023-10-10T17:52:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.713 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2023-10-10T17:52:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.714 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.715 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2023-10-10T17:52:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2023-10-10T17:52:44.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2023-10-10T17:52:44.717 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.718 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2023-10-10T17:52:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2023-10-10T17:52:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.719 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.720 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2023-10-10T17:52:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2023-10-10T17:52:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2023-10-10T17:52:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2023-10-10T17:52:44.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2023-10-10T17:52:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2023-10-10T17:52:44.725 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.726 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.728 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.729 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.730 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.731 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.732 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.733 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.734 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.738 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.739 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.740 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.741 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2023-10-10T17:52:44.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2023-10-10T17:52:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.747 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.748 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2023-10-10T17:52:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2023-10-10T17:52:44.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.750 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.751 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.751 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.751 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.753 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2023-10-10T17:52:44.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2023-10-10T17:52:44.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.754 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.756 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.757 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2023-10-10T17:52:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.758 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2023-10-10T17:52:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2023-10-10T17:52:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.760 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.762 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2023-10-10T17:52:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2023-10-10T17:52:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.764 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.765 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.766 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.767 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.768 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.768 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.769 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.774 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.776 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.778 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.779 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.780 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.781 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.785 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.786 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:44.787 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.788 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.789 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:44.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.792 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.793 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2023-10-10T17:52:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.796 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.798 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.799 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.800 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.802 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.806 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.809 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.810 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.811 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.813 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.815 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2023-10-10T17:52:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.817 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "drive_group_interval": { 2023-10-10T17:52:44.818 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "drive_group_interval", 2023-10-10T17:52:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.819 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300.0", 2023-10-10T17:52:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-10-10T17:52:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.821 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.824 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.825 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.826 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.829 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.832 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.834 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.835 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.836 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.837 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.839 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2023-10-10T17:52:44.841 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2023-10-10T17:52:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2023-10-10T17:52:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.843 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-10-10T17:52:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.844 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2023-10-10T17:52:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.846 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.849 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.850 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.851 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.854 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.860 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.861 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.864 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.865 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2023-10-10T17:52:44.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2023-10-10T17:52:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.867 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.868 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2023-10-10T17:52:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2023-10-10T17:52:44.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2023-10-10T17:52:44.871 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.872 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2023-10-10T17:52:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2023-10-10T17:52:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.875 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2023-10-10T17:52:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2023-10-10T17:52:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.879 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2023-10-10T17:52:44.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2023-10-10T17:52:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.883 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2023-10-10T17:52:44.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2023-10-10T17:52:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2023-10-10T17:52:44.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2023-10-10T17:52:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2023-10-10T17:52:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2023-10-10T17:52:44.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2023-10-10T17:52:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2023-10-10T17:52:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:44.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:44.898 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2023-10-10T17:52:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.899 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.900 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2023-10-10T17:52:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2023-10-10T17:52:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.902 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.903 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2023-10-10T17:52:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2023-10-10T17:52:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.905 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.907 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.908 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2023-10-10T17:52:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2023-10-10T17:52:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.909 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.910 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2023-10-10T17:52:44.913 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2023-10-10T17:52:44.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2023-10-10T17:52:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2023-10-10T17:52:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.917 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2023-10-10T17:52:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2023-10-10T17:52:44.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.919 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.920 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2023-10-10T17:52:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.923 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.924 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.925 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.927 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.929 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.932 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.933 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.934 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.935 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.938 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2023-10-10T17:52:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.944 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.945 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.946 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.948 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.949 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.955 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.956 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.957 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.958 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.960 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.961 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2023-10-10T17:52:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.963 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.966 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.967 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.968 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.970 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.971 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.972 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.973 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.974 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.975 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.976 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.977 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:44.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:44.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.982 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.983 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2023-10-10T17:52:44.985 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:44.986 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2023-10-10T17:52:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2023-10-10T17:52:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-10-10T17:52:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:44.990 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2023-10-10T17:52:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:44.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:44.995 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:44.996 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:44.997 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:44.998 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:44.999 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.002 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.006 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.007 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.009 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.010 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.011 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2023-10-10T17:52:45.014 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2023-10-10T17:52:45.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2023-10-10T17:52:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2023-10-10T17:52:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2023-10-10T17:52:45.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2023-10-10T17:52:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-10-10T17:52:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2023-10-10T17:52:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2023-10-10T17:52:45.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.024 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-10-10T17:52:45.025 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2023-10-10T17:52:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2023-10-10T17:52:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-10-10T17:52:45.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.030 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2023-10-10T17:52:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2023-10-10T17:52:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.032 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2023-10-10T17:52:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2023-10-10T17:52:45.034 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2023-10-10T17:52:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.035 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.036 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.037 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2023-10-10T17:52:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2023-10-10T17:52:45.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.039 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.040 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2023-10-10T17:52:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2023-10-10T17:52:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.042 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-10-10T17:52:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2023-10-10T17:52:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2023-10-10T17:52:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.047 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.048 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:45.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2023-10-10T17:52:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2023-10-10T17:52:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2023-10-10T17:52:45.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2023-10-10T17:52:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2023-10-10T17:52:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2023-10-10T17:52:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2023-10-10T17:52:45.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.058 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2023-10-10T17:52:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2023-10-10T17:52:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.062 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.064 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.066 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.067 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.068 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.069 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.071 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.072 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.076 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.077 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.078 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.079 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.081 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2023-10-10T17:52:45.083 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2023-10-10T17:52:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2023-10-10T17:52:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2023-10-10T17:52:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2023-10-10T17:52:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2023-10-10T17:52:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-10-10T17:52:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2023-10-10T17:52:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2023-10-10T17:52:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2023-10-10T17:52:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2023-10-10T17:52:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2023-10-10T17:52:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2023-10-10T17:52:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2023-10-10T17:52:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2023-10-10T17:52:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-10-10T17:52:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2023-10-10T17:52:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2023-10-10T17:52:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2023-10-10T17:52:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2023-10-10T17:52:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2023-10-10T17:52:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2023-10-10T17:52:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2023-10-10T17:52:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2023-10-10T17:52:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2023-10-10T17:52:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2023-10-10T17:52:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2023-10-10T17:52:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2023-10-10T17:52:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2023-10-10T17:52:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-10-10T17:52:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "gid": 25162, 2023-10-10T17:52:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "z", 2023-10-10T17:52:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_features": 4540138322906710015, 2023-10-10T17:52:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2023-10-10T17:52:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2023-10-10T17:52:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2023-10-10T17:52:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.200 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2023-10-10T17:52:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2023-10-10T17:52:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2023-10-10T17:52:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2023-10-10T17:52:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2023-10-10T17:52:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2023-10-10T17:52:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2023-10-10T17:52:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2023-10-10T17:52:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2023-10-10T17:52:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2023-10-10T17:52:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2023-10-10T17:52:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2023-10-10T17:52:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2023-10-10T17:52:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2023-10-10T17:52:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2023-10-10T17:52:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2023-10-10T17:52:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2023-10-10T17:52:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2023-10-10T17:52:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2023-10-10T17:52:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2023-10-10T17:52:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2023-10-10T17:52:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2023-10-10T17:52:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2023-10-10T17:52:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2023-10-10T17:52:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2023-10-10T17:52:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2023-10-10T17:52:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2023-10-10T17:52:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2023-10-10T17:52:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2023-10-10T17:52:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2023-10-10T17:52:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2023-10-10T17:52:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2023-10-10T17:52:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2023-10-10T17:52:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2023-10-10T17:52:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-10T17:52:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2023-10-10T17:52:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2023-10-10T17:52:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2023-10-10T17:52:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2023-10-10T17:52:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-10-10T17:52:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-10T17:52:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2023-10-10T17:52:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2023-10-10T17:52:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2023-10-10T17:52:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2023-10-10T17:52:45.260 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-10-10T17:52:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2023-10-10T17:52:45.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2023-10-10T17:52:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.263 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2023-10-10T17:52:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.264 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-10-10T17:52:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-10-10T17:52:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.265 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2023-10-10T17:52:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2023-10-10T17:52:45.266 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2023-10-10T17:52:45.267 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2023-10-10T17:52:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2023-10-10T17:52:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.268 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2023-10-10T17:52:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-10-10T17:52:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2023-10-10T17:52:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2023-10-10T17:52:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.270 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2023-10-10T17:52:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2023-10-10T17:52:45.272 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-10T17:52:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.273 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2023-10-10T17:52:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2023-10-10T17:52:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2023-10-10T17:52:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2023-10-10T17:52:45.275 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-10-10T17:52:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-10T17:52:45.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.279 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.280 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.281 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.283 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.284 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.285 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.286 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.287 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.293 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.294 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.295 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2023-10-10T17:52:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2023-10-10T17:52:45.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:45.297 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.298 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-10-10T17:52:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2023-10-10T17:52:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2023-10-10T17:52:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.300 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2023-10-10T17:52:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.301 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2023-10-10T17:52:45.302 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2023-10-10T17:52:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap" 2023-10-10T17:52:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2023-10-10T17:52:45.303 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2023-10-10T17:52:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2023-10-10T17:52:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.305 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.306 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-10-10T17:52:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.308 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.309 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.310 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-10-10T17:52:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.311 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2023-10-10T17:52:45.312 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2023-10-10T17:52:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.313 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.314 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2023-10-10T17:52:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-10-10T17:52:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.315 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2023-10-10T17:52:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2023-10-10T17:52:45.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:45.317 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.318 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-10-10T17:52:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.320 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2023-10-10T17:52:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.321 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2023-10-10T17:52:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2023-10-10T17:52:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:45.322 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2023-10-10T17:52:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.323 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-10-10T17:52:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.325 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2023-10-10T17:52:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2023-10-10T17:52:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.326 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2023-10-10T17:52:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.327 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-10-10T17:52:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.328 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2023-10-10T17:52:45.329 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2023-10-10T17:52:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.330 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2023-10-10T17:52:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.331 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-10-10T17:52:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.332 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "alertmanager_web_password": { 2023-10-10T17:52:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alertmanager_web_password", 2023-10-10T17:52:45.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:45.334 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Alertmanager web password", 2023-10-10T17:52:45.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "alertmanager_web_user": { 2023-10-10T17:52:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alertmanager_web_user", 2023-10-10T17:52:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.337 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.338 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Alertmanager web user", 2023-10-10T17:52:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.340 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2023-10-10T17:52:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2023-10-10T17:52:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.342 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-10-10T17:52:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-10-10T17:52:45.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2023-10-10T17:52:45.344 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2023-10-10T17:52:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.345 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.346 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2023-10-10T17:52:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.347 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2023-10-10T17:52:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2023-10-10T17:52:45.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.349 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2023-10-10T17:52:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.350 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-10-10T17:52:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2023-10-10T17:52:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2023-10-10T17:52:45.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.353 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-10-10T17:52:45.354 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2023-10-10T17:52:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2023-10-10T17:52:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.357 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-10-10T17:52:45.358 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.359 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2023-10-10T17:52:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2023-10-10T17:52:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.360 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.361 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-10-10T17:52:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2023-10-10T17:52:45.363 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2023-10-10T17:52:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.364 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-10-10T17:52:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.365 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.366 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2023-10-10T17:52:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2023-10-10T17:52:45.367 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.368 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2023-10-10T17:52:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.369 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2023-10-10T17:52:45.370 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2023-10-10T17:52:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2023-10-10T17:52:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.372 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-10-10T17:52:45.373 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2023-10-10T17:52:45.374 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2023-10-10T17:52:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2023-10-10T17:52:45.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-10-10T17:52:45.377 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:45.378 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2023-10-10T17:52:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2023-10-10T17:52:45.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.381 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-10-10T17:52:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.382 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2023-10-10T17:52:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.383 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2023-10-10T17:52:45.384 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2023-10-10T17:52:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.385 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-10-10T17:52:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.386 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2023-10-10T17:52:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.387 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2023-10-10T17:52:45.388 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2023-10-10T17:52:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-10-10T17:52:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2023-10-10T17:52:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2023-10-10T17:52:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2023-10-10T17:52:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-10-10T17:52:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2023-10-10T17:52:45.394 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2023-10-10T17:52:45.397 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2023-10-10T17:52:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-10-10T17:52:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.399 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2023-10-10T17:52:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.400 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2023-10-10T17:52:45.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2023-10-10T17:52:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.402 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-10-10T17:52:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.403 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2023-10-10T17:52:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2023-10-10T17:52:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2023-10-10T17:52:45.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-10-10T17:52:45.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:45.407 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2023-10-10T17:52:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2023-10-10T17:52:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-10-10T17:52:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.410 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:45.411 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.412 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2023-10-10T17:52:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2023-10-10T17:52:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-10-10T17:52:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.414 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2023-10-10T17:52:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2023-10-10T17:52:45.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2023-10-10T17:52:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.417 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-10-10T17:52:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.418 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2023-10-10T17:52:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.419 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2023-10-10T17:52:45.420 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2023-10-10T17:52:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.421 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.422 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2023-10-10T17:52:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2023-10-10T17:52:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2023-10-10T17:52:45.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:45.425 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-10-10T17:52:45.426 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2023-10-10T17:52:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2023-10-10T17:52:45.428 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2023-10-10T17:52:45.429 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.430 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-10-10T17:52:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2023-10-10T17:52:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2023-10-10T17:52:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2023-10-10T17:52:45.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-10-10T17:52:45.434 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.435 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2023-10-10T17:52:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2023-10-10T17:52:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.436 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2023-10-10T17:52:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.437 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2023-10-10T17:52:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.439 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2023-10-10T17:52:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2023-10-10T17:52:45.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.442 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2023-10-10T17:52:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.443 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2023-10-10T17:52:45.444 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2023-10-10T17:52:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.446 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2023-10-10T17:52:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2023-10-10T17:52:45.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2023-10-10T17:52:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.450 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2023-10-10T17:52:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.451 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.454 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.455 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.456 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2023-10-10T17:52:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2023-10-10T17:52:45.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.459 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.460 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-10-10T17:52:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.464 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-10-10T17:52:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.467 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.468 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.469 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.471 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.473 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2023-10-10T17:52:45.475 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-10-10T17:52:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.476 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.477 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-10-10T17:52:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-10-10T17:52:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-10-10T17:52:45.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2023-10-10T17:52:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.481 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-10-10T17:52:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2023-10-10T17:52:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2023-10-10T17:52:45.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:45.484 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.485 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2023-10-10T17:52:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2023-10-10T17:52:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2023-10-10T17:52:45.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:45.488 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.489 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-10-10T17:52:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.490 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2023-10-10T17:52:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2023-10-10T17:52:45.491 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.492 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.493 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2023-10-10T17:52:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.494 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2023-10-10T17:52:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2023-10-10T17:52:45.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.496 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2023-10-10T17:52:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.497 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2023-10-10T17:52:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2023-10-10T17:52:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.498 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2023-10-10T17:52:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2023-10-10T17:52:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2023-10-10T17:52:45.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-10-10T17:52:45.501 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-10-10T17:52:45.502 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_web_password": { 2023-10-10T17:52:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_web_password", 2023-10-10T17:52:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.504 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.505 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus web password", 2023-10-10T17:52:45.506 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.507 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_web_user": { 2023-10-10T17:52:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_web_user", 2023-10-10T17:52:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.508 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.509 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus web user", 2023-10-10T17:52:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2023-10-10T17:52:45.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2023-10-10T17:52:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.512 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-10-10T17:52:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2023-10-10T17:52:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2023-10-10T17:52:45.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-10-10T17:52:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2023-10-10T17:52:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2023-10-10T17:52:45.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-10-10T17:52:45.521 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.522 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2023-10-10T17:52:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2023-10-10T17:52:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.523 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.524 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-10-10T17:52:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.526 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2023-10-10T17:52:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2023-10-10T17:52:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.528 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-10-10T17:52:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.529 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2023-10-10T17:52:45.530 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2023-10-10T17:52:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.531 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2023-10-10T17:52:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.532 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2023-10-10T17:52:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.533 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2023-10-10T17:52:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2023-10-10T17:52:45.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.535 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.536 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-10-10T17:52:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2023-10-10T17:52:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2023-10-10T17:52:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.538 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.539 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-10-10T17:52:45.540 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.541 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2023-10-10T17:52:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2023-10-10T17:52:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.542 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.543 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-10-10T17:52:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.545 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2023-10-10T17:52:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2023-10-10T17:52:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.547 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2023-10-10T17:52:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2023-10-10T17:52:45.549 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2023-10-10T17:52:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.550 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.551 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-10-10T17:52:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.552 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2023-10-10T17:52:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2023-10-10T17:52:45.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.555 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-10-10T17:52:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2023-10-10T17:52:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.558 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.560 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.561 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.562 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.563 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.565 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.566 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.567 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.568 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.569 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.570 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.571 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.572 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.574 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.575 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.576 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.577 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2023-10-10T17:52:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2023-10-10T17:52:45.578 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2023-10-10T17:52:45.579 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.580 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2023-10-10T17:52:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2023-10-10T17:52:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2023-10-10T17:52:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.582 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2023-10-10T17:52:45.583 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2023-10-10T17:52:45.584 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.586 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2023-10-10T17:52:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.587 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-10-10T17:52:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-10-10T17:52:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.589 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2023-10-10T17:52:45.591 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2023-10-10T17:52:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.593 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-10-10T17:52:45.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-10-10T17:52:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.597 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.598 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2023-10-10T17:52:45.600 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2023-10-10T17:52:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.601 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.602 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.603 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-10-10T17:52:45.604 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-10-10T17:52:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.605 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.606 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.607 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2023-10-10T17:52:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2023-10-10T17:52:45.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.609 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.610 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-10-10T17:52:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-10-10T17:52:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.613 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.615 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-10-10T17:52:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-10-10T17:52:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.617 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.618 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-10-10T17:52:45.619 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-10-10T17:52:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.620 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.621 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.622 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-10-10T17:52:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-10-10T17:52:45.623 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.624 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.625 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2023-10-10T17:52:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-10-10T17:52:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.627 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.628 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.629 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.630 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2023-10-10T17:52:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-10-10T17:52:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.632 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.634 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2023-10-10T17:52:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-10-10T17:52:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.636 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.637 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-10-10T17:52:45.638 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-10-10T17:52:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.639 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.640 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2023-10-10T17:52:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2023-10-10T17:52:45.642 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:45.643 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.644 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-10-10T17:52:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-10-10T17:52:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.646 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.647 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.648 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.649 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2023-10-10T17:52:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2023-10-10T17:52:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.651 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.652 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2023-10-10T17:52:45.653 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2023-10-10T17:52:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.655 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.656 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-10-10T17:52:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-10-10T17:52:45.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.658 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.659 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-10-10T17:52:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-10-10T17:52:45.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.662 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.663 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.664 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-10-10T17:52:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-10-10T17:52:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2023-10-10T17:52:45.668 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-10-10T17:52:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.669 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.670 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2023-10-10T17:52:45.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2023-10-10T17:52:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.673 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.674 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.675 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-10-10T17:52:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-10-10T17:52:45.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.677 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.678 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-10-10T17:52:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-10-10T17:52:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.680 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.681 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.682 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.683 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-10-10T17:52:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-10-10T17:52:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.685 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.686 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-10-10T17:52:45.687 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-10-10T17:52:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.688 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.689 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-10-10T17:52:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-10-10T17:52:45.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.692 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.693 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-10-10T17:52:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-10-10T17:52:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.695 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.696 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.697 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.698 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-10-10T17:52:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-10-10T17:52:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.699 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-10-10T17:52:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-10-10T17:52:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2023-10-10T17:52:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2023-10-10T17:52:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-10-10T17:52:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-10-10T17:52:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-10-10T17:52:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-10-10T17:52:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-10-10T17:52:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-10-10T17:52:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-10-10T17:52:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2023-10-10T17:52:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2023-10-10T17:52:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-10-10T17:52:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2023-10-10T17:52:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2023-10-10T17:52:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2023-10-10T17:52:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-10-10T17:52:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-10-10T17:52:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2023-10-10T17:52:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2023-10-10T17:52:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2023-10-10T17:52:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2023-10-10T17:52:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-10-10T17:52:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-10-10T17:52:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-10-10T17:52:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-10-10T17:52:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-10-10T17:52:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-10-10T17:52:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2023-10-10T17:52:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2023-10-10T17:52:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2023-10-10T17:52:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2023-10-10T17:52:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2023-10-10T17:52:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2023-10-10T17:52:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2023-10-10T17:52:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2023-10-10T17:52:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2023-10-10T17:52:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2023-10-10T17:52:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2023-10-10T17:52:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2023-10-10T17:52:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2023-10-10T17:52:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2023-10-10T17:52:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2023-10-10T17:52:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2023-10-10T17:52:45.793 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2023-10-10T17:52:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.795 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:45.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2023-10-10T17:52:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:45.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2023-10-10T17:52:45.802 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.803 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2023-10-10T17:52:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2023-10-10T17:52:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.805 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.806 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.808 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2023-10-10T17:52:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2023-10-10T17:52:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2023-10-10T17:52:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.810 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2023-10-10T17:52:45.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2023-10-10T17:52:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.813 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2023-10-10T17:52:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2023-10-10T17:52:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.815 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2023-10-10T17:52:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2023-10-10T17:52:45.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2023-10-10T17:52:45.818 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2023-10-10T17:52:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2023-10-10T17:52:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.819 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.820 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2023-10-10T17:52:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2023-10-10T17:52:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.822 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2023-10-10T17:52:45.825 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.826 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2023-10-10T17:52:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2023-10-10T17:52:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.828 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2023-10-10T17:52:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.830 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.833 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.834 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.835 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.837 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.838 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.840 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.841 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.842 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.843 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.845 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2023-10-10T17:52:45.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2023-10-10T17:52:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2023-10-10T17:52:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-10-10T17:52:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2023-10-10T17:52:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2023-10-10T17:52:45.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2023-10-10T17:52:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2023-10-10T17:52:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2023-10-10T17:52:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2023-10-10T17:52:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2023-10-10T17:52:45.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2023-10-10T17:52:45.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.860 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2023-10-10T17:52:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2023-10-10T17:52:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2023-10-10T17:52:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2023-10-10T17:52:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.864 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2023-10-10T17:52:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2023-10-10T17:52:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.866 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-10-10T17:52:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:45.868 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.869 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.870 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2023-10-10T17:52:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.871 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2023-10-10T17:52:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2023-10-10T17:52:45.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2023-10-10T17:52:45.873 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.874 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-10-10T17:52:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2023-10-10T17:52:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.877 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.880 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.881 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.882 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.885 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.886 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.888 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.891 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.892 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.894 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.896 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.897 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2023-10-10T17:52:45.898 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2023-10-10T17:52:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.899 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2023-10-10T17:52:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.900 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.901 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2023-10-10T17:52:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2023-10-10T17:52:45.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2023-10-10T17:52:45.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:45.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.908 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2023-10-10T17:52:45.910 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2023-10-10T17:52:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2023-10-10T17:52:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.911 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2023-10-10T17:52:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2023-10-10T17:52:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2023-10-10T17:52:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-10-10T17:52:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2023-10-10T17:52:45.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2023-10-10T17:52:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2023-10-10T17:52:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-10-10T17:52:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2023-10-10T17:52:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2023-10-10T17:52:45.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.921 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2023-10-10T17:52:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:45.923 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2023-10-10T17:52:45.924 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2023-10-10T17:52:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-10-10T17:52:45.925 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.926 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.927 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.928 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.929 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.930 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.931 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.932 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.933 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.934 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.935 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.938 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.939 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.940 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.941 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.943 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2023-10-10T17:52:45.945 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2023-10-10T17:52:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.947 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2023-10-10T17:52:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.948 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2023-10-10T17:52:45.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2023-10-10T17:52:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.950 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2023-10-10T17:52:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.951 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2023-10-10T17:52:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2023-10-10T17:52:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2023-10-10T17:52:45.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2023-10-10T17:52:45.954 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-10-10T17:52:45.955 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2023-10-10T17:52:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2023-10-10T17:52:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:45.957 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:45.958 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2023-10-10T17:52:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-10-10T17:52:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.960 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2023-10-10T17:52:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2023-10-10T17:52:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2023-10-10T17:52:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.963 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2023-10-10T17:52:45.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2023-10-10T17:52:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:45.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2023-10-10T17:52:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-10-10T17:52:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2023-10-10T17:52:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.969 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.970 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.971 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.972 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.973 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.975 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.977 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:45.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.980 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.981 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.982 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.983 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.984 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:45.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.986 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.987 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:45.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2023-10-10T17:52:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:45.990 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:45.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:45.993 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:45.994 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:45.996 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:45.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:45.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:45.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.003 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.009 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.010 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2023-10-10T17:52:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.012 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2023-10-10T17:52:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2023-10-10T17:52:46.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2023-10-10T17:52:46.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.015 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2023-10-10T17:52:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2023-10-10T17:52:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2023-10-10T17:52:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.017 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:46.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2023-10-10T17:52:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-10-10T17:52:46.019 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.020 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.021 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.022 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.023 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.024 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.025 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.026 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.027 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.028 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.031 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.032 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.033 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.034 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.036 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2023-10-10T17:52:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.041 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2023-10-10T17:52:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2023-10-10T17:52:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2023-10-10T17:52:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.043 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2023-10-10T17:52:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-10-10T17:52:46.044 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.046 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.047 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.048 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.049 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.052 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.053 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.055 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.056 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.057 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.058 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.060 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.062 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.063 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2023-10-10T17:52:46.064 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2023-10-10T17:52:46.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.065 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.065 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.066 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.066 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.066 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.066 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2023-10-10T17:52:46.067 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-10-10T17:52:46.067 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.067 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.068 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2023-10-10T17:52:46.068 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2023-10-10T17:52:46.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.069 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.069 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2023-10-10T17:52:46.069 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.070 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.070 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.070 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2023-10-10T17:52:46.070 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.071 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.072 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2023-10-10T17:52:46.072 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2023-10-10T17:52:46.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.072 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.073 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2023-10-10T17:52:46.073 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.073 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.074 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.074 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2023-10-10T17:52:46.074 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.075 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.075 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2023-10-10T17:52:46.076 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2023-10-10T17:52:46.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.076 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2023-10-10T17:52:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2023-10-10T17:52:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2023-10-10T17:52:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2023-10-10T17:52:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2023-10-10T17:52:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-10-10T17:52:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2023-10-10T17:52:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.089 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.089 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.089 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.090 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.090 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.090 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.091 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.092 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.092 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.093 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.093 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.093 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.094 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.094 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.095 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.095 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.097 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.097 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.097 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.097 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.098 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.098 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.098 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.098 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.099 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.099 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.099 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.100 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.100 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.101 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.101 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.102 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.102 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.102 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.102 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.103 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.103 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.104 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.105 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2023-10-10T17:52:46.105 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.106 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.106 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.106 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.106 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.107 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.108 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.109 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.110 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.111 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.114 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.116 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.117 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.118 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.118 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.118 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.119 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.120 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.120 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.120 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.120 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.121 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.121 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.121 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.122 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.123 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.123 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.123 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.123 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.124 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.124 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.124 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.126 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2023-10-10T17:52:46.126 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.126 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.127 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.127 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.127 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.128 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.128 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.128 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.129 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.129 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.129 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.130 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.130 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.130 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.131 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.131 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.131 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.131 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.132 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.133 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.133 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.133 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.134 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.135 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.135 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.135 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.136 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.136 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.137 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.137 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.138 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.138 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.138 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.139 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.139 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.139 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.139 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.140 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.140 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.140 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.140 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.141 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.141 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.141 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.142 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.142 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.143 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.144 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.144 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.144 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.144 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.145 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.145 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.147 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2023-10-10T17:52:46.147 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.147 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.147 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2023-10-10T17:52:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2023-10-10T17:52:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.148 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.149 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.150 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.150 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-10-10T17:52:46.150 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.151 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.151 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.153 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.154 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.155 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.156 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.158 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.159 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.160 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.162 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.163 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.164 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.166 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.167 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2023-10-10T17:52:46.170 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2023-10-10T17:52:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.172 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2023-10-10T17:52:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2023-10-10T17:52:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2023-10-10T17:52:46.173 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2023-10-10T17:52:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2023-10-10T17:52:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.176 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.178 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.179 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.180 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.181 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.183 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.184 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.185 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.186 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.188 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.189 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.190 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.191 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.193 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.195 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2023-10-10T17:52:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.197 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.199 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.200 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.201 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.202 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.203 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.204 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.205 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.206 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.208 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.209 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.210 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.211 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.213 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.214 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.215 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2023-10-10T17:52:46.217 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.218 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.221 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.222 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.226 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.227 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.230 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.231 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.235 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.236 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.237 INFO:teuthology.orchestra.run.smithi078.stdout: "noautoscale": { 2023-10-10T17:52:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "noautoscale", 2023-10-10T17:52:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "global autoscale flag", 2023-10-10T17:52:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-10-10T17:52:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2023-10-10T17:52:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2023-10-10T17:52:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2023-10-10T17:52:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2023-10-10T17:52:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2023-10-10T17:52:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-10-10T17:52:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2023-10-10T17:52:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.251 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2023-10-10T17:52:46.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2023-10-10T17:52:46.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.251 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.252 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.252 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2023-10-10T17:52:46.253 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2023-10-10T17:52:46.255 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2023-10-10T17:52:46.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.255 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.256 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.256 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.256 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.257 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.257 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.257 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.258 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.258 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.259 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.259 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.260 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.260 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.260 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.261 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.261 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.261 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.261 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.262 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.263 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.264 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.265 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.266 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.267 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.269 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.270 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.270 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.270 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.270 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.271 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.271 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.271 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.272 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.272 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.272 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.273 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.273 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.274 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.275 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.275 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.276 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.276 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.276 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.276 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.277 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.277 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2023-10-10T17:52:46.278 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2023-10-10T17:52:46.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.279 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2023-10-10T17:52:46.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.279 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.280 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.280 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2023-10-10T17:52:46.280 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.281 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.281 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:46.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:46.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.282 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:46.283 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.283 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.283 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2023-10-10T17:52:46.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.284 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2023-10-10T17:52:46.286 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.286 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.286 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.287 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2023-10-10T17:52:46.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2023-10-10T17:52:46.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.288 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.288 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.288 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.289 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.289 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.289 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.289 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.290 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.291 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2023-10-10T17:52:46.291 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2023-10-10T17:52:46.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.292 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.292 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.292 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-10-10T17:52:46.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-10-10T17:52:46.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.294 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.294 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.295 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.296 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.296 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.296 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.296 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.297 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.297 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.297 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.298 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.299 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.299 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.300 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.300 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.301 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.301 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.301 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.302 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.302 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.302 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.303 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.303 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.304 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.304 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.305 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.306 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.307 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.308 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.308 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.308 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.309 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.309 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.310 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.310 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.311 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.311 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.311 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.312 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.312 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.312 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.313 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.313 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2023-10-10T17:52:46.314 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2023-10-10T17:52:46.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.314 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.314 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.315 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.315 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.315 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.315 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.316 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.316 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.317 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.317 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2023-10-10T17:52:46.317 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-10-10T17:52:46.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.318 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.318 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2023-10-10T17:52:46.319 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.319 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.319 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.320 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.320 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.320 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.320 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2023-10-10T17:52:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2023-10-10T17:52:46.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.322 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2023-10-10T17:52:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.323 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.325 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2023-10-10T17:52:46.326 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.327 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.327 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.327 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-10-10T17:52:46.327 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.328 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.329 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2023-10-10T17:52:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.330 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.331 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.331 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-10-10T17:52:46.331 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.332 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.332 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2023-10-10T17:52:46.332 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2023-10-10T17:52:46.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.333 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.333 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.334 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2023-10-10T17:52:46.334 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.334 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.334 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.335 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.336 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2023-10-10T17:52:46.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2023-10-10T17:52:46.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.337 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.337 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2023-10-10T17:52:46.338 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.338 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.338 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.338 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2023-10-10T17:52:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2023-10-10T17:52:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.339 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.340 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2023-10-10T17:52:46.341 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2023-10-10T17:52:46.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.341 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2023-10-10T17:52:46.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2023-10-10T17:52:46.342 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2023-10-10T17:52:46.343 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.343 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.344 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2023-10-10T17:52:46.345 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.346 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.347 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.348 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.349 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.350 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.352 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.352 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.353 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.354 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.355 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.356 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.357 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.358 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.359 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.361 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.362 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.362 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.363 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.363 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.364 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.364 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.364 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.364 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.365 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2023-10-10T17:52:46.365 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2023-10-10T17:52:46.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.366 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.366 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:46.367 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.367 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.367 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.368 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.368 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.368 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.369 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2023-10-10T17:52:46.369 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2023-10-10T17:52:46.369 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.370 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.370 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.370 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.371 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.371 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.371 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.371 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.372 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.372 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.373 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2023-10-10T17:52:46.373 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2023-10-10T17:52:46.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.373 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.374 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.374 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.374 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.375 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.375 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.375 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.375 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.376 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.377 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2023-10-10T17:52:46.377 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.378 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.378 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.378 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2023-10-10T17:52:46.379 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2023-10-10T17:52:46.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.379 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.380 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.381 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2023-10-10T17:52:46.382 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2023-10-10T17:52:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.384 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.385 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.386 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.387 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.388 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.389 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.390 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.391 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.391 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.393 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.394 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.394 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.394 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.395 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.395 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.396 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.396 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.396 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.397 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.397 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.397 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.398 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.398 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.398 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.399 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.399 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.399 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.399 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.400 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.400 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.400 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.401 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.401 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.402 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.402 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.403 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.403 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.403 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.403 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.404 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.404 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.404 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.405 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:46.405 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:46.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.406 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.406 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.406 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.407 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.407 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.407 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.408 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.408 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.408 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.409 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:46.409 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:46.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.410 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.410 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.410 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.411 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.411 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.411 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.411 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.413 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.413 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.415 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2023-10-10T17:52:46.415 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.415 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.416 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.417 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.418 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.419 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.420 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.421 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.423 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.425 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.426 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.426 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.427 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.428 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.428 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.428 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.429 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.429 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.429 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.429 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.430 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.430 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.430 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.431 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.433 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.433 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.434 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.434 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.434 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2023-10-10T17:52:46.436 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.436 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.436 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.437 INFO:teuthology.orchestra.run.smithi078.stdout: "drive_group_interval": { 2023-10-10T17:52:46.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "drive_group_interval", 2023-10-10T17:52:46.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.438 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.438 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300.0", 2023-10-10T17:52:46.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.439 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.439 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.439 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-10-10T17:52:46.439 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.440 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.441 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.442 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.442 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.443 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.443 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.443 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.443 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.444 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.444 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.444 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.445 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.445 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.446 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.448 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.449 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.449 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.450 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.451 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.452 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.453 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.454 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.455 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.457 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.458 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.458 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.458 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.458 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.459 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.460 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2023-10-10T17:52:46.460 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2023-10-10T17:52:46.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.460 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.461 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2023-10-10T17:52:46.461 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.461 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.462 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.462 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-10-10T17:52:46.462 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.463 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.463 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.464 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2023-10-10T17:52:46.464 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.465 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.465 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.465 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.465 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.466 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.467 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.467 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.467 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.467 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.468 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.468 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.468 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.468 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.469 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.469 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.469 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.469 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.470 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.471 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.471 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.472 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.472 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.473 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.473 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.473 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.473 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.474 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.474 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.474 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.475 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.475 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.476 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.476 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.476 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.477 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.477 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.477 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.477 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.478 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.478 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.478 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.479 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.479 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.479 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.479 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.480 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.480 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.481 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.481 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.482 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.482 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.482 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.482 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.483 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.483 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.483 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.483 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.484 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.484 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2023-10-10T17:52:46.485 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2023-10-10T17:52:46.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.485 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.486 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.486 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.486 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.487 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.487 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.487 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.488 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.488 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2023-10-10T17:52:46.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2023-10-10T17:52:46.489 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.489 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.489 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2023-10-10T17:52:46.490 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.490 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.490 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.491 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.491 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.491 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.491 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.492 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2023-10-10T17:52:46.492 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2023-10-10T17:52:46.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.493 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.493 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.493 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.494 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.494 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.494 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.495 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.495 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.495 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.496 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2023-10-10T17:52:46.496 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2023-10-10T17:52:46.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.496 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.497 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.497 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.497 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.497 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.498 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.498 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.498 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.499 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2023-10-10T17:52:46.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2023-10-10T17:52:46.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.500 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.500 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.501 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.501 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.502 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.502 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.503 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2023-10-10T17:52:46.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2023-10-10T17:52:46.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.504 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.505 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.506 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.506 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.506 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.506 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.507 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2023-10-10T17:52:46.507 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2023-10-10T17:52:46.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.508 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.508 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.508 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.509 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.509 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.509 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.509 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.510 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.510 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2023-10-10T17:52:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2023-10-10T17:52:46.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.512 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2023-10-10T17:52:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2023-10-10T17:52:46.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.516 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2023-10-10T17:52:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2023-10-10T17:52:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2023-10-10T17:52:46.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.520 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.521 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2023-10-10T17:52:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2023-10-10T17:52:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.523 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.524 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.525 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2023-10-10T17:52:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2023-10-10T17:52:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.527 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.528 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2023-10-10T17:52:46.531 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2023-10-10T17:52:46.532 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2023-10-10T17:52:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.533 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.534 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2023-10-10T17:52:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2023-10-10T17:52:46.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2023-10-10T17:52:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.537 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.538 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2023-10-10T17:52:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.539 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.541 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.542 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.543 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.544 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.545 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.546 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.547 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.549 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.551 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.552 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.553 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.554 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.556 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.557 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.558 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2023-10-10T17:52:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.560 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.562 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.563 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.564 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.565 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.566 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.567 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.568 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.570 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.571 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.572 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.573 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.574 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.577 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.578 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2023-10-10T17:52:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.581 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.582 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.583 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.584 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.585 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.586 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.587 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.588 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.589 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.591 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.592 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.593 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.594 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.595 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.596 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.597 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.598 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.599 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2023-10-10T17:52:46.602 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.602 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.602 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.603 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2023-10-10T17:52:46.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2023-10-10T17:52:46.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.603 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.604 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.604 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-10-10T17:52:46.604 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.605 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.605 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.605 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.605 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.606 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.606 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:46.607 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:46.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.607 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.608 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2023-10-10T17:52:46.608 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.608 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.609 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.609 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.609 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.609 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.610 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.610 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.611 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.612 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.612 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.613 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.613 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.613 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.613 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.614 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.614 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.614 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.615 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.615 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.615 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.617 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.618 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.618 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.618 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.618 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.619 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.619 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.619 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.619 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.620 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.620 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.621 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.621 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.622 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.622 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.623 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.623 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.623 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.623 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.624 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.624 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.624 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.625 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.625 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.625 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.625 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.626 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.626 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.627 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.627 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.628 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.628 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.628 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.629 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.629 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.629 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.630 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.632 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2023-10-10T17:52:46.632 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.632 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.633 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.633 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2023-10-10T17:52:46.633 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2023-10-10T17:52:46.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.634 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.634 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.635 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.635 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.635 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.635 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2023-10-10T17:52:46.636 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.636 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.636 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.637 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2023-10-10T17:52:46.637 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2023-10-10T17:52:46.637 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.637 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.638 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.638 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.638 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.639 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.639 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.639 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-10-10T17:52:46.639 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.640 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.640 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2023-10-10T17:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2023-10-10T17:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.642 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.643 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.643 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-10-10T17:52:46.643 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.643 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.644 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.644 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2023-10-10T17:52:46.644 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2023-10-10T17:52:46.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.645 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.646 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.646 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.646 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.646 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.647 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-10-10T17:52:46.647 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.647 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.647 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2023-10-10T17:52:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2023-10-10T17:52:46.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.649 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.650 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.650 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.650 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.650 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2023-10-10T17:52:46.651 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.651 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.652 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2023-10-10T17:52:46.652 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2023-10-10T17:52:46.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.653 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.653 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.653 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.654 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.654 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.654 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.654 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.655 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.656 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2023-10-10T17:52:46.656 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2023-10-10T17:52:46.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.656 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.657 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.657 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.657 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.658 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.658 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.658 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.659 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.659 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2023-10-10T17:52:46.660 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2023-10-10T17:52:46.660 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.660 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.660 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.661 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-10-10T17:52:46.661 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.661 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.662 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.662 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.662 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.663 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2023-10-10T17:52:46.663 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2023-10-10T17:52:46.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.664 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.665 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.665 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.665 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.666 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.666 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.666 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.667 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:46.667 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:46.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.668 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.668 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2023-10-10T17:52:46.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2023-10-10T17:52:46.669 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.669 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.669 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.670 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.670 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.671 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2023-10-10T17:52:46.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2023-10-10T17:52:46.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.672 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.672 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2023-10-10T17:52:46.672 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.673 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.673 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.673 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.674 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.674 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.674 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.675 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2023-10-10T17:52:46.675 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2023-10-10T17:52:46.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.675 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.676 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.676 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.676 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.677 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.677 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.677 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.678 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.678 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.678 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2023-10-10T17:52:46.679 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2023-10-10T17:52:46.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.679 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.680 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.680 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.680 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.681 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.681 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.681 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.682 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.682 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.683 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.684 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.684 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.684 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.685 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.685 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.685 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.685 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.686 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.686 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.686 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.686 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.687 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.687 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.688 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.688 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.689 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.689 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.689 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.690 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.690 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.690 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.691 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.691 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.692 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.692 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.693 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.694 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.694 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.694 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.694 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.695 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.695 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.695 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.695 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.696 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.696 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.696 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.697 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.697 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.699 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.699 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.699 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.699 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2023-10-10T17:52:46.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2023-10-10T17:52:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.703 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2023-10-10T17:52:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2023-10-10T17:52:46.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.707 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.709 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2023-10-10T17:52:46.709 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2023-10-10T17:52:46.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.709 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.710 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-10-10T17:52:46.710 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.711 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.711 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.711 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.711 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.712 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.712 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2023-10-10T17:52:46.713 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.714 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.717 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.718 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.719 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.722 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.723 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.724 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.724 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.724 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.724 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.725 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.725 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.726 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.726 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.726 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.726 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.727 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.727 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.727 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.727 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.728 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.729 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.730 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.731 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.731 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.731 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.731 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.732 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.732 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.733 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.734 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2023-10-10T17:52:46.734 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.734 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.735 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.735 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.735 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.736 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.737 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.738 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.738 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.738 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.738 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.739 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.739 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.739 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.739 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.740 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.740 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.740 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.741 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.742 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.742 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.742 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.743 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.743 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.743 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.744 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.745 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.746 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.746 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.747 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.747 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.747 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.748 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.748 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.748 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.748 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.749 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.749 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.749 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.750 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.750 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.751 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.751 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.752 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.752 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.752 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.752 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.753 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.753 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.753 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.753 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.754 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2023-10-10T17:52:46.754 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2023-10-10T17:52:46.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.755 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.755 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2023-10-10T17:52:46.756 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.756 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.756 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.756 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2023-10-10T17:52:46.757 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.757 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2023-10-10T17:52:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2023-10-10T17:52:46.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:46.759 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.760 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.760 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.760 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-10-10T17:52:46.760 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.761 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.761 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.762 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2023-10-10T17:52:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.763 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2023-10-10T17:52:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2023-10-10T17:52:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.764 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2023-10-10T17:52:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.765 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.766 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2023-10-10T17:52:46.767 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2023-10-10T17:52:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.768 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.769 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.770 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:46.771 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.772 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.773 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.774 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.776 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.777 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.778 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.779 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.780 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.781 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.782 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.784 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.786 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.787 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.788 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.790 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.791 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.792 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.794 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2023-10-10T17:52:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2023-10-10T17:52:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.796 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.797 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2023-10-10T17:52:46.798 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2023-10-10T17:52:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.799 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2023-10-10T17:52:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.800 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.801 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2023-10-10T17:52:46.802 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2023-10-10T17:52:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.803 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-10-10T17:52:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.804 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.807 INFO:teuthology.orchestra.run.smithi078.stdout: "modules": [ 2023-10-10T17:52:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2023-10-10T17:52:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard", 2023-10-10T17:52:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "iostat", 2023-10-10T17:52:46.808 INFO:teuthology.orchestra.run.smithi078.stdout: "nfs", 2023-10-10T17:52:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "restful", 2023-10-10T17:52:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_schedule" 2023-10-10T17:52:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.809 INFO:teuthology.orchestra.run.smithi078.stdout: "available_modules": [ 2023-10-10T17:52:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alerts", 2023-10-10T17:52:46.810 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:46.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.812 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.813 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How frequently to reexamine health status", 2023-10-10T17:52:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.814 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.816 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.817 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.818 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.819 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.821 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.822 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.823 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.824 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.826 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.827 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.828 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.829 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.830 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.831 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.832 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_destination": { 2023-10-10T17:52:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_destination", 2023-10-10T17:52:46.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.835 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email address to send alerts to", 2023-10-10T17:52:46.836 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_from_name": { 2023-10-10T17:52:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_from_name", 2023-10-10T17:52:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.838 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "Ceph", 2023-10-10T17:52:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.839 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Email From: name", 2023-10-10T17:52:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.841 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_host": { 2023-10-10T17:52:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_host", 2023-10-10T17:52:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.843 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP server", 2023-10-10T17:52:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.844 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_password": { 2023-10-10T17:52:46.845 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_password", 2023-10-10T17:52:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.846 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.847 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Password to authenticate with", 2023-10-10T17:52:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.848 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_port": { 2023-10-10T17:52:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_port", 2023-10-10T17:52:46.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "465", 2023-10-10T17:52:46.850 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP port", 2023-10-10T17:52:46.851 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.852 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.853 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_sender": { 2023-10-10T17:52:46.853 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_sender", 2023-10-10T17:52:46.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.853 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.854 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.854 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.854 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.855 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.855 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SMTP envelope sender", 2023-10-10T17:52:46.855 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.856 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.856 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.856 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_ssl": { 2023-10-10T17:52:46.857 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_ssl", 2023-10-10T17:52:46.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.857 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.858 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.858 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use SSL to connect to SMTP server", 2023-10-10T17:52:46.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.860 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.860 INFO:teuthology.orchestra.run.smithi078.stdout: "smtp_user": { 2023-10-10T17:52:46.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "smtp_user", 2023-10-10T17:52:46.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.861 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.861 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.862 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.862 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.862 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.862 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "User to authenticate as", 2023-10-10T17:52:46.863 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.863 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "balancer", 2023-10-10T17:52:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.865 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "active": { 2023-10-10T17:52:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "active", 2023-10-10T17:52:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.866 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.867 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically balance PGs across cluster", 2023-10-10T17:52:46.868 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.869 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_time": { 2023-10-10T17:52:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_time", 2023-10-10T17:52:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.870 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0000", 2023-10-10T17:52:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.871 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "beginning time of day to automatically balance", 2023-10-10T17:52:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-10T17:52:46.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.873 INFO:teuthology.orchestra.run.smithi078.stdout: "begin_weekday": { 2023-10-10T17:52:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "begin_weekday", 2023-10-10T17:52:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2023-10-10T17:52:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2023-10-10T17:52:46.875 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2023-10-10T17:52:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-10T17:52:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.876 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_max_iterations": { 2023-10-10T17:52:46.877 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_max_iterations", 2023-10-10T17:52:46.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:46.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.878 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "25", 2023-10-10T17:52:46.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:46.879 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "250", 2023-10-10T17:52:46.879 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.879 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum number of iterations to attempt optimization", 2023-10-10T17:52:46.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.880 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.881 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_metrics": { 2023-10-10T17:52:46.881 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_metrics", 2023-10-10T17:52:46.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.882 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.882 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "pgs,objects,bytes", 2023-10-10T17:52:46.882 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.883 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.883 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.883 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "metrics with which to calculate OSD utilization", 2023-10-10T17:52:46.883 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2023-10-10T17:52:46.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.884 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.885 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_compat_step": { 2023-10-10T17:52:46.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crush_compat_step", 2023-10-10T17:52:46.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.885 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.886 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.5", 2023-10-10T17:52:46.886 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0.001", 2023-10-10T17:52:46.886 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "0.999", 2023-10-10T17:52:46.887 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.887 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "aggressiveness of optimization", 2023-10-10T17:52:46.887 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2023-10-10T17:52:46.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.888 INFO:teuthology.orchestra.run.smithi078.stdout: "end_time": { 2023-10-10T17:52:46.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_time", 2023-10-10T17:52:46.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.889 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2359", 2023-10-10T17:52:46.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.890 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ending time of day to automatically balance", 2023-10-10T17:52:46.891 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "This is a time of day in the format HHMM.", 2023-10-10T17:52:46.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.892 INFO:teuthology.orchestra.run.smithi078.stdout: "end_weekday": { 2023-10-10T17:52:46.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "end_weekday", 2023-10-10T17:52:46.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:46.893 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.893 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:46.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "0", 2023-10-10T17:52:46.894 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "6", 2023-10-10T17:52:46.894 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.895 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2023-10-10T17:52:46.895 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2023-10-10T17:52:46.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.895 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.896 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:46.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:46.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.897 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.897 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.898 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.898 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.898 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.899 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.899 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.899 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.899 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.900 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.900 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.900 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.900 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.901 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:46.902 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:46.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.902 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.903 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.903 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.904 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.904 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.904 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.904 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.905 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.905 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:46.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:46.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.906 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.907 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:46.907 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.907 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.907 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.908 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:46.908 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:46.908 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:46.909 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:46.909 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:46.909 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:46.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.910 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.910 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.910 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.911 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:46.911 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:46.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.912 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.912 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.913 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.913 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.913 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:46.914 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.914 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.915 INFO:teuthology.orchestra.run.smithi078.stdout: "min_score": { 2023-10-10T17:52:46.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_score", 2023-10-10T17:52:46.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.916 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:46.916 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "minimum score, below which no optimization is attempted", 2023-10-10T17:52:46.917 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.918 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2023-10-10T17:52:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2023-10-10T17:52:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.919 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "upmap", 2023-10-10T17:52:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.920 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-compat", 2023-10-10T17:52:46.921 INFO:teuthology.orchestra.run.smithi078.stdout: "none", 2023-10-10T17:52:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap" 2023-10-10T17:52:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Balancer mode", 2023-10-10T17:52:46.922 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.923 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_ids": { 2023-10-10T17:52:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_ids", 2023-10-10T17:52:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.925 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "pools which the automatic balancing will be limited to", 2023-10-10T17:52:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.927 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.927 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:46.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:46.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:46.929 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.929 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.929 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.930 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and attempt optimization", 2023-10-10T17:52:46.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.931 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_deviation": { 2023-10-10T17:52:46.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_deviation", 2023-10-10T17:52:46.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.932 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:46.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:46.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.933 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "deviation below which no optimization is attempted", 2023-10-10T17:52:46.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2023-10-10T17:52:46.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.935 INFO:teuthology.orchestra.run.smithi078.stdout: "upmap_max_optimizations": { 2023-10-10T17:52:46.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "upmap_max_optimizations", 2023-10-10T17:52:46.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:46.936 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.936 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.936 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:46.936 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.937 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.937 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.937 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "maximum upmap optimizations to make per attempt", 2023-10-10T17:52:46.937 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.938 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.938 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:46.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:46.939 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cephadm", 2023-10-10T17:52:46.940 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:46.940 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:46.940 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:46.940 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_down_multiplier": { 2023-10-10T17:52:46.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_down_multiplier", 2023-10-10T17:52:46.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.941 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.942 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2023-10-10T17:52:46.942 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.942 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.943 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2023-10-10T17:52:46.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.944 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_refresh_rate": { 2023-10-10T17:52:46.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_refresh_rate", 2023-10-10T17:52:46.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.945 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "20", 2023-10-10T17:52:46.946 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.946 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.946 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.947 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2023-10-10T17:52:46.947 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.947 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.947 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.948 INFO:teuthology.orchestra.run.smithi078.stdout: "agent_starting_port": { 2023-10-10T17:52:46.948 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "agent_starting_port", 2023-10-10T17:52:46.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:46.949 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.949 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.949 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4721", 2023-10-10T17:52:46.950 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.950 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.950 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.950 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2023-10-10T17:52:46.951 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.951 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "alertmanager_web_password": { 2023-10-10T17:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alertmanager_web_password", 2023-10-10T17:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.952 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.953 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.953 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:46.953 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.954 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.954 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.954 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Alertmanager web password", 2023-10-10T17:52:46.954 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.955 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.955 INFO:teuthology.orchestra.run.smithi078.stdout: "alertmanager_web_user": { 2023-10-10T17:52:46.956 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "alertmanager_web_user", 2023-10-10T17:52:46.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.956 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.957 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.957 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:46.957 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.957 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.958 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.958 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Alertmanager web user", 2023-10-10T17:52:46.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.958 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.959 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_ptrace": { 2023-10-10T17:52:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_ptrace", 2023-10-10T17:52:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.961 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.962 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2023-10-10T17:52:46.962 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2023-10-10T17:52:46.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.963 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_interval": { 2023-10-10T17:52:46.963 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_interval", 2023-10-10T17:52:46.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:46.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.964 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:46.965 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.965 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.965 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to autotune daemon memory", 2023-10-10T17:52:46.966 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.966 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.967 INFO:teuthology.orchestra.run.smithi078.stdout: "autotune_memory_target_ratio": { 2023-10-10T17:52:46.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "autotune_memory_target_ratio", 2023-10-10T17:52:46.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:46.968 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0.7", 2023-10-10T17:52:46.969 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.969 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.969 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.970 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2023-10-10T17:52:46.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.971 INFO:teuthology.orchestra.run.smithi078.stdout: "cgroups_split": { 2023-10-10T17:52:46.971 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cgroups_split", 2023-10-10T17:52:46.971 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.972 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.972 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.973 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.973 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.973 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.973 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2023-10-10T17:52:46.974 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.974 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.974 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.975 INFO:teuthology.orchestra.run.smithi078.stdout: "config_checks_enabled": { 2023-10-10T17:52:46.975 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_checks_enabled", 2023-10-10T17:52:46.975 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.976 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.976 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.976 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:46.976 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.977 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.977 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.977 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2023-10-10T17:52:46.977 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "config_dashboard": { 2023-10-10T17:52:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_dashboard", 2023-10-10T17:52:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:46.979 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.980 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:46.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.981 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.981 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.981 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2023-10-10T17:52:46.981 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.982 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.982 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.982 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_alertmanager": { 2023-10-10T17:52:46.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_alertmanager", 2023-10-10T17:52:46.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.983 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.983 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.984 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2023-10-10T17:52:46.984 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.984 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.985 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.985 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:46.985 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.986 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_base": { 2023-10-10T17:52:46.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_base", 2023-10-10T17:52:46.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:46.988 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph", 2023-10-10T17:52:46.988 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.989 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Container image name, without the tag", 2023-10-10T17:52:46.989 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.989 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.990 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_elasticsearch": { 2023-10-10T17:52:46.990 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_elasticsearch", 2023-10-10T17:52:46.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.991 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2023-10-10T17:52:46.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.992 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "elasticsearch container image", 2023-10-10T17:52:46.993 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.993 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.993 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.994 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_grafana": { 2023-10-10T17:52:46.994 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_grafana", 2023-10-10T17:52:46.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.995 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.995 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.995 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.7", 2023-10-10T17:52:46.995 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.996 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:46.996 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:46.996 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:46.996 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:46.997 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:46.997 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:46.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:46.998 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_haproxy": { 2023-10-10T17:52:46.998 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_haproxy", 2023-10-10T17:52:46.998 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:46.998 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:46.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:46.999 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2023-10-10T17:52:46.999 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:46.999 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.000 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.000 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "HAproxy container image", 2023-10-10T17:52:47.000 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.000 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.001 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.001 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_agent": { 2023-10-10T17:52:47.002 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_agent", 2023-10-10T17:52:47.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.002 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.003 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2023-10-10T17:52:47.003 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.003 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.003 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.004 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger agent container image", 2023-10-10T17:52:47.004 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.005 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.005 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_collector": { 2023-10-10T17:52:47.005 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_collector", 2023-10-10T17:52:47.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.006 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.006 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2023-10-10T17:52:47.007 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.007 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.007 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.008 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger collector container image", 2023-10-10T17:52:47.008 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.008 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.009 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_jaeger_query": { 2023-10-10T17:52:47.009 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_jaeger_query", 2023-10-10T17:52:47.010 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.010 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.010 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.010 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2023-10-10T17:52:47.011 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.011 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.011 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.011 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Jaeger query container image", 2023-10-10T17:52:47.012 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.012 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.013 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_keepalived": { 2023-10-10T17:52:47.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_keepalived", 2023-10-10T17:52:47.013 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.014 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.014 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2023-10-10T17:52:47.014 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.015 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.015 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.015 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Keepalived container image", 2023-10-10T17:52:47.016 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.016 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.017 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_loki": { 2023-10-10T17:52:47.017 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_loki", 2023-10-10T17:52:47.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.018 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.018 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.018 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2023-10-10T17:52:47.018 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.019 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.019 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.019 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Loki container image", 2023-10-10T17:52:47.020 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.020 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.020 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.021 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_node_exporter": { 2023-10-10T17:52:47.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_node_exporter", 2023-10-10T17:52:47.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.021 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.022 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2023-10-10T17:52:47.022 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.023 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.023 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.023 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:47.023 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.024 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.024 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_prometheus": { 2023-10-10T17:52:47.025 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_prometheus", 2023-10-10T17:52:47.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.025 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.026 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2023-10-10T17:52:47.026 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.026 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.027 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.027 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus container image", 2023-10-10T17:52:47.027 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.027 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.028 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.028 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_promtail": { 2023-10-10T17:52:47.029 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_promtail", 2023-10-10T17:52:47.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.029 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.030 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2023-10-10T17:52:47.030 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.031 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.031 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.031 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Promtail container image", 2023-10-10T17:52:47.033 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.034 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.034 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image_snmp_gateway": { 2023-10-10T17:52:47.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_image_snmp_gateway", 2023-10-10T17:52:47.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.035 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.036 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2023-10-10T17:52:47.036 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.037 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.037 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "SNMP Gateway container image", 2023-10-10T17:52:47.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.038 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.039 INFO:teuthology.orchestra.run.smithi078.stdout: "container_init": { 2023-10-10T17:52:47.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "container_init", 2023-10-10T17:52:47.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.041 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.041 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.042 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.042 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Run podman/docker with `--init`", 2023-10-10T17:52:47.042 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.043 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.043 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.043 INFO:teuthology.orchestra.run.smithi078.stdout: "daemon_cache_timeout": { 2023-10-10T17:52:47.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "daemon_cache_timeout", 2023-10-10T17:52:47.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.044 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.045 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:47.045 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.046 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache service (daemon) inventory", 2023-10-10T17:52:47.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.047 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.048 INFO:teuthology.orchestra.run.smithi078.stdout: "default_cephadm_command_timeout": { 2023-10-10T17:52:47.048 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_cephadm_command_timeout", 2023-10-10T17:52:47.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.050 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "900", 2023-10-10T17:52:47.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2023-10-10T17:52:47.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.053 INFO:teuthology.orchestra.run.smithi078.stdout: "default_registry": { 2023-10-10T17:52:47.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "default_registry", 2023-10-10T17:52:47.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "docker.io", 2023-10-10T17:52:47.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.055 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2023-10-10T17:52:47.056 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.057 INFO:teuthology.orchestra.run.smithi078.stdout: "device_cache_timeout": { 2023-10-10T17:52:47.057 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_cache_timeout", 2023-10-10T17:52:47.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.058 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.058 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1800", 2023-10-10T17:52:47.059 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.059 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.059 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache device inventory", 2023-10-10T17:52:47.060 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.060 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.060 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "device_enhanced_scan": { 2023-10-10T17:52:47.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_enhanced_scan", 2023-10-10T17:52:47.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.062 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.062 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.063 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.063 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.063 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.064 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use libstoragemgmt during device scans", 2023-10-10T17:52:47.064 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.064 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.065 INFO:teuthology.orchestra.run.smithi078.stdout: "facts_cache_timeout": { 2023-10-10T17:52:47.065 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "facts_cache_timeout", 2023-10-10T17:52:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.067 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:47.067 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.067 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.068 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.068 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "seconds to cache host facts data", 2023-10-10T17:52:47.068 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.069 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.069 INFO:teuthology.orchestra.run.smithi078.stdout: "host_check_interval": { 2023-10-10T17:52:47.070 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "host_check_interval", 2023-10-10T17:52:47.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.070 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.071 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:47.071 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.071 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.072 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.072 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to perform a host check", 2023-10-10T17:52:47.072 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.073 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.074 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.074 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.074 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.075 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.075 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.076 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.076 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.076 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.077 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.077 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.077 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.077 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.078 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.078 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.079 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.079 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.079 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.080 INFO:teuthology.orchestra.run.smithi078.stdout: "log_refresh_metadata": { 2023-10-10T17:52:47.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_refresh_metadata", 2023-10-10T17:52:47.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.081 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.082 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.082 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.082 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.083 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.083 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2023-10-10T17:52:47.083 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.084 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.084 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.085 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.085 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.086 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.086 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.086 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.087 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.087 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2023-10-10T17:52:47.087 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.088 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.089 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.089 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.089 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.090 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.090 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.091 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.091 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.091 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.092 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.093 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.093 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.093 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.094 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.094 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.094 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.095 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.095 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.096 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.096 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.096 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.097 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.097 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.097 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.098 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.098 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.099 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf": { 2023-10-10T17:52:47.099 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf", 2023-10-10T17:52:47.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.100 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.100 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.101 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.101 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.101 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.102 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2023-10-10T17:52:47.102 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.103 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.103 INFO:teuthology.orchestra.run.smithi078.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2023-10-10T17:52:47.103 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2023-10-10T17:52:47.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.104 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.105 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "*", 2023-10-10T17:52:47.105 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.106 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.106 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2023-10-10T17:52:47.106 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.107 INFO:teuthology.orchestra.run.smithi078.stdout: "max_count_per_host": { 2023-10-10T17:52:47.108 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_count_per_host", 2023-10-10T17:52:47.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.108 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.109 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:47.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.109 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.110 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.110 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of daemons per service per host", 2023-10-10T17:52:47.110 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.111 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.111 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.112 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd_draining_count": { 2023-10-10T17:52:47.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_osd_draining_count", 2023-10-10T17:52:47.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.112 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.113 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:47.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.114 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.114 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.114 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2023-10-10T17:52:47.115 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.115 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.116 INFO:teuthology.orchestra.run.smithi078.stdout: "migration_current": { 2023-10-10T17:52:47.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "migration_current", 2023-10-10T17:52:47.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.117 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.117 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.118 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.118 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.118 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.118 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "internal - do not modify", 2023-10-10T17:52:47.119 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.119 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.119 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.120 INFO:teuthology.orchestra.run.smithi078.stdout: "mode": { 2023-10-10T17:52:47.120 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mode", 2023-10-10T17:52:47.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.121 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.121 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "root", 2023-10-10T17:52:47.122 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.122 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.122 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.123 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm-package", 2023-10-10T17:52:47.123 INFO:teuthology.orchestra.run.smithi078.stdout: "root" 2023-10-10T17:52:47.123 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.123 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "mode for remote execution of cephadm", 2023-10-10T17:52:47.124 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.124 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.125 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_alerts_path": { 2023-10-10T17:52:47.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_alerts_path", 2023-10-10T17:52:47.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.126 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.126 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2023-10-10T17:52:47.127 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.127 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.128 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "location of alerts to include in prometheus deployments", 2023-10-10T17:52:47.128 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.128 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.128 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.129 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_web_password": { 2023-10-10T17:52:47.129 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_web_password", 2023-10-10T17:52:47.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.130 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.131 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:47.131 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.131 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.131 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.132 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus web password", 2023-10-10T17:52:47.132 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.132 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.133 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.133 INFO:teuthology.orchestra.run.smithi078.stdout: "prometheus_web_user": { 2023-10-10T17:52:47.134 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus_web_user", 2023-10-10T17:52:47.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.134 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.135 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:47.135 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.135 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Prometheus web user", 2023-10-10T17:52:47.136 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.137 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.137 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_insecure": { 2023-10-10T17:52:47.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_insecure", 2023-10-10T17:52:47.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.138 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.140 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.140 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2023-10-10T17:52:47.141 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.141 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.141 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.142 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_password": { 2023-10-10T17:52:47.142 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_password", 2023-10-10T17:52:47.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.143 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.143 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.144 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2023-10-10T17:52:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.145 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.146 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_url": { 2023-10-10T17:52:47.146 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_url", 2023-10-10T17:52:47.147 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.147 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.147 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.148 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.148 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.148 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.149 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.149 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2023-10-10T17:52:47.149 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.150 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.150 INFO:teuthology.orchestra.run.smithi078.stdout: "registry_username": { 2023-10-10T17:52:47.151 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "registry_username", 2023-10-10T17:52:47.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.151 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.152 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.152 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.152 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.153 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.153 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2023-10-10T17:52:47.153 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.154 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.154 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.154 INFO:teuthology.orchestra.run.smithi078.stdout: "secure_monitoring_stack": { 2023-10-10T17:52:47.155 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "secure_monitoring_stack", 2023-10-10T17:52:47.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.155 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.156 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.156 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.156 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.157 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.157 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2023-10-10T17:52:47.157 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.158 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.158 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.159 INFO:teuthology.orchestra.run.smithi078.stdout: "service_discovery_port": { 2023-10-10T17:52:47.159 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "service_discovery_port", 2023-10-10T17:52:47.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.159 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.160 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8765", 2023-10-10T17:52:47.160 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.161 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.161 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "cephadm service discovery port", 2023-10-10T17:52:47.162 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.162 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.162 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.163 INFO:teuthology.orchestra.run.smithi078.stdout: "ssh_config_file": { 2023-10-10T17:52:47.163 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssh_config_file", 2023-10-10T17:52:47.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.164 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.165 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.165 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "customized SSH config file to connect to managed hosts", 2023-10-10T17:52:47.166 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.166 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.167 INFO:teuthology.orchestra.run.smithi078.stdout: "use_agent": { 2023-10-10T17:52:47.167 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_agent", 2023-10-10T17:52:47.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.168 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.168 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.169 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.169 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.169 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2023-10-10T17:52:47.170 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.170 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.171 INFO:teuthology.orchestra.run.smithi078.stdout: "use_repo_digest": { 2023-10-10T17:52:47.171 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "use_repo_digest", 2023-10-10T17:52:47.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.172 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.172 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.173 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.173 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2023-10-10T17:52:47.174 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.175 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_failed_host_check": { 2023-10-10T17:52:47.175 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_failed_host_check", 2023-10-10T17:52:47.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.176 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.176 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.177 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if the host check fails", 2023-10-10T17:52:47.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.179 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.179 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_daemons": { 2023-10-10T17:52:47.180 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_daemons", 2023-10-10T17:52:47.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.180 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.181 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.182 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.182 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2023-10-10T17:52:47.182 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.183 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.183 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_on_stray_hosts": { 2023-10-10T17:52:47.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_on_stray_hosts", 2023-10-10T17:52:47.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.184 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.185 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.185 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.185 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.186 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2023-10-10T17:52:47.186 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.188 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crash", 2023-10-10T17:52:47.189 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.189 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.189 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.190 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.190 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.191 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.191 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.192 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.192 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.192 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.192 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.193 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.193 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.193 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.194 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.194 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.194 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.194 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.195 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.195 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.196 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.196 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.197 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.197 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.198 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.198 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.198 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.199 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.199 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.199 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.200 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.200 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.201 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.202 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.202 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.202 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.202 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.203 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.203 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.203 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.204 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.204 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.204 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.205 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.205 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.205 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.206 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.206 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.207 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.208 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.208 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.208 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.208 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.209 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.209 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.209 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.210 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.211 INFO:teuthology.orchestra.run.smithi078.stdout: "retain_interval": { 2023-10-10T17:52:47.211 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retain_interval", 2023-10-10T17:52:47.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.211 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.212 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "31536000", 2023-10-10T17:52:47.212 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.213 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.213 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.213 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain crashes before pruning them", 2023-10-10T17:52:47.213 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.214 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.215 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_recent_interval": { 2023-10-10T17:52:47.215 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_recent_interval", 2023-10-10T17:52:47.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.216 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.216 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1209600", 2023-10-10T17:52:47.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.217 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.217 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.217 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "time interval in which to warn about recent crashes", 2023-10-10T17:52:47.218 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.218 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.218 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.220 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dashboard", 2023-10-10T17:52:47.220 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.220 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.221 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.221 INFO:teuthology.orchestra.run.smithi078.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2023-10-10T17:52:47.221 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2023-10-10T17:52:47.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.222 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.223 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_HOST": { 2023-10-10T17:52:47.225 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_HOST", 2023-10-10T17:52:47.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.226 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.226 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.227 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.229 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.229 INFO:teuthology.orchestra.run.smithi078.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2023-10-10T17:52:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2023-10-10T17:52:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.231 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.231 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.232 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.232 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.232 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.232 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.233 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.234 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_ENABLED": { 2023-10-10T17:52:47.234 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_ENABLED", 2023-10-10T17:52:47.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.235 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.235 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.235 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.236 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.236 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.237 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.238 INFO:teuthology.orchestra.run.smithi078.stdout: "AUDIT_API_LOG_PAYLOAD": { 2023-10-10T17:52:47.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2023-10-10T17:52:47.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.239 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.239 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.241 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.242 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.242 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.242 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.243 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.243 INFO:teuthology.orchestra.run.smithi078.stdout: "ENABLE_BROWSABLE_API": { 2023-10-10T17:52:47.244 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ENABLE_BROWSABLE_API", 2023-10-10T17:52:47.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.244 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.245 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.245 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.245 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.246 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.246 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.246 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.247 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_CEPHFS": { 2023-10-10T17:52:47.248 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2023-10-10T17:52:47.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.248 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.249 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.249 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.250 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.250 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.250 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.250 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.251 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.252 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2023-10-10T17:52:47.252 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2023-10-10T17:52:47.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.253 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.253 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.253 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.254 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.254 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.254 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.255 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.255 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.256 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_ISCSI": { 2023-10-10T17:52:47.256 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2023-10-10T17:52:47.256 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.257 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.257 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.257 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.258 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.258 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.258 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.259 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.259 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.259 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.259 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.260 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_MIRRORING": { 2023-10-10T17:52:47.260 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2023-10-10T17:52:47.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.261 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.262 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.262 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.262 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.263 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.263 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.263 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.263 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.264 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.264 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_NFS": { 2023-10-10T17:52:47.265 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_NFS", 2023-10-10T17:52:47.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.265 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.266 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.266 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.266 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.267 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.267 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.267 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.268 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.268 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RBD": { 2023-10-10T17:52:47.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RBD", 2023-10-10T17:52:47.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.269 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.270 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.271 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.271 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.271 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.271 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: "FEATURE_TOGGLE_RGW": { 2023-10-10T17:52:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "FEATURE_TOGGLE_RGW", 2023-10-10T17:52:47.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.274 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.274 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.274 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.275 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.275 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.275 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.276 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.276 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.277 INFO:teuthology.orchestra.run.smithi078.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2023-10-10T17:52:47.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2023-10-10T17:52:47.277 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.278 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.278 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.279 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.279 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.279 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.279 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.280 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.280 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.281 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_PASSWORD": { 2023-10-10T17:52:47.281 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_PASSWORD", 2023-10-10T17:52:47.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.283 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:47.283 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.283 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.285 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_SSL_VERIFY": { 2023-10-10T17:52:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2023-10-10T17:52:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.287 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.288 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_URL": { 2023-10-10T17:52:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_URL", 2023-10-10T17:52:47.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.291 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.292 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.293 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_API_USERNAME": { 2023-10-10T17:52:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_API_USERNAME", 2023-10-10T17:52:47.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.295 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.296 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.297 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_FRONTEND_API_URL": { 2023-10-10T17:52:47.298 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2023-10-10T17:52:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.300 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.301 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2023-10-10T17:52:47.302 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2023-10-10T17:52:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.304 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.305 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.306 INFO:teuthology.orchestra.run.smithi078.stdout: "ISCSI_API_SSL_VERIFICATION": { 2023-10-10T17:52:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2023-10-10T17:52:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.307 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.308 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.309 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ISSUE_TRACKER_API_KEY": { 2023-10-10T17:52:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ISSUE_TRACKER_API_KEY", 2023-10-10T17:52:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.311 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.312 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.313 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_HOST": { 2023-10-10T17:52:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_HOST", 2023-10-10T17:52:47.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.316 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.317 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.318 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2023-10-10T17:52:47.319 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2023-10-10T17:52:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.320 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.321 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.323 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2023-10-10T17:52:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2023-10-10T17:52:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.324 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.325 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.326 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2023-10-10T17:52:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2023-10-10T17:52:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.328 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.329 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.330 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2023-10-10T17:52:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2023-10-10T17:52:47.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.333 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.334 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.335 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2023-10-10T17:52:47.336 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2023-10-10T17:52:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.337 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.338 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2023-10-10T17:52:47.340 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2023-10-10T17:52:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.342 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.343 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.344 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2023-10-10T17:52:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2023-10-10T17:52:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.345 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.346 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.347 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.348 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2023-10-10T17:52:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2023-10-10T17:52:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.350 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.351 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_ENABLED": { 2023-10-10T17:52:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_ENABLED", 2023-10-10T17:52:47.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.354 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.355 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.356 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2023-10-10T17:52:47.357 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2023-10-10T17:52:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2023-10-10T17:52:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.359 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.361 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2023-10-10T17:52:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2023-10-10T17:52:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.362 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.364 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "PWD_POLICY_MIN_LENGTH": { 2023-10-10T17:52:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2023-10-10T17:52:47.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.367 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8", 2023-10-10T17:52:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.368 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.369 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.370 INFO:teuthology.orchestra.run.smithi078.stdout: "REST_REQUESTS_TIMEOUT": { 2023-10-10T17:52:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "REST_REQUESTS_TIMEOUT", 2023-10-10T17:52:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "45", 2023-10-10T17:52:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ACCESS_KEY": { 2023-10-10T17:52:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ACCESS_KEY", 2023-10-10T17:52:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_ADMIN_RESOURCE": { 2023-10-10T17:52:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2023-10-10T17:52:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "admin", 2023-10-10T17:52:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SECRET_KEY": { 2023-10-10T17:52:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SECRET_KEY", 2023-10-10T17:52:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "RGW_API_SSL_VERIFY": { 2023-10-10T17:52:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "RGW_API_SSL_VERIFY", 2023-10-10T17:52:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_SPAN": { 2023-10-10T17:52:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2023-10-10T17:52:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2023-10-10T17:52:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2023-10-10T17:52:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2023-10-10T17:52:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2023-10-10T17:52:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "cross_origin_url": { 2023-10-10T17:52:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cross_origin_url", 2023-10-10T17:52:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "crt_file": { 2023-10-10T17:52:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "crt_file", 2023-10-10T17:52:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "debug": { 2023-10-10T17:52:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "debug", 2023-10-10T17:52:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Enable/disable debug options", 2023-10-10T17:52:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "jwt_token_ttl": { 2023-10-10T17:52:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "jwt_token_ttl", 2023-10-10T17:52:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "28800", 2023-10-10T17:52:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2023-10-10T17:52:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2023-10-10T17:52:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "motd": { 2023-10-10T17:52:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "motd", 2023-10-10T17:52:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "The message of the day", 2023-10-10T17:52:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect_resolve_ip_addr": { 2023-10-10T17:52:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "redirect_resolve_ip_addr", 2023-10-10T17:52:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2023-10-10T17:52:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.459 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.459 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.459 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.460 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.460 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:47.461 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:47.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.461 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.462 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8080", 2023-10-10T17:52:47.462 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.462 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.463 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.463 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.463 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.464 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.465 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2023-10-10T17:52:47.465 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2023-10-10T17:52:47.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.466 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.466 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.466 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.467 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.467 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.467 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.468 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.468 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.468 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.469 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl_server_port": { 2023-10-10T17:52:47.469 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl_server_port", 2023-10-10T17:52:47.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.470 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.470 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.470 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8443", 2023-10-10T17:52:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.471 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.472 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2023-10-10T17:52:47.473 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2023-10-10T17:52:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "redirect", 2023-10-10T17:52:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.475 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.476 INFO:teuthology.orchestra.run.smithi078.stdout: "redirect" 2023-10-10T17:52:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.477 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.478 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2023-10-10T17:52:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2023-10-10T17:52:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.479 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2023-10-10T17:52:47.480 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2023-10-10T17:52:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2023-10-10T17:52:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.481 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.483 INFO:teuthology.orchestra.run.smithi078.stdout: "url_prefix": { 2023-10-10T17:52:47.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url_prefix", 2023-10-10T17:52:47.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.484 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.485 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.485 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.485 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.485 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.486 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.486 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.487 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.487 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.488 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2023-10-10T17:52:47.489 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.489 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.489 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.490 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_monitoring": { 2023-10-10T17:52:47.490 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_monitoring", 2023-10-10T17:52:47.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.490 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.491 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.491 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.491 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.492 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.492 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.492 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "monitor device health metrics", 2023-10-10T17:52:47.493 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.493 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.494 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.507 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.508 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.509 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.512 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.513 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "mark_out_threshold": { 2023-10-10T17:52:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mark_out_threshold", 2023-10-10T17:52:47.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.516 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "2419200", 2023-10-10T17:52:47.517 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.517 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.517 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.517 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "automatically mark OSD if it may fail before this long", 2023-10-10T17:52:47.518 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.518 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.519 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": { 2023-10-10T17:52:47.519 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pool_name", 2023-10-10T17:52:47.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.520 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.520 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "device_health_metrics", 2023-10-10T17:52:47.521 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.521 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.521 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.522 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name of pool in which to store device health metrics", 2023-10-10T17:52:47.522 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.522 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.523 INFO:teuthology.orchestra.run.smithi078.stdout: "retention_period": { 2023-10-10T17:52:47.523 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "retention_period", 2023-10-10T17:52:47.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.524 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.524 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.524 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15552000", 2023-10-10T17:52:47.525 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.525 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.525 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.526 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long to retain device health metrics", 2023-10-10T17:52:47.526 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.527 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.527 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_frequency": { 2023-10-10T17:52:47.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_frequency", 2023-10-10T17:52:47.528 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.528 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.528 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.529 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2023-10-10T17:52:47.529 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.529 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.530 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.530 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to scrape device health metrics", 2023-10-10T17:52:47.530 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.531 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.531 INFO:teuthology.orchestra.run.smithi078.stdout: "self_heal": { 2023-10-10T17:52:47.532 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "self_heal", 2023-10-10T17:52:47.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.532 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.533 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.533 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.533 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.534 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.534 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "preemptively heal cluster around devices that may fail", 2023-10-10T17:52:47.534 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.535 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.535 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:47.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:47.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.536 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.537 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:47.537 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.538 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.538 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.538 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how frequently to wake up and check device health", 2023-10-10T17:52:47.539 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.539 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.540 INFO:teuthology.orchestra.run.smithi078.stdout: "warn_threshold": { 2023-10-10T17:52:47.540 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "warn_threshold", 2023-10-10T17:52:47.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7257600", 2023-10-10T17:52:47.541 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.542 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.542 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.542 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "raise health warning if OSD may fail before this long", 2023-10-10T17:52:47.543 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.543 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.544 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.545 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "diskprediction_local", 2023-10-10T17:52:47.545 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.545 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.546 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.546 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.546 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.547 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.548 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.548 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.548 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.548 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.549 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.549 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.549 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.550 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.550 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.550 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.551 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.551 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.551 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.552 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.552 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.553 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.554 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.554 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.554 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.555 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.555 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.555 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.555 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.556 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.556 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.557 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.557 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.558 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.558 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.558 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.559 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.559 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.559 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.560 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.560 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.560 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.560 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.561 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.561 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.561 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.562 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.562 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.563 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.563 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.564 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.564 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.564 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.565 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.565 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.565 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.566 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.566 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.567 INFO:teuthology.orchestra.run.smithi078.stdout: "predict_interval": { 2023-10-10T17:52:47.567 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predict_interval", 2023-10-10T17:52:47.567 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.568 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.568 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.568 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "86400", 2023-10-10T17:52:47.569 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.569 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.569 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.570 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: "predictor_model": { 2023-10-10T17:52:47.571 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "predictor_model", 2023-10-10T17:52:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.572 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.573 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "prophetstor", 2023-10-10T17:52:47.573 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.573 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.574 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.574 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.574 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.574 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.575 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.575 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:47.576 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:47.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.576 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.577 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.577 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "600", 2023-10-10T17:52:47.577 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.578 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.578 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.578 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.579 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.579 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.581 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "influx", 2023-10-10T17:52:47.581 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": false, 2023-10-10T17:52:47.581 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "influxdb python module not found", 2023-10-10T17:52:47.582 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.582 INFO:teuthology.orchestra.run.smithi078.stdout: "batch_size": { 2023-10-10T17:52:47.582 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "batch_size", 2023-10-10T17:52:47.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.583 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.584 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5000", 2023-10-10T17:52:47.585 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.585 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.585 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.586 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2023-10-10T17:52:47.586 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.587 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.588 INFO:teuthology.orchestra.run.smithi078.stdout: "database": { 2023-10-10T17:52:47.588 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "database", 2023-10-10T17:52:47.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.589 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.590 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "ceph", 2023-10-10T17:52:47.590 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.591 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.591 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.591 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2023-10-10T17:52:47.591 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.592 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.592 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.593 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": { 2023-10-10T17:52:47.593 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "hostname", 2023-10-10T17:52:47.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.593 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.594 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.594 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.595 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.595 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.595 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server hostname", 2023-10-10T17:52:47.596 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.596 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.596 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.597 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:47.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.599 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.599 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "30", 2023-10-10T17:52:47.600 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "5", 2023-10-10T17:52:47.600 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.601 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.601 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2023-10-10T17:52:47.601 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.602 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.603 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.603 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.604 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.604 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.605 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.605 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.605 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.606 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.606 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.607 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.607 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.608 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.608 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.608 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.609 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.609 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.609 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.610 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.610 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.610 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.611 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.611 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.612 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.612 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.613 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.613 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.613 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.613 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.614 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.614 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.614 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.615 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.616 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.616 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.617 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.618 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.619 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.621 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.622 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.623 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.625 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.625 INFO:teuthology.orchestra.run.smithi078.stdout: "password": { 2023-10-10T17:52:47.626 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "password", 2023-10-10T17:52:47.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.626 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.627 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.627 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.627 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.628 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.628 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "password of InfluxDB server user", 2023-10-10T17:52:47.628 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.629 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.629 INFO:teuthology.orchestra.run.smithi078.stdout: "port": { 2023-10-10T17:52:47.630 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "port", 2023-10-10T17:52:47.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.630 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.631 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "8086", 2023-10-10T17:52:47.631 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.632 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.632 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.632 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "InfluxDB server port", 2023-10-10T17:52:47.633 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.633 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.634 INFO:teuthology.orchestra.run.smithi078.stdout: "ssl": { 2023-10-10T17:52:47.634 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ssl", 2023-10-10T17:52:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.635 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "false", 2023-10-10T17:52:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.636 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2023-10-10T17:52:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: "threads": { 2023-10-10T17:52:47.638 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threads", 2023-10-10T17:52:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:47.640 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "32", 2023-10-10T17:52:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2023-10-10T17:52:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.642 INFO:teuthology.orchestra.run.smithi078.stdout: "username": { 2023-10-10T17:52:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "username", 2023-10-10T17:52:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.643 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.644 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.644 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.645 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.645 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.645 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "username of InfluxDB server user", 2023-10-10T17:52:47.645 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.646 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.647 INFO:teuthology.orchestra.run.smithi078.stdout: "verify_ssl": { 2023-10-10T17:52:47.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "verify_ssl", 2023-10-10T17:52:47.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.648 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.648 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.648 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "true", 2023-10-10T17:52:47.649 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.649 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.649 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2023-10-10T17:52:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.654 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "insights", 2023-10-10T17:52:47.654 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.654 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.654 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.655 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.655 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.655 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.656 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.656 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.657 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.657 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.657 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.657 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.658 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.658 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.658 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.658 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.659 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.659 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.659 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.660 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.660 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.661 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.661 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.662 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.662 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.662 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.663 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.663 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.663 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.664 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.664 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "iostat", 2023-10-10T17:52:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.678 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.678 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.678 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.679 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.679 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.679 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.679 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.680 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.680 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.680 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.680 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.681 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.681 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.682 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.683 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.684 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.684 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.684 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.685 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.685 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.685 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.686 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.687 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.687 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.687 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.688 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.688 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.688 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.688 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.689 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.689 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.689 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.690 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.690 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.690 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.690 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.691 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.692 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.692 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.693 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.693 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.693 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.693 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.694 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.694 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.694 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "k8sevents", 2023-10-10T17:52:47.696 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.696 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.696 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.697 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_event_retention_days": { 2023-10-10T17:52:47.697 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "ceph_event_retention_days", 2023-10-10T17:52:47.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.698 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.698 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "7", 2023-10-10T17:52:47.698 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.699 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.699 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.699 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Days to hold ceph event information within local cache", 2023-10-10T17:52:47.700 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.700 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.700 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.701 INFO:teuthology.orchestra.run.smithi078.stdout: "config_check_secs": { 2023-10-10T17:52:47.701 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "config_check_secs", 2023-10-10T17:52:47.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.702 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.702 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:47.702 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "10", 2023-10-10T17:52:47.703 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.703 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.703 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2023-10-10T17:52:47.703 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.704 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.704 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.705 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.705 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.705 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.706 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.706 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.706 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.707 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.707 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.707 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.708 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.708 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.708 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.708 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.709 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.709 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.709 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.710 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.710 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.711 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.712 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.712 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.712 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.712 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.713 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.713 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.714 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.714 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.715 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.715 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.715 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.716 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.716 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.717 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.717 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.717 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.717 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.718 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.718 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.718 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.719 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.719 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.719 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.720 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.721 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.721 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.721 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.722 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.722 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.722 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.723 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.723 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.725 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "localpool", 2023-10-10T17:52:47.725 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.725 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.725 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.726 INFO:teuthology.orchestra.run.smithi078.stdout: "failure_domain": { 2023-10-10T17:52:47.726 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "failure_domain", 2023-10-10T17:52:47.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.726 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.727 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "host", 2023-10-10T17:52:47.727 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.728 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.728 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.728 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "failure domain for any created local pool", 2023-10-10T17:52:47.728 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2023-10-10T17:52:47.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.729 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.729 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.730 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.730 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.730 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.731 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.731 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.731 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.732 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.732 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.732 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.732 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.733 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.733 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.733 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.734 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.734 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.734 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.735 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.735 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.735 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.736 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.737 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.737 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.737 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.737 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.738 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.738 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.738 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.739 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.739 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.740 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.740 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.741 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.741 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.741 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.741 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.742 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.742 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.742 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.742 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.743 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.743 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.743 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.745 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.747 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": { 2023-10-10T17:52:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "min_size", 2023-10-10T17:52:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.749 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.750 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.751 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.751 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default min_size for any created local pool", 2023-10-10T17:52:47.751 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2023-10-10T17:52:47.751 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.752 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.752 INFO:teuthology.orchestra.run.smithi078.stdout: "num_rep": { 2023-10-10T17:52:47.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "num_rep", 2023-10-10T17:52:47.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.753 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.753 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.754 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3", 2023-10-10T17:52:47.754 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.754 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.754 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.755 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default replica count for any created local pool", 2023-10-10T17:52:47.755 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.756 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.756 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": { 2023-10-10T17:52:47.756 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_num", 2023-10-10T17:52:47.757 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.757 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.757 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.757 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "128", 2023-10-10T17:52:47.758 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.758 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.758 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.759 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "default pg_num for any created local pool", 2023-10-10T17:52:47.759 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.759 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.760 INFO:teuthology.orchestra.run.smithi078.stdout: "prefix": { 2023-10-10T17:52:47.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prefix", 2023-10-10T17:52:47.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.761 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.761 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.761 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.761 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.762 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "name prefix for any created local pool", 2023-10-10T17:52:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.764 INFO:teuthology.orchestra.run.smithi078.stdout: "subtree": { 2023-10-10T17:52:47.764 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "subtree", 2023-10-10T17:52:47.764 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.765 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.765 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.765 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "rack", 2023-10-10T17:52:47.765 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.766 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.766 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.766 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "CRUSH level for which to create a local pool", 2023-10-10T17:52:47.766 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2023-10-10T17:52:47.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.767 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.768 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.768 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mds_autoscaler", 2023-10-10T17:52:47.769 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.769 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.769 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.769 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.770 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.770 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.771 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.771 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.772 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.772 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.772 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.773 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.773 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.773 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.773 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.774 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.774 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.774 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.774 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.775 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.775 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.776 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.777 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.777 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.777 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.777 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.778 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.778 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.778 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.779 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.780 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.781 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.781 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.781 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.782 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.782 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.782 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.782 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.783 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.783 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.783 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.783 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.784 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.784 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.785 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.785 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.786 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.786 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.786 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.786 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.787 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.787 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.787 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.787 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.788 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.789 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirroring", 2023-10-10T17:52:47.790 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.790 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.790 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.790 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.791 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.791 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.792 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.792 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.792 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.793 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.794 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.794 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.794 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.795 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.795 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.795 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.796 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.796 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.797 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.797 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.798 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.798 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.798 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.798 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.799 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.799 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.799 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.800 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.800 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.800 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.801 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.801 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.802 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.802 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.802 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.803 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.804 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.806 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.806 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.807 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.807 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.807 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.808 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.808 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.808 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.808 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.809 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.810 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "nfs", 2023-10-10T17:52:47.811 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.811 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.811 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.811 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.812 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.813 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.813 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.813 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.814 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.814 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.814 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.814 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.815 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.815 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.815 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.816 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.816 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.817 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.817 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.817 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.818 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.818 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.818 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.819 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.819 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.819 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.819 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.820 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.820 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2023-10-10T17:52:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.833 INFO:teuthology.orchestra.run.smithi078.stdout: "fail_fs": { 2023-10-10T17:52:47.833 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "fail_fs", 2023-10-10T17:52:47.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.833 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.834 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.834 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.834 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.835 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.835 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.835 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2023-10-10T17:52:47.835 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.836 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.836 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.837 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.838 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.838 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.838 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.839 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.839 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.839 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.839 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.840 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.840 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.840 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.841 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.841 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.841 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.842 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.842 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.843 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.843 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.844 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.845 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.845 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.845 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.846 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.846 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.847 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.847 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.848 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.848 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.848 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.848 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.849 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.849 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.849 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.849 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.850 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.850 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.850 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.851 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.851 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.851 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.852 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.852 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.852 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.853 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.853 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.854 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.854 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.854 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.854 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.855 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.856 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator": { 2023-10-10T17:52:47.856 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "orchestrator", 2023-10-10T17:52:47.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.856 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.857 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.857 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.857 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.858 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.858 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm", 2023-10-10T17:52:47.858 INFO:teuthology.orchestra.run.smithi078.stdout: "rook", 2023-10-10T17:52:47.859 INFO:teuthology.orchestra.run.smithi078.stdout: "test_orchestrator" 2023-10-10T17:52:47.859 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.859 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Orchestrator backend", 2023-10-10T17:52:47.859 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.860 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_perf_query", 2023-10-10T17:52:47.862 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.862 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.862 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.862 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.863 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.863 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.864 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.864 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.864 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.865 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.865 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.865 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.865 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.866 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.867 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.867 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.867 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.868 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.868 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.868 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.869 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.870 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.870 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.870 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.871 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.871 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.871 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.871 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.872 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.873 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.873 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.874 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.874 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.874 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.874 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.875 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.875 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.875 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.876 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.876 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.876 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.876 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.877 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.877 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.878 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.878 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.878 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.879 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.879 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.880 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.880 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.880 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.880 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.881 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.882 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "osd_support", 2023-10-10T17:52:47.883 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.883 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.883 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.883 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.884 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.884 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.885 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.885 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.885 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.886 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.886 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.886 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.886 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.887 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.887 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.887 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.887 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.888 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.888 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.888 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.888 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.889 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.889 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.890 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.890 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.891 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.891 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.891 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.892 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.892 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.894 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.894 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.894 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.895 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.895 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.895 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.895 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.896 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.896 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.896 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.897 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.897 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.897 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.897 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.898 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.899 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.900 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.900 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.900 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.901 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.901 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.901 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.902 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.903 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "pg_autoscaler", 2023-10-10T17:52:47.903 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.904 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.904 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.904 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.904 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.905 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.905 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.905 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.906 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.906 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.907 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.907 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.907 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.907 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.908 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.908 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.908 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.909 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.909 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.909 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.910 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.911 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.912 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.914 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.915 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.916 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.917 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.918 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.918 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.918 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.919 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.919 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.919 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.920 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.920 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.921 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.921 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.921 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.922 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.922 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.922 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.922 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.923 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.923 INFO:teuthology.orchestra.run.smithi078.stdout: "noautoscale": { 2023-10-10T17:52:47.923 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "noautoscale", 2023-10-10T17:52:47.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.924 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.925 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.925 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.925 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.925 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.926 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "global autoscale flag", 2023-10-10T17:52:47.926 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Option to turn on/off the autoscaler for all pools", 2023-10-10T17:52:47.926 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.927 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.927 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:47.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:47.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.928 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.928 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:47.929 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.929 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.929 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.929 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.930 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.930 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.930 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.931 INFO:teuthology.orchestra.run.smithi078.stdout: "threshold": { 2023-10-10T17:52:47.931 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "threshold", 2023-10-10T17:52:47.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:47.932 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.932 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "3.0", 2023-10-10T17:52:47.933 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1.0", 2023-10-10T17:52:47.933 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.933 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.933 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "scaling threshold", 2023-10-10T17:52:47.934 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2023-10-10T17:52:47.934 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.934 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "progress", 2023-10-10T17:52:47.936 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.936 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.936 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_pg_recovery_event": { 2023-10-10T17:52:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_pg_recovery_event", 2023-10-10T17:52:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.937 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.938 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow the module to show pg recovery progress", 2023-10-10T17:52:47.939 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.940 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2023-10-10T17:52:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2023-10-10T17:52:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.942 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.943 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.945 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.946 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.947 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.948 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.949 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.951 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.952 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.954 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.955 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.956 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.957 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.958 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.959 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.960 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.960 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.961 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.961 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.961 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.961 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.962 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.963 INFO:teuthology.orchestra.run.smithi078.stdout: "max_completed_events": { 2023-10-10T17:52:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_completed_events", 2023-10-10T17:52:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "50", 2023-10-10T17:52:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.965 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "number of past completed events to remember", 2023-10-10T17:52:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.966 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "sleep_interval": { 2023-10-10T17:52:47.967 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "sleep_interval", 2023-10-10T17:52:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.968 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "5", 2023-10-10T17:52:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.969 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "how long the module is going to sleep", 2023-10-10T17:52:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.970 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "prometheus", 2023-10-10T17:52:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:47.972 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "cache": { 2023-10-10T17:52:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "cache", 2023-10-10T17:52:47.973 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.974 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.975 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.976 INFO:teuthology.orchestra.run.smithi078.stdout: "exclude_perf_counters": { 2023-10-10T17:52:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "exclude_perf_counters", 2023-10-10T17:52:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.977 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.978 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Do not include perf-counters in the metrics output", 2023-10-10T17:52:47.979 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2023-10-10T17:52:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.980 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.982 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.983 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.984 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:47.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.987 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.988 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.989 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:47.990 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.991 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:47.992 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:47.993 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:47.994 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.995 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:47.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:47.997 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:47.998 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:47.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools": { 2023-10-10T17:52:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools", 2023-10-10T17:52:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.000 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.001 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.002 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.003 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_stats_pools_refresh_interval": { 2023-10-10T17:52:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_stats_pools_refresh_interval", 2023-10-10T17:52:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.004 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300", 2023-10-10T17:52:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.005 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.007 INFO:teuthology.orchestra.run.smithi078.stdout: "scrape_interval": { 2023-10-10T17:52:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "scrape_interval", 2023-10-10T17:52:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15.0", 2023-10-10T17:52:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.009 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:48.011 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.012 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "::", 2023-10-10T17:52:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.013 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2023-10-10T17:52:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.014 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:48.015 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "9283", 2023-10-10T17:52:48.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.017 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "the port on which the module listens for HTTP requests", 2023-10-10T17:52:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "stale_cache_strategy": { 2023-10-10T17:52:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stale_cache_strategy", 2023-10-10T17:52:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.019 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "log", 2023-10-10T17:52:48.020 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.021 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.022 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_behaviour": { 2023-10-10T17:52:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_behaviour", 2023-10-10T17:52:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.023 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "default", 2023-10-10T17:52:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.024 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "default", 2023-10-10T17:52:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: "error" 2023-10-10T17:52:48.025 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: "standby_error_status_code": { 2023-10-10T17:52:48.027 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "standby_error_status_code", 2023-10-10T17:52:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.028 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "500", 2023-10-10T17:52:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "400", 2023-10-10T17:52:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "599", 2023-10-10T17:52:48.029 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.030 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2023-10-10T17:52:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.032 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.033 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.034 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.035 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.036 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.037 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.040 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.041 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.043 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.044 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.044 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.044 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.045 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.045 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.045 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.046 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.047 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.047 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.047 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.048 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.048 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.049 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.050 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.050 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.051 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.051 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.051 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.052 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.052 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_snap_create": { 2023-10-10T17:52:48.053 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_snap_create", 2023-10-10T17:52:48.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.053 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10", 2023-10-10T17:52:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.054 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.056 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.056 INFO:teuthology.orchestra.run.smithi078.stdout: "mirror_snapshot_schedule": { 2023-10-10T17:52:48.056 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mirror_snapshot_schedule", 2023-10-10T17:52:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.057 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.058 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.058 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.058 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.059 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.059 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.059 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: "trash_purge_schedule": { 2023-10-10T17:52:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "trash_purge_schedule", 2023-10-10T17:52:48.060 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.061 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.061 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.062 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.062 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.062 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.062 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "restful", 2023-10-10T17:52:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.066 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.067 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.067 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_auth": { 2023-10-10T17:52:48.067 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enable_auth", 2023-10-10T17:52:48.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.068 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.068 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:48.069 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.069 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.069 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.069 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.070 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.070 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.070 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.071 INFO:teuthology.orchestra.run.smithi078.stdout: "key_file": { 2023-10-10T17:52:48.071 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "key_file", 2023-10-10T17:52:48.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.071 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.072 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.073 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.073 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.073 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.073 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.074 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.074 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.075 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.075 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.076 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.076 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.076 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.077 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.077 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.077 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.077 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.078 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.078 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.078 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.079 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.079 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.079 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.081 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.082 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.082 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.082 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.082 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.083 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.083 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.083 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.084 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.085 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.085 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.086 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.087 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.087 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.087 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.088 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.091 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.091 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.091 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.092 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.092 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.092 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.092 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.093 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.094 INFO:teuthology.orchestra.run.smithi078.stdout: "server_addr": { 2023-10-10T17:52:48.094 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_addr", 2023-10-10T17:52:48.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.094 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.095 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.095 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.095 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.096 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.096 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.096 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.097 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.097 INFO:teuthology.orchestra.run.smithi078.stdout: "server_port": { 2023-10-10T17:52:48.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "server_port", 2023-10-10T17:52:48.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.098 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.099 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.099 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.099 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.100 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.100 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.100 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.101 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rgw", 2023-10-10T17:52:48.102 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.103 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.103 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.103 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.104 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.104 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.104 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.105 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.105 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.105 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.106 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.106 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.106 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.106 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.107 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.107 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.107 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.107 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.108 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.108 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.109 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.109 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.110 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.110 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.111 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.111 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.111 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.112 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.112 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.112 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.114 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.115 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.115 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.115 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.115 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.116 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.116 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.116 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.117 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.117 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.117 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.117 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.118 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.118 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.118 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.119 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.119 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.120 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.120 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.120 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.120 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.121 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.121 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.121 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.121 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.122 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.123 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rook", 2023-10-10T17:52:48.124 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.124 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.124 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.124 INFO:teuthology.orchestra.run.smithi078.stdout: "drive_group_interval": { 2023-10-10T17:52:48.125 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "drive_group_interval", 2023-10-10T17:52:48.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:48.125 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.125 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.126 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "300.0", 2023-10-10T17:52:48.126 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.126 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.127 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.127 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2023-10-10T17:52:48.127 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.128 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.128 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.129 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.130 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.130 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.130 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.130 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.131 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.131 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.131 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.132 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.133 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.134 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.134 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.135 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.135 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.136 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.136 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.136 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.137 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.137 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.137 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.138 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.139 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.139 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.139 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.140 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.140 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.140 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.140 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.141 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.141 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.141 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.142 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.142 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.142 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.143 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.143 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.144 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.145 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.146 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.147 INFO:teuthology.orchestra.run.smithi078.stdout: "storage_class": { 2023-10-10T17:52:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "storage_class", 2023-10-10T17:52:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "local", 2023-10-10T17:52:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.149 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "storage class name for LSO-discovered PVs", 2023-10-10T17:52:48.150 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "selftest", 2023-10-10T17:52:48.152 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.153 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.153 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.153 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.155 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:43 smithi190 ceph-mon[101220]: pgmap v316: 129 pgs: 12 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 90 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:48.155 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:43 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3035753081' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-10-10T17:52:48.155 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:46 smithi190 ceph-mon[101220]: pgmap v317: 129 pgs: 12 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 90 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-10T17:52:48.156 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:43 smithi157 ceph-mon[100896]: pgmap v316: 129 pgs: 12 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 90 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:48.156 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:43 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3035753081' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-10-10T17:52:48.156 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:46 smithi157 ceph-mon[100896]: pgmap v317: 129 pgs: 12 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 90 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-10T17:52:48.157 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.157 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.158 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.158 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.158 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.159 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.160 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.160 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.160 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.161 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.161 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.161 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.162 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.162 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.163 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.163 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.164 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.164 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.164 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.164 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.165 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.165 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.165 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.166 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.166 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.166 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.167 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.168 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.169 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.169 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.169 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.169 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.170 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.170 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.170 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.170 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.171 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.171 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.171 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.172 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.172 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.173 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.173 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.173 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.174 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.174 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.174 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.175 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.175 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.175 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.176 INFO:teuthology.orchestra.run.smithi078.stdout: "roption1": { 2023-10-10T17:52:48.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption1", 2023-10-10T17:52:48.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.176 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.177 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.177 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.177 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.178 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.178 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.178 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.179 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.179 INFO:teuthology.orchestra.run.smithi078.stdout: "roption2": { 2023-10-10T17:52:48.180 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "roption2", 2023-10-10T17:52:48.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.180 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.181 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "xyz", 2023-10-10T17:52:48.181 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.181 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.182 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.182 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.182 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.183 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.183 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption1": { 2023-10-10T17:52:48.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption1", 2023-10-10T17:52:48.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.184 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.185 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.185 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.185 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.186 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.186 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.186 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.186 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.187 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.187 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption2": { 2023-10-10T17:52:48.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption2", 2023-10-10T17:52:48.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.188 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.188 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.189 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.189 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.189 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.189 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.190 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.190 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.190 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.191 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.191 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption3": { 2023-10-10T17:52:48.191 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption3", 2023-10-10T17:52:48.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "float", 2023-10-10T17:52:48.192 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.192 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.193 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.193 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.193 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.193 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.194 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.194 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.195 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption4": { 2023-10-10T17:52:48.195 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption4", 2023-10-10T17:52:48.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.196 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.197 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.198 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.198 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.199 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption5": { 2023-10-10T17:52:48.199 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption5", 2023-10-10T17:52:48.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.199 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.200 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.200 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.200 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.201 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.201 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.201 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.202 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.202 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption6": { 2023-10-10T17:52:48.203 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption6", 2023-10-10T17:52:48.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.203 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.203 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.204 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:48.204 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.204 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.205 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.205 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.205 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.206 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.206 INFO:teuthology.orchestra.run.smithi078.stdout: "rwoption7": { 2023-10-10T17:52:48.207 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rwoption7", 2023-10-10T17:52:48.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.207 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.208 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.208 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "1", 2023-10-10T17:52:48.208 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "42", 2023-10-10T17:52:48.208 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.209 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.209 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.209 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.210 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.210 INFO:teuthology.orchestra.run.smithi078.stdout: "testkey": { 2023-10-10T17:52:48.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testkey", 2023-10-10T17:52:48.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.211 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.211 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.212 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.212 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.212 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.212 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.213 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.213 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.213 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.214 INFO:teuthology.orchestra.run.smithi078.stdout: "testlkey": { 2023-10-10T17:52:48.214 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testlkey", 2023-10-10T17:52:48.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.215 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.215 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.215 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.216 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.216 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.216 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.216 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.217 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.217 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.218 INFO:teuthology.orchestra.run.smithi078.stdout: "testnewline": { 2023-10-10T17:52:48.218 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "testnewline", 2023-10-10T17:52:48.218 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.219 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.219 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.219 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.220 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.220 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.220 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.220 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.221 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.222 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2023-10-10T17:52:48.223 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.223 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.223 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.224 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_m_granularity": { 2023-10-10T17:52:48.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "allow_m_granularity", 2023-10-10T17:52:48.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.224 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.225 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.225 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.226 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.226 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.226 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "allow minute scheduled snapshots", 2023-10-10T17:52:48.226 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.227 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.227 INFO:teuthology.orchestra.run.smithi078.stdout: "dump_on_update": { 2023-10-10T17:52:48.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "dump_on_update", 2023-10-10T17:52:48.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.228 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.229 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "dump database to debug log on update", 2023-10-10T17:52:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.231 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.233 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.234 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.235 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.236 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.236 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.236 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.237 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.238 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.239 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.239 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.239 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.239 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.240 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.240 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.240 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.241 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.241 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.242 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.242 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.242 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.243 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.243 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.244 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.244 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.244 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.245 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.245 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.245 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.245 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.246 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.246 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.248 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.249 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.251 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "stats", 2023-10-10T17:52:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.252 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.254 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.255 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.256 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.257 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.257 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.257 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.258 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.259 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.259 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.260 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.260 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.260 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.260 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.261 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.261 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.262 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.262 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.263 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.263 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.263 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.264 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.264 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.264 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.265 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.265 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.265 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.265 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.266 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.266 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.266 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.266 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.269 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.270 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.270 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.271 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.271 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.272 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.272 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.272 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.272 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.273 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.273 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "status", 2023-10-10T17:52:48.275 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.275 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.275 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.276 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.277 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.277 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.277 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.277 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.278 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.278 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.278 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.279 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.279 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.279 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.279 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.280 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.280 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.280 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.281 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.281 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.281 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.282 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.282 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.283 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.283 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.283 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.283 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.284 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.284 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.284 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.285 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.285 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.286 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.286 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.287 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.287 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.287 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.287 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.288 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.288 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.288 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.288 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.289 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.291 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.291 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.291 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.292 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.292 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.293 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.293 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.293 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.293 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.294 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telegraf", 2023-10-10T17:52:48.296 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.296 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.296 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.296 INFO:teuthology.orchestra.run.smithi078.stdout: "address": { 2023-10-10T17:52:48.297 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "address", 2023-10-10T17:52:48.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.297 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.298 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2023-10-10T17:52:48.298 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.298 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.299 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.299 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.299 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.300 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.300 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:48.301 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:48.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:48.301 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.301 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.302 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "15", 2023-10-10T17:52:48.302 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.302 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.302 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.303 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.303 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.304 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.304 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.304 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.305 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.305 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.306 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.306 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.306 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.307 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.307 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.307 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.307 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.308 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.308 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.308 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.309 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.309 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.309 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.310 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.310 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.311 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.311 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.312 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.312 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.312 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.312 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.313 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.313 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.313 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.314 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.314 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.315 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.315 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.315 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.316 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.316 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.316 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.316 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.317 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.317 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.317 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.318 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.318 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.318 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.318 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.319 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.319 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.320 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.320 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.320 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.321 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.321 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.321 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.322 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.322 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.322 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.323 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "telemetry", 2023-10-10T17:52:48.324 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.325 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.325 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.325 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_basic": { 2023-10-10T17:52:48.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_basic", 2023-10-10T17:52:48.326 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.326 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.326 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.327 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:48.327 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.327 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.327 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.328 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share basic cluster information (size, version)", 2023-10-10T17:52:48.328 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.328 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.328 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.329 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_crash": { 2023-10-10T17:52:48.329 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_crash", 2023-10-10T17:52:48.330 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.330 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.330 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.331 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2023-10-10T17:52:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.332 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.333 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_device": { 2023-10-10T17:52:48.333 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_device", 2023-10-10T17:52:48.333 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.334 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.334 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "True", 2023-10-10T17:52:48.335 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.335 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.335 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.335 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2023-10-10T17:52:48.336 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.336 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.337 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_ident": { 2023-10-10T17:52:48.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_ident", 2023-10-10T17:52:48.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.338 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.338 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.338 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.339 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.339 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.339 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.339 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2023-10-10T17:52:48.340 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.340 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.341 INFO:teuthology.orchestra.run.smithi078.stdout: "channel_perf": { 2023-10-10T17:52:48.341 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "channel_perf", 2023-10-10T17:52:48.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.342 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.342 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.342 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.342 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.343 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Share various performance metrics of a cluster", 2023-10-10T17:52:48.344 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.344 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.344 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.345 INFO:teuthology.orchestra.run.smithi078.stdout: "contact": { 2023-10-10T17:52:48.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "contact", 2023-10-10T17:52:48.345 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.345 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.346 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.346 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.347 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.347 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.347 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.347 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.348 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.349 INFO:teuthology.orchestra.run.smithi078.stdout: "description": { 2023-10-10T17:52:48.349 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "description", 2023-10-10T17:52:48.349 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.349 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.350 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.350 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.351 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.351 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.351 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.351 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.352 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.353 INFO:teuthology.orchestra.run.smithi078.stdout: "device_url": { 2023-10-10T17:52:48.353 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "device_url", 2023-10-10T17:52:48.353 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.353 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.354 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.354 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/device", 2023-10-10T17:52:48.354 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.355 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.355 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.355 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.355 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.356 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.356 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.357 INFO:teuthology.orchestra.run.smithi078.stdout: "enabled": { 2023-10-10T17:52:48.357 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "enabled", 2023-10-10T17:52:48.357 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.357 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.358 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.358 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.358 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.359 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.359 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.359 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.360 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.360 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.360 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:48.361 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:48.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.361 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.362 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "24", 2023-10-10T17:52:48.362 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "8", 2023-10-10T17:52:48.362 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.363 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.363 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.363 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.363 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.364 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.364 INFO:teuthology.orchestra.run.smithi078.stdout: "last_opt_revision": { 2023-10-10T17:52:48.365 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "last_opt_revision", 2023-10-10T17:52:48.365 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.365 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.365 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.366 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "1", 2023-10-10T17:52:48.366 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.366 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.367 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.367 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.367 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.368 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.368 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.369 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard": { 2023-10-10T17:52:48.370 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard", 2023-10-10T17:52:48.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.371 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.371 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.372 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.372 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.373 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.373 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.374 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.374 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.375 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.377 INFO:teuthology.orchestra.run.smithi078.stdout: "leaderboard_description": { 2023-10-10T17:52:48.377 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "leaderboard_description", 2023-10-10T17:52:48.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.378 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.379 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.380 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.380 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.381 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.381 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.384 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.385 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.386 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.387 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.388 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.388 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.389 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.389 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.390 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.390 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.391 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.392 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.394 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.394 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.395 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.395 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.396 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.396 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.397 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.397 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.398 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.398 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.399 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.399 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.400 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.402 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.403 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.404 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.404 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.404 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.405 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.405 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.406 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.406 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.407 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.407 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.408 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.408 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.410 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.411 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.412 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.413 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.414 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.415 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.416 INFO:teuthology.orchestra.run.smithi078.stdout: "organization": { 2023-10-10T17:52:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "organization", 2023-10-10T17:52:48.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.419 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.420 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.421 INFO:teuthology.orchestra.run.smithi078.stdout: "proxy": { 2023-10-10T17:52:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "proxy", 2023-10-10T17:52:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.422 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.424 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.424 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.425 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.425 INFO:teuthology.orchestra.run.smithi078.stdout: "url": { 2023-10-10T17:52:48.425 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "url", 2023-10-10T17:52:48.426 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.426 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.426 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.427 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "https://telemetry.ceph.com/report", 2023-10-10T17:52:48.427 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.427 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.427 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.428 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.428 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.428 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.430 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "test_orchestrator", 2023-10-10T17:52:48.430 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.430 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.431 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.431 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.431 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.432 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.432 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.433 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.433 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.433 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.433 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.434 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.434 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.434 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.434 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.435 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.435 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.435 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.435 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.436 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.437 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.437 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.438 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.438 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.439 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.439 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.439 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.440 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.440 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.441 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.442 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.442 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.442 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.443 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.443 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.443 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.443 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.444 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.444 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.444 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.444 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.445 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.445 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.445 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.446 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.446 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.447 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.447 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.448 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.448 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.448 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.448 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.449 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.449 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.450 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.451 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2023-10-10T17:52:48.451 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.451 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.451 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.452 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.452 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.453 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.453 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.453 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.454 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.454 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.454 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.454 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.455 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.455 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.455 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.456 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.456 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.456 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.456 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.457 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.457 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.458 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.458 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.459 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.459 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.459 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.459 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.460 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.460 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.460 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.461 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.463 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.464 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.465 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.465 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.465 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.466 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.467 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.467 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.469 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.470 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.470 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.471 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.471 INFO:teuthology.orchestra.run.smithi078.stdout: "max_concurrent_clones": { 2023-10-10T17:52:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "max_concurrent_clones", 2023-10-10T17:52:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.473 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.474 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "4", 2023-10-10T17:52:48.474 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Number of asynchronous cloner threads", 2023-10-10T17:52:48.475 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.477 INFO:teuthology.orchestra.run.smithi078.stdout: "snapshot_clone_delay": { 2023-10-10T17:52:48.477 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snapshot_clone_delay", 2023-10-10T17:52:48.477 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.477 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.478 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "0", 2023-10-10T17:52:48.478 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.479 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.479 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.479 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2023-10-10T17:52:48.479 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.480 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.481 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix", 2023-10-10T17:52:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "can_run": true, 2023-10-10T17:52:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "error_string": "", 2023-10-10T17:52:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "module_options": { 2023-10-10T17:52:48.482 INFO:teuthology.orchestra.run.smithi078.stdout: "discovery_interval": { 2023-10-10T17:52:48.483 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "discovery_interval", 2023-10-10T17:52:48.483 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "uint", 2023-10-10T17:52:48.483 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.484 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "100", 2023-10-10T17:52:48.484 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.484 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.485 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.485 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.485 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.486 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.486 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.486 INFO:teuthology.orchestra.run.smithi078.stdout: "identifier": { 2023-10-10T17:52:48.487 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "identifier", 2023-10-10T17:52:48.487 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.487 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.488 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.488 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.488 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.490 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.490 INFO:teuthology.orchestra.run.smithi078.stdout: "interval": { 2023-10-10T17:52:48.491 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "interval", 2023-10-10T17:52:48.491 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "secs", 2023-10-10T17:52:48.491 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.491 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.492 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "60", 2023-10-10T17:52:48.492 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.492 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.492 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.493 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.493 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.493 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.494 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.494 INFO:teuthology.orchestra.run.smithi078.stdout: "log_level": { 2023-10-10T17:52:48.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_level", 2023-10-10T17:52:48.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.495 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.495 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.496 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.496 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.496 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.497 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.497 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.497 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.497 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.498 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.498 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.498 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.499 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.499 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.499 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.500 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster": { 2023-10-10T17:52:48.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster", 2023-10-10T17:52:48.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.501 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.501 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.502 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.502 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.502 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.502 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.503 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.503 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.504 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_cluster_level": { 2023-10-10T17:52:48.504 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_cluster_level", 2023-10-10T17:52:48.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.505 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.505 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.505 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "info", 2023-10-10T17:52:48.505 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.506 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.506 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [ 2023-10-10T17:52:48.506 INFO:teuthology.orchestra.run.smithi078.stdout: "", 2023-10-10T17:52:48.506 INFO:teuthology.orchestra.run.smithi078.stdout: "critical", 2023-10-10T17:52:48.507 INFO:teuthology.orchestra.run.smithi078.stdout: "debug", 2023-10-10T17:52:48.507 INFO:teuthology.orchestra.run.smithi078.stdout: "error", 2023-10-10T17:52:48.507 INFO:teuthology.orchestra.run.smithi078.stdout: "info", 2023-10-10T17:52:48.508 INFO:teuthology.orchestra.run.smithi078.stdout: "warning" 2023-10-10T17:52:48.508 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.508 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.508 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.509 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.509 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": { 2023-10-10T17:52:48.510 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "log_to_file", 2023-10-10T17:52:48.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "bool", 2023-10-10T17:52:48.510 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2023-10-10T17:52:48.511 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "False", 2023-10-10T17:52:48.511 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.511 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.512 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.512 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.512 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.513 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.513 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_host": { 2023-10-10T17:52:48.514 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_host", 2023-10-10T17:52:48.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.514 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.515 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "", 2023-10-10T17:52:48.515 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.515 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.516 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.516 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.516 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_port": { 2023-10-10T17:52:48.517 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_port", 2023-10-10T17:52:48.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "int", 2023-10-10T17:52:48.518 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.519 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "10051", 2023-10-10T17:52:48.519 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.519 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.519 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.520 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.520 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.520 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.521 INFO:teuthology.orchestra.run.smithi078.stdout: "zabbix_sender": { 2023-10-10T17:52:48.521 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "zabbix_sender", 2023-10-10T17:52:48.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "str", 2023-10-10T17:52:48.522 INFO:teuthology.orchestra.run.smithi078.stdout: "level": "advanced", 2023-10-10T17:52:48.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 0, 2023-10-10T17:52:48.522 INFO:teuthology.orchestra.run.smithi078.stdout: "default_value": "/usr/bin/zabbix_sender", 2023-10-10T17:52:48.523 INFO:teuthology.orchestra.run.smithi078.stdout: "min": "", 2023-10-10T17:52:48.523 INFO:teuthology.orchestra.run.smithi078.stdout: "max": "", 2023-10-10T17:52:48.523 INFO:teuthology.orchestra.run.smithi078.stdout: "enum_allowed": [], 2023-10-10T17:52:48.524 INFO:teuthology.orchestra.run.smithi078.stdout: "desc": "", 2023-10-10T17:52:48.524 INFO:teuthology.orchestra.run.smithi078.stdout: "long_desc": "", 2023-10-10T17:52:48.524 INFO:teuthology.orchestra.run.smithi078.stdout: "tags": [], 2023-10-10T17:52:48.524 INFO:teuthology.orchestra.run.smithi078.stdout: "see_also": [] 2023-10-10T17:52:48.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.525 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.526 INFO:teuthology.orchestra.run.smithi078.stdout: "services": { 2023-10-10T17:52:48.526 INFO:teuthology.orchestra.run.smithi078.stdout: "dashboard": "https://172.21.15.78:8443/" 2023-10-10T17:52:48.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.527 INFO:teuthology.orchestra.run.smithi078.stdout: "always_on_modules": { 2023-10-10T17:52:48.527 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus": [ 2023-10-10T17:52:48.527 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2023-10-10T17:52:48.527 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2023-10-10T17:52:48.528 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2023-10-10T17:52:48.528 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2023-10-10T17:52:48.528 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2023-10-10T17:52:48.528 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2023-10-10T17:52:48.529 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2023-10-10T17:52:48.529 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2023-10-10T17:52:48.529 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2023-10-10T17:52:48.529 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2023-10-10T17:52:48.530 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.530 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific": [ 2023-10-10T17:52:48.530 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2023-10-10T17:52:48.531 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2023-10-10T17:52:48.531 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2023-10-10T17:52:48.531 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2023-10-10T17:52:48.531 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2023-10-10T17:52:48.532 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2023-10-10T17:52:48.532 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2023-10-10T17:52:48.532 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2023-10-10T17:52:48.533 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2023-10-10T17:52:48.533 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2023-10-10T17:52:48.533 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.533 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy": [ 2023-10-10T17:52:48.534 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2023-10-10T17:52:48.534 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2023-10-10T17:52:48.534 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2023-10-10T17:52:48.534 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2023-10-10T17:52:48.535 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2023-10-10T17:52:48.535 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2023-10-10T17:52:48.535 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2023-10-10T17:52:48.535 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2023-10-10T17:52:48.536 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2023-10-10T17:52:48.536 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2023-10-10T17:52:48.536 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2023-10-10T17:52:48.537 INFO:teuthology.orchestra.run.smithi078.stdout: "reef": [ 2023-10-10T17:52:48.537 INFO:teuthology.orchestra.run.smithi078.stdout: "balancer", 2023-10-10T17:52:48.537 INFO:teuthology.orchestra.run.smithi078.stdout: "crash", 2023-10-10T17:52:48.537 INFO:teuthology.orchestra.run.smithi078.stdout: "devicehealth", 2023-10-10T17:52:48.538 INFO:teuthology.orchestra.run.smithi078.stdout: "orchestrator", 2023-10-10T17:52:48.538 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscaler", 2023-10-10T17:52:48.538 INFO:teuthology.orchestra.run.smithi078.stdout: "progress", 2023-10-10T17:52:48.539 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_support", 2023-10-10T17:52:48.539 INFO:teuthology.orchestra.run.smithi078.stdout: "status", 2023-10-10T17:52:48.539 INFO:teuthology.orchestra.run.smithi078.stdout: "telemetry", 2023-10-10T17:52:48.539 INFO:teuthology.orchestra.run.smithi078.stdout: "volumes" 2023-10-10T17:52:48.540 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.540 INFO:teuthology.orchestra.run.smithi078.stdout: "last_failure_osd_epoch": 92, 2023-10-10T17:52:48.541 INFO:teuthology.orchestra.run.smithi078.stdout: "active_clients": [ 2023-10-10T17:52:48.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.541 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "devicehealth", 2023-10-10T17:52:48.541 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2023-10-10T17:52:48.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2023-10-10T17:52:48.542 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2023-10-10T17:52:48.543 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 656491535 2023-10-10T17:52:48.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.543 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.544 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "libcephsqlite", 2023-10-10T17:52:48.544 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2023-10-10T17:52:48.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.545 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2023-10-10T17:52:48.545 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2023-10-10T17:52:48.545 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2885956736 2023-10-10T17:52:48.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.547 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "rbd_support", 2023-10-10T17:52:48.547 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2023-10-10T17:52:48.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2023-10-10T17:52:48.548 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2023-10-10T17:52:48.548 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 76780899 2023-10-10T17:52:48.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.549 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.549 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2023-10-10T17:52:48.550 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2023-10-10T17:52:48.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2023-10-10T17:52:48.551 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2023-10-10T17:52:48.551 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 429382075 2023-10-10T17:52:48.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.551 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.552 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "snap_schedule", 2023-10-10T17:52:48.552 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2023-10-10T17:52:48.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2023-10-10T17:52:48.553 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2023-10-10T17:52:48.554 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1158757506 2023-10-10T17:52:48.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.554 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:48.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "volumes", 2023-10-10T17:52:48.555 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2023-10-10T17:52:48.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-10-10T17:52:48.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2023-10-10T17:52:48.556 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:0", 2023-10-10T17:52:48.556 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2851748111 2023-10-10T17:52:48.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.557 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.557 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:48.557 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-10-10T17:52:48.558 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.486+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b600a1410 msgr2=0x7f9b600a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:48.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.486+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b600a1410 0x7f9b600a38c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c009a10 tx=0x7f9b5c009190 comp rx=0 tx=0).stop 2023-10-10T17:52:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.486+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b740fee80 msgr2=0x7f9b74114980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:52:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.486+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b740fee80 0x7f9b74114980 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f9b64012bd0 tx=0x7f9b64012c00 comp rx=0 tx=0).stop 2023-10-10T17:52:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 shutdown_connections 2023-10-10T17:52:48.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9b740fe4d0 0x7f9b74114420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f9b600a1410 0x7f9b600a38c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9b741044e0 0x7f9b7410f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:48.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 --2- 172.21.15.78:0/3035753081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9b740fee80 0x7f9b74114980 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:52:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 >> 172.21.15.78:0/3035753081 conn(0x7f9b740742a0 msgr2=0x7f9b74101fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:52:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 shutdown_connections 2023-10-10T17:52:48.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:52:43.488+0000 7f9b7a8ea700 1 -- 172.21.15.78:0/3035753081 wait complete. 2023-10-10T17:52:48.568 DEBUG:tasks.check_counter:Getting stats from a 2023-10-10T17:52:48.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.a perf dump 2023-10-10T17:52:48.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:43 smithi078 ceph-mon[99494]: pgmap v316: 129 pgs: 12 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 90 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:48.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:43 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3035753081' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2023-10-10T17:52:48.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:46 smithi078 ceph-mon[99494]: pgmap v317: 129 pgs: 12 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 90 active+clean; 799 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-10T17:52:48.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:48 smithi157 ceph-mon[100896]: pgmap v318: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-10T17:52:48.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:48 smithi078 ceph-mon[99494]: pgmap v318: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-10T17:52:48.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:48 smithi190 ceph-mon[101220]: pgmap v318: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2023-10-10T17:52:49.244 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:52:51.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:50 smithi078 ceph-mon[99494]: pgmap v319: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-10T17:52:51.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:50 smithi190 ceph-mon[101220]: pgmap v319: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-10T17:52:51.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:50 smithi157 ceph-mon[100896]: pgmap v319: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2023-10-10T17:52:51.832 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:51.832 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:52:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2023-10-10T17:52:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2023-10-10T17:52:51.833 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2023-10-10T17:52:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2023-10-10T17:52:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2023-10-10T17:52:51.834 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2023-10-10T17:52:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.001297945, 2023-10-10T17:52:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2023-10-10T17:52:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000282538, 2023-10-10T17:52:51.835 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:51.836 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:51.837 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.838 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-10T17:52:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2023-10-10T17:52:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.839 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:52:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 210, 2023-10-10T17:52:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 201, 2023-10-10T17:52:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1013536, 2023-10-10T17:52:51.840 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 113680, 2023-10-10T17:52:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2023-10-10T17:52:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2023-10-10T17:52:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.033595820, 2023-10-10T17:52:51.841 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.011870089, 2023-10-10T17:52:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.021786846, 2023-10-10T17:52:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004389194, 2023-10-10T17:52:51.842 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 201, 2023-10-10T17:52:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.018191610, 2023-10-10T17:52:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000090505 2023-10-10T17:52:51.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:51.844 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1013536, 2023-10-10T17:52:51.845 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 113680 2023-10-10T17:52:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:52:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 2, 2023-10-10T17:52:51.846 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 152, 2023-10-10T17:52:51.847 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 320, 2023-10-10T17:52:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 178480, 2023-10-10T17:52:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 5, 2023-10-10T17:52:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2023-10-10T17:52:51.848 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.011132395, 2023-10-10T17:52:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.007296163, 2023-10-10T17:52:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001815662, 2023-10-10T17:52:51.849 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 149, 2023-10-10T17:52:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.016095960, 2023-10-10T17:52:51.850 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000108026 2023-10-10T17:52:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:51.851 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.852 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:52:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 178480 2023-10-10T17:52:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2023-10-10T17:52:51.853 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2023-10-10T17:52:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2023-10-10T17:52:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.854 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2023-10-10T17:52:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:52:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2023-10-10T17:52:51.855 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:52:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:52:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:52:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:52:51.856 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:52:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:52:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:52:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:52:51.857 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:52:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:52:51.858 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:52:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:52:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:52:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2023-10-10T17:52:51.859 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:52:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2023-10-10T17:52:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:52:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:52:51.860 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:52:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:52:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:52:51.861 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:52:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:52:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2023-10-10T17:52:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:52:51.862 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:52:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:52:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2023-10-10T17:52:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2023-10-10T17:52:51.863 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2023-10-10T17:52:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:52:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:52:51.864 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:52:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:52:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148977, 2023-10-10T17:52:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 30, 2023-10-10T17:52:51.865 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:52:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2023-10-10T17:52:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2023-10-10T17:52:51.866 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2023-10-10T17:52:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:52:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2023-10-10T17:52:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:52:51.867 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2023-10-10T17:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2023-10-10T17:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2023-10-10T17:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:52:51.868 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2023-10-10T17:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2023-10-10T17:52:51.869 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2023-10-10T17:52:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2023-10-10T17:52:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2023-10-10T17:52:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2023-10-10T17:52:51.870 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2023-10-10T17:52:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2023-10-10T17:52:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:52:51.871 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2023-10-10T17:52:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2023-10-10T17:52:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2023-10-10T17:52:51.872 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 208, 2023-10-10T17:52:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 981474, 2023-10-10T17:52:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2023-10-10T17:52:51.873 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 208, 2023-10-10T17:52:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2023-10-10T17:52:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2023-10-10T17:52:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 208, 2023-10-10T17:52:51.874 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 981474, 2023-10-10T17:52:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2023-10-10T17:52:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:51.875 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:51.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:51.876 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:52.209 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2023-10-10T17:52:52.210 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2023-10-10T17:52:52.210 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2023-10-10T17:52:52.210 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2023-10-10T17:52:52.211 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2023-10-10T17:52:52.211 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2023-10-10T17:52:52.211 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2023-10-10T17:52:52.212 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2023-10-10T17:52:52.212 DEBUG:tasks.check_counter:Getting stats from d 2023-10-10T17:52:52.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.d perf dump 2023-10-10T17:52:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:52:53.079 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:52 smithi078 ceph-mon[99494]: pgmap v320: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:53.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:52 smithi190 ceph-mon[101220]: pgmap v320: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:53.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:52 smithi157 ceph-mon[100896]: pgmap v320: 129 pgs: 5 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 117 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:54.198 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:54.198 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:52:54.199 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 4, 2023-10-10T17:52:54.199 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 4, 2023-10-10T17:52:54.199 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 2992, 2023-10-10T17:52:54.199 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 448, 2023-10-10T17:52:54.200 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2023-10-10T17:52:54.200 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2023-10-10T17:52:54.200 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.001855248, 2023-10-10T17:52:54.201 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000053540, 2023-10-10T17:52:54.201 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000756918, 2023-10-10T17:52:54.201 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:54.201 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:54.202 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 3, 2023-10-10T17:52:54.202 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000855904, 2023-10-10T17:52:54.203 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000285301 2023-10-10T17:52:54.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.203 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:54.206 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:54.206 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:54.207 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:54.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.207 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-10T17:52:54.207 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 448 2023-10-10T17:52:54.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.208 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:52:54.208 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 2, 2023-10-10T17:52:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 155, 2023-10-10T17:52:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 320, 2023-10-10T17:52:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 179008, 2023-10-10T17:52:54.209 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 5, 2023-10-10T17:52:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2023-10-10T17:52:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.011362682, 2023-10-10T17:52:54.210 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.007607190, 2023-10-10T17:52:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001740115, 2023-10-10T17:52:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:54.211 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 152, 2023-10-10T17:52:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.035380140, 2023-10-10T17:52:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000232764 2023-10-10T17:52:54.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:54.213 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.214 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:52:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 179008 2023-10-10T17:52:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:52:54.215 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 210, 2023-10-10T17:52:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 203, 2023-10-10T17:52:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1011632, 2023-10-10T17:52:54.216 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 115600, 2023-10-10T17:52:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2023-10-10T17:52:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2023-10-10T17:52:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.035792971, 2023-10-10T17:52:54.217 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.012905416, 2023-10-10T17:52:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.022593330, 2023-10-10T17:52:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004901123, 2023-10-10T17:52:54.218 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 202, 2023-10-10T17:52:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.013880313, 2023-10-10T17:52:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000068714 2023-10-10T17:52:54.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:54.220 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1011632, 2023-10-10T17:52:54.221 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 115600 2023-10-10T17:52:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.222 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2023-10-10T17:52:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2023-10-10T17:52:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2023-10-10T17:52:54.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2023-10-10T17:52:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:52:54.224 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2023-10-10T17:52:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:52:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:52:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:52:54.225 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:52:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:52:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:52:54.226 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:52:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:52:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:52:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:52:54.227 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:52:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:52:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:52:54.228 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2023-10-10T17:52:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:52:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2023-10-10T17:52:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:52:54.229 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:52:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:52:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:52:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:52:54.230 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:52:54.231 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:52:54.231 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2023-10-10T17:52:54.231 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:52:54.232 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:52:54.232 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:52:54.232 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2023-10-10T17:52:54.232 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2023-10-10T17:52:54.233 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2023-10-10T17:52:54.233 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:52:54.233 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:52:54.233 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:52:54.234 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:52:54.234 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 150008, 2023-10-10T17:52:54.234 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 31, 2023-10-10T17:52:54.235 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:52:54.235 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2023-10-10T17:52:54.235 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2023-10-10T17:52:54.236 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2023-10-10T17:52:54.236 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:52:54.236 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2023-10-10T17:52:54.237 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:52:54.237 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2023-10-10T17:52:54.237 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2023-10-10T17:52:54.238 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2023-10-10T17:52:54.238 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:52:54.239 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:52:54.239 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2023-10-10T17:52:54.240 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2023-10-10T17:52:54.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2023-10-10T17:52:54.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2023-10-10T17:52:54.241 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2023-10-10T17:52:54.242 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2023-10-10T17:52:54.242 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2023-10-10T17:52:54.242 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2023-10-10T17:52:54.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:54.243 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:52:54.243 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2023-10-10T17:52:54.243 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2023-10-10T17:52:54.244 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2023-10-10T17:52:54.244 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 212, 2023-10-10T17:52:54.244 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 981994, 2023-10-10T17:52:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2023-10-10T17:52:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 212, 2023-10-10T17:52:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2023-10-10T17:52:54.245 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2023-10-10T17:52:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 212, 2023-10-10T17:52:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 981994, 2023-10-10T17:52:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2023-10-10T17:52:54.246 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:54.247 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:54.247 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:54.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:54.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:54.248 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:54.612 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2023-10-10T17:52:54.612 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2023-10-10T17:52:54.612 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2023-10-10T17:52:54.613 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2023-10-10T17:52:54.613 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2023-10-10T17:52:54.613 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2023-10-10T17:52:54.614 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2023-10-10T17:52:54.614 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2023-10-10T17:52:54.614 DEBUG:tasks.check_counter:Getting stats from g 2023-10-10T17:52:54.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.g perf dump 2023-10-10T17:52:54.877 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:54 smithi078 ceph-mon[99494]: pgmap v321: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:55.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:54 smithi190 ceph-mon[101220]: pgmap v321: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:55.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:54 smithi157 ceph-mon[100896]: pgmap v321: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2023-10-10T17:52:55.255 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:52:56.465 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:56.466 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:52:56.466 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 4, 2023-10-10T17:52:56.466 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 4, 2023-10-10T17:52:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 2992, 2023-10-10T17:52:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 448, 2023-10-10T17:52:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2023-10-10T17:52:56.467 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2023-10-10T17:52:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.001786632, 2023-10-10T17:52:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000039242, 2023-10-10T17:52:56.468 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000721616, 2023-10-10T17:52:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 3, 2023-10-10T17:52:56.469 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000798695, 2023-10-10T17:52:56.470 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000266231 2023-10-10T17:52:56.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.470 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:56.471 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:56.471 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:56.471 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:56.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-10T17:52:56.472 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 448 2023-10-10T17:52:56.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.473 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:52:56.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 208, 2023-10-10T17:52:56.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 201, 2023-10-10T17:52:56.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1011088, 2023-10-10T17:52:56.474 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 114416, 2023-10-10T17:52:56.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2023-10-10T17:52:56.475 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2023-10-10T17:52:56.476 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.033224992, 2023-10-10T17:52:56.476 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.012221910, 2023-10-10T17:52:56.476 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.020403507, 2023-10-10T17:52:56.477 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004527179, 2023-10-10T17:52:56.477 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:56.477 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 200, 2023-10-10T17:52:56.477 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.013216251, 2023-10-10T17:52:56.478 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000066081 2023-10-10T17:52:56.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.478 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:56.479 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:56.479 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:56.479 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:56.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.480 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1011088, 2023-10-10T17:52:56.480 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 114416 2023-10-10T17:52:56.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.480 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:52:56.481 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 2, 2023-10-10T17:52:56.481 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 154, 2023-10-10T17:52:56.481 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 320, 2023-10-10T17:52:56.482 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 178832, 2023-10-10T17:52:56.482 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 5, 2023-10-10T17:52:56.482 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2023-10-10T17:52:56.483 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.011404752, 2023-10-10T17:52:56.483 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.007420572, 2023-10-10T17:52:56.484 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001966889, 2023-10-10T17:52:56.484 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:56.484 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:56.484 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 151, 2023-10-10T17:52:56.485 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.020842703, 2023-10-10T17:52:56.485 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000138031 2023-10-10T17:52:56.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.486 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:56.486 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:56.487 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:56.487 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:56.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.487 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:52:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 178832 2023-10-10T17:52:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.488 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2023-10-10T17:52:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2023-10-10T17:52:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2023-10-10T17:52:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.489 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2023-10-10T17:52:56.490 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:52:56.490 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2023-10-10T17:52:56.490 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:52:56.491 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:52:56.491 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:52:56.491 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:52:56.491 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:52:56.492 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:52:56.492 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:52:56.492 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:52:56.493 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:52:56.493 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:52:56.493 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:52:56.493 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:52:56.494 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:52:56.494 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2023-10-10T17:52:56.494 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:52:56.494 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2023-10-10T17:52:56.495 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:52:56.495 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:52:56.495 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:52:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:52:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:52:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:52:56.496 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:52:56.497 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2023-10-10T17:52:56.497 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:52:56.497 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:52:56.498 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:52:56.498 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2023-10-10T17:52:56.498 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2023-10-10T17:52:56.498 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2023-10-10T17:52:56.499 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:52:56.499 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:52:56.499 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:52:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:52:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 149057, 2023-10-10T17:52:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 32, 2023-10-10T17:52:56.500 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:52:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2023-10-10T17:52:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2023-10-10T17:52:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2023-10-10T17:52:56.501 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:52:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2023-10-10T17:52:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:52:56.502 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2023-10-10T17:52:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2023-10-10T17:52:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2023-10-10T17:52:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:52:56.503 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:52:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2023-10-10T17:52:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2023-10-10T17:52:56.504 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2023-10-10T17:52:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2023-10-10T17:52:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2023-10-10T17:52:56.505 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2023-10-10T17:52:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2023-10-10T17:52:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2023-10-10T17:52:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:56.506 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:52:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2023-10-10T17:52:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2023-10-10T17:52:56.507 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2023-10-10T17:52:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 210, 2023-10-10T17:52:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 981734, 2023-10-10T17:52:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2023-10-10T17:52:56.508 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 210, 2023-10-10T17:52:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2023-10-10T17:52:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2023-10-10T17:52:56.509 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 210, 2023-10-10T17:52:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 981734, 2023-10-10T17:52:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2023-10-10T17:52:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:56.510 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:56.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:56.512 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:56.869 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:56 smithi078 ceph-mon[99494]: pgmap v322: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:56.903 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2023-10-10T17:52:56.904 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2023-10-10T17:52:56.904 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2023-10-10T17:52:56.904 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2023-10-10T17:52:56.905 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2023-10-10T17:52:56.905 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2023-10-10T17:52:56.905 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2023-10-10T17:52:56.906 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2023-10-10T17:52:56.906 DEBUG:tasks.check_counter:Getting stats from j 2023-10-10T17:52:56.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.j perf dump 2023-10-10T17:52:57.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:56 smithi190 ceph-mon[101220]: pgmap v322: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:57.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:56 smithi157 ceph-mon[100896]: pgmap v322: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:52:57.545 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.a/config 2023-10-10T17:52:58.991 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-10-10T17:52:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:52:58.991 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 216, 2023-10-10T17:52:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 207, 2023-10-10T17:52:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 1015168, 2023-10-10T17:52:58.992 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 117232, 2023-10-10T17:52:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 4, 2023-10-10T17:52:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 0, 2023-10-10T17:52:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.036889981, 2023-10-10T17:52:58.993 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.013595168, 2023-10-10T17:52:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.022070322, 2023-10-10T17:52:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.004650580, 2023-10-10T17:52:58.994 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 207, 2023-10-10T17:52:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.012585470, 2023-10-10T17:52:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000060799 2023-10-10T17:52:58.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:58.996 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 1015168, 2023-10-10T17:52:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 117232 2023-10-10T17:52:58.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:52:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 2, 2023-10-10T17:52:58.998 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 157, 2023-10-10T17:52:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 320, 2023-10-10T17:52:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 179360, 2023-10-10T17:52:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 5, 2023-10-10T17:52:58.999 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 3, 2023-10-10T17:52:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.011761735, 2023-10-10T17:52:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.007951530, 2023-10-10T17:52:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.001938926, 2023-10-10T17:52:59.000 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:59.001 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:59.001 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 154, 2023-10-10T17:52:59.001 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.023910724, 2023-10-10T17:52:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000155264 2023-10-10T17:52:59.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:59.003 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.004 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:52:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 179360 2023-10-10T17:52:59.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:52:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_messages": 0, 2023-10-10T17:52:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages": 2, 2023-10-10T17:52:59.006 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_bytes": 0, 2023-10-10T17:52:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_bytes": 0, 2023-10-10T17:52:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_created_connections": 2, 2023-10-10T17:52:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_active_connections": 1, 2023-10-10T17:52:59.007 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_total_time": 0.000417390, 2023-10-10T17:52:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_send_time": 0.000000000, 2023-10-10T17:52:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_recv_time": 0.000346397, 2023-10-10T17:52:59.008 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:52:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:52:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:59.009 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:52:59.010 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:59.011 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:59.011 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:59.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.012 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-10T17:52:59.012 INFO:teuthology.orchestra.run.smithi078.stdout: "msgr_send_encrypted_bytes": 0 2023-10-10T17:52:59.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.012 INFO:teuthology.orchestra.run.smithi078.stdout: "cct": { 2023-10-10T17:52:59.013 INFO:teuthology.orchestra.run.smithi078.stdout: "total_workers": 0, 2023-10-10T17:52:59.013 INFO:teuthology.orchestra.run.smithi078.stdout: "unhealthy_workers": 0 2023-10-10T17:52:59.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool": { 2023-10-10T17:52:59.014 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:52:59.014 INFO:teuthology.orchestra.run.smithi078.stdout: "bloom_filter_items": 0, 2023-10-10T17:52:59.014 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:52:59.015 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:52:59.015 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:52:59.015 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:52:59.015 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:52:59.016 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:52:59.016 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:52:59.016 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:52:59.016 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:52:59.017 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:52:59.017 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:52:59.017 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:52:59.018 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:52:59.018 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_items": 0, 2023-10-10T17:52:59.018 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:52:59.018 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blob_items": 0, 2023-10-10T17:52:59.019 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:52:59.019 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:52:59.019 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:52:59.019 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:52:59.020 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:52:59.020 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:52:59.021 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:52:59.021 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_txc_items": 0, 2023-10-10T17:52:59.021 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:52:59.022 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:52:59.022 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:52:59.022 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_writing_items": 0, 2023-10-10T17:52:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_bytes": 0, 2023-10-10T17:52:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_items": 0, 2023-10-10T17:52:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:52:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:52:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:52:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:52:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_bytes": 148994, 2023-10-10T17:52:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_anon_items": 31, 2023-10-10T17:52:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:52:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "buffer_meta_items": 0, 2023-10-10T17:52:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bytes": 0, 2023-10-10T17:52:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_items": 0, 2023-10-10T17:52:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:52:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mapbl_items": 0, 2023-10-10T17:52:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:52:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pglog_items": 0, 2023-10-10T17:52:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_bytes": 0, 2023-10-10T17:52:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_items": 0, 2023-10-10T17:52:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:52:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:52:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_bytes": 0, 2023-10-10T17:52:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pgmap_items": 0, 2023-10-10T17:52:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_bytes": 928, 2023-10-10T17:52:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_co_items": 2, 2023-10-10T17:52:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_bytes": 0, 2023-10-10T17:52:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_1_items": 0, 2023-10-10T17:52:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_bytes": 0, 2023-10-10T17:52:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "unittest_2_items": 0 2023-10-10T17:52:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-10-10T17:52:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:52:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "val": 0, 2023-10-10T17:52:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "max": 104857600, 2023-10-10T17:52:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "get_started": 0, 2023-10-10T17:52:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "get": 214, 2023-10-10T17:52:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "get_sum": 982254, 2023-10-10T17:52:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_fail": 0, 2023-10-10T17:52:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "get_or_fail_success": 214, 2023-10-10T17:52:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "take": 0, 2023-10-10T17:52:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "take_sum": 0, 2023-10-10T17:52:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "put": 214, 2023-10-10T17:52:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "put_sum": 982254, 2023-10-10T17:52:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "wait": { 2023-10-10T17:52:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "avgcount": 0, 2023-10-10T17:52:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "sum": 0.000000000, 2023-10-10T17:52:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "avgtime": 0.000000000 2023-10-10T17:52:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-10-10T17:52:59.036 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-10-10T17:52:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:58 smithi078 ceph-mon[99494]: pgmap v323: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:59.037 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:52:58 smithi078 ceph-mon[99494]: osdmap e102: 12 total, 12 up, 12 in 2023-10-10T17:52:59.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:58 smithi190 ceph-mon[101220]: pgmap v323: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:59.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:52:58 smithi190 ceph-mon[101220]: osdmap e102: 12 total, 12 up, 12 in 2023-10-10T17:52:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:58 smithi157 ceph-mon[100896]: pgmap v323: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:52:59.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:52:58 smithi157 ceph-mon[100896]: osdmap e102: 12 total, 12 up, 12 in 2023-10-10T17:52:59.510 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2023-10-10T17:52:59.511 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2023-10-10T17:52:59.511 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2023-10-10T17:52:59.511 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2023-10-10T17:52:59.512 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2023-10-10T17:52:59.512 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2023-10-10T17:52:59.512 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2023-10-10T17:52:59.513 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2023-10-10T17:52:59.513 DEBUG:tasks.check_counter:Getting stats from b 2023-10-10T17:52:59.514 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.b perf dump 2023-10-10T17:53:00.178 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:53:01.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:00 smithi078 ceph-mon[99494]: pgmap v325: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:53:01.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:00 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:53:01.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:00 smithi190 ceph-mon[101220]: pgmap v325: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:53:01.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:00 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:53:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:00 smithi157 ceph-mon[100896]: pgmap v325: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-10-10T17:53:01.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:00 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-10T17:53:01.542 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-10-10T17:53:01.542 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:01.543 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 144248, 2023-10-10T17:53:01.543 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 243841, 2023-10-10T17:53:01.544 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 61527603, 2023-10-10T17:53:01.544 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 340876484, 2023-10-10T17:53:01.545 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 15, 2023-10-10T17:53:01.545 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 4, 2023-10-10T17:53:01.546 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 8.121428164, 2023-10-10T17:53:01.547 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 3.372058985, 2023-10-10T17:53:01.547 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 121.084756605, 2023-10-10T17:53:01.548 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 1.091096645, 2023-10-10T17:53:01.548 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:01.549 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 243840, 2023-10-10T17:53:01.549 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 69.877933155, 2023-10-10T17:53:01.550 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000286572 2023-10-10T17:53:01.550 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.551 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:01.551 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 285555, 2023-10-10T17:53:01.552 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.045792396, 2023-10-10T17:53:01.552 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000160 2023-10-10T17:53:01.553 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.553 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-10T17:53:01.554 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 448 2023-10-10T17:53:01.554 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.555 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:01.555 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 37220, 2023-10-10T17:53:01.556 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 36506, 2023-10-10T17:53:01.556 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 17049734, 2023-10-10T17:53:01.557 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 145021869, 2023-10-10T17:53:01.557 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 163, 2023-10-10T17:53:01.558 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 8, 2023-10-10T17:53:01.558 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 2.320568600, 2023-10-10T17:53:01.559 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.662217073, 2023-10-10T17:53:01.559 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.438991867, 2023-10-10T17:53:01.560 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 1.131942411, 2023-10-10T17:53:01.560 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:01.561 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 36503, 2023-10-10T17:53:01.561 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 668.047480117, 2023-10-10T17:53:01.562 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.018301166 2023-10-10T17:53:01.562 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.563 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:01.563 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1237, 2023-10-10T17:53:01.564 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000184660, 2023-10-10T17:53:01.564 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000149 2023-10-10T17:53:01.565 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.565 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 402480, 2023-10-10T17:53:01.566 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 225216 2023-10-10T17:53:01.567 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.567 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:01.568 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 60444, 2023-10-10T17:53:01.568 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 59276, 2023-10-10T17:53:01.569 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 144477790, 2023-10-10T17:53:01.569 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 129755262, 2023-10-10T17:53:01.570 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 20, 2023-10-10T17:53:01.570 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 7, 2023-10-10T17:53:01.571 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 3.464407862, 2023-10-10T17:53:01.571 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 1.057145538, 2023-10-10T17:53:01.572 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 1.092583683, 2023-10-10T17:53:01.572 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 1.370994955, 2023-10-10T17:53:01.573 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:01.573 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 59275, 2023-10-10T17:53:01.574 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 73.620426933, 2023-10-10T17:53:01.574 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001242014 2023-10-10T17:53:01.575 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.575 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:01.576 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 26816, 2023-10-10T17:53:01.576 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.003719809, 2023-10-10T17:53:01.577 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000138 2023-10-10T17:53:01.577 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.578 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 1119104, 2023-10-10T17:53:01.578 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 449424 2023-10-10T17:53:01.579 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.579 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2023-10-10T17:53:01.580 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 1, 2023-10-10T17:53:01.580 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2023-10-10T17:53:01.581 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.581 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-MDSRank": { 2023-10-10T17:53:01.582 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2023-10-10T17:53:01.582 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2023-10-10T17:53:01.583 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 12231, 2023-10-10T17:53:01.583 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 105.907063502, 2023-10-10T17:53:01.584 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.008658904 2023-10-10T17:53:01.584 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.585 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.585 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-PurgeQueue": { 2023-10-10T17:53:01.586 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2023-10-10T17:53:01.586 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2023-10-10T17:53:01.587 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 28446, 2023-10-10T17:53:01.587 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 15.527006756, 2023-10-10T17:53:01.588 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000545841 2023-10-10T17:53:01.588 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.589 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.589 INFO:teuthology.orchestra.run.smithi157.stdout: "mds": { 2023-10-10T17:53:01.590 INFO:teuthology.orchestra.run.smithi157.stdout: "request": 69670, 2023-10-10T17:53:01.590 INFO:teuthology.orchestra.run.smithi157.stdout: "reply": 60124, 2023-10-10T17:53:01.593 INFO:teuthology.orchestra.run.smithi157.stdout: "reply_latency": { 2023-10-10T17:53:01.594 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 60124, 2023-10-10T17:53:01.595 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 285.601274373, 2023-10-10T17:53:01.596 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.004750204 2023-10-10T17:53:01.596 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.597 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_reply": 0, 2023-10-10T17:53:01.597 INFO:teuthology.orchestra.run.smithi157.stdout: "forward": 9546, 2023-10-10T17:53:01.598 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_complete": 621, 2023-10-10T17:53:01.598 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_keys": 0, 2023-10-10T17:53:01.599 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_commit": 11219, 2023-10-10T17:53:01.599 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_split": 174, 2023-10-10T17:53:01.600 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_merge": 22, 2023-10-10T17:53:01.600 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes": 16567, 2023-10-10T17:53:01.601 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_top": 11580, 2023-10-10T17:53:01.601 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_bottom": 4977, 2023-10-10T17:53:01.602 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pin_tail": 10, 2023-10-10T17:53:01.602 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pinned": 23, 2023-10-10T17:53:01.603 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_expired": 63147, 2023-10-10T17:53:01.603 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_with_caps": 4, 2023-10-10T17:53:01.604 INFO:teuthology.orchestra.run.smithi157.stdout: "caps": 5, 2023-10-10T17:53:01.604 INFO:teuthology.orchestra.run.smithi157.stdout: "subtrees": 4, 2023-10-10T17:53:01.605 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse": 88292, 2023-10-10T17:53:01.605 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_hit": 78291, 2023-10-10T17:53:01.606 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_forward": 9495, 2023-10-10T17:53:01.606 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_discover": 2, 2023-10-10T17:53:01.607 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_dir_fetch": 1, 2023-10-10T17:53:01.607 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_remote_ino": 0, 2023-10-10T17:53:01.608 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_lock": 0, 2023-10-10T17:53:01.608 INFO:teuthology.orchestra.run.smithi157.stdout: "load_cent": 0, 2023-10-10T17:53:01.609 INFO:teuthology.orchestra.run.smithi157.stdout: "q": 0, 2023-10-10T17:53:01.609 INFO:teuthology.orchestra.run.smithi157.stdout: "exported": 470, 2023-10-10T17:53:01.610 INFO:teuthology.orchestra.run.smithi157.stdout: "exported_inodes": 50931, 2023-10-10T17:53:01.610 INFO:teuthology.orchestra.run.smithi157.stdout: "imported": 484, 2023-10-10T17:53:01.611 INFO:teuthology.orchestra.run.smithi157.stdout: "imported_inodes": 0, 2023-10-10T17:53:01.611 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_dir_fetch": 0, 2023-10-10T17:53:01.612 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_backtrace_fetch": 12953, 2023-10-10T17:53:01.612 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_peer_discover": 0, 2023-10-10T17:53:01.613 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rfiles": 0, 2023-10-10T17:53:01.613 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rbytes": 0, 2023-10-10T17:53:01.614 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rsnaps": 6, 2023-10-10T17:53:01.614 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_fetch": 12953, 2023-10-10T17:53:01.615 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_set_tag": 0, 2023-10-10T17:53:01.615 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_repaired": 0, 2023-10-10T17:53:01.616 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_inotable_repaired": 0, 2023-10-10T17:53:01.616 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_inodes": 45, 2023-10-10T17:53:01.617 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_base_inodes": 12, 2023-10-10T17:53:01.617 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dirfrag_rstats": 33, 2023-10-10T17:53:01.617 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_file_inodes": 12908, 2023-10-10T17:53:01.617 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_inode_file_caps": 11, 2023-10-10T17:53:01.618 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_revoke": 123, 2023-10-10T17:53:01.618 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_grant": 41784, 2023-10-10T17:53:01.618 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_trunc": 1, 2023-10-10T17:53:01.619 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-10T17:53:01.619 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-10T17:53:01.619 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps": 28896, 2023-10-10T17:53:01.619 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps_dirty": 28770, 2023-10-10T17:53:01.620 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_cap_release": 336, 2023-10-10T17:53:01.620 INFO:teuthology.orchestra.run.smithi157.stdout: "process_request_cap_release": 35529 2023-10-10T17:53:01.620 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.621 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_cache": { 2023-10-10T17:53:01.621 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update": 2003, 2023-10-10T17:53:01.621 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update_receipt": 3027, 2023-10-10T17:53:01.621 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_try_discover": 1514, 2023-10-10T17:53:01.622 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_send_discover": 2188, 2023-10-10T17:53:01.622 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_handle_discover": 2036, 2023-10-10T17:53:01.622 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays": 7, 2023-10-10T17:53:01.622 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_delayed": 0, 2023-10-10T17:53:01.623 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_enqueuing": 0, 2023-10-10T17:53:01.623 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_created": 16821, 2023-10-10T17:53:01.623 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_enqueued": 16814, 2023-10-10T17:53:01.624 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_reintegrated": 0, 2023-10-10T17:53:01.624 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_migrated": 0, 2023-10-10T17:53:01.624 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_processing": 0, 2023-10-10T17:53:01.624 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_enqueued": 0, 2023-10-10T17:53:01.625 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_prioritized": 0, 2023-10-10T17:53:01.625 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_started": 0, 2023-10-10T17:53:01.625 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_completed": 0, 2023-10-10T17:53:01.625 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_enqueue_scrub": 12, 2023-10-10T17:53:01.626 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_exportdir": 498, 2023-10-10T17:53:01.626 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_flush": 0, 2023-10-10T17:53:01.626 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragmentdir": 203, 2023-10-10T17:53:01.627 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragstats": 0, 2023-10-10T17:53:01.627 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_inodestats": 0 2023-10-10T17:53:01.627 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.627 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_log": { 2023-10-10T17:53:01.628 INFO:teuthology.orchestra.run.smithi157.stdout: "evadd": 87130, 2023-10-10T17:53:01.628 INFO:teuthology.orchestra.run.smithi157.stdout: "evex": 87063, 2023-10-10T17:53:01.628 INFO:teuthology.orchestra.run.smithi157.stdout: "evtrm": 87063, 2023-10-10T17:53:01.628 INFO:teuthology.orchestra.run.smithi157.stdout: "ev": 67, 2023-10-10T17:53:01.629 INFO:teuthology.orchestra.run.smithi157.stdout: "evexg": 0, 2023-10-10T17:53:01.629 INFO:teuthology.orchestra.run.smithi157.stdout: "evexd": 33, 2023-10-10T17:53:01.629 INFO:teuthology.orchestra.run.smithi157.stdout: "segadd": 138, 2023-10-10T17:53:01.630 INFO:teuthology.orchestra.run.smithi157.stdout: "segex": 137, 2023-10-10T17:53:01.630 INFO:teuthology.orchestra.run.smithi157.stdout: "segtrm": 137, 2023-10-10T17:53:01.630 INFO:teuthology.orchestra.run.smithi157.stdout: "seg": 2, 2023-10-10T17:53:01.630 INFO:teuthology.orchestra.run.smithi157.stdout: "segexg": 0, 2023-10-10T17:53:01.631 INFO:teuthology.orchestra.run.smithi157.stdout: "segexd": 1, 2023-10-10T17:53:01.631 INFO:teuthology.orchestra.run.smithi157.stdout: "expos": 347607314, 2023-10-10T17:53:01.631 INFO:teuthology.orchestra.run.smithi157.stdout: "wrpos": 347751659, 2023-10-10T17:53:01.631 INFO:teuthology.orchestra.run.smithi157.stdout: "rdpos": 0, 2023-10-10T17:53:01.632 INFO:teuthology.orchestra.run.smithi157.stdout: "jlat": { 2023-10-10T17:53:01.632 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 2068, 2023-10-10T17:53:01.632 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 304.274106977, 2023-10-10T17:53:01.633 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.147134481 2023-10-10T17:53:01.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.633 INFO:teuthology.orchestra.run.smithi157.stdout: "replayed": 0 2023-10-10T17:53:01.633 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.634 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_mem": { 2023-10-10T17:53:01.634 INFO:teuthology.orchestra.run.smithi157.stdout: "ino": 14551, 2023-10-10T17:53:01.634 INFO:teuthology.orchestra.run.smithi157.stdout: "ino+": 79118, 2023-10-10T17:53:01.634 INFO:teuthology.orchestra.run.smithi157.stdout: "ino-": 64567, 2023-10-10T17:53:01.635 INFO:teuthology.orchestra.run.smithi157.stdout: "dir": 523, 2023-10-10T17:53:01.635 INFO:teuthology.orchestra.run.smithi157.stdout: "dir+": 3179, 2023-10-10T17:53:01.635 INFO:teuthology.orchestra.run.smithi157.stdout: "dir-": 2656, 2023-10-10T17:53:01.636 INFO:teuthology.orchestra.run.smithi157.stdout: "dn": 16567, 2023-10-10T17:53:01.636 INFO:teuthology.orchestra.run.smithi157.stdout: "dn+": 101430, 2023-10-10T17:53:01.636 INFO:teuthology.orchestra.run.smithi157.stdout: "dn-": 84863, 2023-10-10T17:53:01.636 INFO:teuthology.orchestra.run.smithi157.stdout: "cap": 5, 2023-10-10T17:53:01.637 INFO:teuthology.orchestra.run.smithi157.stdout: "cap+": 49358, 2023-10-10T17:53:01.637 INFO:teuthology.orchestra.run.smithi157.stdout: "cap-": 49353, 2023-10-10T17:53:01.637 INFO:teuthology.orchestra.run.smithi157.stdout: "rss": 284016, 2023-10-10T17:53:01.637 INFO:teuthology.orchestra.run.smithi157.stdout: "heap": 207156 2023-10-10T17:53:01.638 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.638 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_server": { 2023-10-10T17:53:01.638 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_client_request": 70558, 2023-10-10T17:53:01.639 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_server_request": 70, 2023-10-10T17:53:01.639 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_request": 69670, 2023-10-10T17:53:01.639 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_session": 174, 2023-10-10T17:53:01.639 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_peer_request": 147, 2023-10-10T17:53:01.640 INFO:teuthology.orchestra.run.smithi157.stdout: "req_create_latency": { 2023-10-10T17:53:01.640 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 24235, 2023-10-10T17:53:01.640 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 95.178218563, 2023-10-10T17:53:01.640 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.003927304 2023-10-10T17:53:01.641 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.641 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getattr_latency": { 2023-10-10T17:53:01.641 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 322, 2023-10-10T17:53:01.642 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.994809741, 2023-10-10T17:53:01.642 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.003089471 2023-10-10T17:53:01.642 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.642 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getfilelock_latency": { 2023-10-10T17:53:01.643 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.643 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.643 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.643 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.644 INFO:teuthology.orchestra.run.smithi157.stdout: "req_link_latency": { 2023-10-10T17:53:01.644 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.644 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.645 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.645 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.645 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookup_latency": { 2023-10-10T17:53:01.645 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 16593, 2023-10-10T17:53:01.646 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 59.025895083, 2023-10-10T17:53:01.646 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.003557276 2023-10-10T17:53:01.646 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.646 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookuphash_latency": { 2023-10-10T17:53:01.647 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.647 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.647 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.648 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.648 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupino_latency": { 2023-10-10T17:53:01.648 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.648 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.649 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.649 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.649 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupname_latency": { 2023-10-10T17:53:01.649 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.650 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.650 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.650 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.651 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupparent_latency": { 2023-10-10T17:53:01.651 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.651 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.651 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.652 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.652 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupsnap_latency": { 2023-10-10T17:53:01.652 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.652 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.653 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.653 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.653 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lssnap_latency": { 2023-10-10T17:53:01.654 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 11, 2023-10-10T17:53:01.654 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.003394770, 2023-10-10T17:53:01.654 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000308615 2023-10-10T17:53:01.654 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.655 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mkdir_latency": { 2023-10-10T17:53:01.655 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 119, 2023-10-10T17:53:01.655 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.228889382, 2023-10-10T17:53:01.656 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001923440 2023-10-10T17:53:01.656 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.656 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mknod_latency": { 2023-10-10T17:53:01.656 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.657 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.657 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.657 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.657 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mksnap_latency": { 2023-10-10T17:53:01.658 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 11, 2023-10-10T17:53:01.658 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 3.561728430, 2023-10-10T17:53:01.658 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.323793493 2023-10-10T17:53:01.659 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.659 INFO:teuthology.orchestra.run.smithi157.stdout: "req_open_latency": { 2023-10-10T17:53:01.659 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.659 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.660 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.660 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.660 INFO:teuthology.orchestra.run.smithi157.stdout: "req_readdir_latency": { 2023-10-10T17:53:01.660 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1979, 2023-10-10T17:53:01.661 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 7.352689876, 2023-10-10T17:53:01.661 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.003715356 2023-10-10T17:53:01.661 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.662 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rename_latency": { 2023-10-10T17:53:01.662 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.662 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.662 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.663 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.663 INFO:teuthology.orchestra.run.smithi157.stdout: "req_renamesnap_latency": { 2023-10-10T17:53:01.663 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.663 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.664 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.664 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.664 INFO:teuthology.orchestra.run.smithi157.stdout: "req_snapdiff_latency": { 2023-10-10T17:53:01.665 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.665 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.665 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.665 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.666 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmdir_latency": { 2023-10-10T17:53:01.666 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 112, 2023-10-10T17:53:01.666 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.357012306, 2023-10-10T17:53:01.667 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.003187609 2023-10-10T17:53:01.667 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.667 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmsnap_latency": { 2023-10-10T17:53:01.667 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 5, 2023-10-10T17:53:01.668 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.025824562, 2023-10-10T17:53:01.668 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.005164912 2023-10-10T17:53:01.668 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.669 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmxattr_latency": { 2023-10-10T17:53:01.669 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.669 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.669 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.670 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.670 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setattr_latency": { 2023-10-10T17:53:01.670 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 3, 2023-10-10T17:53:01.671 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.010052281, 2023-10-10T17:53:01.671 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.003350760 2023-10-10T17:53:01.671 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.671 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setdirlayout_latency": { 2023-10-10T17:53:01.672 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.672 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.672 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.672 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.673 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setfilelock_latency": { 2023-10-10T17:53:01.673 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 22, 2023-10-10T17:53:01.673 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 10.025575156, 2023-10-10T17:53:01.674 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.455707961 2023-10-10T17:53:01.674 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.674 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setlayout_latency": { 2023-10-10T17:53:01.674 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.675 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.675 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.675 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.675 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setxattr_latency": { 2023-10-10T17:53:01.676 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 3, 2023-10-10T17:53:01.676 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.005181981, 2023-10-10T17:53:01.676 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001727327 2023-10-10T17:53:01.677 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.677 INFO:teuthology.orchestra.run.smithi157.stdout: "req_symlink_latency": { 2023-10-10T17:53:01.677 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.677 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.678 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.678 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.678 INFO:teuthology.orchestra.run.smithi157.stdout: "req_unlink_latency": { 2023-10-10T17:53:01.679 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 16709, 2023-10-10T17:53:01.679 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 108.832002242, 2023-10-10T17:53:01.679 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.006513376 2023-10-10T17:53:01.679 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.680 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_revoke_eviction": 0, 2023-10-10T17:53:01.680 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_acquisition_throttle": 0, 2023-10-10T17:53:01.680 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getvxattr_latency": { 2023-10-10T17:53:01.680 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.681 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.681 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.681 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.682 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.682 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_sessions": { 2023-10-10T17:53:01.682 INFO:teuthology.orchestra.run.smithi157.stdout: "session_count": 2, 2023-10-10T17:53:01.682 INFO:teuthology.orchestra.run.smithi157.stdout: "session_add": 5, 2023-10-10T17:53:01.683 INFO:teuthology.orchestra.run.smithi157.stdout: "session_remove": 3, 2023-10-10T17:53:01.683 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_open": 2, 2023-10-10T17:53:01.683 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_stale": 0, 2023-10-10T17:53:01.683 INFO:teuthology.orchestra.run.smithi157.stdout: "total_load": 870, 2023-10-10T17:53:01.684 INFO:teuthology.orchestra.run.smithi157.stdout: "average_load": 435, 2023-10-10T17:53:01.684 INFO:teuthology.orchestra.run.smithi157.stdout: "avg_session_uptime": 611 2023-10-10T17:53:01.684 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.685 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2023-10-10T17:53:01.685 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 240, 2023-10-10T17:53:01.685 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 240, 2023-10-10T17:53:01.685 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:01.686 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:01.686 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:01.686 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:01.686 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:01.687 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:01.687 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:01.687 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:01.688 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:01.689 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:01.689 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:01.689 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:01.689 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:01.690 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:01.690 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:01.690 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:01.691 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:01.691 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:01.691 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:01.691 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:01.692 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:01.692 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:01.692 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:01.692 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:01.693 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:01.693 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:01.693 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2023-10-10T17:53:01.694 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:01.694 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:01.694 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:01.694 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:01.695 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 821474, 2023-10-10T17:53:01.695 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 14618, 2023-10-10T17:53:01.695 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:53:01.695 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 0, 2023-10-10T17:53:01.696 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2023-10-10T17:53:01.696 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2023-10-10T17:53:01.696 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:01.697 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:01.697 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:01.697 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:01.698 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 9520, 2023-10-10T17:53:01.698 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 183, 2023-10-10T17:53:01.698 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:01.698 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:01.699 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:01.699 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2023-10-10T17:53:01.699 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 45063127, 2023-10-10T17:53:01.700 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 392075, 2023-10-10T17:53:01.700 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:01.700 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2023-10-10T17:53:01.700 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:01.701 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2023-10-10T17:53:01.701 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.701 INFO:teuthology.orchestra.run.smithi157.stdout: "objecter": { 2023-10-10T17:53:01.701 INFO:teuthology.orchestra.run.smithi157.stdout: "op_active": 0, 2023-10-10T17:53:01.702 INFO:teuthology.orchestra.run.smithi157.stdout: "op_laggy": 0, 2023-10-10T17:53:01.702 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send": 116912, 2023-10-10T17:53:01.702 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send_bytes": 422915320, 2023-10-10T17:53:01.703 INFO:teuthology.orchestra.run.smithi157.stdout: "op_resend": 37, 2023-10-10T17:53:01.703 INFO:teuthology.orchestra.run.smithi157.stdout: "op_reply": 116912, 2023-10-10T17:53:01.703 INFO:teuthology.orchestra.run.smithi157.stdout: "op_latency": { 2023-10-10T17:53:01.703 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 116912, 2023-10-10T17:53:01.704 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 54910.689471594, 2023-10-10T17:53:01.704 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.469675392 2023-10-10T17:53:01.704 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.704 INFO:teuthology.orchestra.run.smithi157.stdout: "op_inflight": 0, 2023-10-10T17:53:01.705 INFO:teuthology.orchestra.run.smithi157.stdout: "oplen_avg": { 2023-10-10T17:53:01.705 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 116912, 2023-10-10T17:53:01.705 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 217702 2023-10-10T17:53:01.705 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.706 INFO:teuthology.orchestra.run.smithi157.stdout: "op": 116912, 2023-10-10T17:53:01.706 INFO:teuthology.orchestra.run.smithi157.stdout: "op_r": 13905, 2023-10-10T17:53:01.706 INFO:teuthology.orchestra.run.smithi157.stdout: "op_w": 103007, 2023-10-10T17:53:01.707 INFO:teuthology.orchestra.run.smithi157.stdout: "op_rmw": 0, 2023-10-10T17:53:01.707 INFO:teuthology.orchestra.run.smithi157.stdout: "op_pg": 0, 2023-10-10T17:53:01.707 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_stat": 9777, 2023-10-10T17:53:01.707 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_create": 55081, 2023-10-10T17:53:01.708 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_read": 319, 2023-10-10T17:53:01.708 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_write": 2375, 2023-10-10T17:53:01.708 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writefull": 249, 2023-10-10T17:53:01.709 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writesame": 0, 2023-10-10T17:53:01.709 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_append": 0, 2023-10-10T17:53:01.709 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_zero": 0, 2023-10-10T17:53:01.709 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_truncate": 5, 2023-10-10T17:53:01.710 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_delete": 33950, 2023-10-10T17:53:01.710 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_mapext": 0, 2023-10-10T17:53:01.710 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_sparse_read": 0, 2023-10-10T17:53:01.710 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_clonerange": 0, 2023-10-10T17:53:01.711 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_getxattr": 12967, 2023-10-10T17:53:01.711 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_setxattr": 83111, 2023-10-10T17:53:01.711 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_cmpxattr": 0, 2023-10-10T17:53:01.712 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_rmxattr": 0, 2023-10-10T17:53:01.712 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_resetxattrs": 0, 2023-10-10T17:53:01.712 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_call": 0, 2023-10-10T17:53:01.712 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_watch": 0, 2023-10-10T17:53:01.713 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_notify": 0, 2023-10-10T17:53:01.713 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_src_cmpxattr": 0, 2023-10-10T17:53:01.713 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls": 0, 2023-10-10T17:53:01.714 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls_filter": 0, 2023-10-10T17:53:01.714 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_other": 1, 2023-10-10T17:53:01.714 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_active": 0, 2023-10-10T17:53:01.714 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_send": 0, 2023-10-10T17:53:01.715 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_resend": 0, 2023-10-10T17:53:01.715 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_ping": 0, 2023-10-10T17:53:01.715 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_active": 0, 2023-10-10T17:53:01.715 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_send": 0, 2023-10-10T17:53:01.716 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_resend": 0, 2023-10-10T17:53:01.716 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_active": 0, 2023-10-10T17:53:01.716 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_send": 0, 2023-10-10T17:53:01.717 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_resend": 0, 2023-10-10T17:53:01.717 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_active": 0, 2023-10-10T17:53:01.717 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_send": 0, 2023-10-10T17:53:01.717 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_resend": 0, 2023-10-10T17:53:01.718 INFO:teuthology.orchestra.run.smithi157.stdout: "command_active": 0, 2023-10-10T17:53:01.718 INFO:teuthology.orchestra.run.smithi157.stdout: "command_send": 0, 2023-10-10T17:53:01.718 INFO:teuthology.orchestra.run.smithi157.stdout: "command_resend": 0, 2023-10-10T17:53:01.719 INFO:teuthology.orchestra.run.smithi157.stdout: "map_epoch": 102, 2023-10-10T17:53:01.719 INFO:teuthology.orchestra.run.smithi157.stdout: "map_full": 0, 2023-10-10T17:53:01.719 INFO:teuthology.orchestra.run.smithi157.stdout: "map_inc": 18, 2023-10-10T17:53:01.719 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_sessions": 12, 2023-10-10T17:53:01.720 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_open": 12, 2023-10-10T17:53:01.720 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_close": 0, 2023-10-10T17:53:01.720 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_laggy": 0, 2023-10-10T17:53:01.720 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_wr": 17504, 2023-10-10T17:53:01.721 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_rd": 1242, 2023-10-10T17:53:01.721 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_del": 1121 2023-10-10T17:53:01.721 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.722 INFO:teuthology.orchestra.run.smithi157.stdout: "oft": { 2023-10-10T17:53:01.722 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_objs": 1, 2023-10-10T17:53:01.722 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_kv_pairs": 0, 2023-10-10T17:53:01.722 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_updates": 0, 2023-10-10T17:53:01.723 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_removes": 0 2023-10-10T17:53:01.723 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.723 INFO:teuthology.orchestra.run.smithi157.stdout: "osd-slow-ops": { 2023-10-10T17:53:01.724 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_ops_count": 70468 2023-10-10T17:53:01.724 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.724 INFO:teuthology.orchestra.run.smithi157.stdout: "purge_queue": { 2023-10-10T17:53:01.724 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops": 0, 2023-10-10T17:53:01.725 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops_high_water": 16, 2023-10-10T17:53:01.725 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing": 0, 2023-10-10T17:53:01.725 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_high_water": 14, 2023-10-10T17:53:01.725 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed": 16814, 2023-10-10T17:53:01.726 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_item_in_journal": 0 2023-10-10T17:53:01.726 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.726 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:01.727 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:01.727 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2023-10-10T17:53:01.727 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:01.727 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 241908, 2023-10-10T17:53:01.728 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 201223035, 2023-10-10T17:53:01.728 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:01.728 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 241908, 2023-10-10T17:53:01.729 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:01.729 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:01.729 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 241908, 2023-10-10T17:53:01.729 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 201223035, 2023-10-10T17:53:01.730 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:01.730 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.730 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.730 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.731 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.731 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.731 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_bytes": { 2023-10-10T17:53:01.732 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:01.732 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2023-10-10T17:53:01.732 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:01.732 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:01.733 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:01.733 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:01.733 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:01.733 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 116912, 2023-10-10T17:53:01.734 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 425005574, 2023-10-10T17:53:01.734 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 116912, 2023-10-10T17:53:01.734 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 425005574, 2023-10-10T17:53:01.735 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:01.735 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.735 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.735 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.736 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.736 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.736 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_ops": { 2023-10-10T17:53:01.736 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:01.737 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 1024, 2023-10-10T17:53:01.737 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:01.737 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:01.738 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:01.738 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:01.738 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:01.738 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 116912, 2023-10-10T17:53:01.739 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 116912, 2023-10-10T17:53:01.739 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 116912, 2023-10-10T17:53:01.739 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 116912, 2023-10-10T17:53:01.739 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:01.740 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.740 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.740 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.740 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.741 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.741 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle": { 2023-10-10T17:53:01.741 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:01.742 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2023-10-10T17:53:01.742 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:01.742 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 16814, 2023-10-10T17:53:01.742 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 2100190, 2023-10-10T17:53:01.743 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:01.743 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 16814, 2023-10-10T17:53:01.743 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:01.743 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:01.744 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 307, 2023-10-10T17:53:01.744 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 2100190, 2023-10-10T17:53:01.744 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:01.745 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.745 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.745 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.745 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.746 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:01.746 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle-0x5632982420a0": { 2023-10-10T17:53:01.746 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 60, 2023-10-10T17:53:01.747 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2023-10-10T17:53:01.747 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:01.747 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 87130, 2023-10-10T17:53:01.747 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 343557355, 2023-10-10T17:53:01.748 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:01.748 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 87130, 2023-10-10T17:53:01.748 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:01.748 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:01.749 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 2068, 2023-10-10T17:53:01.749 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 343557295, 2023-10-10T17:53:01.749 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:01.750 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.751 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:01.751 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-10-10T17:53:01.940 INFO:tasks.check_counter:Daemon mds.b mds.exported=470 2023-10-10T17:53:01.941 INFO:tasks.check_counter:Daemon mds.b mds.imported=484 2023-10-10T17:53:01.941 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=2003 2023-10-10T17:53:01.942 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=3027 2023-10-10T17:53:01.942 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2023-10-10T17:53:01.942 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2023-10-10T17:53:01.943 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2023-10-10T17:53:01.943 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=174 2023-10-10T17:53:01.943 DEBUG:tasks.check_counter:Getting stats from e 2023-10-10T17:53:01.944 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.e perf dump 2023-10-10T17:53:02.647 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:53:03.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:02 smithi078 ceph-mon[99494]: pgmap v326: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:53:03.134 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:02 smithi157 ceph-mon[100896]: pgmap v326: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:53:03.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:02 smithi190 ceph-mon[101220]: pgmap v326: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:53:03.900 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-10-10T17:53:03.901 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:03.901 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 916, 2023-10-10T17:53:03.901 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 916, 2023-10-10T17:53:03.902 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 144009369, 2023-10-10T17:53:03.902 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 284144, 2023-10-10T17:53:03.903 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 8, 2023-10-10T17:53:03.903 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 4, 2023-10-10T17:53:03.903 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.160542762, 2023-10-10T17:53:03.903 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.027198666, 2023-10-10T17:53:03.904 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.081916319, 2023-10-10T17:53:03.904 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.043721549, 2023-10-10T17:53:03.904 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:03.905 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 915, 2023-10-10T17:53:03.905 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.052498229, 2023-10-10T17:53:03.905 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000057375 2023-10-10T17:53:03.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.906 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:03.906 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.906 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.907 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.907 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.907 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-10T17:53:03.907 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 448 2023-10-10T17:53:03.908 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.908 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:03.908 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 596, 2023-10-10T17:53:03.909 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 751, 2023-10-10T17:53:03.909 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 139619784, 2023-10-10T17:53:03.909 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 603635, 2023-10-10T17:53:03.909 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 9, 2023-10-10T17:53:03.910 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 7, 2023-10-10T17:53:03.910 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.122533253, 2023-10-10T17:53:03.910 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.016506066, 2023-10-10T17:53:03.910 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.083825759, 2023-10-10T17:53:03.911 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.016114538, 2023-10-10T17:53:03.911 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:03.911 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 748, 2023-10-10T17:53:03.912 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.041436393, 2023-10-10T17:53:03.912 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000055396 2023-10-10T17:53:03.912 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.913 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:03.913 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.913 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.914 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.914 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:53:03.914 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 418896 2023-10-10T17:53:03.915 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.915 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:03.915 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 1515, 2023-10-10T17:53:03.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 1495, 2023-10-10T17:53:03.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 134658626, 2023-10-10T17:53:03.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 515496, 2023-10-10T17:53:03.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 5, 2023-10-10T17:53:03.917 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 4, 2023-10-10T17:53:03.917 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.227517120, 2023-10-10T17:53:03.917 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.044119258, 2023-10-10T17:53:03.918 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.126730602, 2023-10-10T17:53:03.918 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.050985115, 2023-10-10T17:53:03.918 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:03.919 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1494, 2023-10-10T17:53:03.919 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.157876506, 2023-10-10T17:53:03.919 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000105673 2023-10-10T17:53:03.919 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.920 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:03.920 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.920 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.920 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.921 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 1090448, 2023-10-10T17:53:03.921 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 119904 2023-10-10T17:53:03.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.922 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2023-10-10T17:53:03.922 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 1, 2023-10-10T17:53:03.922 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2023-10-10T17:53:03.923 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.923 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-MDSRank": { 2023-10-10T17:53:03.923 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2023-10-10T17:53:03.924 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2023-10-10T17:53:03.924 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 4321, 2023-10-10T17:53:03.924 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.129300049, 2023-10-10T17:53:03.924 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000029923 2023-10-10T17:53:03.925 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:03.925 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.925 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-PurgeQueue": { 2023-10-10T17:53:03.925 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2023-10-10T17:53:03.926 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2023-10-10T17:53:03.926 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.926 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.927 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.927 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:03.927 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.927 INFO:teuthology.orchestra.run.smithi157.stdout: "mds": { 2023-10-10T17:53:03.928 INFO:teuthology.orchestra.run.smithi157.stdout: "request": 0, 2023-10-10T17:53:03.928 INFO:teuthology.orchestra.run.smithi157.stdout: "reply": 0, 2023-10-10T17:53:03.928 INFO:teuthology.orchestra.run.smithi157.stdout: "reply_latency": { 2023-10-10T17:53:03.929 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.929 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.929 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.930 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_reply": 0, 2023-10-10T17:53:03.930 INFO:teuthology.orchestra.run.smithi157.stdout: "forward": 0, 2023-10-10T17:53:03.931 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_complete": 0, 2023-10-10T17:53:03.931 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_keys": 0, 2023-10-10T17:53:03.931 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_commit": 0, 2023-10-10T17:53:03.932 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_split": 0, 2023-10-10T17:53:03.932 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_merge": 0, 2023-10-10T17:53:03.932 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes": 38722, 2023-10-10T17:53:03.933 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_top": 14800, 2023-10-10T17:53:03.933 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_bottom": 6567, 2023-10-10T17:53:03.933 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pin_tail": 17355, 2023-10-10T17:53:03.934 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pinned": 17578, 2023-10-10T17:53:03.934 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_expired": 0, 2023-10-10T17:53:03.934 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_with_caps": 0, 2023-10-10T17:53:03.934 INFO:teuthology.orchestra.run.smithi157.stdout: "caps": 0, 2023-10-10T17:53:03.935 INFO:teuthology.orchestra.run.smithi157.stdout: "subtrees": 1, 2023-10-10T17:53:03.935 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse": 0, 2023-10-10T17:53:03.935 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_hit": 0, 2023-10-10T17:53:03.936 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_forward": 0, 2023-10-10T17:53:03.936 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_discover": 0, 2023-10-10T17:53:03.936 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_dir_fetch": 0, 2023-10-10T17:53:03.936 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_remote_ino": 0, 2023-10-10T17:53:03.937 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_lock": 0, 2023-10-10T17:53:03.937 INFO:teuthology.orchestra.run.smithi157.stdout: "load_cent": 0, 2023-10-10T17:53:03.937 INFO:teuthology.orchestra.run.smithi157.stdout: "q": 0, 2023-10-10T17:53:03.938 INFO:teuthology.orchestra.run.smithi157.stdout: "exported": 0, 2023-10-10T17:53:03.938 INFO:teuthology.orchestra.run.smithi157.stdout: "exported_inodes": 0, 2023-10-10T17:53:03.938 INFO:teuthology.orchestra.run.smithi157.stdout: "imported": 0, 2023-10-10T17:53:03.939 INFO:teuthology.orchestra.run.smithi157.stdout: "imported_inodes": 0, 2023-10-10T17:53:03.939 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_dir_fetch": 0, 2023-10-10T17:53:03.939 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_backtrace_fetch": 0, 2023-10-10T17:53:03.940 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_peer_discover": 0, 2023-10-10T17:53:03.940 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rfiles": 0, 2023-10-10T17:53:03.940 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rbytes": 0, 2023-10-10T17:53:03.940 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rsnaps": 0, 2023-10-10T17:53:03.941 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_fetch": 0, 2023-10-10T17:53:03.941 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_set_tag": 0, 2023-10-10T17:53:03.941 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_repaired": 0, 2023-10-10T17:53:03.942 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_inotable_repaired": 0, 2023-10-10T17:53:03.942 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_inodes": 0, 2023-10-10T17:53:03.942 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_base_inodes": 0, 2023-10-10T17:53:03.942 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dirfrag_rstats": 0, 2023-10-10T17:53:03.943 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_file_inodes": 0, 2023-10-10T17:53:03.943 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_inode_file_caps": 0, 2023-10-10T17:53:03.943 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_revoke": 0, 2023-10-10T17:53:03.944 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_grant": 0, 2023-10-10T17:53:03.944 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_trunc": 0, 2023-10-10T17:53:03.944 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-10T17:53:03.945 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-10T17:53:03.945 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps": 0, 2023-10-10T17:53:03.945 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps_dirty": 0, 2023-10-10T17:53:03.946 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_cap_release": 0, 2023-10-10T17:53:03.946 INFO:teuthology.orchestra.run.smithi157.stdout: "process_request_cap_release": 0 2023-10-10T17:53:03.946 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.947 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_cache": { 2023-10-10T17:53:03.947 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update": 0, 2023-10-10T17:53:03.947 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update_receipt": 0, 2023-10-10T17:53:03.948 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_try_discover": 0, 2023-10-10T17:53:03.948 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_send_discover": 0, 2023-10-10T17:53:03.948 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_handle_discover": 0, 2023-10-10T17:53:03.949 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays": 0, 2023-10-10T17:53:03.949 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_delayed": 0, 2023-10-10T17:53:03.949 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_enqueuing": 0, 2023-10-10T17:53:03.950 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_created": 0, 2023-10-10T17:53:03.950 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_enqueued": 0, 2023-10-10T17:53:03.950 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_reintegrated": 0, 2023-10-10T17:53:03.951 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_migrated": 0, 2023-10-10T17:53:03.951 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_processing": 0, 2023-10-10T17:53:03.951 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_enqueued": 0, 2023-10-10T17:53:03.951 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_prioritized": 0, 2023-10-10T17:53:03.952 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_started": 0, 2023-10-10T17:53:03.952 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_completed": 0, 2023-10-10T17:53:03.952 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_enqueue_scrub": 0, 2023-10-10T17:53:03.953 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_exportdir": 0, 2023-10-10T17:53:03.953 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_flush": 0, 2023-10-10T17:53:03.953 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragmentdir": 0, 2023-10-10T17:53:03.953 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragstats": 0, 2023-10-10T17:53:03.954 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_inodestats": 0 2023-10-10T17:53:03.954 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.954 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_log": { 2023-10-10T17:53:03.955 INFO:teuthology.orchestra.run.smithi157.stdout: "evadd": 0, 2023-10-10T17:53:03.955 INFO:teuthology.orchestra.run.smithi157.stdout: "evex": 0, 2023-10-10T17:53:03.955 INFO:teuthology.orchestra.run.smithi157.stdout: "evtrm": 0, 2023-10-10T17:53:03.955 INFO:teuthology.orchestra.run.smithi157.stdout: "ev": 0, 2023-10-10T17:53:03.956 INFO:teuthology.orchestra.run.smithi157.stdout: "evexg": 0, 2023-10-10T17:53:03.956 INFO:teuthology.orchestra.run.smithi157.stdout: "evexd": 0, 2023-10-10T17:53:03.956 INFO:teuthology.orchestra.run.smithi157.stdout: "segadd": 0, 2023-10-10T17:53:03.957 INFO:teuthology.orchestra.run.smithi157.stdout: "segex": 0, 2023-10-10T17:53:03.957 INFO:teuthology.orchestra.run.smithi157.stdout: "segtrm": 0, 2023-10-10T17:53:03.957 INFO:teuthology.orchestra.run.smithi157.stdout: "seg": 131, 2023-10-10T17:53:03.957 INFO:teuthology.orchestra.run.smithi157.stdout: "segexg": 0, 2023-10-10T17:53:03.958 INFO:teuthology.orchestra.run.smithi157.stdout: "segexd": 0, 2023-10-10T17:53:03.958 INFO:teuthology.orchestra.run.smithi157.stdout: "expos": 48074336, 2023-10-10T17:53:03.958 INFO:teuthology.orchestra.run.smithi157.stdout: "wrpos": 0, 2023-10-10T17:53:03.959 INFO:teuthology.orchestra.run.smithi157.stdout: "rdpos": 410813738, 2023-10-10T17:53:03.959 INFO:teuthology.orchestra.run.smithi157.stdout: "jlat": { 2023-10-10T17:53:03.959 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.959 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.960 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.960 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.960 INFO:teuthology.orchestra.run.smithi157.stdout: "replayed": 87068 2023-10-10T17:53:03.961 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.961 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_mem": { 2023-10-10T17:53:03.961 INFO:teuthology.orchestra.run.smithi157.stdout: "ino": 11158, 2023-10-10T17:53:03.961 INFO:teuthology.orchestra.run.smithi157.stdout: "ino+": 84697, 2023-10-10T17:53:03.962 INFO:teuthology.orchestra.run.smithi157.stdout: "ino-": 73539, 2023-10-10T17:53:03.962 INFO:teuthology.orchestra.run.smithi157.stdout: "dir": 493, 2023-10-10T17:53:03.962 INFO:teuthology.orchestra.run.smithi157.stdout: "dir+": 2445, 2023-10-10T17:53:03.963 INFO:teuthology.orchestra.run.smithi157.stdout: "dir-": 1952, 2023-10-10T17:53:03.963 INFO:teuthology.orchestra.run.smithi157.stdout: "dn": 38722, 2023-10-10T17:53:03.963 INFO:teuthology.orchestra.run.smithi157.stdout: "dn+": 131524, 2023-10-10T17:53:03.963 INFO:teuthology.orchestra.run.smithi157.stdout: "dn-": 92802, 2023-10-10T17:53:03.964 INFO:teuthology.orchestra.run.smithi157.stdout: "cap": 0, 2023-10-10T17:53:03.964 INFO:teuthology.orchestra.run.smithi157.stdout: "cap+": 0, 2023-10-10T17:53:03.964 INFO:teuthology.orchestra.run.smithi157.stdout: "cap-": 0, 2023-10-10T17:53:03.965 INFO:teuthology.orchestra.run.smithi157.stdout: "rss": 160548, 2023-10-10T17:53:03.965 INFO:teuthology.orchestra.run.smithi157.stdout: "heap": 207156 2023-10-10T17:53:03.965 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.966 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_server": { 2023-10-10T17:53:03.966 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_client_request": 0, 2023-10-10T17:53:03.966 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_server_request": 0, 2023-10-10T17:53:03.966 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_request": 0, 2023-10-10T17:53:03.967 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_session": 0, 2023-10-10T17:53:03.967 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_peer_request": 0, 2023-10-10T17:53:03.967 INFO:teuthology.orchestra.run.smithi157.stdout: "req_create_latency": { 2023-10-10T17:53:03.968 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.968 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.968 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.968 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.969 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getattr_latency": { 2023-10-10T17:53:03.969 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.969 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.970 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.970 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getfilelock_latency": { 2023-10-10T17:53:03.970 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.971 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.971 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.971 INFO:teuthology.orchestra.run.smithi157.stdout: "req_link_latency": { 2023-10-10T17:53:03.972 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.972 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.972 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.973 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookup_latency": { 2023-10-10T17:53:03.973 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.973 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.974 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.974 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.975 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookuphash_latency": { 2023-10-10T17:53:03.975 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.975 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.975 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.976 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.976 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupino_latency": { 2023-10-10T17:53:03.976 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.976 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.977 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.977 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupname_latency": { 2023-10-10T17:53:03.978 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.978 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.978 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.979 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupparent_latency": { 2023-10-10T17:53:03.979 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.979 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.980 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.980 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupsnap_latency": { 2023-10-10T17:53:03.980 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.981 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.981 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.982 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lssnap_latency": { 2023-10-10T17:53:03.982 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.982 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.982 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.983 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.983 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mkdir_latency": { 2023-10-10T17:53:03.983 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.983 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.984 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.984 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.984 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mknod_latency": { 2023-10-10T17:53:03.985 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.986 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.986 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.987 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mksnap_latency": { 2023-10-10T17:53:03.987 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.987 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.987 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.988 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.988 INFO:teuthology.orchestra.run.smithi157.stdout: "req_open_latency": { 2023-10-10T17:53:03.988 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.988 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.989 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.989 INFO:teuthology.orchestra.run.smithi157.stdout: "req_readdir_latency": { 2023-10-10T17:53:03.990 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.990 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.990 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.991 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rename_latency": { 2023-10-10T17:53:03.991 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.991 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.991 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.992 INFO:teuthology.orchestra.run.smithi157.stdout: "req_renamesnap_latency": { 2023-10-10T17:53:03.992 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.993 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.993 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.993 INFO:teuthology.orchestra.run.smithi157.stdout: "req_snapdiff_latency": { 2023-10-10T17:53:03.994 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.994 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.994 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.995 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmdir_latency": { 2023-10-10T17:53:03.995 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.995 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.996 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.996 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmsnap_latency": { 2023-10-10T17:53:03.997 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.997 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.997 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.998 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmxattr_latency": { 2023-10-10T17:53:03.998 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:03.998 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:03.999 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:03.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:03.999 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setattr_latency": { 2023-10-10T17:53:03.999 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.000 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.000 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.001 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setdirlayout_latency": { 2023-10-10T17:53:04.001 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.001 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.001 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.002 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setfilelock_latency": { 2023-10-10T17:53:04.002 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.002 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.003 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.003 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.003 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setlayout_latency": { 2023-10-10T17:53:04.004 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.004 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.004 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.005 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setxattr_latency": { 2023-10-10T17:53:04.005 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.005 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.005 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.006 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.006 INFO:teuthology.orchestra.run.smithi157.stdout: "req_symlink_latency": { 2023-10-10T17:53:04.006 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.007 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.007 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.007 INFO:teuthology.orchestra.run.smithi157.stdout: "req_unlink_latency": { 2023-10-10T17:53:04.008 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.008 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.008 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.009 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_revoke_eviction": 0, 2023-10-10T17:53:04.009 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_acquisition_throttle": 0, 2023-10-10T17:53:04.009 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getvxattr_latency": { 2023-10-10T17:53:04.010 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.010 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.010 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.010 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:04.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.011 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_sessions": { 2023-10-10T17:53:04.011 INFO:teuthology.orchestra.run.smithi157.stdout: "session_count": 1, 2023-10-10T17:53:04.012 INFO:teuthology.orchestra.run.smithi157.stdout: "session_add": 2, 2023-10-10T17:53:04.012 INFO:teuthology.orchestra.run.smithi157.stdout: "session_remove": 1, 2023-10-10T17:53:04.012 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_open": 2, 2023-10-10T17:53:04.012 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_stale": 0, 2023-10-10T17:53:04.013 INFO:teuthology.orchestra.run.smithi157.stdout: "total_load": 0, 2023-10-10T17:53:04.013 INFO:teuthology.orchestra.run.smithi157.stdout: "average_load": 0, 2023-10-10T17:53:04.013 INFO:teuthology.orchestra.run.smithi157.stdout: "avg_session_uptime": 3401 2023-10-10T17:53:04.014 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.014 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2023-10-10T17:53:04.014 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:53:04.015 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 0, 2023-10-10T17:53:04.015 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:04.015 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:04.016 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:04.016 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:04.016 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:04.016 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:04.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:04.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:04.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:04.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:04.018 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:04.018 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:04.018 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:04.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:04.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:04.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:04.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:04.020 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:04.020 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:04.020 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:04.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:04.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:04.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:04.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:04.022 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:04.022 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:04.022 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:04.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:04.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:04.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2023-10-10T17:53:04.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:04.024 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:04.024 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:04.024 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:04.024 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 618507, 2023-10-10T17:53:04.025 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 11228, 2023-10-10T17:53:04.025 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:53:04.025 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 0, 2023-10-10T17:53:04.026 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2023-10-10T17:53:04.026 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2023-10-10T17:53:04.026 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:04.026 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:04.027 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:04.027 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:04.027 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 9520, 2023-10-10T17:53:04.027 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 183, 2023-10-10T17:53:04.028 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:04.028 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:04.028 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:04.029 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2023-10-10T17:53:04.029 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 52657708, 2023-10-10T17:53:04.029 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 360508, 2023-10-10T17:53:04.029 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:04.030 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2023-10-10T17:53:04.030 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:04.030 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2023-10-10T17:53:04.030 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.031 INFO:teuthology.orchestra.run.smithi157.stdout: "objecter": { 2023-10-10T17:53:04.031 INFO:teuthology.orchestra.run.smithi157.stdout: "op_active": 0, 2023-10-10T17:53:04.031 INFO:teuthology.orchestra.run.smithi157.stdout: "op_laggy": 0, 2023-10-10T17:53:04.033 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send": 2778, 2023-10-10T17:53:04.033 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send_bytes": 21, 2023-10-10T17:53:04.033 INFO:teuthology.orchestra.run.smithi157.stdout: "op_resend": 3, 2023-10-10T17:53:04.034 INFO:teuthology.orchestra.run.smithi157.stdout: "op_reply": 2778, 2023-10-10T17:53:04.035 INFO:teuthology.orchestra.run.smithi157.stdout: "op_latency": { 2023-10-10T17:53:04.035 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 2778, 2023-10-10T17:53:04.035 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 7.068871628, 2023-10-10T17:53:04.035 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.002544590 2023-10-10T17:53:04.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.036 INFO:teuthology.orchestra.run.smithi157.stdout: "op_inflight": 0, 2023-10-10T17:53:04.036 INFO:teuthology.orchestra.run.smithi157.stdout: "oplen_avg": { 2023-10-10T17:53:04.036 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 2778, 2023-10-10T17:53:04.037 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 2779 2023-10-10T17:53:04.037 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.037 INFO:teuthology.orchestra.run.smithi157.stdout: "op": 2778, 2023-10-10T17:53:04.038 INFO:teuthology.orchestra.run.smithi157.stdout: "op_r": 2778, 2023-10-10T17:53:04.038 INFO:teuthology.orchestra.run.smithi157.stdout: "op_w": 0, 2023-10-10T17:53:04.038 INFO:teuthology.orchestra.run.smithi157.stdout: "op_rmw": 0, 2023-10-10T17:53:04.038 INFO:teuthology.orchestra.run.smithi157.stdout: "op_pg": 0, 2023-10-10T17:53:04.039 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_stat": 1795, 2023-10-10T17:53:04.039 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_create": 0, 2023-10-10T17:53:04.039 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_read": 981, 2023-10-10T17:53:04.039 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_write": 0, 2023-10-10T17:53:04.040 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writefull": 0, 2023-10-10T17:53:04.040 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writesame": 0, 2023-10-10T17:53:04.040 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_append": 0, 2023-10-10T17:53:04.041 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_zero": 0, 2023-10-10T17:53:04.041 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_truncate": 0, 2023-10-10T17:53:04.041 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_delete": 0, 2023-10-10T17:53:04.041 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_mapext": 0, 2023-10-10T17:53:04.042 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_sparse_read": 0, 2023-10-10T17:53:04.042 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_clonerange": 0, 2023-10-10T17:53:04.042 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_getxattr": 1, 2023-10-10T17:53:04.043 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_setxattr": 0, 2023-10-10T17:53:04.043 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_cmpxattr": 0, 2023-10-10T17:53:04.043 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_rmxattr": 0, 2023-10-10T17:53:04.043 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_resetxattrs": 0, 2023-10-10T17:53:04.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_call": 0, 2023-10-10T17:53:04.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_watch": 0, 2023-10-10T17:53:04.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_notify": 0, 2023-10-10T17:53:04.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_src_cmpxattr": 0, 2023-10-10T17:53:04.045 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls": 0, 2023-10-10T17:53:04.045 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls_filter": 0, 2023-10-10T17:53:04.045 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_other": 0, 2023-10-10T17:53:04.046 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_active": 0, 2023-10-10T17:53:04.046 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_send": 0, 2023-10-10T17:53:04.046 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_resend": 0, 2023-10-10T17:53:04.046 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_ping": 0, 2023-10-10T17:53:04.047 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_active": 0, 2023-10-10T17:53:04.047 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_send": 0, 2023-10-10T17:53:04.047 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_resend": 0, 2023-10-10T17:53:04.047 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_active": 0, 2023-10-10T17:53:04.048 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_send": 0, 2023-10-10T17:53:04.048 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_resend": 0, 2023-10-10T17:53:04.048 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_active": 0, 2023-10-10T17:53:04.048 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_send": 0, 2023-10-10T17:53:04.049 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_resend": 0, 2023-10-10T17:53:04.049 INFO:teuthology.orchestra.run.smithi157.stdout: "command_active": 0, 2023-10-10T17:53:04.049 INFO:teuthology.orchestra.run.smithi157.stdout: "command_send": 0, 2023-10-10T17:53:04.050 INFO:teuthology.orchestra.run.smithi157.stdout: "command_resend": 0, 2023-10-10T17:53:04.050 INFO:teuthology.orchestra.run.smithi157.stdout: "map_epoch": 102, 2023-10-10T17:53:04.050 INFO:teuthology.orchestra.run.smithi157.stdout: "map_full": 0, 2023-10-10T17:53:04.050 INFO:teuthology.orchestra.run.smithi157.stdout: "map_inc": 13, 2023-10-10T17:53:04.051 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_sessions": 11, 2023-10-10T17:53:04.051 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_open": 11, 2023-10-10T17:53:04.051 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_close": 0, 2023-10-10T17:53:04.051 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_laggy": 0, 2023-10-10T17:53:04.052 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_wr": 0, 2023-10-10T17:53:04.052 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_rd": 2, 2023-10-10T17:53:04.052 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_del": 0 2023-10-10T17:53:04.053 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.053 INFO:teuthology.orchestra.run.smithi157.stdout: "oft": { 2023-10-10T17:53:04.053 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_objs": 0, 2023-10-10T17:53:04.053 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_kv_pairs": 0, 2023-10-10T17:53:04.054 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_updates": 0, 2023-10-10T17:53:04.054 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_removes": 0 2023-10-10T17:53:04.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.054 INFO:teuthology.orchestra.run.smithi157.stdout: "osd-slow-ops": { 2023-10-10T17:53:04.055 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_ops_count": 0 2023-10-10T17:53:04.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.055 INFO:teuthology.orchestra.run.smithi157.stdout: "purge_queue": { 2023-10-10T17:53:04.056 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops": 0, 2023-10-10T17:53:04.056 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops_high_water": 0, 2023-10-10T17:53:04.056 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing": 0, 2023-10-10T17:53:04.056 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_high_water": 0, 2023-10-10T17:53:04.057 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed": 0, 2023-10-10T17:53:04.057 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_item_in_journal": 0 2023-10-10T17:53:04.057 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.057 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:04.058 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:04.058 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2023-10-10T17:53:04.058 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:04.059 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 3023, 2023-10-10T17:53:04.059 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 418000285, 2023-10-10T17:53:04.059 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:04.059 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 3023, 2023-10-10T17:53:04.060 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:04.060 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:04.060 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 3023, 2023-10-10T17:53:04.060 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 418000285, 2023-10-10T17:53:04.061 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:04.061 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.061 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.062 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.062 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:04.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.062 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_bytes": { 2023-10-10T17:53:04.063 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:04.063 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2023-10-10T17:53:04.063 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:04.063 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:04.064 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:04.064 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:04.064 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:04.065 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 2778, 2023-10-10T17:53:04.065 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 416425951, 2023-10-10T17:53:04.065 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 2778, 2023-10-10T17:53:04.065 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 416425951, 2023-10-10T17:53:04.066 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:04.066 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.066 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.066 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.067 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:04.067 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.067 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_ops": { 2023-10-10T17:53:04.068 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:04.068 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 1024, 2023-10-10T17:53:04.068 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:04.068 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:04.069 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:04.069 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:04.069 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:04.069 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 2778, 2023-10-10T17:53:04.070 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 2778, 2023-10-10T17:53:04.070 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 2778, 2023-10-10T17:53:04.070 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 2778, 2023-10-10T17:53:04.071 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:04.071 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.071 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.071 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.072 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:04.072 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.072 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle": { 2023-10-10T17:53:04.072 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:04.073 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2023-10-10T17:53:04.073 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:04.073 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:04.074 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:04.074 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:04.074 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:04.074 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:04.075 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:04.075 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 0, 2023-10-10T17:53:04.075 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 0, 2023-10-10T17:53:04.075 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:04.076 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.076 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.076 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.076 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:04.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:04.077 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle-0x5589fc8b4460": { 2023-10-10T17:53:04.077 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:04.078 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2023-10-10T17:53:04.078 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:04.078 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:04.078 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:04.079 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:04.079 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:04.079 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:04.079 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:04.080 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 0, 2023-10-10T17:53:04.080 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 0, 2023-10-10T17:53:04.080 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:04.081 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:04.081 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:04.081 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:04.081 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:04.082 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:04.082 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-10-10T17:53:04.313 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2023-10-10T17:53:04.314 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2023-10-10T17:53:04.314 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2023-10-10T17:53:04.315 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2023-10-10T17:53:04.315 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2023-10-10T17:53:04.315 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2023-10-10T17:53:04.316 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2023-10-10T17:53:04.316 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2023-10-10T17:53:04.316 DEBUG:tasks.check_counter:Getting stats from h 2023-10-10T17:53:04.317 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.h perf dump 2023-10-10T17:53:04.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: pgmap v327: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:04.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:04 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: pgmap v327: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:04.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.878 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.879 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.880 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:04 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: pgmap v327: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:04.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:04.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:04 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:04.970 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:53:05.727 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi078 to 196.2M 2023-10-10T17:53:05.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: Unable to set osd_memory_target on smithi078 to 205745254: error parsing value: Value '205745254' is below minimum 939524096 2023-10-10T17:53:05.728 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:05.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:05.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi157 to 196.2M 2023-10-10T17:53:05.730 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: Unable to set osd_memory_target on smithi157 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-10T17:53:05.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.731 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:53:05.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:53:05.732 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:05 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:05.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi078 to 196.2M 2023-10-10T17:53:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: Unable to set osd_memory_target on smithi078 to 205745254: error parsing value: Value '205745254' is below minimum 939524096 2023-10-10T17:53:05.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:05.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:05.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi157 to 196.2M 2023-10-10T17:53:05.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: Unable to set osd_memory_target on smithi157 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-10T17:53:05.917 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.918 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:05.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:53:05.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:53:05.919 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:05 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:06.057 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-10-10T17:53:06.057 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:06.058 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 0, 2023-10-10T17:53:06.058 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 2, 2023-10-10T17:53:06.058 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 0, 2023-10-10T17:53:06.059 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 0, 2023-10-10T17:53:06.059 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 4, 2023-10-10T17:53:06.059 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 0, 2023-10-10T17:53:06.059 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.001358834, 2023-10-10T17:53:06.060 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.000000000, 2023-10-10T17:53:06.060 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.000282378, 2023-10-10T17:53:06.060 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:53:06.060 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:06.061 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:06.061 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:06.061 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:06.062 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.062 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:06.062 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:06.062 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:06.063 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:06.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.063 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-10T17:53:06.064 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 0 2023-10-10T17:53:06.064 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.064 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:06.064 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 5, 2023-10-10T17:53:06.065 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 86, 2023-10-10T17:53:06.065 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 3152, 2023-10-10T17:53:06.065 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 91008, 2023-10-10T17:53:06.066 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 3, 2023-10-10T17:53:06.066 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 2, 2023-10-10T17:53:06.066 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.005572539, 2023-10-10T17:53:06.066 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.003073679, 2023-10-10T17:53:06.067 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.001724985, 2023-10-10T17:53:06.067 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:53:06.067 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:06.067 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 85, 2023-10-10T17:53:06.068 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.007132104, 2023-10-10T17:53:06.068 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000083907 2023-10-10T17:53:06.068 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.069 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:06.069 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:06.069 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:06.070 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:06.070 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.070 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 3152, 2023-10-10T17:53:06.070 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 91008 2023-10-10T17:53:06.071 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.071 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:06.071 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 111, 2023-10-10T17:53:06.072 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 110, 2023-10-10T17:53:06.072 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 169824, 2023-10-10T17:53:06.072 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 62208, 2023-10-10T17:53:06.072 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 2, 2023-10-10T17:53:06.073 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 1, 2023-10-10T17:53:06.073 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.015626734, 2023-10-10T17:53:06.073 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.005814141, 2023-10-10T17:53:06.073 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.005937636, 2023-10-10T17:53:06.074 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.002980544, 2023-10-10T17:53:06.074 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:06.074 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 109, 2023-10-10T17:53:06.075 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.007783095, 2023-10-10T17:53:06.075 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000071404 2023-10-10T17:53:06.075 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.075 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:06.076 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:06.076 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:06.076 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:06.076 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.077 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 169824, 2023-10-10T17:53:06.077 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 62208 2023-10-10T17:53:06.077 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.077 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2023-10-10T17:53:06.078 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 0, 2023-10-10T17:53:06.078 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2023-10-10T17:53:06.078 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.079 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2023-10-10T17:53:06.079 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:53:06.079 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 0, 2023-10-10T17:53:06.079 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:06.080 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:06.080 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:06.080 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:06.080 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:06.081 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:06.081 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:06.081 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:06.082 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:06.082 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:06.082 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:06.082 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:06.083 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:06.083 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:06.083 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:06.084 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:06.084 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:06.084 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:06.084 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:06.085 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:06.085 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:06.085 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:06.085 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:06.086 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:06.086 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:06.086 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:06.087 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:06.087 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:06.087 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:06.087 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2023-10-10T17:53:06.088 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:06.088 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:06.088 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:06.089 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:06.089 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 148863, 2023-10-10T17:53:06.089 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 28, 2023-10-10T17:53:06.089 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:53:06.090 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 0, 2023-10-10T17:53:06.090 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2023-10-10T17:53:06.090 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2023-10-10T17:53:06.090 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:06.091 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:06.091 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:06.091 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:06.092 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 0, 2023-10-10T17:53:06.092 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 0, 2023-10-10T17:53:06.092 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:06.092 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:06.093 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:06.093 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2023-10-10T17:53:06.093 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 928, 2023-10-10T17:53:06.093 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 2, 2023-10-10T17:53:06.094 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:06.094 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2023-10-10T17:53:06.094 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:06.095 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2023-10-10T17:53:06.095 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:06.095 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:06.095 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:06.096 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2023-10-10T17:53:06.096 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:06.096 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 112, 2023-10-10T17:53:06.096 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 154159, 2023-10-10T17:53:06.097 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:06.097 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 112, 2023-10-10T17:53:06.097 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:06.097 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:06.098 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 112, 2023-10-10T17:53:06.098 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 154159, 2023-10-10T17:53:06.098 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:06.099 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:06.099 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:06.099 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:06.099 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:06.100 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:06.100 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-10-10T17:53:06.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi078 to 196.2M 2023-10-10T17:53:06.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: Unable to set osd_memory_target on smithi078 to 205745254: error parsing value: Value '205745254' is below minimum 939524096 2023-10-10T17:53:06.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:06.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:06.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi157 to 196.2M 2023-10-10T17:53:06.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: Unable to set osd_memory_target on smithi157 to 205769625: error parsing value: Value '205769625' is below minimum 939524096 2023-10-10T17:53:06.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-10-10T17:53:06.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-10T17:53:06.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-10T17:53:06.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:05 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' 2023-10-10T17:53:06.423 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.h 2023-10-10T17:53:06.424 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.h 2023-10-10T17:53:06.424 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.h 2023-10-10T17:53:06.424 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.h 2023-10-10T17:53:06.425 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2023-10-10T17:53:06.425 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2023-10-10T17:53:06.425 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2023-10-10T17:53:06.425 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2023-10-10T17:53:06.426 DEBUG:tasks.check_counter:Getting stats from k 2023-10-10T17:53:06.426 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.k perf dump 2023-10-10T17:53:06.735 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:06 smithi157 ceph-mon[100896]: Adjusting osd_memory_target on smithi190 to 196.2M 2023-10-10T17:53:06.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:06 smithi157 ceph-mon[100896]: Unable to set osd_memory_target on smithi190 to 205773926: error parsing value: Value '205773926' is below minimum 939524096 2023-10-10T17:53:06.736 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:06 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:53:06.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:06 smithi157 ceph-mon[100896]: Service mds.all has no related daemon already placed. 2023-10-10T17:53:06.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:06 smithi157 ceph-mon[100896]: Service mgr has no related daemon already placed. 2023-10-10T17:53:06.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:06 smithi157 ceph-mon[100896]: Service mon has no related daemon already placed. 2023-10-10T17:53:06.737 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:06 smithi157 ceph-mon[100896]: pgmap v328: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:06.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:06 smithi190 ceph-mon[101220]: Adjusting osd_memory_target on smithi190 to 196.2M 2023-10-10T17:53:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:06 smithi190 ceph-mon[101220]: Unable to set osd_memory_target on smithi190 to 205773926: error parsing value: Value '205773926' is below minimum 939524096 2023-10-10T17:53:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:06 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:53:06.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:06 smithi190 ceph-mon[101220]: Service mds.all has no related daemon already placed. 2023-10-10T17:53:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:06 smithi190 ceph-mon[101220]: Service mgr has no related daemon already placed. 2023-10-10T17:53:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:06 smithi190 ceph-mon[101220]: Service mon has no related daemon already placed. 2023-10-10T17:53:06.916 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:06 smithi190 ceph-mon[101220]: pgmap v328: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:07.121 INFO:teuthology.orchestra.run.smithi157.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.b/config 2023-10-10T17:53:07.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:06 smithi078 ceph-mon[99494]: Adjusting osd_memory_target on smithi190 to 196.2M 2023-10-10T17:53:07.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:06 smithi078 ceph-mon[99494]: Unable to set osd_memory_target on smithi190 to 205773926: error parsing value: Value '205773926' is below minimum 939524096 2023-10-10T17:53:07.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:06 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:53:07.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:06 smithi078 ceph-mon[99494]: Service mds.all has no related daemon already placed. 2023-10-10T17:53:07.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:06 smithi078 ceph-mon[99494]: Service mgr has no related daemon already placed. 2023-10-10T17:53:07.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:06 smithi078 ceph-mon[99494]: Service mon has no related daemon already placed. 2023-10-10T17:53:07.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:06 smithi078 ceph-mon[99494]: pgmap v328: 129 pgs: 129 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:07.729 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:07 smithi157 ceph-mon[100896]: osdmap e103: 12 total, 12 up, 12 in 2023-10-10T17:53:08.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:07 smithi078 ceph-mon[99494]: osdmap e103: 12 total, 12 up, 12 in 2023-10-10T17:53:08.170 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:07 smithi190 ceph-mon[101220]: osdmap e103: 12 total, 12 up, 12 in 2023-10-10T17:53:08.899 INFO:teuthology.orchestra.run.smithi157.stdout:{ 2023-10-10T17:53:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 1093, 2023-10-10T17:53:08.900 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 1080, 2023-10-10T17:53:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 47084321, 2023-10-10T17:53:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 387667, 2023-10-10T17:53:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 6, 2023-10-10T17:53:08.901 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 2, 2023-10-10T17:53:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.129408110, 2023-10-10T17:53:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.038729429, 2023-10-10T17:53:08.902 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.059465709, 2023-10-10T17:53:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.029843849, 2023-10-10T17:53:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:08.903 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1079, 2023-10-10T17:53:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.087247606, 2023-10-10T17:53:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000080859 2023-10-10T17:53:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.904 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.905 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 1062256, 2023-10-10T17:53:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 121136 2023-10-10T17:53:08.906 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 160, 2023-10-10T17:53:08.907 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 317, 2023-10-10T17:53:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 51158655, 2023-10-10T17:53:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 363428, 2023-10-10T17:53:08.908 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 8, 2023-10-10T17:53:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 6, 2023-10-10T17:53:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.048888866, 2023-10-10T17:53:08.909 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.008269154, 2023-10-10T17:53:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.033616987, 2023-10-10T17:53:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.003980608, 2023-10-10T17:53:08.910 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 314, 2023-10-10T17:53:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.036860375, 2023-10-10T17:53:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000117389 2023-10-10T17:53:08.911 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.912 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:53:08.913 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 314224 2023-10-10T17:53:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:08.914 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_messages": 523, 2023-10-10T17:53:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages": 523, 2023-10-10T17:53:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_bytes": 75674591, 2023-10-10T17:53:08.915 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_bytes": 161864, 2023-10-10T17:53:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_created_connections": 6, 2023-10-10T17:53:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_active_connections": 5, 2023-10-10T17:53:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_total_time": 0.082847821, 2023-10-10T17:53:08.916 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_send_time": 0.009721338, 2023-10-10T17:53:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_recv_time": 0.053406814, 2023-10-10T17:53:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_running_fast_dispatch_time": 0.016265239, 2023-10-10T17:53:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:08.917 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 522, 2023-10-10T17:53:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.043857707, 2023-10-10T17:53:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000084018 2023-10-10T17:53:08.918 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.919 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-10T17:53:08.920 INFO:teuthology.orchestra.run.smithi157.stdout: "msgr_send_encrypted_bytes": 448 2023-10-10T17:53:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "cct": { 2023-10-10T17:53:08.921 INFO:teuthology.orchestra.run.smithi157.stdout: "total_workers": 1, 2023-10-10T17:53:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "unhealthy_workers": 0 2023-10-10T17:53:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-MDSRank": { 2023-10-10T17:53:08.922 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2023-10-10T17:53:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2023-10-10T17:53:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 2457, 2023-10-10T17:53:08.923 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.316117866, 2023-10-10T17:53:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000128660 2023-10-10T17:53:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.924 INFO:teuthology.orchestra.run.smithi157.stdout: "finisher-PurgeQueue": { 2023-10-10T17:53:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "queue_len": 0, 2023-10-10T17:53:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "complete_latency": { 2023-10-10T17:53:08.925 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:08.926 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "mds": { 2023-10-10T17:53:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "request": 0, 2023-10-10T17:53:08.927 INFO:teuthology.orchestra.run.smithi157.stdout: "reply": 0, 2023-10-10T17:53:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "reply_latency": { 2023-10-10T17:53:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.928 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_reply": 0, 2023-10-10T17:53:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "forward": 0, 2023-10-10T17:53:08.929 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_complete": 0, 2023-10-10T17:53:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_fetch_keys": 0, 2023-10-10T17:53:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_commit": 0, 2023-10-10T17:53:08.930 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_split": 0, 2023-10-10T17:53:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_merge": 0, 2023-10-10T17:53:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes": 34910, 2023-10-10T17:53:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_top": 24419, 2023-10-10T17:53:08.931 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_bottom": 10488, 2023-10-10T17:53:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pin_tail": 3, 2023-10-10T17:53:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_pinned": 25, 2023-10-10T17:53:08.932 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_expired": 1127, 2023-10-10T17:53:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "inodes_with_caps": 0, 2023-10-10T17:53:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "caps": 0, 2023-10-10T17:53:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "subtrees": 2, 2023-10-10T17:53:08.933 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse": 0, 2023-10-10T17:53:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_hit": 0, 2023-10-10T17:53:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_forward": 0, 2023-10-10T17:53:08.934 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_discover": 0, 2023-10-10T17:53:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_dir_fetch": 0, 2023-10-10T17:53:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_remote_ino": 0, 2023-10-10T17:53:08.935 INFO:teuthology.orchestra.run.smithi157.stdout: "traverse_lock": 0, 2023-10-10T17:53:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "load_cent": 0, 2023-10-10T17:53:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "q": 0, 2023-10-10T17:53:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "exported": 0, 2023-10-10T17:53:08.936 INFO:teuthology.orchestra.run.smithi157.stdout: "exported_inodes": 0, 2023-10-10T17:53:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "imported": 0, 2023-10-10T17:53:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "imported_inodes": 0, 2023-10-10T17:53:08.937 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_dir_fetch": 0, 2023-10-10T17:53:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_backtrace_fetch": 0, 2023-10-10T17:53:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "openino_peer_discover": 0, 2023-10-10T17:53:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rfiles": 0, 2023-10-10T17:53:08.938 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rbytes": 0, 2023-10-10T17:53:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "root_rsnaps": 6, 2023-10-10T17:53:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_fetch": 0, 2023-10-10T17:53:08.939 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_set_tag": 0, 2023-10-10T17:53:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_backtrace_repaired": 0, 2023-10-10T17:53:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_inotable_repaired": 0, 2023-10-10T17:53:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_inodes": 0, 2023-10-10T17:53:08.940 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dir_base_inodes": 0, 2023-10-10T17:53:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_dirfrag_rstats": 0, 2023-10-10T17:53:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "scrub_file_inodes": 0, 2023-10-10T17:53:08.941 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_inode_file_caps": 0, 2023-10-10T17:53:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_revoke": 0, 2023-10-10T17:53:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_grant": 0, 2023-10-10T17:53:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_trunc": 0, 2023-10-10T17:53:08.942 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-10T17:53:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-10T17:53:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps": 0, 2023-10-10T17:53:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_caps_dirty": 0, 2023-10-10T17:53:08.943 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_cap_release": 0, 2023-10-10T17:53:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "process_request_cap_release": 0 2023-10-10T17:53:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.944 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_cache": { 2023-10-10T17:53:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update": 0, 2023-10-10T17:53:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_update_receipt": 0, 2023-10-10T17:53:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_try_discover": 0, 2023-10-10T17:53:08.945 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_send_discover": 0, 2023-10-10T17:53:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "dir_handle_discover": 0, 2023-10-10T17:53:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays": 0, 2023-10-10T17:53:08.946 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_delayed": 0, 2023-10-10T17:53:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "num_strays_enqueuing": 0, 2023-10-10T17:53:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_created": 0, 2023-10-10T17:53:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_enqueued": 0, 2023-10-10T17:53:08.947 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_reintegrated": 0, 2023-10-10T17:53:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "strays_migrated": 0, 2023-10-10T17:53:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_processing": 0, 2023-10-10T17:53:08.948 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_enqueued": 0, 2023-10-10T17:53:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "num_recovering_prioritized": 0, 2023-10-10T17:53:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_started": 0, 2023-10-10T17:53:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "recovery_completed": 0, 2023-10-10T17:53:08.949 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_enqueue_scrub": 0, 2023-10-10T17:53:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_exportdir": 0, 2023-10-10T17:53:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_flush": 0, 2023-10-10T17:53:08.950 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragmentdir": 0, 2023-10-10T17:53:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_fragstats": 0, 2023-10-10T17:53:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "ireq_inodestats": 0 2023-10-10T17:53:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.951 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_log": { 2023-10-10T17:53:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "evadd": 0, 2023-10-10T17:53:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "evex": 0, 2023-10-10T17:53:08.952 INFO:teuthology.orchestra.run.smithi157.stdout: "evtrm": 0, 2023-10-10T17:53:08.953 INFO:teuthology.orchestra.run.smithi157.stdout: "ev": 0, 2023-10-10T17:53:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "evexg": 0, 2023-10-10T17:53:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "evexd": 0, 2023-10-10T17:53:08.954 INFO:teuthology.orchestra.run.smithi157.stdout: "segadd": 0, 2023-10-10T17:53:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "segex": 0, 2023-10-10T17:53:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "segtrm": 0, 2023-10-10T17:53:08.955 INFO:teuthology.orchestra.run.smithi157.stdout: "seg": 2, 2023-10-10T17:53:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "segexg": 0, 2023-10-10T17:53:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "segexd": 0, 2023-10-10T17:53:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "expos": 347607314, 2023-10-10T17:53:08.956 INFO:teuthology.orchestra.run.smithi157.stdout: "wrpos": 0, 2023-10-10T17:53:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "rdpos": 347751599, 2023-10-10T17:53:08.957 INFO:teuthology.orchestra.run.smithi157.stdout: "jlat": { 2023-10-10T17:53:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.958 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "replayed": 32402 2023-10-10T17:53:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.959 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_mem": { 2023-10-10T17:53:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "ino": 11229, 2023-10-10T17:53:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "ino+": 42180, 2023-10-10T17:53:08.960 INFO:teuthology.orchestra.run.smithi157.stdout: "ino-": 30951, 2023-10-10T17:53:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "dir": 519, 2023-10-10T17:53:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "dir+": 1098, 2023-10-10T17:53:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "dir-": 579, 2023-10-10T17:53:08.961 INFO:teuthology.orchestra.run.smithi157.stdout: "dn": 34910, 2023-10-10T17:53:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "dn+": 67233, 2023-10-10T17:53:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "dn-": 32323, 2023-10-10T17:53:08.962 INFO:teuthology.orchestra.run.smithi157.stdout: "cap": 0, 2023-10-10T17:53:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: "cap+": 0, 2023-10-10T17:53:08.963 INFO:teuthology.orchestra.run.smithi157.stdout: "cap-": 0, 2023-10-10T17:53:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "rss": 180712, 2023-10-10T17:53:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: "heap": 207156 2023-10-10T17:53:08.964 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_server": { 2023-10-10T17:53:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_client_request": 0, 2023-10-10T17:53:08.965 INFO:teuthology.orchestra.run.smithi157.stdout: "dispatch_server_request": 0, 2023-10-10T17:53:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_request": 0, 2023-10-10T17:53:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_client_session": 0, 2023-10-10T17:53:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "handle_peer_request": 0, 2023-10-10T17:53:08.966 INFO:teuthology.orchestra.run.smithi157.stdout: "req_create_latency": { 2023-10-10T17:53:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.967 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.968 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getattr_latency": { 2023-10-10T17:53:08.969 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.970 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getfilelock_latency": { 2023-10-10T17:53:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.971 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "req_link_latency": { 2023-10-10T17:53:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.972 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.973 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookup_latency": { 2023-10-10T17:53:08.974 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.975 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookuphash_latency": { 2023-10-10T17:53:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.976 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupino_latency": { 2023-10-10T17:53:08.977 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.978 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupname_latency": { 2023-10-10T17:53:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.979 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupparent_latency": { 2023-10-10T17:53:08.980 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.981 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lookupsnap_latency": { 2023-10-10T17:53:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.982 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.983 INFO:teuthology.orchestra.run.smithi157.stdout: "req_lssnap_latency": { 2023-10-10T17:53:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.984 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mkdir_latency": { 2023-10-10T17:53:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.985 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.986 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.986 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.986 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mknod_latency": { 2023-10-10T17:53:08.987 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.987 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.987 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.987 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.988 INFO:teuthology.orchestra.run.smithi157.stdout: "req_mksnap_latency": { 2023-10-10T17:53:08.988 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.988 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.989 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.989 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.989 INFO:teuthology.orchestra.run.smithi157.stdout: "req_open_latency": { 2023-10-10T17:53:08.989 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.990 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.990 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.990 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.991 INFO:teuthology.orchestra.run.smithi157.stdout: "req_readdir_latency": { 2023-10-10T17:53:08.991 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.991 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.992 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.992 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.992 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rename_latency": { 2023-10-10T17:53:08.992 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.993 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.993 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.993 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.994 INFO:teuthology.orchestra.run.smithi157.stdout: "req_renamesnap_latency": { 2023-10-10T17:53:08.994 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.994 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.994 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.995 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.995 INFO:teuthology.orchestra.run.smithi157.stdout: "req_snapdiff_latency": { 2023-10-10T17:53:08.995 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.996 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.996 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.996 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.996 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmdir_latency": { 2023-10-10T17:53:08.997 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.997 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.997 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.997 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.998 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmsnap_latency": { 2023-10-10T17:53:08.998 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:08.998 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:08.999 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:08.999 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:08.999 INFO:teuthology.orchestra.run.smithi157.stdout: "req_rmxattr_latency": { 2023-10-10T17:53:08.999 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.000 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.000 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.000 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.001 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setattr_latency": { 2023-10-10T17:53:09.001 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.001 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.001 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.002 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.002 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setdirlayout_latency": { 2023-10-10T17:53:09.003 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.003 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.003 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.004 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.004 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setfilelock_latency": { 2023-10-10T17:53:09.004 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.005 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.005 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.005 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.005 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setlayout_latency": { 2023-10-10T17:53:09.006 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.006 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.006 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.007 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.007 INFO:teuthology.orchestra.run.smithi157.stdout: "req_setxattr_latency": { 2023-10-10T17:53:09.007 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.007 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.008 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.008 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.008 INFO:teuthology.orchestra.run.smithi157.stdout: "req_symlink_latency": { 2023-10-10T17:53:09.008 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.009 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.009 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.009 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.010 INFO:teuthology.orchestra.run.smithi157.stdout: "req_unlink_latency": { 2023-10-10T17:53:09.010 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.010 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.010 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.011 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.011 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_revoke_eviction": 0, 2023-10-10T17:53:09.011 INFO:teuthology.orchestra.run.smithi157.stdout: "cap_acquisition_throttle": 0, 2023-10-10T17:53:09.012 INFO:teuthology.orchestra.run.smithi157.stdout: "req_getvxattr_latency": { 2023-10-10T17:53:09.012 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.012 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.012 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.013 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:09.013 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.013 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_sessions": { 2023-10-10T17:53:09.013 INFO:teuthology.orchestra.run.smithi157.stdout: "session_count": 2, 2023-10-10T17:53:09.014 INFO:teuthology.orchestra.run.smithi157.stdout: "session_add": 2, 2023-10-10T17:53:09.014 INFO:teuthology.orchestra.run.smithi157.stdout: "session_remove": 0, 2023-10-10T17:53:09.014 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_open": 0, 2023-10-10T17:53:09.015 INFO:teuthology.orchestra.run.smithi157.stdout: "sessions_stale": 0, 2023-10-10T17:53:09.015 INFO:teuthology.orchestra.run.smithi157.stdout: "total_load": 0, 2023-10-10T17:53:09.015 INFO:teuthology.orchestra.run.smithi157.stdout: "average_load": 0, 2023-10-10T17:53:09.015 INFO:teuthology.orchestra.run.smithi157.stdout: "avg_session_uptime": 2021 2023-10-10T17:53:09.016 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.016 INFO:teuthology.orchestra.run.smithi157.stdout: "mempool": { 2023-10-10T17:53:09.016 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:53:09.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bloom_filter_items": 0, 2023-10-10T17:53:09.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:09.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:09.017 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:09.018 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:09.018 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:09.018 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:09.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:09.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:09.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:09.019 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:09.020 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:09.020 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:09.020 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:09.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:09.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:09.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:09.021 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:09.022 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:09.022 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:09.022 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:09.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:09.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:09.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:09.023 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:09.024 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:09.024 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:09.024 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:09.024 INFO:teuthology.orchestra.run.smithi157.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:09.025 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:09.025 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_items": 0, 2023-10-10T17:53:09.025 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:09.025 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:09.026 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:09.026 INFO:teuthology.orchestra.run.smithi157.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:09.026 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_bytes": 604539, 2023-10-10T17:53:09.027 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_anon_items": 11305, 2023-10-10T17:53:09.027 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:53:09.027 INFO:teuthology.orchestra.run.smithi157.stdout: "buffer_meta_items": 0, 2023-10-10T17:53:09.027 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_bytes": 0, 2023-10-10T17:53:09.028 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_items": 0, 2023-10-10T17:53:09.028 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:09.028 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:09.029 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:09.029 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:09.029 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_bytes": 9760, 2023-10-10T17:53:09.029 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_items": 189, 2023-10-10T17:53:09.030 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:09.030 INFO:teuthology.orchestra.run.smithi157.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:09.030 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:09.031 INFO:teuthology.orchestra.run.smithi157.stdout: "pgmap_items": 0, 2023-10-10T17:53:09.031 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_bytes": 50380810, 2023-10-10T17:53:09.031 INFO:teuthology.orchestra.run.smithi157.stdout: "mds_co_items": 374819, 2023-10-10T17:53:09.031 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:09.032 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_1_items": 0, 2023-10-10T17:53:09.032 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:09.033 INFO:teuthology.orchestra.run.smithi157.stdout: "unittest_2_items": 0 2023-10-10T17:53:09.033 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.033 INFO:teuthology.orchestra.run.smithi157.stdout: "objecter": { 2023-10-10T17:53:09.033 INFO:teuthology.orchestra.run.smithi157.stdout: "op_active": 0, 2023-10-10T17:53:09.034 INFO:teuthology.orchestra.run.smithi157.stdout: "op_laggy": 0, 2023-10-10T17:53:09.034 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send": 1534, 2023-10-10T17:53:09.034 INFO:teuthology.orchestra.run.smithi157.stdout: "op_send_bytes": 26, 2023-10-10T17:53:09.035 INFO:teuthology.orchestra.run.smithi157.stdout: "op_resend": 4, 2023-10-10T17:53:09.035 INFO:teuthology.orchestra.run.smithi157.stdout: "op_reply": 1534, 2023-10-10T17:53:09.035 INFO:teuthology.orchestra.run.smithi157.stdout: "op_latency": { 2023-10-10T17:53:09.035 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1534, 2023-10-10T17:53:09.036 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 1.727968741, 2023-10-10T17:53:09.036 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.001126446 2023-10-10T17:53:09.036 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.037 INFO:teuthology.orchestra.run.smithi157.stdout: "op_inflight": 0, 2023-10-10T17:53:09.037 INFO:teuthology.orchestra.run.smithi157.stdout: "oplen_avg": { 2023-10-10T17:53:09.037 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 1534, 2023-10-10T17:53:09.037 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 1535 2023-10-10T17:53:09.038 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.038 INFO:teuthology.orchestra.run.smithi157.stdout: "op": 1534, 2023-10-10T17:53:09.038 INFO:teuthology.orchestra.run.smithi157.stdout: "op_r": 1534, 2023-10-10T17:53:09.039 INFO:teuthology.orchestra.run.smithi157.stdout: "op_w": 0, 2023-10-10T17:53:09.039 INFO:teuthology.orchestra.run.smithi157.stdout: "op_rmw": 0, 2023-10-10T17:53:09.039 INFO:teuthology.orchestra.run.smithi157.stdout: "op_pg": 0, 2023-10-10T17:53:09.039 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_stat": 939, 2023-10-10T17:53:09.040 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_create": 0, 2023-10-10T17:53:09.040 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_read": 592, 2023-10-10T17:53:09.040 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_write": 0, 2023-10-10T17:53:09.040 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writefull": 0, 2023-10-10T17:53:09.041 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_writesame": 0, 2023-10-10T17:53:09.041 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_append": 0, 2023-10-10T17:53:09.041 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_zero": 0, 2023-10-10T17:53:09.042 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_truncate": 0, 2023-10-10T17:53:09.042 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_delete": 0, 2023-10-10T17:53:09.042 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_mapext": 0, 2023-10-10T17:53:09.042 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_sparse_read": 0, 2023-10-10T17:53:09.043 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_clonerange": 0, 2023-10-10T17:53:09.043 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_getxattr": 2, 2023-10-10T17:53:09.043 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_setxattr": 0, 2023-10-10T17:53:09.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_cmpxattr": 0, 2023-10-10T17:53:09.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_rmxattr": 0, 2023-10-10T17:53:09.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_resetxattrs": 0, 2023-10-10T17:53:09.044 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_call": 0, 2023-10-10T17:53:09.045 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_watch": 0, 2023-10-10T17:53:09.045 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_notify": 0, 2023-10-10T17:53:09.045 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_src_cmpxattr": 0, 2023-10-10T17:53:09.046 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls": 0, 2023-10-10T17:53:09.046 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_pgls_filter": 0, 2023-10-10T17:53:09.046 INFO:teuthology.orchestra.run.smithi157.stdout: "osdop_other": 0, 2023-10-10T17:53:09.046 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_active": 0, 2023-10-10T17:53:09.047 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_send": 0, 2023-10-10T17:53:09.047 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_resend": 0, 2023-10-10T17:53:09.047 INFO:teuthology.orchestra.run.smithi157.stdout: "linger_ping": 0, 2023-10-10T17:53:09.048 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_active": 0, 2023-10-10T17:53:09.048 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_send": 0, 2023-10-10T17:53:09.048 INFO:teuthology.orchestra.run.smithi157.stdout: "poolop_resend": 0, 2023-10-10T17:53:09.048 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_active": 0, 2023-10-10T17:53:09.049 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_send": 0, 2023-10-10T17:53:09.049 INFO:teuthology.orchestra.run.smithi157.stdout: "poolstat_resend": 0, 2023-10-10T17:53:09.049 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_active": 0, 2023-10-10T17:53:09.049 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_send": 0, 2023-10-10T17:53:09.050 INFO:teuthology.orchestra.run.smithi157.stdout: "statfs_resend": 0, 2023-10-10T17:53:09.050 INFO:teuthology.orchestra.run.smithi157.stdout: "command_active": 0, 2023-10-10T17:53:09.050 INFO:teuthology.orchestra.run.smithi157.stdout: "command_send": 0, 2023-10-10T17:53:09.051 INFO:teuthology.orchestra.run.smithi157.stdout: "command_resend": 0, 2023-10-10T17:53:09.051 INFO:teuthology.orchestra.run.smithi157.stdout: "map_epoch": 103, 2023-10-10T17:53:09.051 INFO:teuthology.orchestra.run.smithi157.stdout: "map_full": 0, 2023-10-10T17:53:09.051 INFO:teuthology.orchestra.run.smithi157.stdout: "map_inc": 9, 2023-10-10T17:53:09.052 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_sessions": 9, 2023-10-10T17:53:09.052 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_open": 9, 2023-10-10T17:53:09.052 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_session_close": 0, 2023-10-10T17:53:09.053 INFO:teuthology.orchestra.run.smithi157.stdout: "osd_laggy": 0, 2023-10-10T17:53:09.053 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_wr": 0, 2023-10-10T17:53:09.053 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_rd": 2, 2023-10-10T17:53:09.053 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_del": 0 2023-10-10T17:53:09.054 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.054 INFO:teuthology.orchestra.run.smithi157.stdout: "oft": { 2023-10-10T17:53:09.054 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_objs": 0, 2023-10-10T17:53:09.054 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_kv_pairs": 0, 2023-10-10T17:53:09.055 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_updates": 0, 2023-10-10T17:53:09.055 INFO:teuthology.orchestra.run.smithi157.stdout: "omap_total_removes": 0 2023-10-10T17:53:09.055 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.056 INFO:teuthology.orchestra.run.smithi157.stdout: "osd-slow-ops": { 2023-10-10T17:53:09.056 INFO:teuthology.orchestra.run.smithi157.stdout: "slow_ops_count": 0 2023-10-10T17:53:09.056 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.057 INFO:teuthology.orchestra.run.smithi157.stdout: "purge_queue": { 2023-10-10T17:53:09.057 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops": 0, 2023-10-10T17:53:09.057 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_ops_high_water": 0, 2023-10-10T17:53:09.057 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing": 0, 2023-10-10T17:53:09.058 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executing_high_water": 0, 2023-10-10T17:53:09.058 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_executed": 0, 2023-10-10T17:53:09.058 INFO:teuthology.orchestra.run.smithi157.stdout: "pq_item_in_journal": 0 2023-10-10T17:53:09.058 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.059 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:09.059 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:09.059 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2023-10-10T17:53:09.060 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:09.060 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 1772, 2023-10-10T17:53:09.060 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 173743002, 2023-10-10T17:53:09.060 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:09.061 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 1772, 2023-10-10T17:53:09.061 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:09.061 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:09.062 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 1772, 2023-10-10T17:53:09.062 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 173743002, 2023-10-10T17:53:09.062 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:09.062 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.063 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.063 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.063 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:09.063 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.064 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_bytes": { 2023-10-10T17:53:09.064 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:09.064 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 104857600, 2023-10-10T17:53:09.065 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:09.065 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:09.065 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:09.065 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:09.066 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:09.066 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 1534, 2023-10-10T17:53:09.066 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 172417201, 2023-10-10T17:53:09.067 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 1534, 2023-10-10T17:53:09.067 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 172417201, 2023-10-10T17:53:09.067 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:09.067 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.068 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.068 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.068 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:09.069 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.069 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-objecter_ops": { 2023-10-10T17:53:09.069 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:09.069 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 1024, 2023-10-10T17:53:09.070 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:09.070 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:09.070 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:09.071 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:09.071 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:09.071 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 1534, 2023-10-10T17:53:09.071 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 1534, 2023-10-10T17:53:09.072 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 1534, 2023-10-10T17:53:09.072 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 1534, 2023-10-10T17:53:09.072 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:09.073 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.073 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.073 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.073 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:09.074 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.074 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle": { 2023-10-10T17:53:09.074 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:09.075 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2023-10-10T17:53:09.075 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:09.075 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:09.075 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:09.076 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:09.077 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 0, 2023-10-10T17:53:09.077 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 0, 2023-10-10T17:53:09.077 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:09.078 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.078 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.078 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.078 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:09.079 INFO:teuthology.orchestra.run.smithi157.stdout: }, 2023-10-10T17:53:09.079 INFO:teuthology.orchestra.run.smithi157.stdout: "throttle-write_buf_throttle-0x55cd13fb6460": { 2023-10-10T17:53:09.079 INFO:teuthology.orchestra.run.smithi157.stdout: "val": 0, 2023-10-10T17:53:09.080 INFO:teuthology.orchestra.run.smithi157.stdout: "max": 3758096384, 2023-10-10T17:53:09.080 INFO:teuthology.orchestra.run.smithi157.stdout: "get_started": 0, 2023-10-10T17:53:09.080 INFO:teuthology.orchestra.run.smithi157.stdout: "get": 0, 2023-10-10T17:53:09.080 INFO:teuthology.orchestra.run.smithi157.stdout: "get_sum": 0, 2023-10-10T17:53:09.081 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:09.081 INFO:teuthology.orchestra.run.smithi157.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:09.081 INFO:teuthology.orchestra.run.smithi157.stdout: "take": 0, 2023-10-10T17:53:09.081 INFO:teuthology.orchestra.run.smithi157.stdout: "take_sum": 0, 2023-10-10T17:53:09.082 INFO:teuthology.orchestra.run.smithi157.stdout: "put": 0, 2023-10-10T17:53:09.082 INFO:teuthology.orchestra.run.smithi157.stdout: "put_sum": 0, 2023-10-10T17:53:09.082 INFO:teuthology.orchestra.run.smithi157.stdout: "wait": { 2023-10-10T17:53:09.083 INFO:teuthology.orchestra.run.smithi157.stdout: "avgcount": 0, 2023-10-10T17:53:09.083 INFO:teuthology.orchestra.run.smithi157.stdout: "sum": 0.000000000, 2023-10-10T17:53:09.083 INFO:teuthology.orchestra.run.smithi157.stdout: "avgtime": 0.000000000 2023-10-10T17:53:09.084 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:09.084 INFO:teuthology.orchestra.run.smithi157.stdout: } 2023-10-10T17:53:09.084 INFO:teuthology.orchestra.run.smithi157.stdout:} 2023-10-10T17:53:09.085 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:08 smithi157 ceph-mon[100896]: pgmap v330: 129 pgs: 8 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 95 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2023-10-10T17:53:09.129 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:08 smithi078 ceph-mon[99494]: pgmap v330: 129 pgs: 8 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 95 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2023-10-10T17:53:09.167 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:08 smithi190 ceph-mon[101220]: pgmap v330: 129 pgs: 8 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 95 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 KiB/s wr, 27 op/s 2023-10-10T17:53:09.347 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2023-10-10T17:53:09.348 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2023-10-10T17:53:09.348 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update=0 2023-10-10T17:53:09.348 INFO:tasks.check_counter:Daemon mds.k mds_cache.dir_update_receipt=0 2023-10-10T17:53:09.349 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=6 2023-10-10T17:53:09.349 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2023-10-10T17:53:09.349 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2023-10-10T17:53:09.350 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2023-10-10T17:53:09.350 DEBUG:tasks.check_counter:Getting stats from c 2023-10-10T17:53:09.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.c perf dump 2023-10-10T17:53:10.397 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:53:11.003 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:10 smithi190 ceph-mon[101220]: pgmap v331: 129 pgs: 8 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 95 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:11.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:10 smithi078 ceph-mon[99494]: pgmap v331: 129 pgs: 8 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 95 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:11.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:10 smithi157 ceph-mon[100896]: pgmap v331: 129 pgs: 8 active+clean+snaptrim, 26 active+clean+snaptrim_wait, 95 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:12.293 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-10-10T17:53:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 30766, 2023-10-10T17:53:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 31079, 2023-10-10T17:53:12.294 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 44567734, 2023-10-10T17:53:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 194996611, 2023-10-10T17:53:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 29, 2023-10-10T17:53:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 6, 2023-10-10T17:53:12.295 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 2.155469406, 2023-10-10T17:53:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.826605953, 2023-10-10T17:53:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.778529480, 2023-10-10T17:53:12.296 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.606719769, 2023-10-10T17:53:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 31079, 2023-10-10T17:53:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 31.244362144, 2023-10-10T17:53:12.297 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001005320 2023-10-10T17:53:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:12.298 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 21811, 2023-10-10T17:53:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.003436496, 2023-10-10T17:53:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000157 2023-10-10T17:53:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.299 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-10T17:53:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 0 2023-10-10T17:53:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:12.300 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 108867, 2023-10-10T17:53:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 113426, 2023-10-10T17:53:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 173115367, 2023-10-10T17:53:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 198172652, 2023-10-10T17:53:12.301 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 15, 2023-10-10T17:53:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 6, 2023-10-10T17:53:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 6.458381428, 2023-10-10T17:53:12.302 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 2.787935046, 2023-10-10T17:53:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 3.893656751, 2023-10-10T17:53:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.595348690, 2023-10-10T17:53:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:12.303 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 113424, 2023-10-10T17:53:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 15.294065107, 2023-10-10T17:53:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000134839 2023-10-10T17:53:12.304 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:12.305 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 177747, 2023-10-10T17:53:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.025298234, 2023-10-10T17:53:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000142 2023-10-10T17:53:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.306 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 4752, 2023-10-10T17:53:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 118944 2023-10-10T17:53:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.307 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 80682, 2023-10-10T17:53:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 139196, 2023-10-10T17:53:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 33454185, 2023-10-10T17:53:12.308 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 183166200, 2023-10-10T17:53:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 12, 2023-10-10T17:53:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2023-10-10T17:53:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 4.730664971, 2023-10-10T17:53:12.309 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 1.985894027, 2023-10-10T17:53:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 68.051881073, 2023-10-10T17:53:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.420831001, 2023-10-10T17:53:12.310 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 139193, 2023-10-10T17:53:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 26.325312234, 2023-10-10T17:53:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000189128 2023-10-10T17:53:12.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 172655, 2023-10-10T17:53:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.029553524, 2023-10-10T17:53:12.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000171 2023-10-10T17:53:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1107776, 2023-10-10T17:53:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 454528 2023-10-10T17:53:12.313 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2023-10-10T17:53:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2023-10-10T17:53:12.314 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2023-10-10T17:53:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2023-10-10T17:53:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2023-10-10T17:53:12.315 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2023-10-10T17:53:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 6977, 2023-10-10T17:53:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 66.346986822, 2023-10-10T17:53:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.009509386 2023-10-10T17:53:12.316 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2023-10-10T17:53:12.317 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2023-10-10T17:53:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2023-10-10T17:53:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 26882, 2023-10-10T17:53:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 16.491185112, 2023-10-10T17:53:12.318 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000613465 2023-10-10T17:53:12.319 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.319 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2023-10-10T17:53:12.319 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 50242, 2023-10-10T17:53:12.320 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 41489, 2023-10-10T17:53:12.320 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2023-10-10T17:53:12.320 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 41489, 2023-10-10T17:53:12.321 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 207.788457910, 2023-10-10T17:53:12.321 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.005008278 2023-10-10T17:53:12.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.321 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2023-10-10T17:53:12.322 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 8753, 2023-10-10T17:53:12.322 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 1335, 2023-10-10T17:53:12.322 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2023-10-10T17:53:12.322 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 771, 2023-10-10T17:53:12.323 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 87, 2023-10-10T17:53:12.323 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 1, 2023-10-10T17:53:12.323 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 19972, 2023-10-10T17:53:12.324 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 13970, 2023-10-10T17:53:12.324 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 5991, 2023-10-10T17:53:12.324 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 11, 2023-10-10T17:53:12.324 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 12, 2023-10-10T17:53:12.325 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 107217, 2023-10-10T17:53:12.325 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 2, 2023-10-10T17:53:12.325 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 2, 2023-10-10T17:53:12.325 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 4, 2023-10-10T17:53:12.326 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 70624, 2023-10-10T17:53:12.326 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 60771, 2023-10-10T17:53:12.326 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 8610, 2023-10-10T17:53:12.327 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 11, 2023-10-10T17:53:12.327 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 818, 2023-10-10T17:53:12.327 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2023-10-10T17:53:12.327 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 2, 2023-10-10T17:53:12.328 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2023-10-10T17:53:12.328 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2023-10-10T17:53:12.328 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 1822, 2023-10-10T17:53:12.328 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 89094, 2023-10-10T17:53:12.329 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 1220, 2023-10-10T17:53:12.329 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2023-10-10T17:53:12.329 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2023-10-10T17:53:12.330 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 2004, 2023-10-10T17:53:12.330 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2023-10-10T17:53:12.330 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 0, 2023-10-10T17:53:12.330 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 0, 2023-10-10T17:53:12.331 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 6, 2023-10-10T17:53:12.331 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 2004, 2023-10-10T17:53:12.331 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2023-10-10T17:53:12.332 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2023-10-10T17:53:12.332 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2023-10-10T17:53:12.332 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 1, 2023-10-10T17:53:12.332 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2023-10-10T17:53:12.333 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 1, 2023-10-10T17:53:12.333 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 2003, 2023-10-10T17:53:12.333 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2023-10-10T17:53:12.333 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 1038, 2023-10-10T17:53:12.334 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 24631, 2023-10-10T17:53:12.334 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2023-10-10T17:53:12.334 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-10T17:53:12.335 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-10T17:53:12.335 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 19699, 2023-10-10T17:53:12.335 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 18658, 2023-10-10T17:53:12.335 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 649, 2023-10-10T17:53:12.336 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 35942 2023-10-10T17:53:12.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.336 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2023-10-10T17:53:12.336 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 691, 2023-10-10T17:53:12.337 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 3700, 2023-10-10T17:53:12.337 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 1851, 2023-10-10T17:53:12.337 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 1861, 2023-10-10T17:53:12.337 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 691, 2023-10-10T17:53:12.338 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 1, 2023-10-10T17:53:12.338 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2023-10-10T17:53:12.338 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2023-10-10T17:53:12.339 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 17568, 2023-10-10T17:53:12.339 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 17567, 2023-10-10T17:53:12.339 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2023-10-10T17:53:12.339 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2023-10-10T17:53:12.340 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2023-10-10T17:53:12.340 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2023-10-10T17:53:12.340 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2023-10-10T17:53:12.340 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2023-10-10T17:53:12.341 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2023-10-10T17:53:12.341 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2023-10-10T17:53:12.341 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 1859, 2023-10-10T17:53:12.342 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2023-10-10T17:53:12.342 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 89, 2023-10-10T17:53:12.342 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2023-10-10T17:53:12.342 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2023-10-10T17:53:12.343 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.343 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2023-10-10T17:53:12.343 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 87068, 2023-10-10T17:53:12.343 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 14675, 2023-10-10T17:53:12.344 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 14675, 2023-10-10T17:53:12.344 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 72393, 2023-10-10T17:53:12.344 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2023-10-10T17:53:12.344 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 2843, 2023-10-10T17:53:12.345 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 151, 2023-10-10T17:53:12.345 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 23, 2023-10-10T17:53:12.345 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 23, 2023-10-10T17:53:12.346 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 131, 2023-10-10T17:53:12.346 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2023-10-10T17:53:12.346 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 3, 2023-10-10T17:53:12.346 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 48074336, 2023-10-10T17:53:12.347 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 410820087, 2023-10-10T17:53:12.347 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2023-10-10T17:53:12.347 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2023-10-10T17:53:12.347 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4419, 2023-10-10T17:53:12.348 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 230.393394480, 2023-10-10T17:53:12.348 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.052136998 2023-10-10T17:53:12.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.349 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2023-10-10T17:53:12.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.349 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2023-10-10T17:53:12.349 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 14313, 2023-10-10T17:53:12.350 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 100817, 2023-10-10T17:53:12.350 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 86504, 2023-10-10T17:53:12.350 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 500, 2023-10-10T17:53:12.351 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 3236, 2023-10-10T17:53:12.351 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 2736, 2023-10-10T17:53:12.351 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 19972, 2023-10-10T17:53:12.351 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 144567, 2023-10-10T17:53:12.352 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 124595, 2023-10-10T17:53:12.352 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 2, 2023-10-10T17:53:12.352 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 35460, 2023-10-10T17:53:12.352 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 35458, 2023-10-10T17:53:12.353 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 248856, 2023-10-10T17:53:12.353 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 198964 2023-10-10T17:53:12.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.353 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2023-10-10T17:53:12.354 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 52039, 2023-10-10T17:53:12.354 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 3, 2023-10-10T17:53:12.354 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 50242, 2023-10-10T17:53:12.355 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 13, 2023-10-10T17:53:12.355 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 313, 2023-10-10T17:53:12.355 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2023-10-10T17:53:12.355 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 5933, 2023-10-10T17:53:12.356 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 14.982512107, 2023-10-10T17:53:12.356 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002525284 2023-10-10T17:53:12.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.357 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2023-10-10T17:53:12.357 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 381, 2023-10-10T17:53:12.357 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1.013489142, 2023-10-10T17:53:12.357 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002660076 2023-10-10T17:53:12.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.358 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2023-10-10T17:53:12.358 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.359 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.359 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.359 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2023-10-10T17:53:12.360 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.360 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.360 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.361 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2023-10-10T17:53:12.361 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 16912, 2023-10-10T17:53:12.361 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 68.157358342, 2023-10-10T17:53:12.362 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.004030118 2023-10-10T17:53:12.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.362 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2023-10-10T17:53:12.362 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.363 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.363 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.364 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2023-10-10T17:53:12.364 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.364 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.364 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.365 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2023-10-10T17:53:12.365 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.365 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.366 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.366 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2023-10-10T17:53:12.367 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.367 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.367 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.368 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2023-10-10T17:53:12.368 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.368 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.368 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.369 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2023-10-10T17:53:12.369 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.370 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.370 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.370 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2023-10-10T17:53:12.371 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.371 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.371 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.372 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2023-10-10T17:53:12.372 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.372 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.373 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.373 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2023-10-10T17:53:12.373 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.374 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.374 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.374 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2023-10-10T17:53:12.375 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.375 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.375 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.376 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2023-10-10T17:53:12.376 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 695, 2023-10-10T17:53:12.376 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 1.151586744, 2023-10-10T17:53:12.377 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.001656959 2023-10-10T17:53:12.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.377 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2023-10-10T17:53:12.377 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.378 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.378 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.379 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2023-10-10T17:53:12.379 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.379 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.379 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.380 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2023-10-10T17:53:12.380 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.380 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.381 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.381 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2023-10-10T17:53:12.382 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1, 2023-10-10T17:53:12.382 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.027211188, 2023-10-10T17:53:12.382 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.027211188 2023-10-10T17:53:12.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.383 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2023-10-10T17:53:12.383 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.383 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.383 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.384 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2023-10-10T17:53:12.384 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.385 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.385 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.385 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2023-10-10T17:53:12.386 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.386 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.386 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.387 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2023-10-10T17:53:12.387 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.387 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.388 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.388 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2023-10-10T17:53:12.388 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.389 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.389 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.389 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2023-10-10T17:53:12.390 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.390 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.390 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.391 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2023-10-10T17:53:12.391 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.391 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.392 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.392 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2023-10-10T17:53:12.393 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.393 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.393 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.394 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2023-10-10T17:53:12.394 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 17567, 2023-10-10T17:53:12.394 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 122.456300387, 2023-10-10T17:53:12.394 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.006970814 2023-10-10T17:53:12.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.395 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2023-10-10T17:53:12.395 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2023-10-10T17:53:12.396 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2023-10-10T17:53:12.396 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.396 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.396 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.397 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.397 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2023-10-10T17:53:12.398 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2023-10-10T17:53:12.398 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 2, 2023-10-10T17:53:12.398 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 1, 2023-10-10T17:53:12.398 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2023-10-10T17:53:12.399 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2023-10-10T17:53:12.399 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 13656, 2023-10-10T17:53:12.399 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 13656, 2023-10-10T17:53:12.399 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 649 2023-10-10T17:53:12.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.400 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2023-10-10T17:53:12.400 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:53:12.400 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2023-10-10T17:53:12.401 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:12.401 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:12.401 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:12.402 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:12.402 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:12.402 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:12.402 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:12.403 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:12.403 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:12.403 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:12.403 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:12.404 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:12.404 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:12.404 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:12.405 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:12.405 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:12.405 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:12.405 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:12.406 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:12.406 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:12.406 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:12.406 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:12.407 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:12.407 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:12.407 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:12.408 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:12.408 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:12.408 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:12.408 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:12.409 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2023-10-10T17:53:12.409 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:12.409 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:12.409 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:12.410 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:12.410 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 766961, 2023-10-10T17:53:12.410 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 14405, 2023-10-10T17:53:12.411 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 176, 2023-10-10T17:53:12.411 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 2, 2023-10-10T17:53:12.411 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2023-10-10T17:53:12.411 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2023-10-10T17:53:12.412 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:12.412 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:12.412 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:12.412 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:12.413 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 9760, 2023-10-10T17:53:12.413 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 189, 2023-10-10T17:53:12.413 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:12.413 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:12.414 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:12.414 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2023-10-10T17:53:12.414 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 46829770, 2023-10-10T17:53:12.415 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 387599, 2023-10-10T17:53:12.415 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:12.415 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2023-10-10T17:53:12.415 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:12.416 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2023-10-10T17:53:12.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.416 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2023-10-10T17:53:12.417 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2023-10-10T17:53:12.417 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2023-10-10T17:53:12.417 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 47237, 2023-10-10T17:53:12.417 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 419206667, 2023-10-10T17:53:12.418 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 33, 2023-10-10T17:53:12.418 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 47237, 2023-10-10T17:53:12.418 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2023-10-10T17:53:12.419 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 47237, 2023-10-10T17:53:12.419 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 872.027558961, 2023-10-10T17:53:12.419 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.018460688 2023-10-10T17:53:12.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.420 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2023-10-10T17:53:12.420 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2023-10-10T17:53:12.420 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 47237, 2023-10-10T17:53:12.421 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 53279 2023-10-10T17:53:12.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.421 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 47237, 2023-10-10T17:53:12.421 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 3660, 2023-10-10T17:53:12.422 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 43577, 2023-10-10T17:53:12.422 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2023-10-10T17:53:12.422 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2023-10-10T17:53:12.422 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 63, 2023-10-10T17:53:12.423 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 2516, 2023-10-10T17:53:12.423 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 321, 2023-10-10T17:53:12.423 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 4740, 2023-10-10T17:53:12.424 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 145, 2023-10-10T17:53:12.424 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2023-10-10T17:53:12.424 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2023-10-10T17:53:12.424 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2023-10-10T17:53:12.425 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 1, 2023-10-10T17:53:12.425 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 35346, 2023-10-10T17:53:12.425 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2023-10-10T17:53:12.425 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2023-10-10T17:53:12.426 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2023-10-10T17:53:12.426 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 2004, 2023-10-10T17:53:12.426 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 3779, 2023-10-10T17:53:12.427 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2023-10-10T17:53:12.427 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2023-10-10T17:53:12.427 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2023-10-10T17:53:12.427 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2023-10-10T17:53:12.428 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2023-10-10T17:53:12.428 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2023-10-10T17:53:12.428 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2023-10-10T17:53:12.429 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2023-10-10T17:53:12.429 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2023-10-10T17:53:12.429 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2023-10-10T17:53:12.429 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2023-10-10T17:53:12.430 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2023-10-10T17:53:12.430 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2023-10-10T17:53:12.430 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2023-10-10T17:53:12.430 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2023-10-10T17:53:12.431 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2023-10-10T17:53:12.431 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2023-10-10T17:53:12.431 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2023-10-10T17:53:12.432 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2023-10-10T17:53:12.432 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2023-10-10T17:53:12.432 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2023-10-10T17:53:12.432 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2023-10-10T17:53:12.433 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2023-10-10T17:53:12.433 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2023-10-10T17:53:12.433 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2023-10-10T17:53:12.433 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2023-10-10T17:53:12.434 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 103, 2023-10-10T17:53:12.434 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2023-10-10T17:53:12.434 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 14, 2023-10-10T17:53:12.435 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 12, 2023-10-10T17:53:12.435 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 12, 2023-10-10T17:53:12.435 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2023-10-10T17:53:12.435 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2023-10-10T17:53:12.436 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 1635, 2023-10-10T17:53:12.436 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 2670, 2023-10-10T17:53:12.436 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 59 2023-10-10T17:53:12.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.437 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2023-10-10T17:53:12.437 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2023-10-10T17:53:12.437 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2023-10-10T17:53:12.437 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2023-10-10T17:53:12.438 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2023-10-10T17:53:12.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.438 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2023-10-10T17:53:12.439 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 52193 2023-10-10T17:53:12.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.439 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2023-10-10T17:53:12.439 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2023-10-10T17:53:12.440 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 16, 2023-10-10T17:53:12.440 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2023-10-10T17:53:12.440 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 8, 2023-10-10T17:53:12.440 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 17567, 2023-10-10T17:53:12.441 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2023-10-10T17:53:12.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.441 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:12.442 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:12.442 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2023-10-10T17:53:12.442 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:12.442 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 220311, 2023-10-10T17:53:12.443 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 231290525, 2023-10-10T17:53:12.443 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:12.443 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 220311, 2023-10-10T17:53:12.443 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:12.444 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:12.444 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 220311, 2023-10-10T17:53:12.444 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 231290525, 2023-10-10T17:53:12.445 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:12.445 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.445 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.445 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.446 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.446 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2023-10-10T17:53:12.446 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:12.447 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2023-10-10T17:53:12.447 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:12.447 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:12.448 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:12.448 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:12.448 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:12.448 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 47237, 2023-10-10T17:53:12.449 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 421403310, 2023-10-10T17:53:12.449 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 47237, 2023-10-10T17:53:12.449 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 421403310, 2023-10-10T17:53:12.449 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:12.450 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.450 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.450 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.451 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.451 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2023-10-10T17:53:12.451 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:12.452 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2023-10-10T17:53:12.452 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:12.452 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:12.452 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:12.453 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:12.453 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:12.453 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 47237, 2023-10-10T17:53:12.453 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 47237, 2023-10-10T17:53:12.454 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 47237, 2023-10-10T17:53:12.454 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 47237, 2023-10-10T17:53:12.454 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:12.455 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.455 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.455 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.455 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.456 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2023-10-10T17:53:12.456 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:12.456 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2023-10-10T17:53:12.457 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:12.457 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 17567, 2023-10-10T17:53:12.457 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 2218003, 2023-10-10T17:53:12.458 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:12.458 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 17567, 2023-10-10T17:53:12.458 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:12.458 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:12.459 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 321, 2023-10-10T17:53:12.459 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 2218003, 2023-10-10T17:53:12.459 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:12.459 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.460 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.460 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.460 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:12.461 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x559d32ade0a0": { 2023-10-10T17:53:12.461 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:12.461 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2023-10-10T17:53:12.462 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:12.462 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 87068, 2023-10-10T17:53:12.462 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 406625783, 2023-10-10T17:53:12.462 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:12.463 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 87068, 2023-10-10T17:53:12.463 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:12.463 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:12.464 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 4419, 2023-10-10T17:53:12.464 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 406625783, 2023-10-10T17:53:12.464 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:12.464 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:12.465 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:12.465 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:12.465 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.465 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:12.466 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-10-10T17:53:13.133 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:12 smithi078 ceph-mon[99494]: pgmap v332: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:13.173 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:12 smithi190 ceph-mon[101220]: pgmap v332: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:13.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:12 smithi157 ceph-mon[100896]: pgmap v332: 129 pgs: 12 active+clean+snaptrim, 31 active+clean+snaptrim_wait, 86 active+clean; 799 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 23 op/s 2023-10-10T17:53:13.347 INFO:tasks.check_counter:Daemon mds.c mds.exported=1822 2023-10-10T17:53:13.348 INFO:tasks.check_counter:Daemon mds.c mds.imported=1220 2023-10-10T17:53:13.348 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=691 2023-10-10T17:53:13.348 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=3700 2023-10-10T17:53:13.349 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2023-10-10T17:53:13.349 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2023-10-10T17:53:13.349 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2023-10-10T17:53:13.350 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=87 2023-10-10T17:53:13.350 DEBUG:tasks.check_counter:Getting stats from f 2023-10-10T17:53:13.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.f perf dump 2023-10-10T17:53:15.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:14 smithi078 ceph-mon[99494]: pgmap v333: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:53:15.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:14 smithi190 ceph-mon[101220]: pgmap v333: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:53:15.231 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:14 smithi157 ceph-mon[100896]: pgmap v333: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:53:15.402 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:53:16.975 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:16 smithi190 ceph-mon[101220]: pgmap v334: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:53:17.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:16 smithi078 ceph-mon[99494]: pgmap v334: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:53:17.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:16 smithi157 ceph-mon[100896]: pgmap v334: 129 pgs: 24 active+clean+snaptrim, 72 active+clean+snaptrim_wait, 33 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-10-10T17:53:18.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:17 smithi157 ceph-mon[100896]: pgmap v335: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 95 B/s wr, 6 op/s 2023-10-10T17:53:18.233 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-10-10T17:53:18.234 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:18.234 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 1583, 2023-10-10T17:53:18.235 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 1563, 2023-10-10T17:53:18.235 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 130757622, 2023-10-10T17:53:18.235 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 537352, 2023-10-10T17:53:18.235 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 7, 2023-10-10T17:53:18.236 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 3, 2023-10-10T17:53:18.236 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.227050849, 2023-10-10T17:53:18.236 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.047077686, 2023-10-10T17:53:18.237 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.118710969, 2023-10-10T17:53:18.237 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.053406017, 2023-10-10T17:53:18.237 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:18.237 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1562, 2023-10-10T17:53:18.238 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.127550602, 2023-10-10T17:53:18.238 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000081658 2023-10-10T17:53:18.238 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.238 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:18.239 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.239 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.239 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.240 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.240 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1092928, 2023-10-10T17:53:18.240 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 121856 2023-10-10T17:53:18.241 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.241 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:18.241 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 633, 2023-10-10T17:53:18.241 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 791, 2023-10-10T17:53:18.242 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 143587929, 2023-10-10T17:53:18.242 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 621935, 2023-10-10T17:53:18.242 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 9, 2023-10-10T17:53:18.243 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 7, 2023-10-10T17:53:18.243 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.136115798, 2023-10-10T17:53:18.243 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.018042129, 2023-10-10T17:53:18.243 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.090035234, 2023-10-10T17:53:18.244 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.021037652, 2023-10-10T17:53:18.244 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:18.244 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 788, 2023-10-10T17:53:18.244 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.110241113, 2023-10-10T17:53:18.245 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000139899 2023-10-10T17:53:18.245 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.245 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:18.245 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.246 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.246 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.246 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.247 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:53:18.247 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 425632 2023-10-10T17:53:18.247 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.247 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:18.248 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 1046, 2023-10-10T17:53:18.248 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 1045, 2023-10-10T17:53:18.248 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 247798461, 2023-10-10T17:53:18.248 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 324206, 2023-10-10T17:53:18.249 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 6, 2023-10-10T17:53:18.249 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 5, 2023-10-10T17:53:18.249 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.290642989, 2023-10-10T17:53:18.250 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.020811460, 2023-10-10T17:53:18.250 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.131007473, 2023-10-10T17:53:18.250 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.132845266, 2023-10-10T17:53:18.250 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:18.251 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 1044, 2023-10-10T17:53:18.251 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.063500837, 2023-10-10T17:53:18.251 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000060824 2023-10-10T17:53:18.251 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.252 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:18.252 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.252 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.253 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.253 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.253 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-10T17:53:18.253 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 448 2023-10-10T17:53:18.254 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.254 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2023-10-10T17:53:18.254 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2023-10-10T17:53:18.254 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2023-10-10T17:53:18.255 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.255 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2023-10-10T17:53:18.255 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2023-10-10T17:53:18.256 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2023-10-10T17:53:18.256 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 4576, 2023-10-10T17:53:18.256 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.129931401, 2023-10-10T17:53:18.256 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000028394 2023-10-10T17:53:18.257 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.257 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.257 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2023-10-10T17:53:18.257 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2023-10-10T17:53:18.258 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2023-10-10T17:53:18.258 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.258 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.259 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.259 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.259 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.259 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2023-10-10T17:53:18.260 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 0, 2023-10-10T17:53:18.260 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 0, 2023-10-10T17:53:18.260 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2023-10-10T17:53:18.261 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.261 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.261 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.261 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.262 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2023-10-10T17:53:18.262 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 0, 2023-10-10T17:53:18.262 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 0, 2023-10-10T17:53:18.262 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2023-10-10T17:53:18.263 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 0, 2023-10-10T17:53:18.263 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 0, 2023-10-10T17:53:18.263 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 0, 2023-10-10T17:53:18.264 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 89398, 2023-10-10T17:53:18.264 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 2461, 2023-10-10T17:53:18.264 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 77056, 2023-10-10T17:53:18.264 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 9881, 2023-10-10T17:53:18.265 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 85882, 2023-10-10T17:53:18.265 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 0, 2023-10-10T17:53:18.265 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 0, 2023-10-10T17:53:18.265 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 0, 2023-10-10T17:53:18.266 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 1, 2023-10-10T17:53:18.266 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 0, 2023-10-10T17:53:18.266 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 0, 2023-10-10T17:53:18.267 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 0, 2023-10-10T17:53:18.267 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 0, 2023-10-10T17:53:18.267 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 0, 2023-10-10T17:53:18.267 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2023-10-10T17:53:18.268 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 0, 2023-10-10T17:53:18.268 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2023-10-10T17:53:18.268 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2023-10-10T17:53:18.268 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 0, 2023-10-10T17:53:18.269 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 0, 2023-10-10T17:53:18.269 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 0, 2023-10-10T17:53:18.269 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2023-10-10T17:53:18.270 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2023-10-10T17:53:18.270 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 0, 2023-10-10T17:53:18.270 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2023-10-10T17:53:18.270 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 0, 2023-10-10T17:53:18.271 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 0, 2023-10-10T17:53:18.271 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 0, 2023-10-10T17:53:18.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 0, 2023-10-10T17:53:18.271 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2023-10-10T17:53:18.272 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2023-10-10T17:53:18.272 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2023-10-10T17:53:18.272 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 0, 2023-10-10T17:53:18.273 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2023-10-10T17:53:18.273 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 0, 2023-10-10T17:53:18.273 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 0, 2023-10-10T17:53:18.273 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 0, 2023-10-10T17:53:18.274 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 0, 2023-10-10T17:53:18.274 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 0, 2023-10-10T17:53:18.274 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2023-10-10T17:53:18.274 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-10T17:53:18.275 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-10T17:53:18.275 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 0, 2023-10-10T17:53:18.275 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 0, 2023-10-10T17:53:18.276 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 0, 2023-10-10T17:53:18.276 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 0 2023-10-10T17:53:18.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.276 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2023-10-10T17:53:18.277 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 0, 2023-10-10T17:53:18.277 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 0, 2023-10-10T17:53:18.277 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 0, 2023-10-10T17:53:18.277 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 0, 2023-10-10T17:53:18.278 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 0, 2023-10-10T17:53:18.278 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 0, 2023-10-10T17:53:18.278 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2023-10-10T17:53:18.279 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2023-10-10T17:53:18.279 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 0, 2023-10-10T17:53:18.279 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 0, 2023-10-10T17:53:18.279 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2023-10-10T17:53:18.280 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2023-10-10T17:53:18.280 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2023-10-10T17:53:18.280 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2023-10-10T17:53:18.280 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2023-10-10T17:53:18.281 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2023-10-10T17:53:18.281 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2023-10-10T17:53:18.281 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2023-10-10T17:53:18.282 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 0, 2023-10-10T17:53:18.282 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2023-10-10T17:53:18.282 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 0, 2023-10-10T17:53:18.282 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2023-10-10T17:53:18.283 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2023-10-10T17:53:18.283 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.283 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2023-10-10T17:53:18.283 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 0, 2023-10-10T17:53:18.284 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 0, 2023-10-10T17:53:18.284 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 0, 2023-10-10T17:53:18.284 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 0, 2023-10-10T17:53:18.285 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2023-10-10T17:53:18.285 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 0, 2023-10-10T17:53:18.285 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 0, 2023-10-10T17:53:18.285 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 0, 2023-10-10T17:53:18.286 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 0, 2023-10-10T17:53:18.286 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 129, 2023-10-10T17:53:18.286 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2023-10-10T17:53:18.286 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 0, 2023-10-10T17:53:18.287 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 58725022, 2023-10-10T17:53:18.287 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 0, 2023-10-10T17:53:18.287 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 520834089, 2023-10-10T17:53:18.288 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2023-10-10T17:53:18.288 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.288 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.288 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.289 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 93861 2023-10-10T17:53:18.289 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.290 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2023-10-10T17:53:18.290 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 43681, 2023-10-10T17:53:18.290 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 96082, 2023-10-10T17:53:18.291 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 52401, 2023-10-10T17:53:18.291 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 1519, 2023-10-10T17:53:18.291 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 1844, 2023-10-10T17:53:18.292 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 325, 2023-10-10T17:53:18.292 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 89398, 2023-10-10T17:53:18.292 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 139231, 2023-10-10T17:53:18.293 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 49833, 2023-10-10T17:53:18.293 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 0, 2023-10-10T17:53:18.293 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 0, 2023-10-10T17:53:18.294 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 0, 2023-10-10T17:53:18.294 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 282440, 2023-10-10T17:53:18.294 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 207156 2023-10-10T17:53:18.294 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.295 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2023-10-10T17:53:18.295 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 0, 2023-10-10T17:53:18.295 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 0, 2023-10-10T17:53:18.296 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 0, 2023-10-10T17:53:18.296 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 0, 2023-10-10T17:53:18.296 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 0, 2023-10-10T17:53:18.297 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2023-10-10T17:53:18.297 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.297 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.298 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.298 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.298 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2023-10-10T17:53:18.299 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.299 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.299 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.300 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.300 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2023-10-10T17:53:18.300 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.301 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.301 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.301 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.302 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2023-10-10T17:53:18.302 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.303 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.303 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.303 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.303 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2023-10-10T17:53:18.304 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.304 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.304 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.305 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.305 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2023-10-10T17:53:18.305 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.306 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.306 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.306 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.307 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2023-10-10T17:53:18.307 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.307 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.307 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.308 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.308 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2023-10-10T17:53:18.308 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.309 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.309 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.309 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.310 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2023-10-10T17:53:18.310 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.310 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.310 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.311 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.311 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2023-10-10T17:53:18.311 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.312 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.312 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.312 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.313 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2023-10-10T17:53:18.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.313 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.313 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.314 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.314 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2023-10-10T17:53:18.314 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.315 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.315 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:17 smithi190 ceph-mon[101220]: pgmap v335: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 95 B/s wr, 6 op/s 2023-10-10T17:53:18.316 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.316 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.317 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2023-10-10T17:53:18.317 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.317 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.318 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.318 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.318 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2023-10-10T17:53:18.318 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.319 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.319 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.319 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.320 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2023-10-10T17:53:18.320 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.320 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.321 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.321 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.321 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2023-10-10T17:53:18.321 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.322 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.322 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.322 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.323 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2023-10-10T17:53:18.323 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.323 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.324 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.324 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.324 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2023-10-10T17:53:18.324 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.325 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.325 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.325 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.326 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2023-10-10T17:53:18.326 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.326 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.327 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.327 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.327 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2023-10-10T17:53:18.328 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.328 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.328 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.329 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2023-10-10T17:53:18.329 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.329 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.330 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.330 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2023-10-10T17:53:18.331 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.331 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.331 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.332 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2023-10-10T17:53:18.332 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.333 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.333 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.333 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2023-10-10T17:53:18.334 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.334 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.334 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.335 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2023-10-10T17:53:18.335 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.336 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.336 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.337 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2023-10-10T17:53:18.337 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.337 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.338 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.338 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2023-10-10T17:53:18.339 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.339 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.339 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.340 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2023-10-10T17:53:18.340 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.340 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.341 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.341 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2023-10-10T17:53:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.342 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.343 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2023-10-10T17:53:18.343 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2023-10-10T17:53:18.343 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2023-10-10T17:53:18.344 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.344 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.344 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.345 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.345 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2023-10-10T17:53:18.346 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2023-10-10T17:53:18.346 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 2, 2023-10-10T17:53:18.347 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 1, 2023-10-10T17:53:18.347 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 2, 2023-10-10T17:53:18.347 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2023-10-10T17:53:18.347 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 0, 2023-10-10T17:53:18.348 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 0, 2023-10-10T17:53:18.348 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 3498 2023-10-10T17:53:18.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.349 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2023-10-10T17:53:18.349 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:53:18.349 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2023-10-10T17:53:18.350 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:18.350 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:18.350 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:18.351 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:18.351 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:18.351 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:18.352 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:18.352 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:18.352 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:18.352 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:18.353 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:18.353 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:18.353 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:18.354 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:18.354 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:18.354 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:18.355 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:18.355 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:18.355 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:18.355 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:18.356 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:18.356 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:18.356 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:18.357 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:18.357 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:18.357 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:18.358 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:18.358 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:18.358 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:18.359 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2023-10-10T17:53:18.359 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:18.359 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:18.359 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:18.360 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:18.360 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 1723845, 2023-10-10T17:53:18.360 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 43759, 2023-10-10T17:53:18.361 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:53:18.361 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2023-10-10T17:53:18.361 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2023-10-10T17:53:18.361 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2023-10-10T17:53:18.362 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:18.362 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:18.362 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:18.363 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:18.363 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 9760, 2023-10-10T17:53:18.363 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 189, 2023-10-10T17:53:18.364 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:18.364 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:18.364 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:18.364 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2023-10-10T17:53:18.365 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 163122371, 2023-10-10T17:53:18.365 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 1339717, 2023-10-10T17:53:18.365 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:18.366 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2023-10-10T17:53:18.366 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:18.366 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2023-10-10T17:53:18.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.367 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2023-10-10T17:53:18.367 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2023-10-10T17:53:18.368 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2023-10-10T17:53:18.368 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 3008, 2023-10-10T17:53:18.368 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 21, 2023-10-10T17:53:18.368 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 3, 2023-10-10T17:53:18.369 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 3008, 2023-10-10T17:53:18.369 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2023-10-10T17:53:18.369 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3008, 2023-10-10T17:53:18.370 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 8.293767948, 2023-10-10T17:53:18.370 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002757236 2023-10-10T17:53:18.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.371 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2023-10-10T17:53:18.371 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2023-10-10T17:53:18.371 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3008, 2023-10-10T17:53:18.372 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 3009 2023-10-10T17:53:18.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.372 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 3008, 2023-10-10T17:53:18.373 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 3008, 2023-10-10T17:53:18.373 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 0, 2023-10-10T17:53:18.373 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2023-10-10T17:53:18.374 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2023-10-10T17:53:18.374 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 1956, 2023-10-10T17:53:18.374 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 0, 2023-10-10T17:53:18.375 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 1050, 2023-10-10T17:53:18.375 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 0, 2023-10-10T17:53:18.375 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 0, 2023-10-10T17:53:18.376 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2023-10-10T17:53:18.376 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2023-10-10T17:53:18.376 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2023-10-10T17:53:18.377 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 0, 2023-10-10T17:53:18.377 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 0, 2023-10-10T17:53:18.377 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2023-10-10T17:53:18.378 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2023-10-10T17:53:18.378 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2023-10-10T17:53:18.378 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 1, 2023-10-10T17:53:18.379 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 0, 2023-10-10T17:53:18.379 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2023-10-10T17:53:18.379 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2023-10-10T17:53:18.380 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2023-10-10T17:53:18.380 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2023-10-10T17:53:18.380 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2023-10-10T17:53:18.381 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2023-10-10T17:53:18.381 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2023-10-10T17:53:18.381 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2023-10-10T17:53:18.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2023-10-10T17:53:18.382 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2023-10-10T17:53:18.382 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2023-10-10T17:53:18.383 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2023-10-10T17:53:18.383 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2023-10-10T17:53:18.383 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2023-10-10T17:53:18.383 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2023-10-10T17:53:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2023-10-10T17:53:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2023-10-10T17:53:18.384 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2023-10-10T17:53:18.385 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2023-10-10T17:53:18.385 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2023-10-10T17:53:18.385 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2023-10-10T17:53:18.386 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2023-10-10T17:53:18.386 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2023-10-10T17:53:18.386 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2023-10-10T17:53:18.386 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2023-10-10T17:53:18.387 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2023-10-10T17:53:18.387 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 103, 2023-10-10T17:53:18.387 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2023-10-10T17:53:18.388 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 14, 2023-10-10T17:53:18.388 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 11, 2023-10-10T17:53:18.388 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 11, 2023-10-10T17:53:18.389 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2023-10-10T17:53:18.389 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2023-10-10T17:53:18.389 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 0, 2023-10-10T17:53:18.390 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 2, 2023-10-10T17:53:18.390 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 0 2023-10-10T17:53:18.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.390 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2023-10-10T17:53:18.391 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 0, 2023-10-10T17:53:18.391 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2023-10-10T17:53:18.391 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2023-10-10T17:53:18.392 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2023-10-10T17:53:18.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.392 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2023-10-10T17:53:18.393 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 0 2023-10-10T17:53:18.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.393 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2023-10-10T17:53:18.394 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2023-10-10T17:53:18.394 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 0, 2023-10-10T17:53:18.394 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2023-10-10T17:53:18.394 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 0, 2023-10-10T17:53:18.395 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 0, 2023-10-10T17:53:18.395 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2023-10-10T17:53:18.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.396 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:18.396 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:18.396 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2023-10-10T17:53:18.397 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:18.397 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 3258, 2023-10-10T17:53:18.397 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 521835166, 2023-10-10T17:53:18.397 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:18.398 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 3258, 2023-10-10T17:53:18.398 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:18.398 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:18.399 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 3258, 2023-10-10T17:53:18.399 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 521835166, 2023-10-10T17:53:18.399 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:18.400 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.400 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.400 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.401 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.401 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2023-10-10T17:53:18.401 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:18.402 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2023-10-10T17:53:18.402 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:18.402 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:18.403 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:18.403 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:18.403 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:18.404 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 3008, 2023-10-10T17:53:18.404 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 520218582, 2023-10-10T17:53:18.404 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 3008, 2023-10-10T17:53:18.405 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 520218582, 2023-10-10T17:53:18.405 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:18.405 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.405 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.406 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.406 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.409 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2023-10-10T17:53:18.409 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:18.410 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2023-10-10T17:53:18.410 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:18.410 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:18.411 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:18.411 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:18.411 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:18.412 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 3008, 2023-10-10T17:53:18.412 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 3008, 2023-10-10T17:53:18.412 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 3008, 2023-10-10T17:53:18.412 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 3008, 2023-10-10T17:53:18.413 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:18.413 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.413 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.414 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.415 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2023-10-10T17:53:18.415 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:18.415 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2023-10-10T17:53:18.415 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:18.416 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:18.416 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:18.416 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:18.417 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:18.417 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:18.417 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:18.418 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2023-10-10T17:53:18.418 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2023-10-10T17:53:18.418 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:18.418 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.419 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.419 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.419 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:18.420 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x55604481a460": { 2023-10-10T17:53:18.420 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:18.421 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2023-10-10T17:53:18.421 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:18.421 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:18.421 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:18.422 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:18.422 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:18.422 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:18.423 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:18.423 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 0, 2023-10-10T17:53:18.423 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 0, 2023-10-10T17:53:18.424 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:18.424 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:18.424 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:18.425 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:18.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:18.425 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-10-10T17:53:18.426 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:17 smithi078 ceph-mon[99494]: pgmap v335: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 95 B/s wr, 6 op/s 2023-10-10T17:53:19.251 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2023-10-10T17:53:19.252 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2023-10-10T17:53:19.252 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2023-10-10T17:53:19.252 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2023-10-10T17:53:19.253 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2023-10-10T17:53:19.253 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2023-10-10T17:53:19.253 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2023-10-10T17:53:19.254 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2023-10-10T17:53:19.254 DEBUG:tasks.check_counter:Getting stats from i 2023-10-10T17:53:19.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.i perf dump 2023-10-10T17:53:20.280 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:53:21.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:20 smithi078 ceph-mon[99494]: pgmap v336: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 4 op/s 2023-10-10T17:53:21.133 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:20 smithi190 ceph-mon[101220]: pgmap v336: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 4 op/s 2023-10-10T17:53:21.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:20 smithi157 ceph-mon[100896]: pgmap v336: 129 pgs: 22 active+clean+snaptrim, 70 active+clean+snaptrim_wait, 37 active+clean; 799 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 4 op/s 2023-10-10T17:53:22.664 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-10-10T17:53:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 209524, 2023-10-10T17:53:22.665 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 299849, 2023-10-10T17:53:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 103269524, 2023-10-10T17:53:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 412542547, 2023-10-10T17:53:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 28, 2023-10-10T17:53:22.666 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 8, 2023-10-10T17:53:22.667 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 10.899253313, 2023-10-10T17:53:22.667 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 4.927572472, 2023-10-10T17:53:22.667 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 324.424119579, 2023-10-10T17:53:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.505568795, 2023-10-10T17:53:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 299845, 2023-10-10T17:53:22.668 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 29.998189974, 2023-10-10T17:53:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000100045 2023-10-10T17:53:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.669 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 456875, 2023-10-10T17:53:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.067610885, 2023-10-10T17:53:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000147 2023-10-10T17:53:22.670 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 0, 2023-10-10T17:53:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 0 2023-10-10T17:53:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.671 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 25656, 2023-10-10T17:53:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 26907, 2023-10-10T17:53:22.672 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 29877191, 2023-10-10T17:53:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 228064249, 2023-10-10T17:53:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 11, 2023-10-10T17:53:22.673 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 5, 2023-10-10T17:53:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 1.928181031, 2023-10-10T17:53:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.734230994, 2023-10-10T17:53:22.674 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.684136444, 2023-10-10T17:53:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.539194341, 2023-10-10T17:53:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 26909, 2023-10-10T17:53:22.675 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 6.745820845, 2023-10-10T17:53:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000250690 2023-10-10T17:53:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:22.676 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 26912, 2023-10-10T17:53:22.677 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.004165606, 2023-10-10T17:53:22.677 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000154 2023-10-10T17:53:22.677 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1109936, 2023-10-10T17:53:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 129776 2023-10-10T17:53:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.678 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 26054, 2023-10-10T17:53:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 26213, 2023-10-10T17:53:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 21327801, 2023-10-10T17:53:22.679 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 219142333, 2023-10-10T17:53:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 17, 2023-10-10T17:53:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 6, 2023-10-10T17:53:22.680 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 2.229552238, 2023-10-10T17:53:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.869162187, 2023-10-10T17:53:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.376097766, 2023-10-10T17:53:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.883201589, 2023-10-10T17:53:22.681 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 26210, 2023-10-10T17:53:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 6.975867761, 2023-10-10T17:53:22.682 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000266152 2023-10-10T17:53:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 15, 2023-10-10T17:53:22.683 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000002284, 2023-10-10T17:53:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000152 2023-10-10T17:53:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1232, 2023-10-10T17:53:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 430336 2023-10-10T17:53:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.685 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2023-10-10T17:53:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 1, 2023-10-10T17:53:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2023-10-10T17:53:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.686 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-MDSRank": { 2023-10-10T17:53:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2023-10-10T17:53:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2023-10-10T17:53:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 8153, 2023-10-10T17:53:22.687 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 106.810511083, 2023-10-10T17:53:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.013100761 2023-10-10T17:53:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.688 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "finisher-PurgeQueue": { 2023-10-10T17:53:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "queue_len": 0, 2023-10-10T17:53:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "complete_latency": { 2023-10-10T17:53:22.689 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 24242, 2023-10-10T17:53:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 14.769226107, 2023-10-10T17:53:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000609241 2023-10-10T17:53:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.690 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "mds": { 2023-10-10T17:53:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "request": 65335, 2023-10-10T17:53:22.691 INFO:teuthology.orchestra.run.smithi190.stdout: "reply": 53834, 2023-10-10T17:53:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "reply_latency": { 2023-10-10T17:53:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 53834, 2023-10-10T17:53:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 321.756637477, 2023-10-10T17:53:22.692 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.005976829 2023-10-10T17:53:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_reply": 0, 2023-10-10T17:53:22.693 INFO:teuthology.orchestra.run.smithi190.stdout: "forward": 11501, 2023-10-10T17:53:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_complete": 2745, 2023-10-10T17:53:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_fetch_keys": 0, 2023-10-10T17:53:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_commit": 899, 2023-10-10T17:53:22.694 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_split": 109, 2023-10-10T17:53:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_merge": 16, 2023-10-10T17:53:22.695 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes": 74536, 2023-10-10T17:53:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_top": 4550, 2023-10-10T17:53:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_bottom": 12603, 2023-10-10T17:53:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pin_tail": 57383, 2023-10-10T17:53:22.696 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_pinned": 68033, 2023-10-10T17:53:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_expired": 52462, 2023-10-10T17:53:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "inodes_with_caps": 1, 2023-10-10T17:53:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "caps": 1, 2023-10-10T17:53:22.697 INFO:teuthology.orchestra.run.smithi190.stdout: "subtrees": 3, 2023-10-10T17:53:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse": 82244, 2023-10-10T17:53:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_hit": 70388, 2023-10-10T17:53:22.698 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_forward": 9552, 2023-10-10T17:53:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_discover": 6, 2023-10-10T17:53:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_dir_fetch": 401, 2023-10-10T17:53:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_remote_ino": 0, 2023-10-10T17:53:22.699 INFO:teuthology.orchestra.run.smithi190.stdout: "traverse_lock": 2, 2023-10-10T17:53:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "load_cent": 0, 2023-10-10T17:53:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "q": 0, 2023-10-10T17:53:22.700 INFO:teuthology.orchestra.run.smithi190.stdout: "exported": 791, 2023-10-10T17:53:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "exported_inodes": 49868, 2023-10-10T17:53:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "imported": 1379, 2023-10-10T17:53:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "imported_inodes": 0, 2023-10-10T17:53:22.701 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_dir_fetch": 0, 2023-10-10T17:53:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_backtrace_fetch": 8621, 2023-10-10T17:53:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "openino_peer_discover": 0, 2023-10-10T17:53:22.702 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rfiles": 0, 2023-10-10T17:53:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rbytes": 0, 2023-10-10T17:53:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "root_rsnaps": 6, 2023-10-10T17:53:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_fetch": 8621, 2023-10-10T17:53:22.703 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_set_tag": 0, 2023-10-10T17:53:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_backtrace_repaired": 0, 2023-10-10T17:53:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_inotable_repaired": 0, 2023-10-10T17:53:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_inodes": 6, 2023-10-10T17:53:22.704 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dir_base_inodes": 0, 2023-10-10T17:53:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_dirfrag_rstats": 6, 2023-10-10T17:53:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "scrub_file_inodes": 8615, 2023-10-10T17:53:22.705 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_inode_file_caps": 14, 2023-10-10T17:53:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_revoke": 53, 2023-10-10T17:53:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_grant": 35899, 2023-10-10T17:53:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_trunc": 0, 2023-10-10T17:53:22.706 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flushsnap_ack": 0, 2023-10-10T17:53:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_cap_op_flush_ack": 0, 2023-10-10T17:53:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps": 38669, 2023-10-10T17:53:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_caps_dirty": 38615, 2023-10-10T17:53:22.707 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_cap_release": 257, 2023-10-10T17:53:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "process_request_cap_release": 34095 2023-10-10T17:53:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.708 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_cache": { 2023-10-10T17:53:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update": 1684, 2023-10-10T17:53:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_update_receipt": 2035, 2023-10-10T17:53:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_try_discover": 1019, 2023-10-10T17:53:22.709 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_send_discover": 1040, 2023-10-10T17:53:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "dir_handle_discover": 2362, 2023-10-10T17:53:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays": 3, 2023-10-10T17:53:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_delayed": 0, 2023-10-10T17:53:22.710 INFO:teuthology.orchestra.run.smithi190.stdout: "num_strays_enqueuing": 0, 2023-10-10T17:53:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_created": 15735, 2023-10-10T17:53:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_enqueued": 15732, 2023-10-10T17:53:22.711 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_reintegrated": 0, 2023-10-10T17:53:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "strays_migrated": 0, 2023-10-10T17:53:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_processing": 0, 2023-10-10T17:53:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_enqueued": 0, 2023-10-10T17:53:22.712 INFO:teuthology.orchestra.run.smithi190.stdout: "num_recovering_prioritized": 0, 2023-10-10T17:53:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_started": 0, 2023-10-10T17:53:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "recovery_completed": 0, 2023-10-10T17:53:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_enqueue_scrub": 0, 2023-10-10T17:53:22.713 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_exportdir": 791, 2023-10-10T17:53:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_flush": 0, 2023-10-10T17:53:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragmentdir": 126, 2023-10-10T17:53:22.714 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_fragstats": 0, 2023-10-10T17:53:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "ireq_inodestats": 0 2023-10-10T17:53:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_log": { 2023-10-10T17:53:22.715 INFO:teuthology.orchestra.run.smithi190.stdout: "evadd": 93861, 2023-10-10T17:53:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "evex": 20140, 2023-10-10T17:53:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "evtrm": 20140, 2023-10-10T17:53:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "ev": 73721, 2023-10-10T17:53:22.716 INFO:teuthology.orchestra.run.smithi190.stdout: "evexg": 0, 2023-10-10T17:53:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "evexd": 568, 2023-10-10T17:53:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "segadd": 153, 2023-10-10T17:53:22.717 INFO:teuthology.orchestra.run.smithi190.stdout: "segex": 25, 2023-10-10T17:53:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "segtrm": 25, 2023-10-10T17:53:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "seg": 129, 2023-10-10T17:53:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "segexg": 0, 2023-10-10T17:53:22.718 INFO:teuthology.orchestra.run.smithi190.stdout: "segexd": 1, 2023-10-10T17:53:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "expos": 58725022, 2023-10-10T17:53:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "wrpos": 520840005, 2023-10-10T17:53:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "rdpos": 0, 2023-10-10T17:53:22.719 INFO:teuthology.orchestra.run.smithi190.stdout: "jlat": { 2023-10-10T17:53:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3080, 2023-10-10T17:53:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 192.475932931, 2023-10-10T17:53:22.720 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.062492186 2023-10-10T17:53:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: "replayed": 0 2023-10-10T17:53:22.721 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_mem": { 2023-10-10T17:53:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "ino": 45706, 2023-10-10T17:53:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "ino+": 107812, 2023-10-10T17:53:22.722 INFO:teuthology.orchestra.run.smithi190.stdout: "ino-": 62106, 2023-10-10T17:53:22.725 INFO:teuthology.orchestra.run.smithi190.stdout: "dir": 1524, 2023-10-10T17:53:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "dir+": 2935, 2023-10-10T17:53:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "dir-": 1411, 2023-10-10T17:53:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "dn": 74536, 2023-10-10T17:53:22.726 INFO:teuthology.orchestra.run.smithi190.stdout: "dn+": 143664, 2023-10-10T17:53:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "dn-": 69128, 2023-10-10T17:53:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "cap": 1, 2023-10-10T17:53:22.727 INFO:teuthology.orchestra.run.smithi190.stdout: "cap+": 44783, 2023-10-10T17:53:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "cap-": 44782, 2023-10-10T17:53:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "rss": 364564, 2023-10-10T17:53:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: "heap": 198964 2023-10-10T17:53:22.728 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_server": { 2023-10-10T17:53:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_client_request": 68780, 2023-10-10T17:53:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "dispatch_server_request": 246, 2023-10-10T17:53:22.729 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_request": 65335, 2023-10-10T17:53:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_client_session": 148, 2023-10-10T17:53:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "handle_peer_request": 501, 2023-10-10T17:53:22.730 INFO:teuthology.orchestra.run.smithi190.stdout: "req_create_latency": { 2023-10-10T17:53:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 19840, 2023-10-10T17:53:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 127.719238070, 2023-10-10T17:53:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.006437461 2023-10-10T17:53:22.731 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getattr_latency": { 2023-10-10T17:53:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 163, 2023-10-10T17:53:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.459602521, 2023-10-10T17:53:22.732 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.002819647 2023-10-10T17:53:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getfilelock_latency": { 2023-10-10T17:53:22.733 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.734 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "req_link_latency": { 2023-10-10T17:53:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.735 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookup_latency": { 2023-10-10T17:53:22.736 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 15208, 2023-10-10T17:53:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 58.038613087, 2023-10-10T17:53:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.003816321 2023-10-10T17:53:22.737 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookuphash_latency": { 2023-10-10T17:53:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.738 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupino_latency": { 2023-10-10T17:53:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.739 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.740 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupname_latency": { 2023-10-10T17:53:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.741 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupparent_latency": { 2023-10-10T17:53:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.742 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.743 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lookupsnap_latency": { 2023-10-10T17:53:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.744 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "req_lssnap_latency": { 2023-10-10T17:53:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.745 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mkdir_latency": { 2023-10-10T17:53:22.746 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.747 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mknod_latency": { 2023-10-10T17:53:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.748 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "req_mksnap_latency": { 2023-10-10T17:53:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.749 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.750 INFO:teuthology.orchestra.run.smithi190.stdout: "req_open_latency": { 2023-10-10T17:53:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.751 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "req_readdir_latency": { 2023-10-10T17:53:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 2888, 2023-10-10T17:53:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 45.555110497, 2023-10-10T17:53:22.752 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.015773930 2023-10-10T17:53:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rename_latency": { 2023-10-10T17:53:22.753 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.754 INFO:teuthology.orchestra.run.smithi190.stdout: "req_renamesnap_latency": { 2023-10-10T17:53:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.755 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "req_snapdiff_latency": { 2023-10-10T17:53:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.756 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmdir_latency": { 2023-10-10T17:53:22.757 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3, 2023-10-10T17:53:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.162215109, 2023-10-10T17:53:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.054071703 2023-10-10T17:53:22.758 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmsnap_latency": { 2023-10-10T17:53:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.759 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "req_rmxattr_latency": { 2023-10-10T17:53:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.760 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.761 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setattr_latency": { 2023-10-10T17:53:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.762 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setdirlayout_latency": { 2023-10-10T17:53:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.763 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.764 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setfilelock_latency": { 2023-10-10T17:53:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.765 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setlayout_latency": { 2023-10-10T17:53:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.766 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "req_setxattr_latency": { 2023-10-10T17:53:22.767 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.768 INFO:teuthology.orchestra.run.smithi190.stdout: "req_symlink_latency": { 2023-10-10T17:53:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.769 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "req_unlink_latency": { 2023-10-10T17:53:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 15732, 2023-10-10T17:53:22.770 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 89.821858193, 2023-10-10T17:53:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.005709500 2023-10-10T17:53:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.771 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_revoke_eviction": 0, 2023-10-10T17:53:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "cap_acquisition_throttle": 0, 2023-10-10T17:53:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "req_getvxattr_latency": { 2023-10-10T17:53:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.772 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.773 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_sessions": { 2023-10-10T17:53:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "session_count": 1, 2023-10-10T17:53:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "session_add": 2, 2023-10-10T17:53:22.774 INFO:teuthology.orchestra.run.smithi190.stdout: "session_remove": 1, 2023-10-10T17:53:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_open": 1, 2023-10-10T17:53:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "sessions_stale": 0, 2023-10-10T17:53:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "total_load": 11024, 2023-10-10T17:53:22.775 INFO:teuthology.orchestra.run.smithi190.stdout: "average_load": 11024, 2023-10-10T17:53:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "avg_session_uptime": 597 2023-10-10T17:53:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.776 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2023-10-10T17:53:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:53:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2023-10-10T17:53:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:22.777 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:22.778 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:22.779 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:22.780 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:22.781 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:22.782 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:22.783 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:22.784 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:22.785 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2023-10-10T17:53:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:22.786 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 1750109, 2023-10-10T17:53:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 42579, 2023-10-10T17:53:22.787 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 792, 2023-10-10T17:53:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 9, 2023-10-10T17:53:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2023-10-10T17:53:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2023-10-10T17:53:22.788 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:22.789 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 9760, 2023-10-10T17:53:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 189, 2023-10-10T17:53:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:22.790 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2023-10-10T17:53:22.791 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 158018182, 2023-10-10T17:53:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 1445616, 2023-10-10T17:53:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:22.792 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2023-10-10T17:53:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2023-10-10T17:53:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.793 INFO:teuthology.orchestra.run.smithi190.stdout: "objecter": { 2023-10-10T17:53:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "op_active": 0, 2023-10-10T17:53:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "op_laggy": 0, 2023-10-10T17:53:22.794 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send": 49116, 2023-10-10T17:53:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "op_send_bytes": 529883841, 2023-10-10T17:53:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "op_resend": 33, 2023-10-10T17:53:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "op_reply": 49116, 2023-10-10T17:53:22.795 INFO:teuthology.orchestra.run.smithi190.stdout: "op_latency": { 2023-10-10T17:53:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 49116, 2023-10-10T17:53:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 514.007590436, 2023-10-10T17:53:22.796 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.010465176 2023-10-10T17:53:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "op_inflight": 0, 2023-10-10T17:53:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "oplen_avg": { 2023-10-10T17:53:22.797 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 49116, 2023-10-10T17:53:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 54703 2023-10-10T17:53:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "op": 49116, 2023-10-10T17:53:22.798 INFO:teuthology.orchestra.run.smithi190.stdout: "op_r": 11645, 2023-10-10T17:53:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "op_w": 37471, 2023-10-10T17:53:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "op_rmw": 0, 2023-10-10T17:53:22.799 INFO:teuthology.orchestra.run.smithi190.stdout: "op_pg": 0, 2023-10-10T17:53:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_stat": 0, 2023-10-10T17:53:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_create": 1246, 2023-10-10T17:53:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_read": 279, 2023-10-10T17:53:22.800 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_write": 3359, 2023-10-10T17:53:22.801 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writefull": 140, 2023-10-10T17:53:22.801 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_writesame": 0, 2023-10-10T17:53:22.801 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_append": 0, 2023-10-10T17:53:22.802 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_zero": 0, 2023-10-10T17:53:22.802 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_truncate": 1, 2023-10-10T17:53:22.802 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_delete": 31771, 2023-10-10T17:53:22.802 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_mapext": 0, 2023-10-10T17:53:22.803 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_sparse_read": 0, 2023-10-10T17:53:22.803 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_clonerange": 0, 2023-10-10T17:53:22.803 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_getxattr": 8621, 2023-10-10T17:53:22.803 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_setxattr": 1874, 2023-10-10T17:53:22.804 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_cmpxattr": 0, 2023-10-10T17:53:22.804 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_rmxattr": 0, 2023-10-10T17:53:22.804 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_resetxattrs": 0, 2023-10-10T17:53:22.805 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_call": 0, 2023-10-10T17:53:22.805 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_watch": 0, 2023-10-10T17:53:22.805 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_notify": 0, 2023-10-10T17:53:22.805 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_src_cmpxattr": 0, 2023-10-10T17:53:22.806 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls": 0, 2023-10-10T17:53:22.806 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_pgls_filter": 0, 2023-10-10T17:53:22.807 INFO:teuthology.orchestra.run.smithi190.stdout: "osdop_other": 0, 2023-10-10T17:53:22.807 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_active": 0, 2023-10-10T17:53:22.807 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_send": 0, 2023-10-10T17:53:22.808 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_resend": 0, 2023-10-10T17:53:22.808 INFO:teuthology.orchestra.run.smithi190.stdout: "linger_ping": 0, 2023-10-10T17:53:22.808 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_active": 0, 2023-10-10T17:53:22.808 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_send": 0, 2023-10-10T17:53:22.809 INFO:teuthology.orchestra.run.smithi190.stdout: "poolop_resend": 0, 2023-10-10T17:53:22.809 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_active": 0, 2023-10-10T17:53:22.809 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_send": 0, 2023-10-10T17:53:22.809 INFO:teuthology.orchestra.run.smithi190.stdout: "poolstat_resend": 0, 2023-10-10T17:53:22.810 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_active": 0, 2023-10-10T17:53:22.810 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_send": 0, 2023-10-10T17:53:22.810 INFO:teuthology.orchestra.run.smithi190.stdout: "statfs_resend": 0, 2023-10-10T17:53:22.811 INFO:teuthology.orchestra.run.smithi190.stdout: "command_active": 0, 2023-10-10T17:53:22.811 INFO:teuthology.orchestra.run.smithi190.stdout: "command_send": 0, 2023-10-10T17:53:22.811 INFO:teuthology.orchestra.run.smithi190.stdout: "command_resend": 0, 2023-10-10T17:53:22.811 INFO:teuthology.orchestra.run.smithi190.stdout: "map_epoch": 103, 2023-10-10T17:53:22.812 INFO:teuthology.orchestra.run.smithi190.stdout: "map_full": 0, 2023-10-10T17:53:22.812 INFO:teuthology.orchestra.run.smithi190.stdout: "map_inc": 14, 2023-10-10T17:53:22.812 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_sessions": 12, 2023-10-10T17:53:22.812 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_open": 12, 2023-10-10T17:53:22.813 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_session_close": 0, 2023-10-10T17:53:22.813 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_laggy": 0, 2023-10-10T17:53:22.813 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_wr": 1881, 2023-10-10T17:53:22.814 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_rd": 5490, 2023-10-10T17:53:22.814 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_del": 41 2023-10-10T17:53:22.814 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.814 INFO:teuthology.orchestra.run.smithi190.stdout: "oft": { 2023-10-10T17:53:22.815 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_objs": 1, 2023-10-10T17:53:22.815 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_kv_pairs": 0, 2023-10-10T17:53:22.815 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_updates": 0, 2023-10-10T17:53:22.815 INFO:teuthology.orchestra.run.smithi190.stdout: "omap_total_removes": 0 2023-10-10T17:53:22.816 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.816 INFO:teuthology.orchestra.run.smithi190.stdout: "osd-slow-ops": { 2023-10-10T17:53:22.816 INFO:teuthology.orchestra.run.smithi190.stdout: "slow_ops_count": 66501 2023-10-10T17:53:22.817 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.817 INFO:teuthology.orchestra.run.smithi190.stdout: "purge_queue": { 2023-10-10T17:53:22.817 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops": 0, 2023-10-10T17:53:22.817 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_ops_high_water": 16, 2023-10-10T17:53:22.818 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing": 0, 2023-10-10T17:53:22.818 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executing_high_water": 8, 2023-10-10T17:53:22.818 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_executed": 15732, 2023-10-10T17:53:22.819 INFO:teuthology.orchestra.run.smithi190.stdout: "pq_item_in_journal": 0 2023-10-10T17:53:22.819 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.819 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:22.819 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:22.820 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2023-10-10T17:53:22.820 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:22.820 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 261230, 2023-10-10T17:53:22.820 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 130946195, 2023-10-10T17:53:22.821 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:22.821 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 261230, 2023-10-10T17:53:22.821 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:22.822 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:22.822 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 261230, 2023-10-10T17:53:22.822 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 130946195, 2023-10-10T17:53:22.822 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:22.823 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.823 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.823 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.823 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.824 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.824 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_bytes": { 2023-10-10T17:53:22.824 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:22.825 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2023-10-10T17:53:22.825 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:22.825 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:22.825 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:22.826 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:22.826 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:22.826 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 49116, 2023-10-10T17:53:22.827 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 531824437, 2023-10-10T17:53:22.827 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 49116, 2023-10-10T17:53:22.827 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 531824437, 2023-10-10T17:53:22.827 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:22.828 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.828 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.828 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.828 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.829 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.829 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-objecter_ops": { 2023-10-10T17:53:22.829 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:22.830 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 1024, 2023-10-10T17:53:22.830 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:22.830 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 0, 2023-10-10T17:53:22.830 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 0, 2023-10-10T17:53:22.831 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:22.831 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 0, 2023-10-10T17:53:22.831 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 49116, 2023-10-10T17:53:22.831 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 49116, 2023-10-10T17:53:22.832 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 49116, 2023-10-10T17:53:22.832 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 49116, 2023-10-10T17:53:22.832 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:22.833 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.833 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.833 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.833 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.834 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.835 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle": { 2023-10-10T17:53:22.835 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:22.835 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2023-10-10T17:53:22.836 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:22.836 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 15732, 2023-10-10T17:53:22.837 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 1984516, 2023-10-10T17:53:22.837 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:22.837 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 15732, 2023-10-10T17:53:22.838 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:22.838 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:22.838 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 279, 2023-10-10T17:53:22.838 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 1984516, 2023-10-10T17:53:22.839 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:22.839 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.839 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.840 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.840 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.840 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:22.840 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-write_buf_throttle-0x5633327ce0a0": { 2023-10-10T17:53:22.841 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:22.841 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 3758096384, 2023-10-10T17:53:22.841 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:22.842 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 93861, 2023-10-10T17:53:22.842 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 516645701, 2023-10-10T17:53:22.842 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:22.843 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 93861, 2023-10-10T17:53:22.843 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:22.843 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:22.843 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 3080, 2023-10-10T17:53:22.844 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 516645701, 2023-10-10T17:53:22.844 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:22.844 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:22.845 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:22.845 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:22.845 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.846 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:22.846 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-10-10T17:53:23.010 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:22 smithi190 ceph-mon[101220]: pgmap v337: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:23.107 INFO:tasks.check_counter:Daemon mds.i mds.exported=791 2023-10-10T17:53:23.108 INFO:tasks.check_counter:Daemon mds.i mds.imported=1379 2023-10-10T17:53:23.108 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=1684 2023-10-10T17:53:23.108 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=2035 2023-10-10T17:53:23.109 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2023-10-10T17:53:23.109 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2023-10-10T17:53:23.109 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2023-10-10T17:53:23.109 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=109 2023-10-10T17:53:23.110 DEBUG:tasks.check_counter:Getting stats from l 2023-10-10T17:53:23.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:05a43c8fb3988914c879a50970f9054a098e0612 shell --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 -- ceph daemon mds.l perf dump 2023-10-10T17:53:23.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:22 smithi078 ceph-mon[99494]: pgmap v337: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:23.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:22 smithi157 ceph-mon[100896]: pgmap v337: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:23.796 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/mon.c/config 2023-10-10T17:53:25.075 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:24 smithi190 ceph-mon[101220]: pgmap v338: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:25.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:24 smithi078 ceph-mon[99494]: pgmap v338: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:25.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:24 smithi157 ceph-mon[100896]: pgmap v338: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:26.095 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2023-10-10T17:53:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-0": { 2023-10-10T17:53:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 4, 2023-10-10T17:53:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 4, 2023-10-10T17:53:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 2992, 2023-10-10T17:53:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 448, 2023-10-10T17:53:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 4, 2023-10-10T17:53:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 0, 2023-10-10T17:53:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.001439766, 2023-10-10T17:53:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.000040272, 2023-10-10T17:53:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.000607547, 2023-10-10T17:53:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:53:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 3, 2023-10-10T17:53:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000740651, 2023-10-10T17:53:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000246883 2023-10-10T17:53:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 2992, 2023-10-10T17:53:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 448 2023-10-10T17:53:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-1": { 2023-10-10T17:53:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 2, 2023-10-10T17:53:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 163, 2023-10-10T17:53:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 320, 2023-10-10T17:53:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 180416, 2023-10-10T17:53:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 5, 2023-10-10T17:53:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 3, 2023-10-10T17:53:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.009823420, 2023-10-10T17:53:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.006295490, 2023-10-10T17:53:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.001567616, 2023-10-10T17:53:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2023-10-10T17:53:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 160, 2023-10-10T17:53:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.015176415, 2023-10-10T17:53:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000094852 2023-10-10T17:53:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 320, 2023-10-10T17:53:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 180416 2023-10-10T17:53:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "AsyncMessenger::Worker-2": { 2023-10-10T17:53:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_messages": 220, 2023-10-10T17:53:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages": 213, 2023-10-10T17:53:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_bytes": 1014352, 2023-10-10T17:53:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_bytes": 121520, 2023-10-10T17:53:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_created_connections": 2, 2023-10-10T17:53:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_active_connections": 1, 2023-10-10T17:53:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_total_time": 0.037458177, 2023-10-10T17:53:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_send_time": 0.012298381, 2023-10-10T17:53:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_recv_time": 0.023813097, 2023-10-10T17:53:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_running_fast_dispatch_time": 0.005738595, 2023-10-10T17:53:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_messages_queue_lat": { 2023-10-10T17:53:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 212, 2023-10-10T17:53:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.014234339, 2023-10-10T17:53:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000067143 2023-10-10T17:53:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_handle_ack_lat": { 2023-10-10T17:53:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_recv_encrypted_bytes": 1014352, 2023-10-10T17:53:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "msgr_send_encrypted_bytes": 121520 2023-10-10T17:53:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "cct": { 2023-10-10T17:53:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "total_workers": 0, 2023-10-10T17:53:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "unhealthy_workers": 0 2023-10-10T17:53:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "mempool": { 2023-10-10T17:53:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_bytes": 0, 2023-10-10T17:53:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "bloom_filter_items": 0, 2023-10-10T17:53:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_bytes": 0, 2023-10-10T17:53:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_alloc_items": 0, 2023-10-10T17:53:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_bytes": 0, 2023-10-10T17:53:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_data_items": 0, 2023-10-10T17:53:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_bytes": 0, 2023-10-10T17:53:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_onode_items": 0, 2023-10-10T17:53:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_bytes": 0, 2023-10-10T17:53:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_meta_items": 0, 2023-10-10T17:53:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_bytes": 0, 2023-10-10T17:53:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_other_items": 0, 2023-10-10T17:53:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_bytes": 0, 2023-10-10T17:53:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_cache_buffer_items": 0, 2023-10-10T17:53:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_bytes": 0, 2023-10-10T17:53:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_extent_items": 0, 2023-10-10T17:53:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_bytes": 0, 2023-10-10T17:53:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_blob_items": 0, 2023-10-10T17:53:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_bytes": 0, 2023-10-10T17:53:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_shared_blob_items": 0, 2023-10-10T17:53:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_bytes": 0, 2023-10-10T17:53:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_inline_bl_items": 0, 2023-10-10T17:53:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_bytes": 0, 2023-10-10T17:53:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_fsck_items": 0, 2023-10-10T17:53:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_bytes": 0, 2023-10-10T17:53:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_txc_items": 0, 2023-10-10T17:53:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_bytes": 0, 2023-10-10T17:53:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_deferred_items": 0, 2023-10-10T17:53:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_bytes": 0, 2023-10-10T17:53:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "bluestore_writing_items": 0, 2023-10-10T17:53:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_bytes": 0, 2023-10-10T17:53:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_items": 0, 2023-10-10T17:53:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_bytes": 0, 2023-10-10T17:53:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_reader_items": 0, 2023-10-10T17:53:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_bytes": 0, 2023-10-10T17:53:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "bluefs_file_writer_items": 0, 2023-10-10T17:53:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_bytes": 149057, 2023-10-10T17:53:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_anon_items": 32, 2023-10-10T17:53:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_bytes": 0, 2023-10-10T17:53:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "buffer_meta_items": 0, 2023-10-10T17:53:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_bytes": 0, 2023-10-10T17:53:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_items": 0, 2023-10-10T17:53:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_bytes": 0, 2023-10-10T17:53:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_mapbl_items": 0, 2023-10-10T17:53:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_bytes": 0, 2023-10-10T17:53:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "osd_pglog_items": 0, 2023-10-10T17:53:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_bytes": 0, 2023-10-10T17:53:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_items": 0, 2023-10-10T17:53:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_bytes": 0, 2023-10-10T17:53:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "osdmap_mapping_items": 0, 2023-10-10T17:53:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_bytes": 0, 2023-10-10T17:53:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "pgmap_items": 0, 2023-10-10T17:53:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_bytes": 928, 2023-10-10T17:53:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "mds_co_items": 2, 2023-10-10T17:53:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_bytes": 0, 2023-10-10T17:53:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_1_items": 0, 2023-10-10T17:53:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_bytes": 0, 2023-10-10T17:53:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "unittest_2_items": 0 2023-10-10T17:53:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2023-10-10T17:53:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "throttle-msgr_dispatch_throttler-mds": { 2023-10-10T17:53:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "val": 0, 2023-10-10T17:53:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "max": 104857600, 2023-10-10T17:53:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "get_started": 0, 2023-10-10T17:53:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "get": 222, 2023-10-10T17:53:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "get_sum": 983294, 2023-10-10T17:53:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_fail": 0, 2023-10-10T17:53:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "get_or_fail_success": 222, 2023-10-10T17:53:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "take": 0, 2023-10-10T17:53:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "take_sum": 0, 2023-10-10T17:53:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "put": 222, 2023-10-10T17:53:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "put_sum": 983294, 2023-10-10T17:53:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "wait": { 2023-10-10T17:53:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "avgcount": 0, 2023-10-10T17:53:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "sum": 0.000000000, 2023-10-10T17:53:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "avgtime": 0.000000000 2023-10-10T17:53:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2023-10-10T17:53:26.138 INFO:teuthology.orchestra.run.smithi190.stdout:} 2023-10-10T17:53:26.857 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:26 smithi190 ceph-mon[101220]: pgmap v339: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:53:27.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:26 smithi078 ceph-mon[99494]: pgmap v339: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:53:27.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:26 smithi157 ceph-mon[100896]: pgmap v339: 129 pgs: 21 active+clean+snaptrim, 67 active+clean+snaptrim_wait, 41 active+clean; 799 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:53:27.288 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2023-10-10T17:53:27.289 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2023-10-10T17:53:27.289 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2023-10-10T17:53:27.290 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2023-10-10T17:53:27.290 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2023-10-10T17:53:27.290 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2023-10-10T17:53:27.291 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2023-10-10T17:53:27.291 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2023-10-10T17:53:27.291 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-10T17:53:27.304 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-10T17:53:27.305 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:53:27.305 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-10T17:53:27.382 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-10T17:53:27.383 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-10T17:53:27.433 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:53:27.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.672+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/711688438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8ac0ff420 msgr2=0x7ff8ac0ff7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:27.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.672+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/711688438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac0ff7f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00d360 tx=0x7ff89c00d670 comp rx=0 tx=0).stop 2023-10-10T17:53:27.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.677+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/711688438 shutdown_connections 2023-10-10T17:53:27.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.677+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/711688438 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac0ff7f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:27.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.677+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/711688438 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac104460 0x7ff8ac10c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:27.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.677+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/711688438 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8ac0ffdc0 0x7ff8ac103d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:27.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.677+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/711688438 >> 172.21.15.78:0/711688438 conn(0x7ff8ac076310 msgr2=0x7ff8ac076710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:27.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.684+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/711688438 shutdown_connections 2023-10-10T17:53:27.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.684+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/711688438 wait complete. 2023-10-10T17:53:27.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.685+0000 7ff8b3d9b700 1 Processor -- start 2023-10-10T17:53:27.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.692+0000 7ff8b3d9b700 1 -- start start 2023-10-10T17:53:27.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b3d9b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac1a28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:27.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b3d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8ac0ffdc0 0x7ff8ac1a2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:27.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b3d9b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8ac104460 0x7ff8ac19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:27.702 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b3d9b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac19cfc0 con 0x7ff8ac0ffdc0 2023-10-10T17:53:27.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b3d9b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac19d130 con 0x7ff8ac0ff420 2023-10-10T17:53:27.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b3d9b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac19d2f0 con 0x7ff8ac104460 2023-10-10T17:53:27.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b1b37700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac1a28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:27.703 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b1b37700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac1a28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50970/0 (socket says 172.21.15.78:50970) 2023-10-10T17:53:27.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.693+0000 7ff8b1b37700 1 -- 172.21.15.78:0/4217309992 learned_addr learned my addr 172.21.15.78:0/4217309992 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:27.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.695+0000 7ff8b1b37700 1 -- 172.21.15.78:0/4217309992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8ac104460 msgr2=0x7ff8ac19c950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:53:27.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.695+0000 7ff8b1b37700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8ac104460 0x7ff8ac19c950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:27.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.695+0000 7ff8b1b37700 1 -- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8ac0ffdc0 msgr2=0x7ff8ac1a2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:27.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.695+0000 7ff8b1b37700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8ac0ffdc0 0x7ff8ac1a2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:27.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.695+0000 7ff8b1b37700 1 -- 172.21.15.78:0/4217309992 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c009d00 con 0x7ff8ac0ff420 2023-10-10T17:53:27.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.695+0000 7ff8b1b37700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac1a28e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00d330 tx=0x7ff89c006230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:27.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.696+0000 7ff8aaffd700 1 -- 172.21.15.78:0/4217309992 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c011860 con 0x7ff8ac0ff420 2023-10-10T17:53:27.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.696+0000 7ff8aaffd700 1 -- 172.21.15.78:0/4217309992 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff89c011e80 con 0x7ff8ac0ff420 2023-10-10T17:53:27.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.696+0000 7ff8aaffd700 1 -- 172.21.15.78:0/4217309992 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c02ce40 con 0x7ff8ac0ff420 2023-10-10T17:53:27.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.701+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8ac10e2b0 con 0x7ff8ac0ff420 2023-10-10T17:53:27.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.701+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac10e120 con 0x7ff8ac0ff420 2023-10-10T17:53:27.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.702+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac10e440 con 0x7ff8ac0ff420 2023-10-10T17:53:27.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.708+0000 7ff8aaffd700 1 -- 172.21.15.78:0/4217309992 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7ff89c02c5a0 con 0x7ff8ac0ff420 2023-10-10T17:53:27.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.708+0000 7ff8aaffd700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8940a1400 0x7ff8940a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:27.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.709+0000 7ff8aaffd700 1 -- 172.21.15.78:0/4217309992 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7ff89c0dec70 con 0x7ff8ac0ff420 2023-10-10T17:53:27.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.709+0000 7ff8b1336700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8940a1400 0x7ff8940a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:27.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.710+0000 7ff8b1336700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8940a1400 0x7ff8940a38b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac1041d0 tx=0x7ff8a0006d20 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:27.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:27.713+0000 7ff8aaffd700 1 -- 172.21.15.78:0/4217309992 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7ff89c0a7cf0 con 0x7ff8ac0ff420 2023-10-10T17:53:28.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.063+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8ac19df00 con 0x7ff8ac0ff420 2023-10-10T17:53:28.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.072+0000 7ff8aaffd700 1 -- 172.21.15.78:0/4217309992 <== mon.2 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25283 (secure 0 0 0) 0x7ff89c0ab310 con 0x7ff8ac0ff420 2023-10-10T17:53:28.078 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:53:28.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8940a1400 msgr2=0x7ff8940a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:28.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8940a1400 0x7ff8940a38b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac1041d0 tx=0x7ff8a0006d20 comp rx=0 tx=0).stop 2023-10-10T17:53:28.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac0ff420 msgr2=0x7ff8ac1a28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:28.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac1a28e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00d330 tx=0x7ff89c006230 comp rx=0 tx=0).stop 2023-10-10T17:53:28.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 shutdown_connections 2023-10-10T17:53:28.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7ff8ac104460 0x7ff8ac19c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7ff8940a1400 0x7ff8940a38b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7ff8ac0ff420 0x7ff8ac1a28e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 --2- 172.21.15.78:0/4217309992 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8ac0ffdc0 0x7ff8ac1a2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 >> 172.21.15.78:0/4217309992 conn(0x7ff8ac076310 msgr2=0x7ff8ac102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:28.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 shutdown_connections 2023-10-10T17:53:28.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.077+0000 7ff8b3d9b700 1 -- 172.21.15.78:0/4217309992 wait complete. 2023-10-10T17:53:28.082 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":103,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:53:06.650410+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1054117069":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/1411676003":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/540877351":"2023-10-11T17:42:35.200450+0000","172.21.15.78:6801/321866266":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/951456648":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/321866266":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:53:28.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-10T17:36:46.460785+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-10T17:40:39.520570+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-10T17:40:40.752674+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-10-10T17:40:43.919172+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-10T17:53:28.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-10T17:53:28.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.360+0000 7f7218223700 1 -- 172.21.15.78:0/306805446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210104180 msgr2=0x7f7210104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:28.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.360+0000 7f7218223700 1 --2- 172.21.15.78:0/306805446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210104180 0x7f7210104550 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f7204009990 tx=0x7f7204009ca0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.366+0000 7f7218223700 1 -- 172.21.15.78:0/306805446 shutdown_connections 2023-10-10T17:53:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.366+0000 7f7218223700 1 --2- 172.21.15.78:0/306805446 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7210105840 0x7f721010c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.366+0000 7f7218223700 1 --2- 172.21.15.78:0/306805446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210104180 0x7f7210104550 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.366+0000 7f7218223700 1 --2- 172.21.15.78:0/306805446 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7210104e90 0x7f7210105300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.366+0000 7f7218223700 1 -- 172.21.15.78:0/306805446 >> 172.21.15.78:0/306805446 conn(0x7f7210075460 msgr2=0x7f7210075860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:28.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.371+0000 7f7218223700 1 -- 172.21.15.78:0/306805446 shutdown_connections 2023-10-10T17:53:28.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.376+0000 7f7218223700 1 -- 172.21.15.78:0/306805446 wait complete. 2023-10-10T17:53:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.376+0000 7f7218223700 1 Processor -- start 2023-10-10T17:53:28.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.376+0000 7f7218223700 1 -- start start 2023-10-10T17:53:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f7218223700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7210104180 0x7f7210107c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f7218223700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7210104e90 0x7f7210108140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f7218223700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210105840 0x7f72101a1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:28.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f7218223700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101a1af0 con 0x7f7210105840 2023-10-10T17:53:28.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f7218223700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101a1c60 con 0x7f7210104180 2023-10-10T17:53:28.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f7218223700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101a1e20 con 0x7f7210104e90 2023-10-10T17:53:28.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f72167c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210105840 0x7f72101a1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:28.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f72167c0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210105840 0x7f72101a1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49262/0 (socket says 172.21.15.78:49262) 2023-10-10T17:53:28.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.377+0000 7f72167c0700 1 -- 172.21.15.78:0/210441078 learned_addr learned my addr 172.21.15.78:0/210441078 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:28.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.378+0000 7f72167c0700 1 -- 172.21.15.78:0/210441078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7210104e90 msgr2=0x7f7210108140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:53:28.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.378+0000 7f72167c0700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7210104e90 0x7f7210108140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.378+0000 7f72167c0700 1 -- 172.21.15.78:0/210441078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7210104180 msgr2=0x7f7210107c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T17:53:28.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.378+0000 7f72167c0700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7210104180 0x7f7210107c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.378+0000 7f72167c0700 1 -- 172.21.15.78:0/210441078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040095f0 con 0x7f7210105840 2023-10-10T17:53:28.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.378+0000 7f72167c0700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210105840 0x7f72101a1460 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f7208007ee0 tx=0x7f720800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:28.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.379+0000 7f7202ffd700 1 -- 172.21.15.78:0/210441078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720800ddd0 con 0x7f7210105840 2023-10-10T17:53:28.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.379+0000 7f7202ffd700 1 -- 172.21.15.78:0/210441078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720800f040 con 0x7f7210105840 2023-10-10T17:53:28.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.379+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721010cfe0 con 0x7f7210105840 2023-10-10T17:53:28.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.379+0000 7f7202ffd700 1 -- 172.21.15.78:0/210441078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208015400 con 0x7f7210105840 2023-10-10T17:53:28.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.379+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721010ce50 con 0x7f7210105840 2023-10-10T17:53:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.382+0000 7f7202ffd700 1 -- 172.21.15.78:0/210441078 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f72080040b0 con 0x7f7210105840 2023-10-10T17:53:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.382+0000 7f7202ffd700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71fc0b24a0 0x7f71fc0b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.383+0000 7f7202ffd700 1 -- 172.21.15.78:0/210441078 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f72080cca30 con 0x7f7210105840 2023-10-10T17:53:28.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.383+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721010d170 con 0x7f7210105840 2023-10-10T17:53:28.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.395+0000 7f7215fbf700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71fc0b24a0 0x7f71fc0b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:28.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.395+0000 7f7215fbf700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71fc0b24a0 0x7f71fc0b4950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7204003760 tx=0x7f7204005fb0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:28.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.395+0000 7f7202ffd700 1 -- 172.21.15.78:0/210441078 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f720804e730 con 0x7f7210105840 2023-10-10T17:53:28.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.744+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7210108e70 con 0x7f7210105840 2023-10-10T17:53:28.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.744+0000 7f7202ffd700 1 -- 172.21.15.78:0/210441078 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7208095a20 con 0x7f7210105840 2023-10-10T17:53:28.746 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-10-10T17:53:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.750+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71fc0b24a0 msgr2=0x7f71fc0b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.750+0000 7f7218223700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71fc0b24a0 0x7f71fc0b4950 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7204003760 tx=0x7f7204005fb0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.751+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210105840 msgr2=0x7f72101a1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:28.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.751+0000 7f7218223700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210105840 0x7f72101a1460 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f7208007ee0 tx=0x7f720800d3b0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.752+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 shutdown_connections 2023-10-10T17:53:28.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.752+0000 7f7218223700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f71fc0b24a0 0x7f71fc0b4950 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.752+0000 7f7218223700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7210105840 0x7f72101a1460 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.752+0000 7f7218223700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f7210104180 0x7f7210107c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.752+0000 7f7218223700 1 --2- 172.21.15.78:0/210441078 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f7210104e90 0x7f7210108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.752+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 >> 172.21.15.78:0/210441078 conn(0x7f7210075460 msgr2=0x7f72100fbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:28.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.754+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 shutdown_connections 2023-10-10T17:53:28.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:28.757+0000 7f7218223700 1 -- 172.21.15.78:0/210441078 wait complete. 2023-10-10T17:53:28.779 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-10T17:53:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.020+0000 7fc5eb360700 1 -- 172.21.15.78:0/3745838863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 msgr2=0x7fc5e410b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.020+0000 7fc5eb360700 1 --2- 172.21.15.78:0/3745838863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 0x7fc5e410b790 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00c210 tx=0x7fc5dc00c520 comp rx=0 tx=0).stop 2023-10-10T17:53:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.024+0000 7fc5eb360700 1 -- 172.21.15.78:0/3745838863 shutdown_connections 2023-10-10T17:53:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.024+0000 7fc5eb360700 1 --2- 172.21.15.78:0/3745838863 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 0x7fc5e4104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.024+0000 7fc5eb360700 1 --2- 172.21.15.78:0/3745838863 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 0x7fc5e410b790 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.024+0000 7fc5eb360700 1 --2- 172.21.15.78:0/3745838863 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5e40fe950 0x7fc5e40fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.024+0000 7fc5eb360700 1 -- 172.21.15.78:0/3745838863 >> 172.21.15.78:0/3745838863 conn(0x7fc5e4075920 msgr2=0x7fc5e4075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:29.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.030+0000 7fc5eb360700 1 -- 172.21.15.78:0/3745838863 shutdown_connections 2023-10-10T17:53:29.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.039+0000 7fc5eb360700 1 -- 172.21.15.78:0/3745838863 wait complete. 2023-10-10T17:53:29.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 Processor -- start 2023-10-10T17:53:29.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 -- start start 2023-10-10T17:53:29.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5e40fe950 0x7fc5e406c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 0x7fc5e406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 0x7fc5e4071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4072460 con 0x7fc5e40ff300 2023-10-10T17:53:29.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40725d0 con 0x7fc5e40fe950 2023-10-10T17:53:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5eb360700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4072790 con 0x7fc5e4104960 2023-10-10T17:53:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5e90fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5e40fe950 0x7fc5e406c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5e98fd700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 0x7fc5e4071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5e88fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 0x7fc5e406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5e90fc700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5e40fe950 0x7fc5e406c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:50994/0 (socket says 172.21.15.78:50994) 2023-10-10T17:53:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5e90fc700 1 -- 172.21.15.78:0/1457185066 learned_addr learned my addr 172.21.15.78:0/1457185066 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:29.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.040+0000 7fc5e98fd700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 0x7fc5e4071cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40804/0 (socket says 172.21.15.78:40804) 2023-10-10T17:53:29.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5e88fb700 1 -- 172.21.15.78:0/1457185066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 msgr2=0x7fc5e4071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5e88fb700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 0x7fc5e4071cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5e88fb700 1 -- 172.21.15.78:0/1457185066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5e40fe950 msgr2=0x7fc5e406c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5e88fb700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5e40fe950 0x7fc5e406c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5e88fb700 1 -- 172.21.15.78:0/1457185066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc00ae10 con 0x7fc5e40ff300 2023-10-10T17:53:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5e98fd700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 0x7fc5e4071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:53:29.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5e88fb700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 0x7fc5e406c8d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400b450 tx=0x7fc5d400b760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:29.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5da7fc700 1 -- 172.21.15.78:0/1457185066 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4011830 con 0x7fc5e40ff300 2023-10-10T17:53:29.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e406ade0 con 0x7fc5e40ff300 2023-10-10T17:53:29.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.041+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e406ac50 con 0x7fc5e40ff300 2023-10-10T17:53:29.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.042+0000 7fc5da7fc700 1 -- 172.21.15.78:0/1457185066 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc5d4011e50 con 0x7fc5e40ff300 2023-10-10T17:53:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.042+0000 7fc5da7fc700 1 -- 172.21.15.78:0/1457185066 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4010e40 con 0x7fc5e40ff300 2023-10-10T17:53:29.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.043+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e406af70 con 0x7fc5e40ff300 2023-10-10T17:53:29.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.046+0000 7fc5da7fc700 1 -- 172.21.15.78:0/1457185066 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fc5d4011990 con 0x7fc5e40ff300 2023-10-10T17:53:29.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.046+0000 7fc5da7fc700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc5d00a1340 0x7fc5d00a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.047+0000 7fc5da7fc700 1 -- 172.21.15.78:0/1457185066 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fc5d40ccf70 con 0x7fc5e40ff300 2023-10-10T17:53:29.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.047+0000 7fc5e90fc700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc5d00a1340 0x7fc5d00a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:29.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.047+0000 7fc5e90fc700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc5d00a1340 0x7fc5d00a37f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000bcd0 tx=0x7fc5e000b2c0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:29.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.051+0000 7fc5da7fc700 1 -- 172.21.15.78:0/1457185066 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fc5d4096090 con 0x7fc5e40ff300 2023-10-10T17:53:29.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:28 smithi078 ceph-mon[99494]: pgmap v340: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:53:29.130 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/4217309992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:53:29.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:28 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/210441078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:53:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:28 smithi190 ceph-mon[101220]: pgmap v340: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:53:29.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/4217309992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:53:29.166 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:28 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/210441078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:53:29.227 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:28 smithi157 ceph-mon[100896]: pgmap v340: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-10-10T17:53:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/4217309992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:53:29.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:28 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/210441078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-10T17:53:29.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.398+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc5e406d550 con 0x7fc5e40ff300 2023-10-10T17:53:29.405 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:53:29.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.399+0000 7fc5da7fc700 1 -- 172.21.15.78:0/1457185066 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc5d4015020 con 0x7fc5e40ff300 2023-10-10T17:53:29.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc5d00a1340 msgr2=0x7fc5d00a37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc5d00a1340 0x7fc5d00a37f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc5e000bcd0 tx=0x7fc5e000b2c0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 msgr2=0x7fc5e406c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 0x7fc5e406c8d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400b450 tx=0x7fc5d400b760 comp rx=0 tx=0).stop 2023-10-10T17:53:29.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 shutdown_connections 2023-10-10T17:53:29.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fc5e4104960 0x7fc5e4071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fc5d00a1340 0x7fc5d00a37f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fc5e40fe950 0x7fc5e406c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 --2- 172.21.15.78:0/1457185066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5e40ff300 0x7fc5e406c8d0 secure :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fc5d400b450 tx=0x7fc5d400b760 comp rx=0 tx=0).stop 2023-10-10T17:53:29.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 >> 172.21.15.78:0/1457185066 conn(0x7fc5e4075920 msgr2=0x7fc5e41057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:29.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.405+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 shutdown_connections 2023-10-10T17:53:29.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.406+0000 7fc5eb360700 1 -- 172.21.15.78:0/1457185066 wait complete. 2023-10-10T17:53:29.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-10T17:53:29.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.674+0000 7f5c42148700 1 -- 172.21.15.78:0/412539298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c3c1074b0 msgr2=0x7f5c3c113740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.674+0000 7f5c42148700 1 --2- 172.21.15.78:0/412539298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c3c1074b0 0x7f5c3c113740 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400b0a0 tx=0x7f5c3400b3b0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.681+0000 7f5c42148700 1 -- 172.21.15.78:0/412539298 shutdown_connections 2023-10-10T17:53:29.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.681+0000 7f5c42148700 1 --2- 172.21.15.78:0/412539298 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c10c980 0x7f5c3c10cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.681+0000 7f5c42148700 1 --2- 172.21.15.78:0/412539298 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c3c1074b0 0x7f5c3c113740 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.681+0000 7f5c42148700 1 --2- 172.21.15.78:0/412539298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c106b00 0x7f5c3c106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.681+0000 7f5c42148700 1 -- 172.21.15.78:0/412539298 >> 172.21.15.78:0/412539298 conn(0x7f5c3c073dd0 msgr2=0x7f5c3c0761e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:29.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.684+0000 7f5c42148700 1 -- 172.21.15.78:0/412539298 shutdown_connections 2023-10-10T17:53:29.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.685+0000 7f5c42148700 1 -- 172.21.15.78:0/412539298 wait complete. 2023-10-10T17:53:29.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.685+0000 7f5c42148700 1 Processor -- start 2023-10-10T17:53:29.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c42148700 1 -- start start 2023-10-10T17:53:29.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c42148700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c106b00 0x7f5c3c10ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c42148700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c1074b0 0x7f5c3c10f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c42148700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c3c10c980 0x7f5c3c071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c42148700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c10fa00 con 0x7f5c3c1074b0 2023-10-10T17:53:29.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c42148700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c071a90 con 0x7f5c3c10c980 2023-10-10T17:53:29.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c42148700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c071c50 con 0x7f5c3c106b00 2023-10-10T17:53:29.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c3b7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c106b00 0x7f5c3c10ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:29.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c3b7fe700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c106b00 0x7f5c3c10ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40816/0 (socket says 172.21.15.78:40816) 2023-10-10T17:53:29.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.686+0000 7f5c3b7fe700 1 -- 172.21.15.78:0/1299073680 learned_addr learned my addr 172.21.15.78:0/1299073680 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:29.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.687+0000 7f5c3b7fe700 1 -- 172.21.15.78:0/1299073680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c3c10c980 msgr2=0x7f5c3c071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.687+0000 7f5c3b7fe700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c3c10c980 0x7f5c3c071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.688+0000 7f5c3b7fe700 1 -- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c1074b0 msgr2=0x7f5c3c10f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.688+0000 7f5c3b7fe700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c1074b0 0x7f5c3c10f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.688+0000 7f5c3b7fe700 1 -- 172.21.15.78:0/1299073680 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c34009d00 con 0x7f5c3c106b00 2023-10-10T17:53:29.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.688+0000 7f5c3b7fe700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c106b00 0x7f5c3c10ebd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00d950 tx=0x7f5c2c00dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.691+0000 7f5c38ff9700 1 -- 172.21.15.78:0/1299073680 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c009b40 con 0x7f5c3c106b00 2023-10-10T17:53:29.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.691+0000 7f5c38ff9700 1 -- 172.21.15.78:0/1299073680 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5c2c009ca0 con 0x7f5c3c106b00 2023-10-10T17:53:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.691+0000 7f5c38ff9700 1 -- 172.21.15.78:0/1299073680 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c00f620 con 0x7f5c3c106b00 2023-10-10T17:53:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.693+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3c07baf0 con 0x7f5c3c106b00 2023-10-10T17:53:29.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.693+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3c07b960 con 0x7f5c3c106b00 2023-10-10T17:53:29.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.695+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3c07bc80 con 0x7f5c3c106b00 2023-10-10T17:53:29.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.707+0000 7f5c38ff9700 1 -- 172.21.15.78:0/1299073680 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f5c2c004020 con 0x7f5c3c106b00 2023-10-10T17:53:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.707+0000 7f5c38ff9700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5c240a13c0 0x7f5c240a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.708+0000 7f5c38ff9700 1 -- 172.21.15.78:0/1299073680 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f5c2c0cd350 con 0x7f5c3c106b00 2023-10-10T17:53:29.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.708+0000 7f5c3affd700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5c240a13c0 0x7f5c240a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:29.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.708+0000 7f5c3affd700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5c240a13c0 0x7f5c240a3870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5c30005950 tx=0x7f5c300058e0 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:29.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:29.710+0000 7f5c38ff9700 1 -- 172.21.15.78:0/1299073680 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f5c2c096340 con 0x7f5c3c106b00 2023-10-10T17:53:30.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.066+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5c3c0727d0 con 0x7f5c3c106b00 2023-10-10T17:53:30.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.066+0000 7f5c38ff9700 1 -- 172.21.15.78:0/1299073680 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5c2c0160a0 con 0x7f5c3c106b00 2023-10-10T17:53:30.072 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-10-10T17:53:30.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.079+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5c240a13c0 msgr2=0x7f5c240a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:30.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.079+0000 7f5c42148700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5c240a13c0 0x7f5c240a3870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5c30005950 tx=0x7f5c300058e0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.079+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c106b00 msgr2=0x7f5c3c10ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:30.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.079+0000 7f5c42148700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c106b00 0x7f5c3c10ebd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c00d950 tx=0x7f5c2c00dd10 comp rx=0 tx=0).stop 2023-10-10T17:53:30.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 shutdown_connections 2023-10-10T17:53:30.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f5c3c106b00 0x7f5c3c10ebd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f5c240a13c0 0x7f5c240a3870 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f5c3c10c980 0x7f5c3c071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 --2- 172.21.15.78:0/1299073680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c3c1074b0 0x7f5c3c10f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 >> 172.21.15.78:0/1299073680 conn(0x7f5c3c073dd0 msgr2=0x7f5c3c10a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:30.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 shutdown_connections 2023-10-10T17:53:30.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.081+0000 7f5c42148700 1 -- 172.21.15.78:0/1299073680 wait complete. 2023-10-10T17:53:30.102 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-10-10T17:53:30.165 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:29 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1457185066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:53:30.171 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:29 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1457185066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:53:30.228 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:29 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1457185066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-10-10T17:53:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/321544699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 msgr2=0x7fa8f4113bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/321544699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 0x7fa8f4113bf0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c310 tx=0x7fa8e400c620 comp rx=0 tx=0).stop 2023-10-10T17:53:30.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/321544699 shutdown_connections 2023-10-10T17:53:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/321544699 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8f4107d60 0x7fa8f4108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/321544699 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 0x7fa8f4113bf0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/321544699 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4108700 0x7fa8f410cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/321544699 >> 172.21.15.78:0/321544699 conn(0x7fa8f406c7b0 msgr2=0x7fa8f4071770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/321544699 shutdown_connections 2023-10-10T17:53:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.363+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/321544699 wait complete. 2023-10-10T17:53:30.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.364+0000 7fa8fa6d5700 1 Processor -- start 2023-10-10T17:53:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.364+0000 7fa8fa6d5700 1 -- start start 2023-10-10T17:53:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8fa6d5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4107d60 0x7fa8f41166c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:30.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8fa6d5700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8f4108700 0x7fa8f4116c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:30.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8fa6d5700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 0x7fa8f4123b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:30.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8fa6d5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4124090 con 0x7fa8f4107d60 2023-10-10T17:53:30.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8fa6d5700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f41241d0 con 0x7fa8f410d220 2023-10-10T17:53:30.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8fa6d5700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4124390 con 0x7fa8f4108700 2023-10-10T17:53:30.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4107d60 0x7fa8f41166c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:30.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4107d60 0x7fa8f41166c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49296/0 (socket says 172.21.15.78:49296) 2023-10-10T17:53:30.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f3fff700 1 -- 172.21.15.78:0/1705432880 learned_addr learned my addr 172.21.15.78:0/1705432880 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:30.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f8c72700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 0x7fa8f4123b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:30.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f8c72700 1 -- 172.21.15.78:0/1705432880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8f4108700 msgr2=0x7fa8f4116c00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-10T17:53:30.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f8c72700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8f4108700 0x7fa8f4116c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f8c72700 1 -- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4107d60 msgr2=0x7fa8f41166c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:30.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f8c72700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4107d60 0x7fa8f41166c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f8c72700 1 -- 172.21.15.78:0/1705432880 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e400c050 con 0x7fa8f410d220 2023-10-10T17:53:30.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.365+0000 7fa8f3fff700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4107d60 0x7fa8f41166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:53:30.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.366+0000 7fa8f8c72700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 0x7fa8f4123b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c310 tx=0x7fa8e4012870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:30.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.366+0000 7fa8f17fa700 1 -- 172.21.15.78:0/1705432880 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e400f040 con 0x7fa8f410d220 2023-10-10T17:53:30.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.366+0000 7fa8f17fa700 1 -- 172.21.15.78:0/1705432880 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8e4013d90 con 0x7fa8f410d220 2023-10-10T17:53:30.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.366+0000 7fa8f17fa700 1 -- 172.21.15.78:0/1705432880 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4005940 con 0x7fa8f410d220 2023-10-10T17:53:30.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.367+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f411f280 con 0x7fa8f410d220 2023-10-10T17:53:30.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.367+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f411f0f0 con 0x7fa8f410d220 2023-10-10T17:53:30.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.369+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d4005310 con 0x7fa8f410d220 2023-10-10T17:53:30.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.379+0000 7fa8f17fa700 1 -- 172.21.15.78:0/1705432880 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7fa8e40050a0 con 0x7fa8f410d220 2023-10-10T17:53:30.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.380+0000 7fa8f17fa700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa8e00a1380 0x7fa8e00a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:30.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.380+0000 7fa8f17fa700 1 -- 172.21.15.78:0/1705432880 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7fa8e40ceb80 con 0x7fa8f410d220 2023-10-10T17:53:30.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.380+0000 7fa8f3fff700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa8e00a1380 0x7fa8e00a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:30.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.381+0000 7fa8f17fa700 1 -- 172.21.15.78:0/1705432880 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7fa8e40fa7d0 con 0x7fa8f410d220 2023-10-10T17:53:30.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.391+0000 7fa8f3fff700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa8e00a1380 0x7fa8e00a3830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009d80 tx=0x7fa8ec009210 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.733+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa8d4005f60 con 0x7fa8f410d220 2023-10-10T17:53:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.734+0000 7fa8f17fa700 1 -- 172.21.15.78:0/1705432880 <== mon.2 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa8e4097cc0 con 0x7fa8f410d220 2023-10-10T17:53:30.736 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-10-10T17:53:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.739+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa8e00a1380 msgr2=0x7fa8e00a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.739+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa8e00a1380 0x7fa8e00a3830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009d80 tx=0x7fa8ec009210 comp rx=0 tx=0).stop 2023-10-10T17:53:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.739+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 msgr2=0x7fa8f4123b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.739+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 0x7fa8f4123b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c310 tx=0x7fa8e4012870 comp rx=0 tx=0).stop 2023-10-10T17:53:30.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.748+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 shutdown_connections 2023-10-10T17:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.748+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7fa8e00a1380 0x7fa8e00a3830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.748+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7fa8f410d220 0x7fa8f4123b50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.748+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8f4107d60 0x7fa8f41166c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.748+0000 7fa8fa6d5700 1 --2- 172.21.15.78:0/1705432880 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7fa8f4108700 0x7fa8f4116c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:30.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.748+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 >> 172.21.15.78:0/1705432880 conn(0x7fa8f406c7b0 msgr2=0x7fa8f4070dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.749+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 shutdown_connections 2023-10-10T17:53:30.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:30.749+0000 7fa8fa6d5700 1 -- 172.21.15.78:0/1705432880 wait complete. 2023-10-10T17:53:30.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-10T17:53:31.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.051+0000 7f6233eb1700 1 -- 172.21.15.78:0/2126763290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 msgr2=0x7f622c113650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.051+0000 7f6233eb1700 1 --2- 172.21.15.78:0/2126763290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 0x7f622c113650 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f6224009f90 tx=0x7f622400b330 comp rx=0 tx=0).stop 2023-10-10T17:53:31.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.051+0000 7f6233eb1700 1 -- 172.21.15.78:0/2126763290 shutdown_connections 2023-10-10T17:53:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.051+0000 7f6233eb1700 1 --2- 172.21.15.78:0/2126763290 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c10c890 0x7f622c10cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.051+0000 7f6233eb1700 1 --2- 172.21.15.78:0/2126763290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 0x7f622c113650 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.051+0000 7f6233eb1700 1 --2- 172.21.15.78:0/2126763290 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f622c10d1a0 0x7f622c106960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.051+0000 7f6233eb1700 1 -- 172.21.15.78:0/2126763290 >> 172.21.15.78:0/2126763290 conn(0x7f622c074840 msgr2=0x7f622c076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:31.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.056+0000 7f6233eb1700 1 -- 172.21.15.78:0/2126763290 shutdown_connections 2023-10-10T17:53:31.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.058+0000 7f6233eb1700 1 -- 172.21.15.78:0/2126763290 wait complete. 2023-10-10T17:53:31.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.058+0000 7f6233eb1700 1 Processor -- start 2023-10-10T17:53:31.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6233eb1700 1 -- start start 2023-10-10T17:53:31.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6233eb1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 0x7f622c071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:31.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6233eb1700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f622c10c890 0x7f622c072530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:31.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6233eb1700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c10d1a0 0x7f622c06c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:31.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6233eb1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c06c7f0 con 0x7f622c107090 2023-10-10T17:53:31.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6233eb1700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c06c960 con 0x7f622c10c890 2023-10-10T17:53:31.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6233eb1700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f622c06cb20 con 0x7f622c10d1a0 2023-10-10T17:53:31.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f623144c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f622c10c890 0x7f622c072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:31.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f623144c700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f622c10c890 0x7f622c072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:51030/0 (socket says 172.21.15.78:51030) 2023-10-10T17:53:31.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f623144c700 1 -- 172.21.15.78:0/95808669 learned_addr learned my addr 172.21.15.78:0/95808669 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:31.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f623244e700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c10d1a0 0x7f622c06c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:31.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.059+0000 7f6231c4d700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 0x7f622c071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:31.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f623244e700 1 -- 172.21.15.78:0/95808669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f622c10c890 msgr2=0x7f622c072530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:31.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f623244e700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f622c10c890 0x7f622c072530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f623244e700 1 -- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 msgr2=0x7f622c071ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:31.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f623244e700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 0x7f622c071ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f623244e700 1 -- 172.21.15.78:0/95808669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6224009c40 con 0x7f622c10d1a0 2023-10-10T17:53:31.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f623244e700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c10d1a0 0x7f622c06c020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6224015000 tx=0x7f622400bc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:31.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f6222ffd700 1 -- 172.21.15.78:0/95808669 <== mon.1 v2:172.21.15.190:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622400e040 con 0x7f622c10d1a0 2023-10-10T17:53:31.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.060+0000 7f6222ffd700 1 -- 172.21.15.78:0/95808669 <== mon.1 v2:172.21.15.190:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6224007700 con 0x7f622c10d1a0 2023-10-10T17:53:31.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.061+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f622c06ae20 con 0x7f622c10d1a0 2023-10-10T17:53:31.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.061+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f622c06ac90 con 0x7f622c10d1a0 2023-10-10T17:53:31.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.061+0000 7f6222ffd700 1 -- 172.21.15.78:0/95808669 <== mon.1 v2:172.21.15.190:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f622401b460 con 0x7f622c10d1a0 2023-10-10T17:53:31.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.064+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6210005310 con 0x7f622c10d1a0 2023-10-10T17:53:31.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.066+0000 7f6222ffd700 1 -- 172.21.15.78:0/95808669 <== mon.1 v2:172.21.15.190:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f622401d470 con 0x7f622c10d1a0 2023-10-10T17:53:31.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.067+0000 7f6222ffd700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f62180a1380 0x7f62180a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:31.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.067+0000 7f6231c4d700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f62180a1380 0x7f62180a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:31.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.067+0000 7f6222ffd700 1 -- 172.21.15.78:0/95808669 <== mon.1 v2:172.21.15.190:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f62240ce0f0 con 0x7f622c10d1a0 2023-10-10T17:53:31.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.068+0000 7f6231c4d700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f62180a1380 0x7f62180a3830 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f621c009a10 tx=0x7f621c009190 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:31.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.073+0000 7f6222ffd700 1 -- 172.21.15.78:0/95808669 <== mon.1 v2:172.21.15.190:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f6224097170 con 0x7f622c10d1a0 2023-10-10T17:53:31.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:30 smithi078 ceph-mon[99494]: pgmap v341: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:53:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:30 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1299073680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:53:31.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:30 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1705432880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:53:31.229 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:30 smithi157 ceph-mon[100896]: pgmap v341: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:53:31.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:30 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1299073680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:53:31.230 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:30 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1705432880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:53:31.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:30 smithi190 ceph-mon[101220]: pgmap v341: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-10-10T17:53:31.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:30 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1299073680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-10-10T17:53:31.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:30 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1705432880' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-10-10T17:53:31.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.450+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f62100059e0 con 0x7f622c10d1a0 2023-10-10T17:53:31.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.458+0000 7f6222ffd700 1 -- 172.21.15.78:0/95808669 <== mon.1 v2:172.21.15.190:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v103) v1 ==== 81+0+939 (secure 0 0 0) 0x7f622409a790 con 0x7f622c10d1a0 2023-10-10T17:53:31.460 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1054117069 2023-10-11T17:42:35.200450+0000 2023-10-10T17:53:31.460 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1411676003 2023-10-11T17:42:35.200450+0000 2023-10-10T17:53:31.460 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/540877351 2023-10-11T17:42:35.200450+0000 2023-10-10T17:53:31.461 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/321866266 2023-10-11T17:42:35.200450+0000 2023-10-10T17:53:31.461 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2447900312 2023-10-11T17:34:36.143528+0000 2023-10-10T17:53:31.461 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/942965343 2023-10-11T17:34:36.143528+0000 2023-10-10T17:53:31.461 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/1662641434 2023-10-11T17:34:36.143528+0000 2023-10-10T17:53:31.462 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3346226185 2023-10-11T17:34:13.659771+0000 2023-10-10T17:53:31.462 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1720027577 2023-10-11T17:34:36.143528+0000 2023-10-10T17:53:31.462 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3346226185 2023-10-11T17:34:13.659771+0000 2023-10-10T17:53:31.463 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/951456648 2023-10-11T17:42:35.200450+0000 2023-10-10T17:53:31.463 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/843244395 2023-10-11T17:34:13.659771+0000 2023-10-10T17:53:31.463 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/667827327 2023-10-11T17:34:13.659771+0000 2023-10-10T17:53:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/321866266 2023-10-11T17:42:35.200450+0000 2023-10-10T17:53:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2252830072 2023-10-11T17:34:13.659771+0000 2023-10-10T17:53:31.464 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/1662641434 2023-10-11T17:34:36.143528+0000 2023-10-10T17:53:31.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f62180a1380 msgr2=0x7f62180a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:31.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f62180a1380 0x7f62180a3830 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f621c009a10 tx=0x7f621c009190 comp rx=0 tx=0).stop 2023-10-10T17:53:31.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c10d1a0 msgr2=0x7f622c06c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:31.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c10d1a0 0x7f622c06c020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6224015000 tx=0x7f622400bc20 comp rx=0 tx=0).stop 2023-10-10T17:53:31.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 shutdown_connections 2023-10-10T17:53:31.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f622c10d1a0 0x7f622c06c020 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f62180a1380 0x7f62180a3830 secure :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f621c009a10 tx=0x7f621c009190 comp rx=0 tx=0).stop 2023-10-10T17:53:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f622c107090 0x7f622c071ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 --2- 172.21.15.78:0/95808669 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f622c10c890 0x7f622c072530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:31.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.462+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 >> 172.21.15.78:0/95808669 conn(0x7f622c074840 msgr2=0x7f622c075dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:31.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.466+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 shutdown_connections 2023-10-10T17:53:31.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:31.469+0000 7f6233eb1700 1 -- 172.21.15.78:0/95808669 wait complete. 2023-10-10T17:53:31.486 INFO:teuthology.orchestra.run.smithi078.stderr:listed 16 entries 2023-10-10T17:53:31.503 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-10T17:53:31.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:53:31.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-10T17:53:31.738 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T17:53:31.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:53:31.739 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-10T17:53:31.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:53:31.857 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-10-10T17:53:32.061 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-10T17:53:32.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:53:32.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-10T17:53:32.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-10T17:53:32.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2023-10-10T17:53:32.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:31 smithi078 ceph-mon[99494]: pgmap v342: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:53:32.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:31 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/95808669' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:53:32.310 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-10-10T17:53:32.321 INFO:tasks.cephadm:Teardown begin 2023-10-10T17:53:32.322 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-10T17:53:32.323 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:53:32.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:31 smithi190 ceph-mon[101220]: pgmap v342: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:53:32.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:31 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/95808669' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:53:32.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:31 smithi157 ceph-mon[100896]: pgmap v342: 129 pgs: 18 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 56 active+clean; 799 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-10-10T17:53:32.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:31 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/95808669' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-10-10T17:53:32.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.577+0000 7f3d1664f700 1 -- 172.21.15.78:0/2963739155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10119a40 msgr2=0x7f3d1011be20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:32.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.577+0000 7f3d1664f700 1 --2- 172.21.15.78:0/2963739155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10119a40 0x7f3d1011be20 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f3d0800b0a0 tx=0x7f3d0800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:53:32.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.580+0000 7f3d1664f700 1 -- 172.21.15.78:0/2963739155 shutdown_connections 2023-10-10T17:53:32.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.580+0000 7f3d1664f700 1 --2- 172.21.15.78:0/2963739155 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d10107af0 0x7f3d1010eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:32.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.580+0000 7f3d1664f700 1 --2- 172.21.15.78:0/2963739155 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10119a40 0x7f3d1011be20 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:32.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.580+0000 7f3d1664f700 1 --2- 172.21.15.78:0/2963739155 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d10107ec0 0x7f3d10108330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:32.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.580+0000 7f3d1664f700 1 -- 172.21.15.78:0/2963739155 >> 172.21.15.78:0/2963739155 conn(0x7f3d1006c550 msgr2=0x7f3d100716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:32.584 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.582+0000 7f3d1664f700 1 -- 172.21.15.78:0/2963739155 shutdown_connections 2023-10-10T17:53:32.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.584+0000 7f3d1664f700 1 -- 172.21.15.78:0/2963739155 wait complete. 2023-10-10T17:53:32.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.584+0000 7f3d1664f700 1 Processor -- start 2023-10-10T17:53:32.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d1664f700 1 -- start start 2023-10-10T17:53:32.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d1664f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10107af0 0x7f3d10114a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:32.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d1664f700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d10107ec0 0x7f3d10114fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:32.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d1664f700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d10119a40 0x7f3d1010fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:32.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d1664f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d101102e0 con 0x7f3d10107af0 2023-10-10T17:53:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d1664f700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d10110450 con 0x7f3d10107ec0 2023-10-10T17:53:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d1664f700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d10110610 con 0x7f3d10119a40 2023-10-10T17:53:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d0ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10107af0 0x7f3d10114a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d0ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10107af0 0x7f3d10114a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49322/0 (socket says 172.21.15.78:49322) 2023-10-10T17:53:32.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.585+0000 7f3d0ffff700 1 -- 172.21.15.78:0/1678753885 learned_addr learned my addr 172.21.15.78:0/1678753885 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:32.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.586+0000 7f3d0ffff700 1 -- 172.21.15.78:0/1678753885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d10119a40 msgr2=0x7f3d1010fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:32.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.586+0000 7f3d0ffff700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d10119a40 0x7f3d1010fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:32.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.586+0000 7f3d0ffff700 1 -- 172.21.15.78:0/1678753885 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d10107ec0 msgr2=0x7f3d10114fc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:53:32.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.586+0000 7f3d0ffff700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d10107ec0 0x7f3d10114fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.586+0000 7f3d0ffff700 1 -- 172.21.15.78:0/1678753885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08009d00 con 0x7f3d10107af0 2023-10-10T17:53:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.586+0000 7f3d0ffff700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10107af0 0x7f3d10114a80 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000d950 tx=0x7f3d0000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.586+0000 7f3d0d7fa700 1 -- 172.21.15.78:0/1678753885 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d00009b40 con 0x7f3d10107af0 2023-10-10T17:53:32.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.587+0000 7f3d0d7fa700 1 -- 172.21.15.78:0/1678753885 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d0000b6e0 con 0x7f3d10107af0 2023-10-10T17:53:32.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.587+0000 7f3d0d7fa700 1 -- 172.21.15.78:0/1678753885 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d0000f620 con 0x7f3d10107af0 2023-10-10T17:53:32.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.593+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1011f1a0 con 0x7f3d10107af0 2023-10-10T17:53:32.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.593+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1011f010 con 0x7f3d10107af0 2023-10-10T17:53:32.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.594+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1011f330 con 0x7f3d10107af0 2023-10-10T17:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.599+0000 7f3d0d7fa700 1 -- 172.21.15.78:0/1678753885 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f3d00009ca0 con 0x7f3d10107af0 2023-10-10T17:53:32.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.599+0000 7f3d0d7fa700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3cf80a1400 0x7f3cf80a38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:32.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.600+0000 7f3d0d7fa700 1 -- 172.21.15.78:0/1678753885 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f3d000cd580 con 0x7f3d10107af0 2023-10-10T17:53:32.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.605+0000 7f3d0f7fe700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3cf80a1400 0x7f3cf80a38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:32.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.605+0000 7f3d0f7fe700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3cf80a1400 0x7f3cf80a38b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3d101114a0 tx=0x7f3d04009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:32.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:32.606+0000 7f3d0d7fa700 1 -- 172.21.15.78:0/1678753885 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f3d00096570 con 0x7f3d10107af0 2023-10-10T17:53:33.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.026+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d10067a60 con 0x7f3d10107af0 2023-10-10T17:53:33.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.029+0000 7f3d0d7fa700 1 -- 172.21.15.78:0/1678753885 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f3d00016090 con 0x7f3d10107af0 2023-10-10T17:53:33.030 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:53:33.031 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:52:25.779986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:53:33.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.033+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3cf80a1400 msgr2=0x7f3cf80a38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.033+0000 7f3d1664f700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3cf80a1400 0x7f3cf80a38b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3d101114a0 tx=0x7f3d04009260 comp rx=0 tx=0).stop 2023-10-10T17:53:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.033+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10107af0 msgr2=0x7f3d10114a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:33.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.033+0000 7f3d1664f700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10107af0 0x7f3d10114a80 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000d950 tx=0x7f3d0000dd10 comp rx=0 tx=0).stop 2023-10-10T17:53:33.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.034+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 shutdown_connections 2023-10-10T17:53:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.034+0000 7f3d1664f700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f3d10119a40 0x7f3d1010fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.034+0000 7f3d1664f700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f3cf80a1400 0x7f3cf80a38b0 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3d101114a0 tx=0x7f3d04009260 comp rx=0 tx=0).stop 2023-10-10T17:53:33.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.034+0000 7f3d1664f700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3d10107af0 0x7f3d10114a80 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.034+0000 7f3d1664f700 1 --2- 172.21.15.78:0/1678753885 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f3d10107ec0 0x7f3d10114fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.034+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 >> 172.21.15.78:0/1678753885 conn(0x7f3d1006c550 msgr2=0x7f3d1011a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:33.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.039+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 shutdown_connections 2023-10-10T17:53:33.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.047+0000 7f3d1664f700 1 -- 172.21.15.78:0/1678753885 wait complete. 2023-10-10T17:53:33.056 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 38 2023-10-10T17:53:33.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-10T17:53:33.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.353+0000 7f4074ff8700 1 -- 172.21.15.78:0/257216549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4070108c00 msgr2=0x7f4070108fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.353+0000 7f4074ff8700 1 --2- 172.21.15.78:0/257216549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4070108c00 0x7f4070108fd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f40600099c0 tx=0x7f4060009cd0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.356+0000 7f4074ff8700 1 -- 172.21.15.78:0/257216549 shutdown_connections 2023-10-10T17:53:33.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.356+0000 7f4074ff8700 1 --2- 172.21.15.78:0/257216549 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f4070108c00 0x7f4070108fd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.356+0000 7f4074ff8700 1 --2- 172.21.15.78:0/257216549 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f40701121b0 0x7f4070114590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.356+0000 7f4074ff8700 1 --2- 172.21.15.78:0/257216549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4070109510 0x7f4070111c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.356+0000 7f4074ff8700 1 -- 172.21.15.78:0/257216549 >> 172.21.15.78:0/257216549 conn(0x7f407006c550 msgr2=0x7f40700716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:33.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.357+0000 7f4074ff8700 1 -- 172.21.15.78:0/257216549 shutdown_connections 2023-10-10T17:53:33.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.357+0000 7f4074ff8700 1 -- 172.21.15.78:0/257216549 wait complete. 2023-10-10T17:53:33.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 Processor -- start 2023-10-10T17:53:33.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 -- start start 2023-10-10T17:53:33.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4070108c00 0x7f407007a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:33.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4070109510 0x7f407007a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:33.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40701121b0 0x7f40700752b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:33.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4070075a80 con 0x7f4070108c00 2023-10-10T17:53:33.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4070075bf0 con 0x7f4070109510 2023-10-10T17:53:33.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f4074ff8700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4070075db0 con 0x7f40701121b0 2023-10-10T17:53:33.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406dd9b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4070109510 0x7f407007a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:33.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4070108c00 0x7f407007a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:33.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406dd9b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4070109510 0x7f407007a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:51056/0 (socket says 172.21.15.78:51056) 2023-10-10T17:53:33.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406dd9b700 1 -- 172.21.15.78:0/1294960443 learned_addr learned my addr 172.21.15.78:0/1294960443 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:33.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406ed9d700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40701121b0 0x7f40700752b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:33.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406e59c700 1 -- 172.21.15.78:0/1294960443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40701121b0 msgr2=0x7f40700752b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:33.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406e59c700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40701121b0 0x7f40700752b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406e59c700 1 -- 172.21.15.78:0/1294960443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4070109510 msgr2=0x7f407007a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:33.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.358+0000 7f406e59c700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4070109510 0x7f407007a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.359+0000 7f406e59c700 1 -- 172.21.15.78:0/1294960443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40600095f0 con 0x7f4070108c00 2023-10-10T17:53:33.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.359+0000 7f406e59c700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4070108c00 0x7f407007a1a0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f40600098e0 tx=0x7f4060005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:33.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.359+0000 7f405f7fe700 1 -- 172.21.15.78:0/1294960443 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406001d070 con 0x7f4070108c00 2023-10-10T17:53:33.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.359+0000 7f405f7fe700 1 -- 172.21.15.78:0/1294960443 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4060003fa0 con 0x7f4070108c00 2023-10-10T17:53:33.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.359+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40700730c0 con 0x7f4070108c00 2023-10-10T17:53:33.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.359+0000 7f405f7fe700 1 -- 172.21.15.78:0/1294960443 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406000fcc0 con 0x7f4070108c00 2023-10-10T17:53:33.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.359+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4070072f30 con 0x7f4070108c00 2023-10-10T17:53:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.360+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4070073250 con 0x7f4070108c00 2023-10-10T17:53:33.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.362+0000 7f405f7fe700 1 -- 172.21.15.78:0/1294960443 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f406000f460 con 0x7f4070108c00 2023-10-10T17:53:33.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.363+0000 7f405f7fe700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f40580a1380 0x7f40580a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.363+0000 7f405f7fe700 1 -- 172.21.15.78:0/1294960443 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f40600cdea0 con 0x7f4070108c00 2023-10-10T17:53:33.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.363+0000 7f406dd9b700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f40580a1380 0x7f40580a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:33.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.364+0000 7f406dd9b700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f40580a1380 0x7f40580a3830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4070076a30 tx=0x7f4064007480 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:33.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.370+0000 7f405f7fe700 1 -- 172.21.15.78:0/1294960443 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f4060096f60 con 0x7f4070108c00 2023-10-10T17:53:33.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:33 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1678753885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:53:33.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:33 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1678753885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:53:33.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:33 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1678753885' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:53:33.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.799+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40700825c0 con 0x7f4070108c00 2023-10-10T17:53:33.814 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:53:33.814 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/1904698406","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":1904698406},{"type":"v1","addr":"172.21.15.78:6835","nonce":1904698406}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14565,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4004555102","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4004555102},{"type":"v1","addr":"172.21.15.78:6837","nonce":4004555102}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14625,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/943709499","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":943709499},{"type":"v1","addr":"172.21.15.78:6839","nonce":943709499}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14637,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/504515550","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":504515550},{"type":"v1","addr":"172.21.15.78:6841","nonce":504515550}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.190:6833/1708598976","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6832","nonce":1708598976},{"type":"v1","addr":"172.21.15.190:6833","nonce":1708598976}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":25802,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.157:6837/3534679839","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6836","nonce":3534679839},{"type":"v1","addr":"172.21.15.157:6837","nonce":3534679839}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":32}],"filesystems":[{"mdsmap":{"epoch":38,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-10-10T17:40:41.574125+0000","modified":"2023-10-10T17:52:25.779986+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24500,"mds_1":24490,"mds_2":24472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24419":{"gid":24419,"name":"k","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6833/1712500424","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6832","nonce":1712500424},{"type":"v1","addr":"172.21.15.157:6833","nonce":1712500424}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24437":{"gid":24437,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.157:6835/2501992479","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6834","nonce":2501992479},{"type":"v1","addr":"172.21.15.157:6835","nonce":2501992479}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.190:6835/1269112994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6834","nonce":1269112994},{"type":"v1","addr":"172.21.15.190:6835","nonce":1269112994}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24472":{"gid":24472,"name":"c","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.190:6837/661255149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6836","nonce":661255149},{"type":"v1","addr":"172.21.15.190:6837","nonce":661255149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24490":{"gid":24490,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.190:6839/486459111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6838","nonce":486459111},{"type":"v1","addr":"172.21.15.190:6839","nonce":486459111}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.157:6839/2554315829","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6838","nonce":2554315829},{"type":"v1","addr":"172.21.15.157:6839","nonce":2554315829}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-10T17:53:33.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.809+0000 7f405f7fe700 1 -- 172.21.15.78:0/1294960443 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f4060030020 con 0x7f4070108c00 2023-10-10T17:53:33.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.823+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f40580a1380 msgr2=0x7f40580a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:33.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.823+0000 7f4074ff8700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f40580a1380 0x7f40580a3830 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4070076a30 tx=0x7f4064007480 comp rx=0 tx=0).stop 2023-10-10T17:53:33.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.823+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4070108c00 msgr2=0x7f407007a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:33.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.823+0000 7f4074ff8700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4070108c00 0x7f407007a1a0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f40600098e0 tx=0x7f4060005740 comp rx=0 tx=0).stop 2023-10-10T17:53:33.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.827+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 shutdown_connections 2023-10-10T17:53:33.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.827+0000 7f4074ff8700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f40701121b0 0x7f40700752b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.827+0000 7f4074ff8700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f40580a1380 0x7f40580a3830 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.827+0000 7f4074ff8700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4070108c00 0x7f407007a1a0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.827+0000 7f4074ff8700 1 --2- 172.21.15.78:0/1294960443 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f4070109510 0x7f407007a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:33.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.827+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 >> 172.21.15.78:0/1294960443 conn(0x7f407006c550 msgr2=0x7f407006fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:33.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.838+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 shutdown_connections 2023-10-10T17:53:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:33.840+0000 7f4074ff8700 1 -- 172.21.15.78:0/1294960443 wait complete. 2023-10-10T17:53:33.841 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 38 2023-10-10T17:53:33.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-10T17:53:34.115 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:34 smithi078 ceph-mon[99494]: pgmap v343: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 799 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:34.116 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:34 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1294960443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:53:34.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.121+0000 7f055cb95700 1 -- 172.21.15.78:0/3225070295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 msgr2=0x7f0558072910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:34.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.121+0000 7f055cb95700 1 --2- 172.21.15.78:0/3225070295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558072910 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f0548009990 tx=0x7f0548009ca0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.129+0000 7f055cb95700 1 -- 172.21.15.78:0/3225070295 shutdown_connections 2023-10-10T17:53:34.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.129+0000 7f055cb95700 1 --2- 172.21.15.78:0/3225070295 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0558072ff0 0x7f05580799e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.129+0000 7f055cb95700 1 --2- 172.21.15.78:0/3225070295 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0558107d60 0x7f0558108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.129+0000 7f055cb95700 1 --2- 172.21.15.78:0/3225070295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558072910 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.129+0000 7f055cb95700 1 -- 172.21.15.78:0/3225070295 >> 172.21.15.78:0/3225070295 conn(0x7f055806c550 msgr2=0x7f05580716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:34.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.135+0000 7f055cb95700 1 -- 172.21.15.78:0/3225070295 shutdown_connections 2023-10-10T17:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.143+0000 7f055cb95700 1 -- 172.21.15.78:0/3225070295 wait complete. 2023-10-10T17:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 Processor -- start 2023-10-10T17:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 -- start start 2023-10-10T17:53:34.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0558072ff0 0x7f055807cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0558107d60 0x7f055807d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558081780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0558081dd0 con 0x7f0558108700 2023-10-10T17:53:34.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0558081f40 con 0x7f0558107d60 2023-10-10T17:53:34.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f055cb95700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0558082100 con 0x7f0558072ff0 2023-10-10T17:53:34.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f0556d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558081780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:34.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f0556d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558081780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49346/0 (socket says 172.21.15.78:49346) 2023-10-10T17:53:34.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.144+0000 7f0556d9d700 1 -- 172.21.15.78:0/3959734832 learned_addr learned my addr 172.21.15.78:0/3959734832 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:34.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.145+0000 7f0556d9d700 1 -- 172.21.15.78:0/3959734832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0558072ff0 msgr2=0x7f055807cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:34.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.145+0000 7f0556d9d700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0558072ff0 0x7f055807cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.145+0000 7f0556d9d700 1 -- 172.21.15.78:0/3959734832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0558107d60 msgr2=0x7f055807d2b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:53:34.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.145+0000 7f0556d9d700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0558107d60 0x7f055807d2b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.145+0000 7f0556d9d700 1 -- 172.21.15.78:0/3959734832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05480095f0 con 0x7f0558108700 2023-10-10T17:53:34.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.146+0000 7f0556d9d700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558081780 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f054c00b4a0 tx=0x7f054c00b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:34.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.146+0000 7f05477fe700 1 -- 172.21.15.78:0/3959734832 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c011830 con 0x7f0558108700 2023-10-10T17:53:34.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.146+0000 7f05477fe700 1 -- 172.21.15.78:0/3959734832 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f054c011e50 con 0x7f0558108700 2023-10-10T17:53:34.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.146+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f055807ba70 con 0x7f0558108700 2023-10-10T17:53:34.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.147+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f055807b8e0 con 0x7f0558108700 2023-10-10T17:53:34.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.147+0000 7f05477fe700 1 -- 172.21.15.78:0/3959734832 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c00f2e0 con 0x7f0558108700 2023-10-10T17:53:34.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.148+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0538005310 con 0x7f0558108700 2023-10-10T17:53:34.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.156+0000 7f05477fe700 1 -- 172.21.15.78:0/3959734832 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f054c010580 con 0x7f0558108700 2023-10-10T17:53:34.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.157+0000 7f05477fe700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f05400a13b0 0x7f05400a3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.157+0000 7f05477fe700 1 -- 172.21.15.78:0/3959734832 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f054c0cd000 con 0x7f0558108700 2023-10-10T17:53:34.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.158+0000 7f05477fe700 1 -- 172.21.15.78:0/3959734832 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f054c0cd3e0 con 0x7f0558108700 2023-10-10T17:53:34.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.158+0000 7f055659c700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f05400a13b0 0x7f05400a3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:34.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.159+0000 7f055659c700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f05400a13b0 0x7f05400a3860 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f055000e350 tx=0x7f055000b040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:34.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:34 smithi190 ceph-mon[101220]: pgmap v343: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 799 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:34.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:34 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1294960443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:53:34.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:34 smithi157 ceph-mon[100896]: pgmap v343: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 799 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 6 op/s 2023-10-10T17:53:34.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:34 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1294960443' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-10-10T17:53:34.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.509+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0538005cb0 con 0x7f0558108700 2023-10-10T17:53:34.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.512+0000 7f05477fe700 1 -- 172.21.15.78:0/3959734832 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25283 (secure 0 0 0) 0x7f054c015090 con 0x7f0558108700 2023-10-10T17:53:34.515 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-10T17:53:34.516 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":103,"fsid":"0ba2616a-6793-11ee-8db6-212e2dc638e7","created":"2023-10-10T17:33:42.140151+0000","modified":"2023-10-10T17:53:06.650410+0000","last_up_change":"2023-10-10T17:40:00.167418+0000","last_in_change":"2023-10-10T17:39:42.501678+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-10T17:36:46.460785+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-10T17:40:39.520570+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-10T17:40:40.752674+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-10-10T17:40:43.919172+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"d74c4b96-ee1f-4f21-9424-d693d38f0335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6803","nonce":857914992}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6805","nonce":857914992}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6809","nonce":857914992}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":857914992},{"type":"v1","addr":"172.21.15.78:6807","nonce":857914992}]},"public_addr":"172.21.15.78:6803/857914992","cluster_addr":"172.21.15.78:6805/857914992","heartbeat_back_addr":"172.21.15.78:6809/857914992","heartbeat_front_addr":"172.21.15.78:6807/857914992","state":["exists","up"]},{"osd":1,"uuid":"c01e2eb5-d98b-4e22-9339-06aaa290b7ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6800","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6801","nonce":4215285299}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6802","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6803","nonce":4215285299}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6806","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6807","nonce":4215285299}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6804","nonce":4215285299},{"type":"v1","addr":"172.21.15.157:6805","nonce":4215285299}]},"public_addr":"172.21.15.157:6801/4215285299","cluster_addr":"172.21.15.157:6803/4215285299","heartbeat_back_addr":"172.21.15.157:6807/4215285299","heartbeat_front_addr":"172.21.15.157:6805/4215285299","state":["exists","up"]},{"osd":2,"uuid":"4c48ed5d-d817-45b6-9c2c-1c371df35f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6801","nonce":678998123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6803","nonce":678998123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6807","nonce":678998123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":678998123},{"type":"v1","addr":"172.21.15.190:6805","nonce":678998123}]},"public_addr":"172.21.15.190:6801/678998123","cluster_addr":"172.21.15.190:6803/678998123","heartbeat_back_addr":"172.21.15.190:6807/678998123","heartbeat_front_addr":"172.21.15.190:6805/678998123","state":["exists","up"]},{"osd":3,"uuid":"0b2e185b-92ed-4d3e-8d87-e8f34fc640e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6811","nonce":2152346860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6813","nonce":2152346860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6817","nonce":2152346860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":2152346860},{"type":"v1","addr":"172.21.15.78:6815","nonce":2152346860}]},"public_addr":"172.21.15.78:6811/2152346860","cluster_addr":"172.21.15.78:6813/2152346860","heartbeat_back_addr":"172.21.15.78:6817/2152346860","heartbeat_front_addr":"172.21.15.78:6815/2152346860","state":["exists","up"]},{"osd":4,"uuid":"11514abf-4ffd-437b-9b35-e92d90c3a963","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6808","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6809","nonce":1484677990}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6810","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6811","nonce":1484677990}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6814","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6815","nonce":1484677990}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6812","nonce":1484677990},{"type":"v1","addr":"172.21.15.157:6813","nonce":1484677990}]},"public_addr":"172.21.15.157:6809/1484677990","cluster_addr":"172.21.15.157:6811/1484677990","heartbeat_back_addr":"172.21.15.157:6815/1484677990","heartbeat_front_addr":"172.21.15.157:6813/1484677990","state":["exists","up"]},{"osd":5,"uuid":"e1a3f83a-3fe5-470e-a55b-76a851412013","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6809","nonce":934325798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6811","nonce":934325798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6815","nonce":934325798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":934325798},{"type":"v1","addr":"172.21.15.190:6813","nonce":934325798}]},"public_addr":"172.21.15.190:6809/934325798","cluster_addr":"172.21.15.190:6811/934325798","heartbeat_back_addr":"172.21.15.190:6815/934325798","heartbeat_front_addr":"172.21.15.190:6813/934325798","state":["exists","up"]},{"osd":6,"uuid":"149dd00c-dd70-4d62-91db-f9a6830d918f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6819","nonce":2340006051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6821","nonce":2340006051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6825","nonce":2340006051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2340006051},{"type":"v1","addr":"172.21.15.78:6823","nonce":2340006051}]},"public_addr":"172.21.15.78:6819/2340006051","cluster_addr":"172.21.15.78:6821/2340006051","heartbeat_back_addr":"172.21.15.78:6825/2340006051","heartbeat_front_addr":"172.21.15.78:6823/2340006051","state":["exists","up"]},{"osd":7,"uuid":"2c72dbb4-e18a-4627-9871-2730cc29fcd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6816","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6817","nonce":1149014920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6818","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6819","nonce":1149014920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6822","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6823","nonce":1149014920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6820","nonce":1149014920},{"type":"v1","addr":"172.21.15.157:6821","nonce":1149014920}]},"public_addr":"172.21.15.157:6817/1149014920","cluster_addr":"172.21.15.157:6819/1149014920","heartbeat_back_addr":"172.21.15.157:6823/1149014920","heartbeat_front_addr":"172.21.15.157:6821/1149014920","state":["exists","up"]},{"osd":8,"uuid":"82521098-f51f-4c32-ad81-5d1c12c67a24","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6816","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6817","nonce":2390054900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6818","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6819","nonce":2390054900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6822","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6823","nonce":2390054900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6820","nonce":2390054900},{"type":"v1","addr":"172.21.15.190:6821","nonce":2390054900}]},"public_addr":"172.21.15.190:6817/2390054900","cluster_addr":"172.21.15.190:6819/2390054900","heartbeat_back_addr":"172.21.15.190:6823/2390054900","heartbeat_front_addr":"172.21.15.190:6821/2390054900","state":["exists","up"]},{"osd":9,"uuid":"7670b692-faa9-4a49-8d19-32418fa64d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6827","nonce":2313179218}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6829","nonce":2313179218}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6833","nonce":2313179218}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2313179218},{"type":"v1","addr":"172.21.15.78:6831","nonce":2313179218}]},"public_addr":"172.21.15.78:6827/2313179218","cluster_addr":"172.21.15.78:6829/2313179218","heartbeat_back_addr":"172.21.15.78:6833/2313179218","heartbeat_front_addr":"172.21.15.78:6831/2313179218","state":["exists","up"]},{"osd":10,"uuid":"998b645c-fda2-4969-9917-da58dede0c03","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6824","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6825","nonce":936155371}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6826","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6827","nonce":936155371}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6830","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6831","nonce":936155371}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.157:6828","nonce":936155371},{"type":"v1","addr":"172.21.15.157:6829","nonce":936155371}]},"public_addr":"172.21.15.157:6825/936155371","cluster_addr":"172.21.15.157:6827/936155371","heartbeat_back_addr":"172.21.15.157:6831/936155371","heartbeat_front_addr":"172.21.15.157:6829/936155371","state":["exists","up"]},{"osd":11,"uuid":"d638b1e0-6b53-45f2-9ea0-719cdf4ff100","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6824","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6825","nonce":2501616180}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6826","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6827","nonce":2501616180}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6830","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6831","nonce":2501616180}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6828","nonce":2501616180},{"type":"v1","addr":"172.21.15.190:6829","nonce":2501616180}]},"public_addr":"172.21.15.190:6825/2501616180","cluster_addr":"172.21.15.190:6827/2501616180","heartbeat_back_addr":"172.21.15.190:6831/2501616180","heartbeat_front_addr":"172.21.15.190:6829/2501616180","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:00.201058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:21.502256+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:36:43.435816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:05.486650+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:26.368117+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:37:48.671045+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:10.753665+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:32.684833+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:38:54.096135+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:15.905761+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:37.345196+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-10T17:39:58.247248+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":1}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.10","mappings":[{"from":5,"to":9}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]},{"pgid":"4.b","mappings":[{"from":9,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.1b","mappings":[{"from":9,"to":10}]},{"pgid":"4.1c","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:0/1054117069":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/1411676003":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/540877351":"2023-10-11T17:42:35.200450+0000","172.21.15.78:6801/321866266":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/2447900312":"2023-10-11T17:34:36.143528+0000","172.21.15.78:0/942965343":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/1662641434":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6801/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/1720027577":"2023-10-11T17:34:36.143528+0000","172.21.15.78:6800/3346226185":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/951456648":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/843244395":"2023-10-11T17:34:13.659771+0000","172.21.15.78:0/667827327":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/321866266":"2023-10-11T17:42:35.200450+0000","172.21.15.78:0/2252830072":"2023-10-11T17:34:13.659771+0000","172.21.15.78:6800/1662641434":"2023-10-11T17:34:36.143528+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-10T17:53:34.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.516+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f05400a13b0 msgr2=0x7f05400a3860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:34.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.516+0000 7f055cb95700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f05400a13b0 0x7f05400a3860 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f055000e350 tx=0x7f055000b040 comp rx=0 tx=0).stop 2023-10-10T17:53:34.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.516+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 msgr2=0x7f0558081780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:34.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.516+0000 7f055cb95700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558081780 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f054c00b4a0 tx=0x7f054c00b860 comp rx=0 tx=0).stop 2023-10-10T17:53:34.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.517+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 shutdown_connections 2023-10-10T17:53:34.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.517+0000 7f055cb95700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f0558072ff0 0x7f055807cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.517+0000 7f055cb95700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f05400a13b0 0x7f05400a3860 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.517+0000 7f055cb95700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0558108700 0x7f0558081780 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.517+0000 7f055cb95700 1 --2- 172.21.15.78:0/3959734832 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f0558107d60 0x7f055807d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.517+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 >> 172.21.15.78:0/3959734832 conn(0x7f055806c550 msgr2=0x7f0558070930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:34.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.519+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 shutdown_connections 2023-10-10T17:53:34.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.520+0000 7f055cb95700 1 -- 172.21.15.78:0/3959734832 wait complete. 2023-10-10T17:53:34.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-10T17:53:34.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.812+0000 7f287845b700 1 -- 172.21.15.78:0/3689587723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 msgr2=0x7f2870102a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:34.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.812+0000 7f287845b700 1 --2- 172.21.15.78:0/3689587723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f2870102a90 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f28600099c0 tx=0x7f2860009cd0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.816+0000 7f287845b700 1 -- 172.21.15.78:0/3689587723 shutdown_connections 2023-10-10T17:53:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.816+0000 7f287845b700 1 --2- 172.21.15.78:0/3689587723 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 0x7f287010f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.816+0000 7f287845b700 1 --2- 172.21.15.78:0/3689587723 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f2870102a90 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.816+0000 7f287845b700 1 --2- 172.21.15.78:0/3689587723 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2870103060 0x7f2870107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.816+0000 7f287845b700 1 -- 172.21.15.78:0/3689587723 >> 172.21.15.78:0/3689587723 conn(0x7f28700fdf50 msgr2=0x7f2870100360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:34.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.816+0000 7f287845b700 1 -- 172.21.15.78:0/3689587723 shutdown_connections 2023-10-10T17:53:34.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.816+0000 7f287845b700 1 -- 172.21.15.78:0/3689587723 wait complete. 2023-10-10T17:53:34.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 Processor -- start 2023-10-10T17:53:34.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 -- start start 2023-10-10T17:53:34.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f287006b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2870103060 0x7f287006b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 0x7f28701acba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28701ad170 con 0x7f28701026c0 2023-10-10T17:53:34.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28701ad2e0 con 0x7f2870103060 2023-10-10T17:53:34.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f287845b700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28701ad4a0 con 0x7f2870107700 2023-10-10T17:53:34.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f28761f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f287006b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:34.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f28769f8700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 0x7f28701acba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:34.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f28761f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f287006b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49358/0 (socket says 172.21.15.78:49358) 2023-10-10T17:53:34.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f28761f7700 1 -- 172.21.15.78:0/1662719583 learned_addr learned my addr 172.21.15.78:0/1662719583 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:34.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.817+0000 7f28769f8700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 0x7f28701acba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40890/0 (socket says 172.21.15.78:40890) 2023-10-10T17:53:34.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28761f7700 1 -- 172.21.15.78:0/1662719583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 msgr2=0x7f28701acba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:34.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28761f7700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 0x7f28701acba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28761f7700 1 -- 172.21.15.78:0/1662719583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2870103060 msgr2=0x7f287006b540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:53:34.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28761f7700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2870103060 0x7f287006b540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:34.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28761f7700 1 -- 172.21.15.78:0/1662719583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600095f0 con 0x7f28701026c0 2023-10-10T17:53:34.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28769f8700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 0x7f28701acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-10T17:53:34.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28761f7700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f287006b000 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2860005f50 tx=0x7f28600052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:34.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28677fe700 1 -- 172.21.15.78:0/1662719583 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286001d070 con 0x7f28701026c0 2023-10-10T17:53:34.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28677fe700 1 -- 172.21.15.78:0/1662719583 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f286000bc50 con 0x7f28701026c0 2023-10-10T17:53:34.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f28677fe700 1 -- 172.21.15.78:0/1662719583 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286000fc50 con 0x7f28701026c0 2023-10-10T17:53:34.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28701125b0 con 0x7f28701026c0 2023-10-10T17:53:34.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.818+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2870112420 con 0x7f28701026c0 2023-10-10T17:53:34.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.820+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2870112740 con 0x7f28701026c0 2023-10-10T17:53:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.821+0000 7f28677fe700 1 -- 172.21.15.78:0/1662719583 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f286000b790 con 0x7f28701026c0 2023-10-10T17:53:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.822+0000 7f28677fe700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f285c0a13c0 0x7f285c0a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:34.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.822+0000 7f28677fe700 1 -- 172.21.15.78:0/1662719583 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9520+0+0 (secure 0 0 0) 0x7f28600cdf70 con 0x7f28701026c0 2023-10-10T17:53:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.822+0000 7f28759f6700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f285c0a13c0 0x7f285c0a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.822+0000 7f28759f6700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f285c0a13c0 0x7f285c0a3870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f287006c610 tx=0x7f286c009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:34.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:34.828+0000 7f28677fe700 1 -- 172.21.15.78:0/1662719583 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f2860096ff0 con 0x7f28701026c0 2023-10-10T17:53:35.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:35.258+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f28700679f0 con 0x7f28701026c0 2023-10-10T17:53:35.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:35 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3959734832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:53:35.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:35 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3959734832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:53:35.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:35 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3959734832' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-10T17:53:36.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.138+0000 7f28677fe700 1 -- 172.21.15.78:0/1662719583 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v39) v1 ==== 161+0+0 (secure 0 0 0) 0x7f286009a610 con 0x7f28701026c0 2023-10-10T17:53:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.143+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f285c0a13c0 msgr2=0x7f285c0a3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:36.143 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.143+0000 7f287845b700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f285c0a13c0 0x7f285c0a3870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f287006c610 tx=0x7f286c009260 comp rx=0 tx=0).stop 2023-10-10T17:53:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.143+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 msgr2=0x7f287006b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:36.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.143+0000 7f287845b700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f287006b000 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2860005f50 tx=0x7f28600052f0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.146 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.146+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 shutdown_connections 2023-10-10T17:53:36.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.146+0000 7f287845b700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f2870107700 0x7f28701acba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.146+0000 7f287845b700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f285c0a13c0 0x7f285c0a3870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.146+0000 7f287845b700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f28701026c0 0x7f287006b000 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.146+0000 7f287845b700 1 --2- 172.21.15.78:0/1662719583 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f2870103060 0x7f287006b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.146+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 >> 172.21.15.78:0/1662719583 conn(0x7f28700fdf50 msgr2=0x7f28700fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:36.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.147+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 shutdown_connections 2023-10-10T17:53:36.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.150+0000 7f287845b700 1 -- 172.21.15.78:0/1662719583 wait complete. 2023-10-10T17:53:36.150 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-10T17:53:36.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-10T17:53:36.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:36 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1662719583' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-10-10T17:53:36.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:36 smithi078 ceph-mon[99494]: osdmap e104: 12 total, 12 up, 12 in 2023-10-10T17:53:36.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:36 smithi078 ceph-mon[99494]: pgmap v345: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 799 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:53:36.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:36 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-a[99470]: 2023-10-10T17:53:36.112+0000 7f54b1b2f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-10T17:53:36.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.407+0000 7f1bb6510700 1 -- 172.21.15.78:0/410354199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 msgr2=0x7f1bb010b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:36.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.407+0000 7f1bb6510700 1 --2- 172.21.15.78:0/410354199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 0x7f1bb010b790 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800b0a0 tx=0x7f1ba800b3b0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.411+0000 7f1bb6510700 1 -- 172.21.15.78:0/410354199 shutdown_connections 2023-10-10T17:53:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.411+0000 7f1bb6510700 1 --2- 172.21.15.78:0/410354199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 0x7f1bb010b790 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.411+0000 7f1bb6510700 1 --2- 172.21.15.78:0/410354199 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bb0104960 0x7f1bb0104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.411+0000 7f1bb6510700 1 --2- 172.21.15.78:0/410354199 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb00fe950 0x7f1bb00fedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.411+0000 7f1bb6510700 1 -- 172.21.15.78:0/410354199 >> 172.21.15.78:0/410354199 conn(0x7f1bb0075920 msgr2=0x7f1bb0075d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:36.414 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-i[127457]: ignoring --setuser ceph since I am not root 2023-10-10T17:53:36.414 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-i[127457]: ignoring --setgroup ceph since I am not root 2023-10-10T17:53:36.415 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-i[127457]: starting mds.i at 2023-10-10T17:53:36.415 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-c[127115]: ignoring --setuser ceph since I am not root 2023-10-10T17:53:36.416 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-c[127115]: ignoring --setgroup ceph since I am not root 2023-10-10T17:53:36.416 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-mds[127139]: -- 172.21.15.190:0/1757985877 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564cb0cd3520 con 0x564cb0ca8c00 2023-10-10T17:53:36.416 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-c[127115]: starting mds.c at 2023-10-10T17:53:36.417 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-f[126774]: ignoring --setuser ceph since I am not root 2023-10-10T17:53:36.417 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-f[126774]: ignoring --setgroup ceph since I am not root 2023-10-10T17:53:36.417 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-f[126774]: starting mds.f at 2023-10-10T17:53:36.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1662719583' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-10-10T17:53:36.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-mon[101220]: osdmap e104: 12 total, 12 up, 12 in 2023-10-10T17:53:36.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:36 smithi190 ceph-mon[101220]: pgmap v345: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 799 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:53:36.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.420+0000 7f1bb6510700 1 -- 172.21.15.78:0/410354199 shutdown_connections 2023-10-10T17:53:36.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.424+0000 7f1bb6510700 1 -- 172.21.15.78:0/410354199 wait complete. 2023-10-10T17:53:36.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.424+0000 7f1bb6510700 1 Processor -- start 2023-10-10T17:53:36.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.424+0000 7f1bb6510700 1 -- start start 2023-10-10T17:53:36.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb6510700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bb00fe950 0x7f1bb01070e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:36.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb6510700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 0x7f1bb0107620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb6510700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb0104960 0x7f1bb0191fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb6510700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb01925a0 con 0x7f1bb00ff300 2023-10-10T17:53:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb6510700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb0192710 con 0x7f1bb00fe950 2023-10-10T17:53:36.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb6510700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb01928d0 con 0x7f1bb0104960 2023-10-10T17:53:36.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb4aad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb0104960 0x7f1bb0191fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:36.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb4aad700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb0104960 0x7f1bb0191fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.190:3300/0 says I am v2:172.21.15.78:40900/0 (socket says 172.21.15.78:40900) 2023-10-10T17:53:36.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.425+0000 7f1bb4aad700 1 -- 172.21.15.78:0/2083499323 learned_addr learned my addr 172.21.15.78:0/2083499323 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:36.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.427+0000 7f1baf7fe700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 0x7f1bb0107620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:36.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.428+0000 7f1baf7fe700 1 -- 172.21.15.78:0/2083499323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb0104960 msgr2=0x7f1bb0191fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:36.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.428+0000 7f1baf7fe700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb0104960 0x7f1bb0191fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.428+0000 7f1baf7fe700 1 -- 172.21.15.78:0/2083499323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bb00fe950 msgr2=0x7f1bb01070e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-10T17:53:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.428+0000 7f1baf7fe700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bb00fe950 0x7f1bb01070e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.428+0000 7f1baf7fe700 1 -- 172.21.15.78:0/2083499323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8009d00 con 0x7f1bb00ff300 2023-10-10T17:53:36.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.428+0000 7f1baf7fe700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 0x7f1bb0107620 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1ba001c970 tx=0x7f1ba001cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.432+0000 7f1bad7fa700 1 -- 172.21.15.78:0/2083499323 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000bb60 con 0x7f1bb00ff300 2023-10-10T17:53:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.432+0000 7f1bad7fa700 1 -- 172.21.15.78:0/2083499323 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ba001ce60 con 0x7f1bb00ff300 2023-10-10T17:53:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.432+0000 7f1bad7fa700 1 -- 172.21.15.78:0/2083499323 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000e640 con 0x7f1bb00ff300 2023-10-10T17:53:36.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.433+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb006cc80 con 0x7f1bb00ff300 2023-10-10T17:53:36.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.433+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb006caf0 con 0x7f1bb00ff300 2023-10-10T17:53:36.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.437+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b940052e0 con 0x7f1bb00ff300 2023-10-10T17:53:36.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.442+0000 7f1bad7fa700 1 -- 172.21.15.78:0/2083499323 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f1ba000bcc0 con 0x7f1bb00ff300 2023-10-10T17:53:36.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.443+0000 7f1bad7fa700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f1b980c78b0 0x7f1b980c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:36.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.443+0000 7f1bad7fa700 1 -- 172.21.15.78:0/2083499323 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9690+0+0 (secure 0 0 0) 0x7f1ba00cda10 con 0x7f1bb00ff300 2023-10-10T17:53:36.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.445+0000 7f1bad7fa700 1 -- 172.21.15.78:0/2083499323 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f1ba00969b0 con 0x7f1bb00ff300 2023-10-10T17:53:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.447+0000 7f1baffff700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f1b980c78b0 0x7f1b980c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:36.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.448+0000 7f1baffff700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f1b980c78b0 0x7f1b980c9d60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00ffa90 tx=0x7f1ba4009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:36.478 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: ignoring --setuser ceph since I am not root 2023-10-10T17:53:36.479 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: ignoring --setgroup ceph since I am not root 2023-10-10T17:53:36.479 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-mds[126230]: -- 172.21.15.157:0/2488780700 <== mon.2 v2:172.21.15.157:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e8a6e55520 con 0x55e8a6e2b000 2023-10-10T17:53:36.479 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: starting mds.b at 2023-10-10T17:53:36.480 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-k[125187]: ignoring --setuser ceph since I am not root 2023-10-10T17:53:36.480 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-k[125187]: ignoring --setgroup ceph since I am not root 2023-10-10T17:53:36.480 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-k[125187]: starting mds.k at 2023-10-10T17:53:36.481 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-e[125527]: ignoring --setuser ceph since I am not root 2023-10-10T17:53:36.482 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-e[125527]: ignoring --setgroup ceph since I am not root 2023-10-10T17:53:36.482 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-e[125527]: starting mds.e at 2023-10-10T17:53:36.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1662719583' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-10-10T17:53:36.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-mon[100896]: osdmap e104: 12 total, 12 up, 12 in 2023-10-10T17:53:36.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:36 smithi157 ceph-mon[100896]: pgmap v345: 129 pgs: 17 active+clean+snaptrim, 55 active+clean+snaptrim_wait, 57 active+clean; 799 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-10-10T17:53:36.871 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:36.869+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1b94005f30 con 0x7f1bb00ff300 2023-10-10T17:53:37.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.139+0000 7f1bad7fa700 1 -- 172.21.15.78:0/2083499323 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v40) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1ba0099fd0 con 0x7f1bb00ff300 2023-10-10T17:53:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.144+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f1b980c78b0 msgr2=0x7f1b980c9d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:37.144 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.144+0000 7f1bb6510700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f1b980c78b0 0x7f1b980c9d60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1bb00ffa90 tx=0x7f1ba4009260 comp rx=0 tx=0).stop 2023-10-10T17:53:37.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.144+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 msgr2=0x7f1bb0107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:37.145 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.144+0000 7f1bb6510700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 0x7f1bb0107620 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1ba001c970 tx=0x7f1ba001cc80 comp rx=0 tx=0).stop 2023-10-10T17:53:37.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.147+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 shutdown_connections 2023-10-10T17:53:37.147 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.147+0000 7f1bb6510700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f1bb0104960 0x7f1bb0191fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.147+0000 7f1bb6510700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f1b980c78b0 0x7f1b980c9d60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.147+0000 7f1bb6510700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f1bb00fe950 0x7f1bb01070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.147+0000 7f1bb6510700 1 --2- 172.21.15.78:0/2083499323 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1bb00ff300 0x7f1bb0107620 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.147+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 >> 172.21.15.78:0/2083499323 conn(0x7f1bb0075920 msgr2=0x7f1bb00fcf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:37.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.150+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 shutdown_connections 2023-10-10T17:53:37.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.153+0000 7f1bb6510700 1 -- 172.21.15.78:0/2083499323 wait complete. 2023-10-10T17:53:37.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-10T17:53:37.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:37 smithi078 ceph-mon[99494]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-10-10T17:53:37.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:37 smithi078 ceph-mon[99494]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-10T17:53:37.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:37 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/1662719583' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-10-10T17:53:37.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:37 smithi078 ceph-mon[99494]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-10-10T17:53:37.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:37 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2083499323' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:53:37.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:37 smithi190 ceph-mon[101220]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-10-10T17:53:37.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:37 smithi190 ceph-mon[101220]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-10T17:53:37.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:37 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/1662719583' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-10-10T17:53:37.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:37 smithi190 ceph-mon[101220]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-10-10T17:53:37.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:37 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2083499323' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:53:37.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.420+0000 7f82dc50d700 1 -- 172.21.15.78:0/2905917156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 msgr2=0x7f82d4103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:37.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.420+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2905917156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 0x7f82d4103db0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00b0a0 tx=0x7f82cc00b3b0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.425+0000 7f82dc50d700 1 -- 172.21.15.78:0/2905917156 shutdown_connections 2023-10-10T17:53:37.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.425+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2905917156 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82d40ff450 0x7f82d40ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.425+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2905917156 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d4104490 0x7f82d410c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.425+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2905917156 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 0x7f82d4103db0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.425+0000 7f82dc50d700 1 -- 172.21.15.78:0/2905917156 >> 172.21.15.78:0/2905917156 conn(0x7f82d40770b0 msgr2=0x7f82d40774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:37.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.428+0000 7f82dc50d700 1 -- 172.21.15.78:0/2905917156 shutdown_connections 2023-10-10T17:53:37.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.430+0000 7f82dc50d700 1 -- 172.21.15.78:0/2905917156 wait complete. 2023-10-10T17:53:37.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.430+0000 7f82dc50d700 1 Processor -- start 2023-10-10T17:53:37.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82dc50d700 1 -- start start 2023-10-10T17:53:37.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82dc50d700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82d40ff450 0x7f82d410b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82dc50d700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 0x7f82d410b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82dc50d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d4104490 0x7f82d4199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:37.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82dc50d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d419a5c0 con 0x7f82d4104490 2023-10-10T17:53:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82dc50d700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d419a730 con 0x7f82d40ffdf0 2023-10-10T17:53:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82dc50d700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d419a8f0 con 0x7f82d40ff450 2023-10-10T17:53:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82daaaa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d4104490 0x7f82d4199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:37.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82daaaa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d4104490 0x7f82d4199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49384/0 (socket says 172.21.15.78:49384) 2023-10-10T17:53:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82daaaa700 1 -- 172.21.15.78:0/2918943570 learned_addr learned my addr 172.21.15.78:0/2918943570 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.431+0000 7f82d9aa8700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 0x7f82d410b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:37.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82da2a9700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82d40ff450 0x7f82d410b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82d9aa8700 1 -- 172.21.15.78:0/2918943570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82d40ff450 msgr2=0x7f82d410b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82d9aa8700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82d40ff450 0x7f82d410b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82d9aa8700 1 -- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d4104490 msgr2=0x7f82d4199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82d9aa8700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d4104490 0x7f82d4199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82d9aa8700 1 -- 172.21.15.78:0/2918943570 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc009d00 con 0x7f82d40ffdf0 2023-10-10T17:53:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82da2a9700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82d40ff450 0x7f82d410b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-10T17:53:37.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.432+0000 7f82d9aa8700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 0x7f82d410b590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00b070 tx=0x7f82cc007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:37.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.434+0000 7f82cb7fe700 1 -- 172.21.15.78:0/2918943570 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc00e040 con 0x7f82d40ffdf0 2023-10-10T17:53:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.434+0000 7f82cb7fe700 1 -- 172.21.15.78:0/2918943570 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82cc00be20 con 0x7f82d40ffdf0 2023-10-10T17:53:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.434+0000 7f82cb7fe700 1 -- 172.21.15.78:0/2918943570 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82cc01b6e0 con 0x7f82d40ffdf0 2023-10-10T17:53:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.434+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d4110400 con 0x7f82d40ffdf0 2023-10-10T17:53:37.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.434+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d4110270 con 0x7f82d40ffdf0 2023-10-10T17:53:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.437+0000 7f82cb7fe700 1 -- 172.21.15.78:0/2918943570 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f82cc01b840 con 0x7f82d40ffdf0 2023-10-10T17:53:37.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.437+0000 7f82cb7fe700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f82c00a1410 0x7f82c00a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:37.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.438+0000 7f82cb7fe700 1 -- 172.21.15.78:0/2918943570 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9690+0+0 (secure 0 0 0) 0x7f82cc0cf050 con 0x7f82d40ffdf0 2023-10-10T17:53:37.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.441+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b8005310 con 0x7f82d40ffdf0 2023-10-10T17:53:37.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.448+0000 7f82da2a9700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f82c00a1410 0x7f82c00a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:37.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.449+0000 7f82da2a9700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f82c00a1410 0x7f82c00a38c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009be0 tx=0x7f82d0009260 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:37.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.449+0000 7f82cb7fe700 1 -- 172.21.15.78:0/2918943570 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f82cc098060 con 0x7f82d40ffdf0 2023-10-10T17:53:37.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:37 smithi157 ceph-mon[100896]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-10-10T17:53:37.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:37 smithi157 ceph-mon[100896]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-10T17:53:37.480 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:37 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/1662719583' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-10-10T17:53:37.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:37 smithi157 ceph-mon[100896]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-10-10T17:53:37.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:37 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2083499323' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-10-10T17:53:37.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:37.791+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f82b80059e0 con 0x7f82d40ffdf0 2023-10-10T17:53:38.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.158+0000 7f82cb7fe700 1 -- 172.21.15.78:0/2918943570 <== mon.2 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v105) v1 ==== 181+0+0 (secure 0 0 0) 0x7f82cc09b680 con 0x7f82d40ffdf0 2023-10-10T17:53:38.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.166+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f82c00a1410 msgr2=0x7f82c00a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:38.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.166+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f82c00a1410 0x7f82c00a38c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f82d0009be0 tx=0x7f82d0009260 comp rx=0 tx=0).stop 2023-10-10T17:53:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.166+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 msgr2=0x7f82d410b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.166+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 0x7f82d410b590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f82cc00b070 tx=0x7f82cc007870 comp rx=0 tx=0).stop 2023-10-10T17:53:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 shutdown_connections 2023-10-10T17:53:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f82d40ff450 0x7f82d410b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f82c00a1410 0x7f82c00a38c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82d4104490 0x7f82d4199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 --2- 172.21.15.78:0/2918943570 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f82d40ffdf0 0x7f82d410b590 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 >> 172.21.15.78:0/2918943570 conn(0x7f82d40770b0 msgr2=0x7f82d40fcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 shutdown_connections 2023-10-10T17:53:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.167+0000 7f82dc50d700 1 -- 172.21.15.78:0/2918943570 wait complete. 2023-10-10T17:53:38.174 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2023-10-10T17:53:38.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-10T17:53:38.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-10-10T17:53:38.378 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-10-10T17:53:38.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: Cluster is now healthy 2023-10-10T17:53:38.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6832/2077523948,v1:172.21.15.157:6833/2077523948] up:boot 2023-10-10T17:53:38.379 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6834/1680134895,v1:172.21.15.157:6839/1680134895] up:boot 2023-10-10T17:53:38.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.157:6835/349284786,v1:172.21.15.157:6838/349284786] up:boot 2023-10-10T17:53:38.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6836/179689905,v1:172.21.15.190:6837/179689905] up:boot 2023-10-10T17:53:38.380 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6834/1469847069,v1:172.21.15.190:6835/1469847069] up:boot 2023-10-10T17:53:38.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: mds.? [v2:172.21.15.190:6838/3207680032,v1:172.21.15.190:6839/3207680032] up:boot 2023-10-10T17:53:38.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2083499323' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-10-10T17:53:38.381 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: fsmap 12 up:standby 2023-10-10T17:53:38.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:53:38.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:53:38.382 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:53:38.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:53:38.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:53:38.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:53:38.383 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: pgmap v346: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 799 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2023-10-10T17:53:38.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/2918943570' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:38.384 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:38 smithi078 ceph-mon[99494]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:38.414 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-10-10T17:53:38.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-10-10T17:53:38.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: Cluster is now healthy 2023-10-10T17:53:38.415 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6832/2077523948,v1:172.21.15.157:6833/2077523948] up:boot 2023-10-10T17:53:38.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6834/1680134895,v1:172.21.15.157:6839/1680134895] up:boot 2023-10-10T17:53:38.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.157:6835/349284786,v1:172.21.15.157:6838/349284786] up:boot 2023-10-10T17:53:38.416 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6836/179689905,v1:172.21.15.190:6837/179689905] up:boot 2023-10-10T17:53:38.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6834/1469847069,v1:172.21.15.190:6835/1469847069] up:boot 2023-10-10T17:53:38.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: mds.? [v2:172.21.15.190:6838/3207680032,v1:172.21.15.190:6839/3207680032] up:boot 2023-10-10T17:53:38.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2083499323' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-10-10T17:53:38.417 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: fsmap 12 up:standby 2023-10-10T17:53:38.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:53:38.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:53:38.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:53:38.418 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:53:38.419 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:53:38.419 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:53:38.419 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: pgmap v346: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 799 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2023-10-10T17:53:38.420 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/2918943570' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:38.420 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:38 smithi190 ceph-mon[101220]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.431+0000 7f2923418700 1 -- 172.21.15.78:0/1340927024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c1015f0 msgr2=0x7f291c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.431+0000 7f2923418700 1 --2- 172.21.15.78:0/1340927024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c1015f0 0x7f291c104f50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f290c01a9d0 tx=0x7f290c01ace0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.436+0000 7f2923418700 1 -- 172.21.15.78:0/1340927024 shutdown_connections 2023-10-10T17:53:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.436+0000 7f2923418700 1 --2- 172.21.15.78:0/1340927024 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291c100c50 0x7f291c101020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.436+0000 7f2923418700 1 --2- 172.21.15.78:0/1340927024 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c105630 0x7f291c10d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.436+0000 7f2923418700 1 --2- 172.21.15.78:0/1340927024 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c1015f0 0x7f291c104f50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.436+0000 7f2923418700 1 -- 172.21.15.78:0/1340927024 >> 172.21.15.78:0/1340927024 conn(0x7f291c0775c0 msgr2=0x7f291c0779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.436+0000 7f2923418700 1 -- 172.21.15.78:0/1340927024 shutdown_connections 2023-10-10T17:53:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.436+0000 7f2923418700 1 -- 172.21.15.78:0/1340927024 wait complete. 2023-10-10T17:53:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 Processor -- start 2023-10-10T17:53:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 -- start start 2023-10-10T17:53:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c100c50 0x7f291c10c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:38.440 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291c1015f0 0x7f291c10cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c105630 0x7f291c072d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c10d5c0 con 0x7f291c100c50 2023-10-10T17:53:38.441 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c073290 con 0x7f291c105630 2023-10-10T17:53:38.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f2923418700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c073450 con 0x7f291c1015f0 2023-10-10T17:53:38.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f29211b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c100c50 0x7f291c10c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:38.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f29211b4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c100c50 0x7f291c10c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:49394/0 (socket says 172.21.15.78:49394) 2023-10-10T17:53:38.442 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.437+0000 7f29211b4700 1 -- 172.21.15.78:0/3742486512 learned_addr learned my addr 172.21.15.78:0/3742486512 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:38.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29219b5700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c105630 0x7f291c072d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:38.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29211b4700 1 -- 172.21.15.78:0/3742486512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291c1015f0 msgr2=0x7f291c10cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:38.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29211b4700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291c1015f0 0x7f291c10cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29211b4700 1 -- 172.21.15.78:0/3742486512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c105630 msgr2=0x7f291c072d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:38.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29211b4700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c105630 0x7f291c072d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:38.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29211b4700 1 -- 172.21.15.78:0/3742486512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c01a600 con 0x7f291c100c50 2023-10-10T17:53:38.444 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29211b4700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c100c50 0x7f291c10c8b0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f291800e970 tx=0x7f291800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:38.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29127fc700 1 -- 172.21.15.78:0/3742486512 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2918009790 con 0x7f291c100c50 2023-10-10T17:53:38.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29127fc700 1 -- 172.21.15.78:0/3742486512 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f291800eec0 con 0x7f291c100c50 2023-10-10T17:53:38.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c06cd70 con 0x7f291c100c50 2023-10-10T17:53:38.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f29127fc700 1 -- 172.21.15.78:0/3742486512 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2918003cf0 con 0x7f291c100c50 2023-10-10T17:53:38.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.438+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c06cbe0 con 0x7f291c100c50 2023-10-10T17:53:38.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.441+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c06cf00 con 0x7f291c100c50 2023-10-10T17:53:38.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.442+0000 7f29127fc700 1 -- 172.21.15.78:0/3742486512 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f291800ccd0 con 0x7f291c100c50 2023-10-10T17:53:38.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.442+0000 7f29127fc700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29080a1380 0x7f29080a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:38.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.443+0000 7f29127fc700 1 -- 172.21.15.78:0/3742486512 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9157+0+0 (secure 0 0 0) 0x7f2918014070 con 0x7f291c100c50 2023-10-10T17:53:38.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.448+0000 7f29209b3700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29080a1380 0x7f29080a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:38.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.451+0000 7f29127fc700 1 -- 172.21.15.78:0/3742486512 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f29180963a0 con 0x7f291c100c50 2023-10-10T17:53:38.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.454+0000 7f29209b3700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29080a1380 0x7f29080a3830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f290c004800 tx=0x7f290c003680 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:38.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-10-10T17:53:38.481 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-10-10T17:53:38.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: Cluster is now healthy 2023-10-10T17:53:38.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6832/2077523948,v1:172.21.15.157:6833/2077523948] up:boot 2023-10-10T17:53:38.482 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6834/1680134895,v1:172.21.15.157:6839/1680134895] up:boot 2023-10-10T17:53:38.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.157:6835/349284786,v1:172.21.15.157:6838/349284786] up:boot 2023-10-10T17:53:38.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6836/179689905,v1:172.21.15.190:6837/179689905] up:boot 2023-10-10T17:53:38.483 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6834/1469847069,v1:172.21.15.190:6835/1469847069] up:boot 2023-10-10T17:53:38.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: mds.? [v2:172.21.15.190:6838/3207680032,v1:172.21.15.190:6839/3207680032] up:boot 2023-10-10T17:53:38.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2083499323' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-10-10T17:53:38.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: fsmap 12 up:standby 2023-10-10T17:53:38.484 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-10-10T17:53:38.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-10-10T17:53:38.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-10-10T17:53:38.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-10-10T17:53:38.485 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-10-10T17:53:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='mgr.15588 172.21.15.78:0/1801043258' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-10-10T17:53:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: pgmap v346: 129 pgs: 12 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 88 active+clean; 799 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 921 B/s wr, 7 op/s 2023-10-10T17:53:38.486 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/2918943570' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:38.487 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:38 smithi157 ceph-mon[100896]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:38.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:38.790+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f291c073c00 con 0x7f291c100c50 2023-10-10T17:53:39.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.153+0000 7f29127fc700 1 -- 172.21.15.78:0/3742486512 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v106) v1 ==== 169+0+0 (secure 0 0 0) 0x7f29180999c0 con 0x7f291c100c50 2023-10-10T17:53:39.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29080a1380 msgr2=0x7f29080a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:39.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29080a1380 0x7f29080a3830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f290c004800 tx=0x7f290c003680 comp rx=0 tx=0).stop 2023-10-10T17:53:39.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c100c50 msgr2=0x7f291c10c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:39.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c100c50 0x7f291c10c8b0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f291800e970 tx=0x7f291800ec80 comp rx=0 tx=0).stop 2023-10-10T17:53:39.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 shutdown_connections 2023-10-10T17:53:39.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f29080a1380 0x7f29080a3830 secure :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f290c004800 tx=0x7f290c003680 comp rx=0 tx=0).stop 2023-10-10T17:53:39.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f291c105630 0x7f291c072d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f291c100c50 0x7f291c10c8b0 secure :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f291800e970 tx=0x7f291800ec80 comp rx=0 tx=0).stop 2023-10-10T17:53:39.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 --2- 172.21.15.78:0/3742486512 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f291c1015f0 0x7f291c10cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.158+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 >> 172.21.15.78:0/3742486512 conn(0x7f291c0775c0 msgr2=0x7f291c0fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:39.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.164+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 shutdown_connections 2023-10-10T17:53:39.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.168+0000 7f2923418700 1 -- 172.21.15.78:0/3742486512 wait complete. 2023-10-10T17:53:39.193 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2023-10-10T17:53:39.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-10-10T17:53:39.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 -- 172.21.15.78:0/4235518120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 msgr2=0x7f9298103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 --2- 172.21.15.78:0/4235518120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 0x7f9298103db0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f928800b0d0 tx=0x7f928800b3e0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 -- 172.21.15.78:0/4235518120 shutdown_connections 2023-10-10T17:53:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 --2- 172.21.15.78:0/4235518120 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f92980ff450 0x7f92980ff820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 --2- 172.21.15.78:0/4235518120 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f9298104490 0x7f929810c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 --2- 172.21.15.78:0/4235518120 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 0x7f9298103db0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 -- 172.21.15.78:0/4235518120 >> 172.21.15.78:0/4235518120 conn(0x7f9298077080 msgr2=0x7f9298077480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:39.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 -- 172.21.15.78:0/4235518120 shutdown_connections 2023-10-10T17:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.462+0000 7f929e4fa700 1 -- 172.21.15.78:0/4235518120 wait complete. 2023-10-10T17:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 Processor -- start 2023-10-10T17:53:39.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 -- start start 2023-10-10T17:53:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f92980ff450 0x7f92981953e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:39.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 0x7f9298195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 --2- >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9298104490 0x7f9298199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929819a3c0 con 0x7f92980ffdf0 2023-10-10T17:53:39.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 -- --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929819a530 con 0x7f92980ff450 2023-10-10T17:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.463+0000 7f929e4fa700 1 -- --> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929819a6f0 con 0x7f9298104490 2023-10-10T17:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f92980ff450 0x7f92981953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:39.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 --2- >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f92980ff450 0x7f92981953e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.157:3300/0 says I am v2:172.21.15.78:51128/0 (socket says 172.21.15.78:51128) 2023-10-10T17:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 -- 172.21.15.78:0/3512888659 learned_addr learned my addr 172.21.15.78:0/3512888659 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-10-10T17:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f929ca97700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9298104490 0x7f9298199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f92977fe700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 0x7f9298195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:39.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 -- 172.21.15.78:0/3512888659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9298104490 msgr2=0x7f9298199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9298104490 0x7f9298199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 -- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 msgr2=0x7f9298195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:39.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 0x7f9298195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 -- 172.21.15.78:0/3512888659 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9288009d00 con 0x7f92980ff450 2023-10-10T17:53:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f9297fff700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f92980ff450 0x7f92981953e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f928c00b550 tx=0x7f928c00b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f92957fa700 1 -- 172.21.15.78:0/3512888659 <== mon.2 v2:172.21.15.157:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c011830 con 0x7f92980ff450 2023-10-10T17:53:39.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f92957fa700 1 -- 172.21.15.78:0/3512888659 <== mon.2 v2:172.21.15.157:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f928c011e50 con 0x7f92980ff450 2023-10-10T17:53:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929810f350 con 0x7f92980ff450 2023-10-10T17:53:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f92957fa700 1 -- 172.21.15.78:0/3512888659 <== mon.2 v2:172.21.15.157:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c010e40 con 0x7f92980ff450 2023-10-10T17:53:39.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.464+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929810f1c0 con 0x7f92980ff450 2023-10-10T17:53:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.465+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929810f4e0 con 0x7f92980ff450 2023-10-10T17:53:39.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.468+0000 7f92957fa700 1 -- 172.21.15.78:0/3512888659 <== mon.2 v2:172.21.15.157:3300/0 4 ==== mgrmap(e 22) v1 ==== 135846+0+0 (secure 0 0 0) 0x7f928c010380 con 0x7f92980ff450 2023-10-10T17:53:39.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.468+0000 7f92957fa700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f92800a1300 0x7f92800a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-10T17:53:39.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.469+0000 7f92957fa700 1 -- 172.21.15.78:0/3512888659 <== mon.2 v2:172.21.15.157:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8598+0+0 (secure 0 0 0) 0x7f928c0ccc20 con 0x7f92980ff450 2023-10-10T17:53:39.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.469+0000 7f92977fe700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f92800a1300 0x7f92800a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-10T17:53:39.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.470+0000 7f92977fe700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f92800a1300 0x7f92800a37b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9288000f80 tx=0x7f928800b040 comp rx=0 tx=0).ready entity=mgr.15588 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-10T17:53:39.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.471+0000 7f92957fa700 1 -- 172.21.15.78:0/3512888659 <== mon.2 v2:172.21.15.157:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177948 (secure 0 0 0) 0x7f928c0960f0 con 0x7f92980ff450 2023-10-10T17:53:39.477 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:39 smithi157 ceph-mon[100896]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-10-10T17:53:39.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:39 smithi157 ceph-mon[100896]: osdmap e105: 12 total, 12 up, 12 in 2023-10-10T17:53:39.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:39 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3742486512' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:39.628 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:39 smithi078 ceph-mon[99494]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-10-10T17:53:39.629 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:39 smithi078 ceph-mon[99494]: osdmap e105: 12 total, 12 up, 12 in 2023-10-10T17:53:39.630 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:39 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3742486512' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:39.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:39 smithi190 ceph-mon[101220]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-10-10T17:53:39.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:39 smithi190 ceph-mon[101220]: osdmap e105: 12 total, 12 up, 12 in 2023-10-10T17:53:39.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:39 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3742486512' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:39.799+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 --> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9298196830 con 0x7f92980ff450 2023-10-10T17:53:40.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.199+0000 7f92957fa700 1 -- 172.21.15.78:0/3512888659 <== mon.2 v2:172.21.15.157:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v107) v1 ==== 178+0+0 (secure 0 0 0) 0x7f928c0150d0 con 0x7f92980ff450 2023-10-10T17:53:40.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f92800a1300 msgr2=0x7f92800a37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:40.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f92800a1300 0x7f92800a37b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9288000f80 tx=0x7f928800b040 comp rx=0 tx=0).stop 2023-10-10T17:53:40.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f92980ff450 msgr2=0x7f92981953e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-10T17:53:40.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f92980ff450 0x7f92981953e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f928c00b550 tx=0x7f928c00b860 comp rx=0 tx=0).stop 2023-10-10T17:53:40.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 shutdown_connections 2023-10-10T17:53:40.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] conn(0x7f9298104490 0x7f9298199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:40.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:6800/2386743294,v1:172.21.15.78:6801/2386743294] conn(0x7f92800a1300 0x7f92800a37b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:40.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.157:3300/0,v1:172.21.15.157:6789/0] conn(0x7f92980ff450 0x7f92981953e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:40.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 --2- 172.21.15.78:0/3512888659 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f92980ffdf0 0x7f9298195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-10T17:53:40.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 >> 172.21.15.78:0/3512888659 conn(0x7f9298077080 msgr2=0x7f92980fcc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-10T17:53:40.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 shutdown_connections 2023-10-10T17:53:40.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-10-10T17:53:40.202+0000 7f929e4fa700 1 -- 172.21.15.78:0/3512888659 wait complete. 2023-10-10T17:53:40.210 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' removed 2023-10-10T17:53:40.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:53:40.264 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:53:40.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:53:40.365 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-10-10T17:53:40.366 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-10T17:53:40.383 DEBUG:teuthology.orchestra.run.smithi157:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-10T17:53:40.399 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-10T17:53:40.419 INFO:tasks.cephadm:Stopping all daemons... 2023-10-10T17:53:40.420 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-10-10T17:53:40.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a 2023-10-10T17:53:40.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3742486512' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-10-10T17:53:40.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 ceph-mon[99494]: osdmap e106: 12 total, 12 up, 12 in 2023-10-10T17:53:40.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 ceph-mon[99494]: pgmap v349: 33 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 14 active+clean; 577 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:53:40.448 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 ceph-mon[99494]: from='client.? 172.21.15.78:0/3512888659' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:40.449 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 ceph-mon[99494]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:40.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:40 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3742486512' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-10-10T17:53:40.478 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:40 smithi157 ceph-mon[100896]: osdmap e106: 12 total, 12 up, 12 in 2023-10-10T17:53:40.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:40 smithi157 ceph-mon[100896]: pgmap v349: 33 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 14 active+clean; 577 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:53:40.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:40 smithi157 ceph-mon[100896]: from='client.? 172.21.15.78:0/3512888659' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:40.479 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:40 smithi157 ceph-mon[100896]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:40.664 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:40 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3742486512' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-10-10T17:53:40.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:40 smithi190 ceph-mon[101220]: osdmap e106: 12 total, 12 up, 12 in 2023-10-10T17:53:40.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:40 smithi190 ceph-mon[101220]: pgmap v349: 33 pgs: 9 active+clean+snaptrim, 10 active+clean+snaptrim_wait, 14 active+clean; 577 KiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail 2023-10-10T17:53:40.665 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:40 smithi190 ceph-mon[101220]: from='client.? 172.21.15.78:0/3512888659' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:40.666 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:40 smithi190 ceph-mon[101220]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-10-10T17:53:40.851 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 systemd[1]: Stopping Ceph mon.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:53:41.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-a[99470]: 2023-10-10T17:53:40.850+0000 7f54b4b35700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:53:41.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-a[99470]: 2023-10-10T17:53:40.850+0000 7f54b4b35700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-10-10T17:53:41.489 INFO:journalctl@ceph.mon.a.smithi078.stdout:Oct 10 17:53:41 smithi078 bash[265292]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-a 2023-10-10T17:53:41.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.a.service' 2023-10-10T17:53:41.774 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:53:41.774 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-10-10T17:53:41.775 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-10-10T17:53:41.775 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.b 2023-10-10T17:53:42.130 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:41 smithi157 systemd[1]: Stopping Ceph mon.b for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:53:42.131 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-b[100872]: 2023-10-10T17:53:42.127+0000 7f1674cc9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:53:42.409 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-b[100872]: 2023-10-10T17:53:42.127+0000 7f1674cc9700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-10-10T17:53:42.715 INFO:journalctl@ceph.mon.b.smithi157.stdout:Oct 10 17:53:42 smithi157 bash[132542]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-b 2023-10-10T17:53:42.721 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.b.service' 2023-10-10T17:53:42.844 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:53:42.846 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-10-10T17:53:42.846 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-10-10T17:53:42.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.c 2023-10-10T17:53:43.164 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:42 smithi190 systemd[1]: Stopping Ceph mon.c for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:53:43.646 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-c[101193]: 2023-10-10T17:53:43.244+0000 7f4b3e6e9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:53:43.646 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-c[101193]: 2023-10-10T17:53:43.244+0000 7f4b3e6e9700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-10-10T17:53:43.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.c.service' 2023-10-10T17:53:43.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:43 smithi190 bash[133337]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mon-c 2023-10-10T17:53:43.914 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:43 smithi190 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mon.c.service: Succeeded. 2023-10-10T17:53:43.915 INFO:journalctl@ceph.mon.c.smithi190.stdout:Oct 10 17:53:43 smithi190 systemd[1]: Stopped Ceph mon.c for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:53:43.945 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:53:43.946 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-10-10T17:53:43.946 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-10-10T17:53:43.946 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.x 2023-10-10T17:53:44.378 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:53:43 smithi078 systemd[1]: Stopping Ceph mgr.x for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:53:45.127 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:53:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:53:44.868+0000 7f47ba42e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:53:45.128 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:53:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x[99737]: 2023-10-10T17:53:44.868+0000 7f47ba42e700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-10-10T17:53:46.377 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Oct 10 17:53:46 smithi078 bash[265431]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-x 2023-10-10T17:53:48.035 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.x.service' 2023-10-10T17:53:48.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:53:48.085 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-10-10T17:53:48.085 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-10-10T17:53:48.086 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.y 2023-10-10T17:53:48.478 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:53:48 smithi157 systemd[1]: Stopping Ceph mgr.y for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:53:48.794 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:53:48 smithi157 bash[132688]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-y 2023-10-10T17:53:48.795 INFO:journalctl@ceph.mgr.y.smithi157.stdout:Oct 10 17:53:48 smithi157 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-10-10T17:53:48.979 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.y.service' 2023-10-10T17:53:49.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:53:49.016 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-10-10T17:53:49.017 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-10-10T17:53:49.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.z 2023-10-10T17:53:49.414 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:53:49 smithi190 systemd[1]: Stopping Ceph mgr.z for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:53:52.224 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:53:51 smithi190 bash[133477]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mgr-z 2023-10-10T17:53:52.224 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:53:52 smithi190 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-10-10T17:53:52.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.z.service' 2023-10-10T17:53:52.253 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:53:52 smithi190 systemd[1]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mgr.z.service: Failed with result 'exit-code'. 2023-10-10T17:53:52.254 INFO:journalctl@ceph.mgr.z.smithi190.stdout:Oct 10 17:53:52 smithi190 systemd[1]: Stopped Ceph mgr.z for 0ba2616a-6793-11ee-8db6-212e2dc638e7. 2023-10-10T17:53:52.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:53:52.267 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-10-10T17:53:52.267 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-10-10T17:53:52.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.0 2023-10-10T17:53:52.627 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:53:52 smithi078 systemd[1]: Stopping Ceph osd.0 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:53:53.127 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:53:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-0[112519]: 2023-10-10T17:53:52.640+0000 7f60a24b2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:53:53.128 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:53:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-0[112519]: 2023-10-10T17:53:52.640+0000 7f60a24b2700 -1 osd.0 107 *** Got signal Terminated *** 2023-10-10T17:53:53.128 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:53:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-0[112519]: 2023-10-10T17:53:52.640+0000 7f60a24b2700 -1 osd.0 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:53:58.628 INFO:journalctl@ceph.osd.0.smithi078.stdout:Oct 10 17:53:58 smithi078 bash[265583]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-0 2023-10-10T17:54:01.362 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.0.service' 2023-10-10T17:54:01.403 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:01.404 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-10-10T17:54:01.404 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-10-10T17:54:01.405 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.1 2023-10-10T17:54:01.727 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:54:01 smithi157 systemd[1]: Stopping Ceph osd.1 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:02.227 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:54:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-1[105957]: 2023-10-10T17:54:01.766+0000 7fbc2ee83700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:02.228 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:54:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-1[105957]: 2023-10-10T17:54:01.766+0000 7fbc2ee83700 -1 osd.1 107 *** Got signal Terminated *** 2023-10-10T17:54:02.228 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:54:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-1[105957]: 2023-10-10T17:54:01.766+0000 7fbc2ee83700 -1 osd.1 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:07.978 INFO:journalctl@ceph.osd.1.smithi157.stdout:Oct 10 17:54:07 smithi157 bash[132834]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-1 2023-10-10T17:54:08.954 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.1.service' 2023-10-10T17:54:08.996 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:08.996 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-10-10T17:54:08.997 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-10-10T17:54:08.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.2 2023-10-10T17:54:09.353 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:54:09 smithi190 systemd[1]: Stopping Ceph osd.2 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:09.664 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:54:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-2[107039]: 2023-10-10T17:54:09.351+0000 7f7f37bd8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:09.664 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:54:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-2[107039]: 2023-10-10T17:54:09.351+0000 7f7f37bd8700 -1 osd.2 107 *** Got signal Terminated *** 2023-10-10T17:54:09.665 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:54:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-2[107039]: 2023-10-10T17:54:09.351+0000 7f7f37bd8700 -1 osd.2 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:15.233 INFO:journalctl@ceph.osd.2.smithi190.stdout:Oct 10 17:54:15 smithi190 bash[133634]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-2 2023-10-10T17:54:16.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.2.service' 2023-10-10T17:54:16.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:16.652 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-10-10T17:54:16.652 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-10-10T17:54:16.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.3 2023-10-10T17:54:17.021 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:16 smithi078 systemd[1]: Stopping Ceph osd.3 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:17.378 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:17 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:54:17.020+0000 7f41e4497700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:17.378 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:17 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:54:17.020+0000 7f41e4497700 -1 osd.3 107 *** Got signal Terminated *** 2023-10-10T17:54:17.378 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:17 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:54:17.020+0000 7f41e4497700 -1 osd.3 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:20.378 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:20 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:54:20.032+0000 7f41de288700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.929538+0000 front 2023-10-10T17:53:56.929564+0000 (oldest deadline 2023-10-10T17:54:19.229645+0000) 2023-10-10T17:54:20.379 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:20 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:20.080+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:20.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:20.168+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:21.377 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:21 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:54:21.012+0000 7f41de288700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.929538+0000 front 2023-10-10T17:53:56.929564+0000 (oldest deadline 2023-10-10T17:54:19.229645+0000) 2023-10-10T17:54:21.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:20 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:20.992+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:21.379 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:21 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:21.063+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:21.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:21.130+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:21.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:21 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:21.416+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:22.378 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:21 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3[117891]: 2023-10-10T17:54:21.987+0000 7f41de288700 -1 osd.3 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.929538+0000 front 2023-10-10T17:53:56.929564+0000 (oldest deadline 2023-10-10T17:54:19.229645+0000) 2023-10-10T17:54:22.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:22 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:22.051+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:22.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:21 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:21.986+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:22.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:22.132+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:22.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:22 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:22.451+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:23.127 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:23 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:23.036+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:23.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:22 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:22.973+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:23.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:23 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:23.119+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:23.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:23 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:23.266+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:23.627 INFO:journalctl@ceph.osd.3.smithi078.stdout:Oct 10 17:54:23 smithi078 bash[265866]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-3 2023-10-10T17:54:23.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:23 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:23.493+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:24.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:24 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:24.066+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:24.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:24 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:24.020+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:24.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:24 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:24.216+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:24.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:24 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:24.076+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:24.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:24 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:24.472+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:24.783 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.3.service' 2023-10-10T17:54:24.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:24.824 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-10-10T17:54:24.825 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-10-10T17:54:24.825 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.4 2023-10-10T17:54:25.058 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:25 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:25.022+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:25.178 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:54:24 smithi157 systemd[1]: Stopping Ceph osd.4 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:25.377 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:25 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:25.058+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:25.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:25 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:25.187+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:25.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:25 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:25.030+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:25.477 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:54:25 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:54:25.177+0000 7f1ebdb60700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:25.478 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:54:25 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:54:25.177+0000 7f1ebdb60700 -1 osd.4 107 *** Got signal Terminated *** 2023-10-10T17:54:25.478 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:54:25 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:54:25.177+0000 7f1ebdb60700 -1 osd.4 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:25.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:25 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:25.485+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:26.377 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:26 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:26.090+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:26.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:26 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:26.049+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:26.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:26 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:26.194+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:26.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:26 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:26.034+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:26.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:26 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:26.463+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:27.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:27 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:27.088+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:27.379 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:27 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:27.096+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:27.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:27 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:27.168+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:27.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:27 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:27.006+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:27.779 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:27 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:27.503+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:28.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:27 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:27.778+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:28.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:28 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:28.085+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:28.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:28 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:28.048+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:28.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:28.127+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:28.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:28.127+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:28.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:28.031+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:28.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:28 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:28.031+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:28.727 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:54:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:54:28.581+0000 7f1eba156700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.019315+0000 front 2023-10-10T17:54:05.019358+0000 (oldest deadline 2023-10-10T17:54:28.420112+0000) 2023-10-10T17:54:28.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:28.456+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:28.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:28.456+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:29.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:28 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:28.822+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:29.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:29.011+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:29.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:29.124+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:29.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:29.157+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:29.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:29.157+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:29.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:29.074+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:29.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:29 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:29.074+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:29.727 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:54:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4[111465]: 2023-10-10T17:54:29.534+0000 7f1eba156700 -1 osd.4 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.019315+0000 front 2023-10-10T17:54:05.019358+0000 (oldest deadline 2023-10-10T17:54:28.420112+0000) 2023-10-10T17:54:29.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:29.427+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:29.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:29.427+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:30.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:29 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:29.775+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:30.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:29.968+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:30.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:29 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:29.968+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:30.379 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:30.111+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:30.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:30 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:30.116+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:30.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:30 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:30.116+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:30.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:30 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:30.122+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:30.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:30 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:30.122+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:30.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:30 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:30.469+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:30.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:30 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:30.469+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:31.227 INFO:journalctl@ceph.osd.4.smithi157.stdout:Oct 10 17:54:30 smithi157 bash[133104]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-4 2023-10-10T17:54:31.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:30 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:30.818+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:31.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:30.989+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:31.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:30.989+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:31.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:31.077+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:31.379 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:31.077+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:31.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:31 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:31.121+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:31.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:31 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:31.121+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:31.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:31 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:31.082+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:31.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:31 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:31.082+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:31.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:31 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:31.430+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:31.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:31 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:31.430+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:32.117 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:31 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:31.819+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:32.124 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.4.service' 2023-10-10T17:54:32.125 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:31 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:31.761+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:32.164 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:32.165 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-10-10T17:54:32.165 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-10-10T17:54:32.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.5 2023-10-10T17:54:32.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:31.960+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:32.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:31 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:31.960+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:32.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:32.125+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:32.379 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:32.125+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:32.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:32.125+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:32.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:32.125+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:32.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:32.151+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:32.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:32.151+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:32.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:32 smithi190 systemd[1]: Stopping Ceph osd.5 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:32.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:32.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:32.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:32.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:32.808 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:32.530+0000 7ff6eb1a2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:32.809 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:32.530+0000 7ff6eb1a2700 -1 osd.5 107 *** Got signal Terminated *** 2023-10-10T17:54:32.809 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:32.530+0000 7ff6eb1a2700 -1 osd.5 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:33.121 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:32 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:32.807+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:33.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:32 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:32.811+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:33.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:32.964+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:33.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:32 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:32.964+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:33.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:33.104+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:33.379 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:33.104+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:33.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:33.125+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:33.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:33.126+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:33.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:33.120+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:33.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:33.120+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:33.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:33 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:33.443+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:33.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:33 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:33.443+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:34.121 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:33 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:33.773+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:34.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:33 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:33.791+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:34.377 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:34.147+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:34.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:34.147+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:34.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:33.962+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:34.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:33 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:33.962+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:34.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:34 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:34.121+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:34.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:34 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:34.121+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:34.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:34 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:34.126+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:34.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:34 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:34.126+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:34.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:34 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:34.459+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:34.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:34 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:34.459+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:35.113 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:34 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:34.789+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:35.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:34 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:34.805+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:35.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:35.173+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:35.378 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:35.173+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:35.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:34.941+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:35.380 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:34 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:34.941+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:35.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:35 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:35.113+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:35.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:35 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:35.113+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:35.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:35 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:35.150+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:35.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:35 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:35.150+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:35.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:35 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:35.462+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:35.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:35 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:35.462+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:36.112 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:35 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:35.787+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:36.184 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:35.927+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:36.185 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:35.927+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:36.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:35 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:35.807+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:36.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:36.112+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:36.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:36.112+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:36.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:36.112+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.995424+0000 front 2023-10-10T17:54:13.995444+0000 (oldest deadline 2023-10-10T17:54:35.695600+0000) 2023-10-10T17:54:36.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:36.183+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:36.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:36.183+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:36.627 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:36 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:36.184+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:36.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:36 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:36.184+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:36.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:36.441+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:36.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:36.441+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:37.143 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:36.818+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:37.144 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:36 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:36.818+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:37.169 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:36 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:36.883+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:37.169 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:36 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:36.883+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:37.170 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:36 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:36.883+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:37.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:36 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:36.842+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:37.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:37.145+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:37.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:37.145+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:37.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:37.143+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:57.593255+0000 front 2023-10-10T17:53:57.593198+0000 (oldest deadline 2023-10-10T17:54:22.293280+0000) 2023-10-10T17:54:37.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:37.143+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.293824+0000 front 2023-10-10T17:54:02.293718+0000 (oldest deadline 2023-10-10T17:54:27.593841+0000) 2023-10-10T17:54:37.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5[112576]: 2023-10-10T17:54:37.143+0000 7ff6e5f95700 -1 osd.5 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.995424+0000 front 2023-10-10T17:54:13.995444+0000 (oldest deadline 2023-10-10T17:54:35.695600+0000) 2023-10-10T17:54:37.627 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:37 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:37.169+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:37.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:37 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:37.169+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:37.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:37 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:37.467+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:37.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:37 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:37.467+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:38.108 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:37.851+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:38.109 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:37 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:37.851+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:38.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:37 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:37.849+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:38.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:37 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:37.849+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:38.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:37 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:37.849+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:38.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:37 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:37.824+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:38.371 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:38 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:38.107+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:38.371 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:38 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:38.107+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:38.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:38 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:38.190+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:38.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:38 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:38.190+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:38.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:38 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:38.190+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.425795+0000 front 2023-10-10T17:54:13.425816+0000 (oldest deadline 2023-10-10T17:54:38.125822+0000) 2023-10-10T17:54:38.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Oct 10 17:54:38 smithi190 bash[133898]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-5 2023-10-10T17:54:38.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:38 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:38.427+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:38.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:38 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:38.427+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:38.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:38 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:38.427+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:39.117 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:38 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:38.815+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:39.118 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:38 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:38.815+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:39.180 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:38 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:38.892+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:39.181 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:38 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:38.892+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:39.181 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:38 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:38.892+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:39.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:38 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:38.815+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:39.367 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:39.116+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:39.368 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:39.116+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:39.368 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:39.116+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:39.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:39 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:39.180+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:39.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:39 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:39.180+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:39.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:39 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:39.180+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.425795+0000 front 2023-10-10T17:54:13.425816+0000 (oldest deadline 2023-10-10T17:54:38.125822+0000) 2023-10-10T17:54:39.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:39.463+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:39.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:39.464+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:39.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:39.464+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:39.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.5.service' 2023-10-10T17:54:39.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:39.810 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-10-10T17:54:39.811 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-10-10T17:54:39.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.6 2023-10-10T17:54:40.115 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:39.809+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:40.115 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:39 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:39.809+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:40.127 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:39 smithi078 systemd[1]: Stopping Ceph osd.6 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:40.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:39 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:39.877+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:40.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:39 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:39.878+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:40.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:39 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:39.878+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:40.231 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:39.806+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:40.231 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:39 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:39.806+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:40.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:40.114+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:40.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:40.114+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:40.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:40.114+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:40.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:40.142+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:40.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:40.142+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:40.629 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:40.142+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.425795+0000 front 2023-10-10T17:54:13.425816+0000 (oldest deadline 2023-10-10T17:54:38.125822+0000) 2023-10-10T17:54:40.629 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:40.176+0000 7fd2e101f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:40.629 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:40.176+0000 7fd2e101f700 -1 osd.6 107 *** Got signal Terminated *** 2023-10-10T17:54:40.629 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:40.176+0000 7fd2e101f700 -1 osd.6 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:40.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:40.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:40.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:40.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:40.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:40.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:41.081 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:40.824+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:41.081 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:40 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:40.824+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:41.173 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:40.902+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:41.174 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:40.902+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:41.174 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:40.902+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:41.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:40.798+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:41.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:40 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:40.798+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:41.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:41.080+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:41.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:41.080+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:41.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:41.080+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:41.627 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:41 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:41.173+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:41.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:41 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:41.173+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:41.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:41 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:41.173+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.425795+0000 front 2023-10-10T17:54:13.425816+0000 (oldest deadline 2023-10-10T17:54:38.125822+0000) 2023-10-10T17:54:41.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:41 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:41.447+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:41.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:41 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:41.447+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:41.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:41 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:41.447+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:42.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:41.868+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:42.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:41 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:41.868+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:42.165 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:42.109+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:42.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:42.109+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:42.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:42.109+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:42.209 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:41 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:41.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:42.210 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:41 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:41.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:42.210 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:41 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:41.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:42.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:41 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:41.791+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:42.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:41 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:41.791+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:42.627 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:42 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:42.209+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:42.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:42 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:42.209+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:42.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:42 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:42.209+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.425795+0000 front 2023-10-10T17:54:13.425816+0000 (oldest deadline 2023-10-10T17:54:38.125822+0000) 2023-10-10T17:54:42.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:42.472+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:42.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:42.472+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:42.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:42.472+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:43.112 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:42.861+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:43.112 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:42 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:42.861+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:43.191 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:42 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:42.929+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:43.192 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:42 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:42.929+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:43.192 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:42 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:42.929+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:43.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:42.758+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:43.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:42 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:42.758+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:43.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:43.111+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:43.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:43.111+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:43.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:43.111+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:43.627 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:43 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:43.191+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:43.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:43 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:43.191+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:43.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:43 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:43.191+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.425795+0000 front 2023-10-10T17:54:13.425816+0000 (oldest deadline 2023-10-10T17:54:38.125822+0000) 2023-10-10T17:54:43.629 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:43 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:43.191+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.126222+0000 front 2023-10-10T17:54:18.126439+0000 (oldest deadline 2023-10-10T17:54:42.826368+0000) 2023-10-10T17:54:43.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:43.452+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:43.740 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:43.452+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:43.741 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:43.452+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:44.128 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:43.876+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:44.129 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:43 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:43.876+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:44.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:43.767+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:44.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:43.767+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:44.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:43 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:43.767+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:44.239 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:43 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:43.888+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:44.239 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:43 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:43.888+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:44.239 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:43 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:43.888+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:44.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:44 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:44.128+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:44.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:44 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:44.128+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:44.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:44 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:44.128+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:44.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:44.238+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.322936+0000 front 2023-10-10T17:53:55.322818+0000 (oldest deadline 2023-10-10T17:54:19.422933+0000) 2023-10-10T17:54:44.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:44.238+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.624674+0000 front 2023-10-10T17:54:04.624552+0000 (oldest deadline 2023-10-10T17:54:30.524641+0000) 2023-10-10T17:54:44.628 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:44.238+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.425795+0000 front 2023-10-10T17:54:13.425816+0000 (oldest deadline 2023-10-10T17:54:38.125822+0000) 2023-10-10T17:54:44.629 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6[123358]: 2023-10-10T17:54:44.238+0000 7fd2dd615700 -1 osd.6 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.126222+0000 front 2023-10-10T17:54:18.126439+0000 (oldest deadline 2023-10-10T17:54:42.826368+0000) 2023-10-10T17:54:44.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:44.460+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:44.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:44.460+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:44.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:44.460+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:45.143 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:44 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:44.834+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:45.143 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:44 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:44.834+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:45.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:44.778+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:45.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:44.778+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:45.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:44 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:44.779+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:45.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:44.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:45.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:44.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:45.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:44.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:45.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:44 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:44.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:45.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:45 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:45.142+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:45.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:45 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:45.142+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:45.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:45 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:45.142+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:45.760 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:45.501+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:45.760 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:45.501+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:45.760 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:45.501+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:45.761 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:45.501+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:46.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:45 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:45.817+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:46.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:45 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:45.817+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:46.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:45 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:45.817+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:46.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:45.759+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:46.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:45.759+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:46.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:45.759+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:46.377 INFO:journalctl@ceph.osd.6.smithi078.stdout:Oct 10 17:54:45 smithi078 bash[266135]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-6 2023-10-10T17:54:46.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:45 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:45.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:46.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:45 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:45.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:46.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:45 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:45.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:46.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:45 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:45.890+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:46.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:46 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:46.165+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:46.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:46 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:46.165+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:46.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:46 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:46.165+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:46.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:46 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:46.454+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:46.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:46 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:46.454+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:46.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:46 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:46.454+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:46.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:46 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:46.454+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:47.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:46 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:46.924+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:47.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:46 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:46.924+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:47.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:46 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:46.924+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:47.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:46 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:46.924+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:47.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:46 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:46.801+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:47.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:46 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:46.801+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:47.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:46 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:46.801+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:47.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:46 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:46.808+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:47.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:46 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:46.808+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:47.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:46 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:46.808+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:47.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.6.service' 2023-10-10T17:54:47.635 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:47.635 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-10-10T17:54:47.636 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-10-10T17:54:47.636 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.7 2023-10-10T17:54:47.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:47 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:47.179+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:47.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:47 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:47.179+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:47.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:47 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:47.179+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:47.828 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:47 smithi157 systemd[1]: Stopping Ceph osd.7 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:47.828 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:47.491+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:47.828 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:47.491+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:47.829 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:47.491+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:47.829 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:47.491+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:48.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:47 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:47.839+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:48.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:47 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:47.839+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:48.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:47 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:47.839+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:48.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:47.826+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:48.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:47.826+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:48.229 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:47.826+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:48.229 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:47.990+0000 7f425c3d5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:48.229 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:47.990+0000 7f425c3d5700 -1 osd.7 107 *** Got signal Terminated *** 2023-10-10T17:54:48.229 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:47 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:47.990+0000 7f425c3d5700 -1 osd.7 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:48.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:47 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:47.958+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:48.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:47 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:47.958+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:48.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:47 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:47.958+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:48.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:47 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:47.958+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:48.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:48 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:48.206+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:48.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:48 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:48.206+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:48.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:48 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:48.206+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:48.781 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:48 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:48.518+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:48.782 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:48 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:48.518+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:48.782 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:48 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:48.518+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:48.783 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:48 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:48.518+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:49.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:48 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:48.829+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:49.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:48 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:48.829+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:49.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:48 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:48.829+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:49.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:48 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:48.780+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:49.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:48 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:48.780+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:49.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:48 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:48.780+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:49.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:48 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:48.935+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:49.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:48 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:48.935+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:49.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:48 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:48.935+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:49.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:48 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:48.935+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:49.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:49.193+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:49.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:49.193+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:49.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:49.193+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:49.823 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:49.508+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:49.823 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:49.508+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:49.824 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:49.508+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:49.825 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:49.508+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:50.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:49.872+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:50.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:49.872+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:50.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:49 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:49.872+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:50.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:49.821+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:50.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:49.821+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:50.229 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:49.821+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:50.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:49 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:49.917+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:50.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:49 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:49.917+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:50.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:49 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:49.917+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:50.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:49 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:49.917+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:50.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:50 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:50.228+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:50.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:50 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:50.228+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:50.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:50 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:50.228+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:50.818 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:50 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:50.539+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:50.819 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:50 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:50.539+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:50.819 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:50 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:50.539+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:50.819 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:50 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:50.539+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:51.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:50 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:50.848+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:51.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:50 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:50.848+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:51.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:50 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:50.848+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:51.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:50 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:50.818+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:51.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:50 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:50.818+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:51.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:50 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:50.818+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:51.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:50.933+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:51.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:50.933+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:51.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:50.933+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:51.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:50 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:50.933+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:51.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:51 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:51.180+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:51.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:51 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:51.180+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:51.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:51 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:51.180+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:51.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:51 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:51.180+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:51.855 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:51 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:51.580+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:51.856 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:51 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:51.580+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:51.856 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:51 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:51.580+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:51.856 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:51 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:51.580+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:52.151 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:51 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:51.869+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:52.152 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:51 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:51.869+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:52.152 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:51 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:51.869+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:52.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:51 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:51.855+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:52.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:51 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:51.855+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:52.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:51 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:51.855+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:52.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:51.945+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:52.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:51.945+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:52.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:51.945+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:52.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:51 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:51.945+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:52.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:52 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:52.151+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:52.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:52 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:52.151+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:52.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:52 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:52.151+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:52.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:52 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:52.151+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:52.869 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:52 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:52.583+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:52.870 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:52 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:52.583+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:52.870 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:52 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:52.583+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:52.870 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:52 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:52.583+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:53.134 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:52 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:52.852+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:53.134 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:52 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:52.852+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:53.135 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:52 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:52.852+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:53.227 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:52 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:52.869+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:02.146028+0000 front 2023-10-10T17:54:02.146050+0000 (oldest deadline 2023-10-10T17:54:27.446262+0000) 2023-10-10T17:54:53.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:52 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:52.869+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.847919+0000 front 2023-10-10T17:54:13.847993+0000 (oldest deadline 2023-10-10T17:54:39.748044+0000) 2023-10-10T17:54:53.228 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:52 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7[117008]: 2023-10-10T17:54:52.869+0000 7f42551c4700 -1 osd.7 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:19.748609+0000 front 2023-10-10T17:54:19.748621+0000 (oldest deadline 2023-10-10T17:54:43.248603+0000) 2023-10-10T17:54:53.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:52.916+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:53.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:52.916+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:53.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:52.916+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:53.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:52 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:52.916+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:53.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:53.133+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:53.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:53.133+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:53.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:53.133+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:53.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:53.133+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:53.877 INFO:journalctl@ceph.osd.7.smithi157.stdout:Oct 10 17:54:53 smithi157 bash[133376]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-7 2023-10-10T17:54:53.878 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:53 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:53.548+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:53.878 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:53 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:53.548+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:53.879 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:53 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:53.548+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:53.879 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:53 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:53.548+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:53.879 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:53 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:53.548+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:54:54.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:53.873+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:54.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:53.873+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:54.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:53.873+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:54.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:53 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:53.873+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:54.159 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:53.895+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:54.159 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:53.895+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:54.160 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:53.895+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:54.160 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:53 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:53.895+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:24.915078+0000 front 2023-10-10T17:54:24.915091+0000 (oldest deadline 2023-10-10T17:54:53.715574+0000) 2023-10-10T17:54:54.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:54.158+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:54.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:54.158+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:54.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:54.158+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:54.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:54.158+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:54.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:54 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:54.525+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:54.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:54 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:54.525+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:54.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:54 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:54.525+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:54.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:54 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:54.525+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:54.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:54 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:54.525+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:54:55.143 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.7.service' 2023-10-10T17:54:55.157 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:54.861+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:55.158 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:54.861+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:55.158 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:54.861+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:55.158 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:54 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:54.861+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:24.915078+0000 front 2023-10-10T17:54:24.915091+0000 (oldest deadline 2023-10-10T17:54:53.715574+0000) 2023-10-10T17:54:55.184 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:54:55.185 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-10-10T17:54:55.185 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-10-10T17:54:55.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.8 2023-10-10T17:54:55.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:54.900+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:55.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:54.900+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:55.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:54.900+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:55.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:54.900+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:55.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:54 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:54.900+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:54:55.414 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 systemd[1]: Stopping Ceph osd.8 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:54:55.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:55.157+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:55.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:55.157+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:55.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:55.157+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:55.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:55.157+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:55.820 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:55.552+0000 7fefe9db4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:54:55.821 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:55.552+0000 7fefe9db4700 -1 osd.8 107 *** Got signal Terminated *** 2023-10-10T17:54:55.821 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:55.552+0000 7fefe9db4700 -1 osd.8 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:54:55.821 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:55.820+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:55.822 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:55.820+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:55.822 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:55.820+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:55.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:55 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:55.542+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:55.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:55 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:55.542+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:55.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:55 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:55.542+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:55.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:55 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:55.542+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:55.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:55 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:55.542+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:54:56.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:55 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:55.820+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:24.915078+0000 front 2023-10-10T17:54:24.915091+0000 (oldest deadline 2023-10-10T17:54:53.715574+0000) 2023-10-10T17:54:56.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:55.914+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:56.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:55.914+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:56.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:55.914+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:56.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:55.914+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:56.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:55 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:55.914+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:54:56.664 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:56.173+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:56.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:56.173+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:56.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:56.173+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:56.665 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:56.173+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:56.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:56 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:56.588+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:56.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:56 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:56.588+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:56.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:56 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:56.588+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:56.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:56 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:56.588+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:56.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:56 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:56.588+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:54:57.136 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:56.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:57.136 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:56.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:57.137 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:56.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:57.137 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:56 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:56.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:24.915078+0000 front 2023-10-10T17:54:24.915091+0000 (oldest deadline 2023-10-10T17:54:53.715574+0000) 2023-10-10T17:54:57.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:56 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:56.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:57.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:56 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:56.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:57.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:56 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:56.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:57.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:56 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:56.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:57.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:56 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:56.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:54:57.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:57.134+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:57.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:57.134+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:57.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:57.134+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:57.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:57.134+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:57.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:57 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:57.561+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:57.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:57 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:57.561+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:57.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:57 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:57.561+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:57.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:57 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:57.561+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:57.980 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:57 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:57.561+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:54:58.100 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:57.762+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:58.100 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:57.762+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:58.100 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:57.762+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:58.101 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:57 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:57.762+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:24.915078+0000 front 2023-10-10T17:54:24.915091+0000 (oldest deadline 2023-10-10T17:54:53.715574+0000) 2023-10-10T17:54:58.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:57 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:57.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:58.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:57 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:57.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:58.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:57 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:57.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:58.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:57 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:57.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:58.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:57 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:57.895+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:54:58.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:58.098+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:58.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:58.098+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:58.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:58.098+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:58.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:58.098+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:58.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:58 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:58.611+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:58.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:58 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:58.611+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:58.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:58 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:58.611+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:58.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:58 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:58.611+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:58.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:58 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:58.611+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:54:59.127 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:58 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:58.859+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:54:59.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:58 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:58.859+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:54:59.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:58 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:58.859+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:54:59.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:58 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:58.859+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:54:59.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:58 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:58.859+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:54:59.165 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:59.053+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:54:59.165 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:59.053+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:54:59.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:59.053+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:54:59.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:59.053+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:54:59.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:54:59.053+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:54:59.167 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:58.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:54:59.167 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:58.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:54:59.167 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:58.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:54:59.168 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:58 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:58.805+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:24.915078+0000 front 2023-10-10T17:54:24.915091+0000 (oldest deadline 2023-10-10T17:54:53.715574+0000) 2023-10-10T17:54:59.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:59 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:59.575+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:54:59.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:59 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:59.575+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:54:59.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:59 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:59.575+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:54:59.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:59 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:59.575+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:54:59.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:59 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:59.575+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:54:59.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:54:59 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:54:59.575+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:00.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:59 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:59.827+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:00.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:59 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:59.827+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:00.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:59 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:59.827+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:00.130 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:59 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:59.827+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:00.130 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:54:59 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:54:59.827+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:00.164 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:59.826+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:03.610490+0000 front 2023-10-10T17:54:03.610242+0000 (oldest deadline 2023-10-10T17:54:31.210883+0000) 2023-10-10T17:55:00.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:59.826+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:11.211307+0000 front 2023-10-10T17:54:11.211409+0000 (oldest deadline 2023-10-10T17:54:35.911539+0000) 2023-10-10T17:55:00.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:59.826+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:22.014610+0000 front 2023-10-10T17:54:22.014705+0000 (oldest deadline 2023-10-10T17:54:44.914459+0000) 2023-10-10T17:55:00.165 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:54:59 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8[118076]: 2023-10-10T17:54:59.826+0000 7fefe3ba5700 -1 osd.8 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:24.915078+0000 front 2023-10-10T17:54:24.915091+0000 (oldest deadline 2023-10-10T17:54:53.715574+0000) 2023-10-10T17:55:00.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:00 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:00.065+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:00.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:00 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:00.065+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:00.167 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:00 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:00.065+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:00.167 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:00 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:00.065+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:00.167 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:00 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:00.065+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:00.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:00 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:00.566+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:00.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:00 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:00.566+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:00.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:00 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:00.566+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:00.980 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:00 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:00.566+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:00.980 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:00 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:00.566+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:00.980 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:00 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:00.566+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:01.127 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:00.829+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:01.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:00.829+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:01.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:00.829+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:01.128 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:00.829+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:01.129 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:00 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:00.829+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:01.415 INFO:journalctl@ceph.osd.8.smithi190.stdout:Oct 10 17:55:01 smithi190 bash[134163]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-8 2023-10-10T17:55:01.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.021+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:01.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.021+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:01.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.021+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:01.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.021+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:01.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.021+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:01.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:01.531+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:01.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:01.531+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:01.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:01.531+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:01.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:01.531+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:01.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:01.531+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:01.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:01 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:01.531+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:02.164 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.996+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:02.165 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.996+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:02.165 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.996+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:02.165 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.996+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:02.166 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:01.996+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:02.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:01 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:01.877+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:02.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:01 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:01.877+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:02.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:01 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:01.877+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:02.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:01 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:01.877+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:02.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:01 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:01.877+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:02.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:01 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:01.877+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:34.178221+0000 front 2023-10-10T17:54:34.178104+0000 (oldest deadline 2023-10-10T17:55:01.778608+0000) 2023-10-10T17:55:02.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.8.service' 2023-10-10T17:55:02.709 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:02.710 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-10-10T17:55:02.711 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-10-10T17:55:02.711 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.9 2023-10-10T17:55:02.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:02 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:02.504+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:02.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:02 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:02.504+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:02.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:02 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:02.504+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:02.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:02 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:02.504+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:02.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:02 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:02.504+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:02.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:02 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:02.504+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:03.082 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:02 smithi078 systemd[1]: Stopping Ceph osd.9 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:03.083 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:02 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:02.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:03.083 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:02 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:02.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:03.083 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:02 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:02.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:03.084 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:02 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:02.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:03.084 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:02 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:02.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:03.084 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:02 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:02.913+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:34.178221+0000 front 2023-10-10T17:54:34.178104+0000 (oldest deadline 2023-10-10T17:55:01.778608+0000) 2023-10-10T17:55:03.084 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.080+0000 7f8d7a958700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:03.085 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.080+0000 7f8d7a958700 -1 osd.9 107 *** Got signal Terminated *** 2023-10-10T17:55:03.085 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.080+0000 7f8d7a958700 -1 osd.9 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:55:03.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:02 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:02.986+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:03.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:02 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:02.986+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:03.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:02 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:02.986+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:03.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:02 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:02.986+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:03.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:02 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:02.986+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:03.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:03 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:03.530+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:03.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:03 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:03.530+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:03.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:03 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:03.530+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:03.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:03 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:03.530+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:03.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:03 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:03.530+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:03.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:03 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:03.530+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:04.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.907+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:04.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.907+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:04.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.907+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:04.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.907+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:04.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.907+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:04.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:03 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:03.907+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:34.178221+0000 front 2023-10-10T17:54:34.178104+0000 (oldest deadline 2023-10-10T17:55:01.778608+0000) 2023-10-10T17:55:04.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:03 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:03.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:04.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:03 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:03.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:04.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:03 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:03.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:04.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:03 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:03.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:04.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:03 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:03.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:04.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:04 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:04.489+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:04.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:04 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:04.489+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:04.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:04 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:04.489+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:04.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:04 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:04.489+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:04.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:04 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:04.489+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:04.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:04 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:04.489+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:05.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:04 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:04.949+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:05.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:04 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:04.949+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:05.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:04 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:04.949+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:05.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:04 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:04.949+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:05.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:04 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:04.949+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:05.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:04 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:04.949+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:34.178221+0000 front 2023-10-10T17:54:34.178104+0000 (oldest deadline 2023-10-10T17:55:01.778608+0000) 2023-10-10T17:55:05.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:04 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:04.981+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:05.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:04 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:04.981+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:05.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:04 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:04.981+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:05.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:04 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:04.981+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:05.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:04 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:04.981+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:05.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:05 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:05.492+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:05.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:05 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:05.492+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:05.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:05 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:05.492+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:05.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:05 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:05.492+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:05.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:05 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:05.492+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:05.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:05 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:05.492+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:06.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:05.985+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:06.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:05.985+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:06.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:05.985+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:06.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:05.985+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:06.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:05.985+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:06.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:05 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:05.985+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:34.178221+0000 front 2023-10-10T17:54:34.178104+0000 (oldest deadline 2023-10-10T17:55:01.778608+0000) 2023-10-10T17:55:06.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:06 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:06.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:06.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:06 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:06.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:06.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:06 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:06.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:06.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:06 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:06.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:06.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:06 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:06.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:06.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:06 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:06.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:06.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:06 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:06.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:06.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:06 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:06.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:06.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:06 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:06.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:06.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:06 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:06.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:06.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:06 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:06.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:06.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:06 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:06.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:07.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:06 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:06.948+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:07.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:06 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:06.948+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:07.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:06 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:06.948+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:07.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:06 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:06.948+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:07.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:06 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:06.948+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:07.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:06 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:06.948+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:34.178221+0000 front 2023-10-10T17:54:34.178104+0000 (oldest deadline 2023-10-10T17:55:01.778608+0000) 2023-10-10T17:55:07.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.007+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:07.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.007+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:07.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.007+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:07.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.007+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:07.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.007+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:07.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.007+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:07.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:07 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:07.445+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:07.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:07 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:07.445+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:07.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:07 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:07.445+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:07.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:07 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:07.445+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:07.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:07 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:07.445+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:07.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:07 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:07.445+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:08.377 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:07.904+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:55.071771+0000 front 2023-10-10T17:53:55.071758+0000 (oldest deadline 2023-10-10T17:54:20.971925+0000) 2023-10-10T17:55:08.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:07.904+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:05.573723+0000 front 2023-10-10T17:54:05.573680+0000 (oldest deadline 2023-10-10T17:54:29.073846+0000) 2023-10-10T17:55:08.378 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:07.904+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:09.074467+0000 front 2023-10-10T17:54:09.074377+0000 (oldest deadline 2023-10-10T17:54:36.075091+0000) 2023-10-10T17:55:08.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:07.904+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:21.876605+0000 front 2023-10-10T17:54:21.876577+0000 (oldest deadline 2023-10-10T17:54:44.176724+0000) 2023-10-10T17:55:08.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:07.904+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.877813+0000 front 2023-10-10T17:54:28.877915+0000 (oldest deadline 2023-10-10T17:54:54.177617+0000) 2023-10-10T17:55:08.379 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:07 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9[128847]: 2023-10-10T17:55:07.904+0000 7f8d74f4a700 -1 osd.9 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:34.178221+0000 front 2023-10-10T17:54:34.178104+0000 (oldest deadline 2023-10-10T17:55:01.778608+0000) 2023-10-10T17:55:08.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.967+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:08.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.967+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:08.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.967+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:08.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.967+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:08.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.967+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:08.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:07 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:07.967+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:08.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:08.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:08.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:08.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:08.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:08.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:08.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:08.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:08.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:08.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:08.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:08.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:08.730 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:08 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:08.422+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:08.953 INFO:journalctl@ceph.osd.9.smithi078.stdout:Oct 10 17:55:08 smithi078 bash[266412]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-9 2023-10-10T17:55:09.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:08 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:08.992+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:09.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:08 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:08.992+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:09.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:08 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:08.992+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:09.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:08 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:08.992+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:09.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:08 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:08.992+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:09.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:08 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:08.992+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:09.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:09 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:09.411+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:09.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:09 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:09.411+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:09.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:09 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:09.411+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:09.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:09 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:09.411+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:09.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:09 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:09.411+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:09.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:09 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:09.411+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:09.730 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:09 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:09.411+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:10.311 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.9.service' 2023-10-10T17:55:10.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:10.352 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-10-10T17:55:10.353 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-10-10T17:55:10.353 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.10 2023-10-10T17:55:10.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:09.985+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:10.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:09.985+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:10.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:09.985+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:10.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:09.985+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:10.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:09.985+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:10.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:09.985+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:10.697 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 systemd[1]: Stopping Ceph osd.10 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:10.697 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.392+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:10.697 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.392+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:10.698 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.392+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:10.698 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.392+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:10.698 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.392+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:10.699 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.392+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:10.699 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.392+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:10.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.696+0000 7fc82258e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:10.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.696+0000 7fc82258e700 -1 osd.10 107 *** Got signal Terminated *** 2023-10-10T17:55:10.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:10 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:10.696+0000 7fc82258e700 -1 osd.10 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:55:11.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:11 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:11.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:11.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:11 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:11.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:11.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:11 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:11.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:11.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:11 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:11.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:11.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:11 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:11.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:11.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:11 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:11.020+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:11.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:11 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:11.426+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:11.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:11 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:11.426+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:11.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:11 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:11.426+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:11.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:11 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:11.426+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:11.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:11 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:11.426+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:11.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:11 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:11.426+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:11.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:11 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:11.426+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:12.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:12 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:12.043+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:12.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:12 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:12.043+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:12.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:12 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:12.043+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:12.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:12 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:12.043+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:12.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:12 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:12.043+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:12.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:12 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:12.043+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:12.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:12 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:12.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:12.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:12 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:12.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:12.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:12 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:12.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:12.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:12 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:12.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:12.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:12 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:12.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:12.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:12 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:12.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:12.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:12 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:12.444+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:13.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:13.052+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:13.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:13.052+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:13.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:13.052+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:13.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:13.052+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:13.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:13.052+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:13.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:13.052+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:13.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:13.461+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:13.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:13.461+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:13.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:13.461+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:13.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:13.461+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:13.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:13.461+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:13.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:13.461+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:13.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:13 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:13.461+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:14.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.003+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:14.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.003+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:14.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.003+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:14.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.003+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:14.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.003+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:14.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.003+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:14.727 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:14 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:14.465+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:14.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:14 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:14.465+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:14.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:14 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:14.465+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:14.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:14 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:14.465+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:14.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:14 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:14.465+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:14.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:14 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:14.465+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:14.729 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:14 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:14.465+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:15.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:15.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:15.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:15.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:15.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:15.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:14 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:14.954+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:15.977 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:15 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:15.496+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.611614+0000 front 2023-10-10T17:53:56.611639+0000 (oldest deadline 2023-10-10T17:54:21.311525+0000) 2023-10-10T17:55:15.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:15 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:15.496+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:04.812764+0000 front 2023-10-10T17:54:04.812799+0000 (oldest deadline 2023-10-10T17:54:27.712923+0000) 2023-10-10T17:55:15.978 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:15 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:15.496+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:12.413979+0000 front 2023-10-10T17:54:12.414011+0000 (oldest deadline 2023-10-10T17:54:38.313997+0000) 2023-10-10T17:55:15.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:15 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:15.496+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-10-10T17:54:18.815196+0000 front 2023-10-10T17:54:18.815323+0000 (oldest deadline 2023-10-10T17:54:44.715180+0000) 2023-10-10T17:55:15.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:15 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:15.496+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:30.016114+0000 front 2023-10-10T17:54:30.016139+0000 (oldest deadline 2023-10-10T17:54:52.916296+0000) 2023-10-10T17:55:15.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:15 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:15.496+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:36.918273+0000 front 2023-10-10T17:54:36.918308+0000 (oldest deadline 2023-10-10T17:54:58.618097+0000) 2023-10-10T17:55:15.979 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:15 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10[122435]: 2023-10-10T17:55:15.496+0000 7fc81eb84700 -1 osd.10 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:43.319260+0000 front 2023-10-10T17:54:43.319181+0000 (oldest deadline 2023-10-10T17:55:08.019096+0000) 2023-10-10T17:55:16.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:15 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:15.944+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:16.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:15 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:15.944+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:16.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:15 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:15.944+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:16.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:15 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:15.944+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:16.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:15 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:15.944+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:16.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:15 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:15.944+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:16.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:15 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:15.944+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6820 osd.7 since back 2023-10-10T17:54:51.581534+0000 front 2023-10-10T17:54:51.581480+0000 (oldest deadline 2023-10-10T17:55:14.981754+0000) 2023-10-10T17:55:16.728 INFO:journalctl@ceph.osd.10.smithi157.stdout:Oct 10 17:55:16 smithi157 bash[133637]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-10 2023-10-10T17:55:17.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:16 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:16.976+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:17.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:16 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:16.976+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:17.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:16 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:16.976+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:17.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:16 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:16.976+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:17.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:16 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:16.976+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:17.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:16 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:16.976+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:17.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:16 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:16.976+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6820 osd.7 since back 2023-10-10T17:54:51.581534+0000 front 2023-10-10T17:54:51.581480+0000 (oldest deadline 2023-10-10T17:55:14.981754+0000) 2023-10-10T17:55:17.666 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.10.service' 2023-10-10T17:55:17.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:17.708 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-10-10T17:55:17.709 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-10-10T17:55:17.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.11 2023-10-10T17:55:18.010 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:17 smithi190 systemd[1]: Stopping Ceph osd.11 for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:18.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.009+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:18.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.009+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:18.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.009+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:18.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.009+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:18.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.009+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:18.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.009+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:18.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.009+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6820 osd.7 since back 2023-10-10T17:54:51.581534+0000 front 2023-10-10T17:54:51.581480+0000 (oldest deadline 2023-10-10T17:55:14.981754+0000) 2023-10-10T17:55:18.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.065+0000 7fe1b8828700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:18.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.065+0000 7fe1b8828700 -1 osd.11 107 *** Got signal Terminated *** 2023-10-10T17:55:18.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:18 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:18.065+0000 7fe1b8828700 -1 osd.11 107 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-10T17:55:19.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:19 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:19.051+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:19.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:19 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:19.051+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:19.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:19 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:19.051+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:19.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:19 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:19.051+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:19.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:19 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:19.051+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:19.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:19 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:19.051+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:19.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:19 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:19.051+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6820 osd.7 since back 2023-10-10T17:54:51.581534+0000 front 2023-10-10T17:54:51.581480+0000 (oldest deadline 2023-10-10T17:55:14.981754+0000) 2023-10-10T17:55:20.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:20.064+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:20.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:20.064+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:20.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:20.064+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:20.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:20.064+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:20.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:20.064+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:20.417 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:20.064+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:20.418 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:20 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:20.064+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6820 osd.7 since back 2023-10-10T17:54:51.581534+0000 front 2023-10-10T17:54:51.581480+0000 (oldest deadline 2023-10-10T17:55:14.981754+0000) 2023-10-10T17:55:21.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:21.059+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:21.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:21.059+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:21.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:21.059+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:21.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:21.059+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:21.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:21.059+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:21.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:21.059+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:21.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:21 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:21.059+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6820 osd.7 since back 2023-10-10T17:54:51.581534+0000 front 2023-10-10T17:54:51.581480+0000 (oldest deadline 2023-10-10T17:55:14.981754+0000) 2023-10-10T17:55:22.414 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:22.110+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-10-10T17:53:56.672918+0000 front 2023-10-10T17:53:56.672782+0000 (oldest deadline 2023-10-10T17:54:19.572933+0000) 2023-10-10T17:55:22.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:22.110+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6804 osd.1 since back 2023-10-10T17:54:01.874158+0000 front 2023-10-10T17:54:01.874174+0000 (oldest deadline 2023-10-10T17:54:27.774187+0000) 2023-10-10T17:55:22.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:22.110+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6804 osd.2 since back 2023-10-10T17:54:13.675223+0000 front 2023-10-10T17:54:13.675201+0000 (oldest deadline 2023-10-10T17:54:38.975405+0000) 2023-10-10T17:55:22.415 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:22.110+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6812 osd.4 since back 2023-10-10T17:54:28.277560+0000 front 2023-10-10T17:54:28.277670+0000 (oldest deadline 2023-10-10T17:54:51.177601+0000) 2023-10-10T17:55:22.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:22.110+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.190:6812 osd.5 since back 2023-10-10T17:54:32.878549+0000 front 2023-10-10T17:54:32.878504+0000 (oldest deadline 2023-10-10T17:54:58.778805+0000) 2023-10-10T17:55:22.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:22.110+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-10-10T17:54:40.479826+0000 front 2023-10-10T17:54:40.479901+0000 (oldest deadline 2023-10-10T17:55:05.779890+0000) 2023-10-10T17:55:22.416 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:22 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11[123523]: 2023-10-10T17:55:22.110+0000 7fe1b461d700 -1 osd.11 107 heartbeat_check: no reply from 172.21.15.157:6820 osd.7 since back 2023-10-10T17:54:51.581534+0000 front 2023-10-10T17:54:51.581480+0000 (oldest deadline 2023-10-10T17:55:14.981754+0000) 2023-10-10T17:55:24.164 INFO:journalctl@ceph.osd.11.smithi190.stdout:Oct 10 17:55:23 smithi190 bash[134449]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-osd-11 2023-10-10T17:55:25.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@osd.11.service' 2023-10-10T17:55:25.158 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:25.158 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-10-10T17:55:25.159 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-10-10T17:55:25.159 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.a 2023-10-10T17:55:25.521 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:55:25 smithi078 systemd[1]: Stopping Ceph mds.a for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:25.877 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:55:25 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-a[134095]: 2023-10-10T17:55:25.520+0000 7f42b2d6d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:25.878 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:55:25 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-a[134095]: 2023-10-10T17:55:25.520+0000 7f42b2d6d700 -1 mds.a *** got signal Terminated *** 2023-10-10T17:55:29.878 INFO:journalctl@ceph.mds.a.smithi078.stdout:Oct 10 17:55:29 smithi078 bash[266674]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-a 2023-10-10T17:55:30.080 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.a.service' 2023-10-10T17:55:30.115 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:30.115 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-10-10T17:55:30.116 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-10-10T17:55:30.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.d 2023-10-10T17:55:30.377 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:55:30 smithi078 systemd[1]: Stopping Ceph mds.d for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:30.877 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:55:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-d[133102]: 2023-10-10T17:55:30.494+0000 7f5c64255700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:30.878 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:55:30 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-d[133102]: 2023-10-10T17:55:30.494+0000 7f5c64255700 -1 mds.d *** got signal Terminated *** 2023-10-10T17:55:34.764 INFO:journalctl@ceph.mds.d.smithi078.stdout:Oct 10 17:55:34 smithi078 bash[266817]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-d 2023-10-10T17:55:34.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.d.service' 2023-10-10T17:55:34.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:34.988 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-10-10T17:55:34.988 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-10-10T17:55:34.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.g 2023-10-10T17:55:35.369 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:55:35 smithi078 systemd[1]: Stopping Ceph mds.g for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:35.627 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:55:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-g[133712]: 2023-10-10T17:55:35.368+0000 7f61e6854700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:35.628 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:55:35 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-g[133712]: 2023-10-10T17:55:35.368+0000 7f61e6854700 -1 mds.g *** got signal Terminated *** 2023-10-10T17:55:39.736 INFO:journalctl@ceph.mds.g.smithi078.stdout:Oct 10 17:55:39 smithi078 bash[266958]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-g 2023-10-10T17:55:39.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.g.service' 2023-10-10T17:55:39.956 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:39.956 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-10-10T17:55:39.957 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-10-10T17:55:39.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.j 2023-10-10T17:55:40.335 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:55:40 smithi078 systemd[1]: Stopping Ceph mds.j for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:40.627 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:55:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-j[132743]: 2023-10-10T17:55:40.334+0000 7f0408ce2700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:40.628 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:55:40 smithi078 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-j[132743]: 2023-10-10T17:55:40.334+0000 7f0408ce2700 -1 mds.j *** got signal Terminated *** 2023-10-10T17:55:44.758 INFO:journalctl@ceph.mds.j.smithi078.stdout:Oct 10 17:55:44 smithi078 bash[267100]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-j 2023-10-10T17:55:44.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.j.service' 2023-10-10T17:55:44.928 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:44.929 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-10-10T17:55:44.929 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-10-10T17:55:44.930 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.b 2023-10-10T17:55:45.229 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:55:44 smithi157 systemd[1]: Stopping Ceph mds.b for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:45.727 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:55:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: 2023-10-10T17:55:45.281+0000 7f2e42572700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:45.728 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:55:45 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b[126206]: 2023-10-10T17:55:45.281+0000 7f2e42572700 -1 mds.b *** got signal Terminated *** 2023-10-10T17:55:48.557 INFO:journalctl@ceph.mds.b.smithi157.stdout:Oct 10 17:55:48 smithi157 bash[133913]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-b 2023-10-10T17:55:48.750 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.b.service' 2023-10-10T17:55:48.785 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:48.786 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-10-10T17:55:48.786 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-10-10T17:55:48.787 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.e 2023-10-10T17:55:49.165 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:55:48 smithi157 systemd[1]: Stopping Ceph mds.e for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:49.477 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:55:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-e[125527]: 2023-10-10T17:55:49.165+0000 7f39b9102700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:49.478 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:55:49 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-e[125527]: 2023-10-10T17:55:49.165+0000 7f39b9102700 -1 mds.e *** got signal Terminated *** 2023-10-10T17:55:52.532 INFO:journalctl@ceph.mds.e.smithi157.stdout:Oct 10 17:55:52 smithi157 bash[134055]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-e 2023-10-10T17:55:52.721 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.e.service' 2023-10-10T17:55:52.756 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:52.757 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-10-10T17:55:52.758 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-10-10T17:55:52.758 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.h 2023-10-10T17:55:52.977 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:55:52 smithi157 systemd[1]: Stopping Ceph mds.h for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:53.478 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:55:53 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-h[129959]: 2023-10-10T17:55:53.129+0000 7efe3dacc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:53.478 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:55:53 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-h[129959]: 2023-10-10T17:55:53.129+0000 7efe3dacc700 -1 mds.h *** got signal Terminated *** 2023-10-10T17:55:56.727 INFO:journalctl@ceph.mds.h.smithi157.stdout:Oct 10 17:55:56 smithi157 bash[134198]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-h 2023-10-10T17:55:56.732 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.h.service' 2023-10-10T17:55:56.765 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:55:56.766 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-10-10T17:55:56.766 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-10-10T17:55:56.767 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.k 2023-10-10T17:55:57.146 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:55:56 smithi157 systemd[1]: Stopping Ceph mds.k for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:55:57.477 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:55:57 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-k[125187]: 2023-10-10T17:55:57.145+0000 7f09365ce700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:55:57.478 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:55:57 smithi157 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-k[125187]: 2023-10-10T17:55:57.145+0000 7f09365ce700 -1 mds.k *** got signal Terminated *** 2023-10-10T17:56:00.727 INFO:journalctl@ceph.mds.k.smithi157.stdout:Oct 10 17:56:00 smithi157 bash[134341]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-k 2023-10-10T17:56:00.803 DEBUG:teuthology.orchestra.run.smithi157:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.k.service' 2023-10-10T17:56:00.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:56:00.837 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-10-10T17:56:00.837 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-10-10T17:56:00.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.c 2023-10-10T17:56:01.164 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:56:00 smithi190 systemd[1]: Stopping Ceph mds.c for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:56:01.664 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:56:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-c[127115]: 2023-10-10T17:56:01.217+0000 7f720ef2f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:56:01.665 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:56:01 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-c[127115]: 2023-10-10T17:56:01.217+0000 7f720ef2f700 -1 mds.c *** got signal Terminated *** 2023-10-10T17:56:04.834 INFO:journalctl@ceph.mds.c.smithi190.stdout:Oct 10 17:56:04 smithi190 bash[134728]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-c 2023-10-10T17:56:04.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.c.service' 2023-10-10T17:56:04.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:56:04.880 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-10-10T17:56:04.880 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-10-10T17:56:04.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.f 2023-10-10T17:56:05.164 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:56:04 smithi190 systemd[1]: Stopping Ceph mds.f for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:56:05.664 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:56:05 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-f[126774]: 2023-10-10T17:56:05.257+0000 7fcc66d45700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:56:05.665 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:56:05 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-f[126774]: 2023-10-10T17:56:05.257+0000 7fcc66d45700 -1 mds.f *** got signal Terminated *** 2023-10-10T17:56:08.811 INFO:journalctl@ceph.mds.f.smithi190.stdout:Oct 10 17:56:08 smithi190 bash[134876]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-f 2023-10-10T17:56:08.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.f.service' 2023-10-10T17:56:08.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:56:08.852 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-10-10T17:56:08.853 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-10-10T17:56:08.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.i 2023-10-10T17:56:09.164 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:56:08 smithi190 systemd[1]: Stopping Ceph mds.i for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:56:09.664 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:56:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-i[127457]: 2023-10-10T17:56:09.220+0000 7fe9b80cd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:56:09.665 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:56:09 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-i[127457]: 2023-10-10T17:56:09.220+0000 7fe9b80cd700 -1 mds.i *** got signal Terminated *** 2023-10-10T17:56:12.599 INFO:journalctl@ceph.mds.i.smithi190.stdout:Oct 10 17:56:12 smithi190 bash[135020]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-i 2023-10-10T17:56:12.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.i.service' 2023-10-10T17:56:12.820 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:56:12.821 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-10-10T17:56:12.821 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-10-10T17:56:12.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.l 2023-10-10T17:56:13.164 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:56:12 smithi190 systemd[1]: Stopping Ceph mds.l for 0ba2616a-6793-11ee-8db6-212e2dc638e7... 2023-10-10T17:56:13.664 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:56:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-l[126436]: 2023-10-10T17:56:13.201+0000 7fc747c0d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-10T17:56:13.665 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:56:13 smithi190 ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-l[126436]: 2023-10-10T17:56:13.201+0000 7fc747c0d700 -1 mds.l *** got signal Terminated *** 2023-10-10T17:56:16.300 INFO:journalctl@ceph.mds.l.smithi190.stdout:Oct 10 17:56:16 smithi190 bash[135159]: ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7-mds-l 2023-10-10T17:56:16.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0ba2616a-6793-11ee-8db6-212e2dc638e7@mds.l.service' 2023-10-10T17:56:16.507 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-10T17:56:16.508 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-10-10T17:56:16.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 --force --keep-logs 2023-10-10T17:56:19.097 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 --force --keep-logs 2023-10-10T17:56:21.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 --force --keep-logs 2023-10-10T17:56:23.125 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:56:23.163 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:56:23.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-10T17:56:23.229 INFO:tasks.cephadm:Archiving crash dumps... 2023-10-10T17:56:23.231 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi078/crash 2023-10-10T17:56:23.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash -- . 2023-10-10T17:56:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-10T17:56:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2023-10-10T17:56:23.281 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi157/crash 2023-10-10T17:56:23.282 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash -- . 2023-10-10T17:56:23.333 INFO:teuthology.orchestra.run.smithi157.stderr:tar: /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-10T17:56:23.334 INFO:teuthology.orchestra.run.smithi157.stderr:tar: Error is not recoverable: exiting now 2023-10-10T17:56:23.336 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi190/crash 2023-10-10T17:56:23.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash -- . 2023-10-10T17:56:23.382 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-10T17:56:23.382 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2023-10-10T17:56:23.383 INFO:tasks.cephadm:Checking cluster log for badness... 2023-10-10T17:56:23.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-10-10T17:56:23.423 INFO:teuthology.orchestra.run.smithi078.stderr:grep: /var/log/ceph/0ba2616a-6793-11ee-8db6-212e2dc638e7/ceph.log: No such file or directory 2023-10-10T17:56:23.426 INFO:tasks.cephadm:Compressing logs... 2023-10-10T17:56:23.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T17:56:23.469 DEBUG:teuthology.orchestra.run.smithi157:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T17:56:23.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T17:56:23.501 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-10-10T17:56:23.502 INFO:teuthology.orchestra.run.smithi157.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-10-10T17:56:23.531 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-10-10T18:05:26.132 INFO:tasks.cephadm:Archiving logs... 2023-10-10T18:05:26.133 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi078/log 2023-10-10T18:05:26.134 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-10T18:05:44.565 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi157/log 2023-10-10T18:05:44.566 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-10T18:06:20.809 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi190/log 2023-10-10T18:06:20.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-10T18:07:08.835 INFO:tasks.cephadm:Removing cluster... 2023-10-10T18:07:08.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 --force 2023-10-10T18:07:09.840 DEBUG:teuthology.orchestra.run.smithi157:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 --force 2023-10-10T18:07:10.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0ba2616a-6793-11ee-8db6-212e2dc638e7 --force 2023-10-10T18:07:12.199 INFO:tasks.cephadm:Removing cephadm ... 2023-10-10T18:07:12.199 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-10T18:07:12.221 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-10T18:07:12.243 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-10T18:07:12.273 INFO:tasks.cephadm:Teardown complete 2023-10-10T18:07:12.274 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-10T18:07:12.286 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-10T18:07:12.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-10T18:07:12.289 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-10T18:07:12.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-10T18:07:12.424 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-10T18:07:12.435 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-10T18:07:12.436 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T18:07:12.439 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T18:07:12.441 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-10T18:07:12.455 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-10-10T18:07:12.457 INFO:teuthology.orchestra.run.smithi190.stderr:bash: ntpq: command not found 2023-10-10T18:07:12.458 INFO:teuthology.orchestra.run.smithi157.stderr:bash: ntpq: command not found 2023-10-10T18:07:12.464 INFO:teuthology.orchestra.run.smithi157.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T18:07:12.464 INFO:teuthology.orchestra.run.smithi157.stdout:=============================================================================== 2023-10-10T18:07:12.464 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -433us[ -426us] +/- 93ms 2023-10-10T18:07:12.465 INFO:teuthology.orchestra.run.smithi157.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -50us[ -43us] +/- 54ms 2023-10-10T18:07:12.465 INFO:teuthology.orchestra.run.smithi157.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +485us[ +485us] +/- 75ms 2023-10-10T18:07:12.465 INFO:teuthology.orchestra.run.smithi157.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:07:12.474 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T18:07:12.475 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-10-10T18:07:12.475 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 24 -429us[ -429us] +/- 93ms 2023-10-10T18:07:12.475 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -59us[ -38us] +/- 54ms 2023-10-10T18:07:12.476 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +484us[ +484us] +/- 75ms 2023-10-10T18:07:12.476 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:07:12.521 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-10T18:07:12.522 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2023-10-10T18:07:12.522 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 19 -444us[ -444us] +/- 93ms 2023-10-10T18:07:12.522 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 20 -69us[ -74us] +/- 54ms 2023-10-10T18:07:12.522 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +452us[ +452us] +/- 75ms 2023-10-10T18:07:12.523 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-10-10T18:07:12.524 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-10T18:07:12.535 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-10T18:07:12.536 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-10T18:07:12.575 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-10T18:07:12.783 DEBUG:teuthology.orchestra.run.smithi157:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-10T18:07:12.889 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-10T18:07:12.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T18:07:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:07:13.039 DEBUG:teuthology.orchestra.run.smithi157:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T18:07:13.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:07:13.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-10T18:07:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:07:13.107 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-10T18:07:13.120 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-10T18:07:13.150 INFO:teuthology.task.internal:Duration was 2794.400474 seconds 2023-10-10T18:07:13.151 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-10T18:07:13.201 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-10T18:07:13.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-10T18:07:13.205 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-10T18:07:13.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-10T18:07:13.257 INFO:teuthology.orchestra.run.smithi157.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T18:07:13.283 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T18:07:13.317 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-10T18:07:13.690 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-10T18:07:13.691 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2023-10-10T18:07:13.691 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-10T18:07:13.731 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2023-10-10T18:07:13.731 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-10T18:07:13.757 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2023-10-10T18:07:13.757 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-10T18:07:13.779 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-10T18:07:13.780 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T18:07:13.783 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T18:07:13.801 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-10T18:07:13.869 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-10T18:07:13.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-10T18:07:13.873 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-10T18:07:13.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-10T18:07:14.000 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-10T18:07:14.012 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-10T18:07:14.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-10T18:07:14.057 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-10T18:07:14.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-10T18:07:14.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-10T18:07:14.126 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-10T18:07:14.128 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-10T18:07:14.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-10T18:07:14.155 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2023-10-10T18:07:14.163 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2023-10-10T18:07:14.186 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2023-10-10T18:07:14.214 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-10T18:07:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:07:14.247 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-10T18:07:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:07:14.264 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-10T18:07:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-10T18:07:14.281 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-10T18:07:14.293 INFO:teuthology.task.internal:Transferring archived files... 2023-10-10T18:07:14.294 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi078 2023-10-10T18:07:14.294 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-10T18:07:14.339 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi157 2023-10-10T18:07:14.340 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-10T18:07:14.379 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361/remote/smithi190 2023-10-10T18:07:14.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-10T18:07:14.422 INFO:teuthology.task.internal:Removing archive directory... 2023-10-10T18:07:14.422 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-10T18:07:14.425 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-10T18:07:14.428 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-10T18:07:14.470 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-10T18:07:14.487 INFO:teuthology.task.internal:Not uploading archives. 2023-10-10T18:07:14.487 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-10T18:07:14.515 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-10T18:07:14.515 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-10T18:07:14.518 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-10T18:07:14.520 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-10T18:07:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 10 18:07 /home/ubuntu/cephtest 2023-10-10T18:07:14.534 INFO:teuthology.orchestra.run.smithi157.stdout: 262181 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 10 18:07 /home/ubuntu/cephtest 2023-10-10T18:07:14.536 INFO:teuthology.orchestra.run.smithi190.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 10 18:07 /home/ubuntu/cephtest 2023-10-10T18:07:14.538 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-10T18:07:14.550 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-10T18:07:14.599 INFO:teuthology.nuke:Checking targets against current locks 2023-10-10T18:07:14.629 DEBUG:teuthology.nuke:shortname: smithi078 2023-10-10T18:07:14.629 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-10T18:07:14.655 DEBUG:teuthology.nuke:shortname: smithi157 2023-10-10T18:07:14.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-10T18:07:14.661 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 17:14:32.194213', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-10T18:07:14.682 DEBUG:teuthology.nuke:shortname: smithi190 2023-10-10T18:07:14.682 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-10T18:07:14.685 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 17:14:32.195393', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-10T18:07:14.711 INFO:teuthology.orchestra.console.smithi078:Power off 2023-10-10T18:07:14.712 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-10T18:07:14.725 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/choffman-2023-10-10_17:11:00-fs:workload-wip-choffman-62793-distro-default-smithi/7420361', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-10 17:14:32.192674', 'locked_by': 'scheduled_choffman@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-10T18:07:14.735 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2023-10-10T18:07:14.736 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:14.758 INFO:teuthology.orchestra.console.smithi157:Power off 2023-10-10T18:07:14.759 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-10T18:07:14.769 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-10-10T18:07:14.781 DEBUG:teuthology.orchestra.console.smithi157:power off output: Chassis Power Control: Down/Off 2023-10-10T18:07:14.781 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:14.800 INFO:teuthology.orchestra.console.smithi190:Power off 2023-10-10T18:07:14.801 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-10T18:07:14.809 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-10-10T18:07:14.824 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2023-10-10T18:07:14.824 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:14.850 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-10-10T18:07:18.770 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:18.811 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:18.852 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:18.880 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-10-10T18:07:18.922 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is on 2023-10-10T18:07:18.962 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2023-10-10T18:07:22.882 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:22.923 DEBUG:teuthology.orchestra.console.smithi157:pexpect command: ipmitool -H smithi157.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:22.963 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:22.995 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-10-10T18:07:23.034 DEBUG:teuthology.orchestra.console.smithi157:check power output: Chassis Power is off 2023-10-10T18:07:23.073 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2023-10-10T18:07:23.135 INFO:teuthology.orchestra.console.smithi157:Power off completed 2023-10-10T18:07:23.175 INFO:teuthology.orchestra.console.smithi190:Power off completed 2023-10-10T18:07:26.998 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-10T18:07:27.109 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2023-10-10T18:07:27.210 INFO:teuthology.orchestra.console.smithi078:Power off completed 2023-10-10T18:07:27.316 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} duration: 2794.400473833084 failure_reason: 'Command failed (workunit test fs/misc/general_vxattrs.sh) on smithi078 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=48b68c107f2b9994cc15db6a2fd7f04cd42ba2cf TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/general_vxattrs.sh''' flavor: default owner: scheduled_choffman@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9a3ebbdf078c4d02844107561eb782e4 status: fail success: false 2023-10-10T18:07:27.317 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-10T18:07:27.417 INFO:teuthology.run:FAIL